AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise
Authors: Dhruv Agarwal
Bodhisattwa Prasad Majumder
Reece Adamson
Megha Chakravorty
Satvika Reddy Gavireddy
Aditya Parashar
Harshit Surana
Bhavana Dalvi Mishra
Andrew McCallum
Ashish Sabharwal
Peter Clark
Venue: NeurIPS 2025 poster ()
Keywords: scientific discovery
LLM
open-ended
Bayesian
surprise
data-driven discovery
autonomous discovery
Primary area: machine_learning_for_sciences
TL;DR: We present AutoDiscovery, a method for open-ended scientific discovery that uses Bayesian surprise and Monte Carlo tree search to sample diverse hypotheses at scale that are likely to lead to discoveries that are suprising both to LLMs and humans.

Abstract: The promise of autonomous scientific discovery (ASD) hinges not only on answering questions, but also on knowing which questions to ask. Most recent works in ASD explore the use of large language models (LLMs) in goal-driven settings, relying on human-specified research questions to guide hypothesis generation. However, scientific discovery may be accelerated further by allowing the AI system to drive exploration by its own criteria. The few existing approaches in open-ended ASD select hypotheses based on diversity heuristics or subjective proxies for human interestingness, but the former struggles to meaningfully navigate the typically vast hypothesis space, and the latter suffers from imprecise definitions. This paper presents AutoDiscovery—a method for open-ended ASD that instead drives scientific exploration using Bayesian surprise. Here, we quantify the epistemic shift from the LLM’s prior beliefs about a hypothesis to its posterior beliefs after gathering experimental results. To efficiently explore the space of nested hypotheses, our method employs a Monte Carlo tree search (MCTS) strategy with progressive widening using surprisal as the reward function. We evaluate AutoDiscovery in the setting of data-driven discovery across 21 real-world datasets spanning domains such as biology, economics, finance, and behavioral science. Our results demonstrate that under a fixed budget, AutoDiscovery substantially outperforms competitors by producing 5-29% more discoveries deemed surprising by the LLM. Our human evaluation further reveals that two-thirds of discoveries made by our system are surprising to domain experts as well, suggesting this is an important step towards building open-ended ASD systems.

AUTODISCOVERY:
Open-ended Scientific Discovery via Bayesian Surprise
Dhruv Agarwal"
Bodhisattwa Prasad Majumder
Reece Adamson Megha Chakravorty To Satvika Reddy Gavireddy
Aditya Parashar"
Harshit Surana"
Bhavana Dalvi Mishra
Andrew McCallum Ashish Sabharwal
8
Peter Clark
University of Massachusetts Amherst
AAlen Institute for AI CCapitll One
dagarwal0cs, umass. edu, bodiisattwan@lllenii org equal contributions
htter//gttabb.oo/lllllaaaaaaddet
Abstract
The promise of autonomous scientific discovery (ASD) hinges not only on answer- ing questions, but also knowing which questions to ask. Most recent works in ASD explore the use of large language models (LLMs) in goal-driven settings, research questions guide hypothesis generation.
relying on numan-speciftie However, scientific discovery may be accelerated further by allowing the Al system to drive exploration by its own criteria. The few existing approaches open- ended ASD select hypotheses based on diversity heuristics or subjective proxies for human interestingness, typically vast hypothesis space, and the latter suffers from imprecise definitions. This paper presents AUTODISCOVERY method for open-ended ASD that stead drives scientific exploration using Bayesian surprise. Here, we quantify the epistemic shift from the LLM's prior beliefs about a hypothesis to its posterior beliefs after gathering experimental results. To efficiently explore the space of nested hypotheses, our method employs Monte Carlo tree search (MCTS) strategy with progressive widening using surprisal as the reward function. We evaluate AU- TODISCOVERY the setting of data-driven discovery across real-world datasets spanning domains such biology, economics, finance, and behavioral science. Our results demonstrate that under a fixed budget, AUTODISCOVERY substantially outperforms competitors by producing 5-29% more discoveries deemed surprising by the LLM. Our human evaluation further reveals that two-thirds of discoveries made by our system surprising to domain experts well, suggesting this is important step towards building open-ended ASD systems.
the former struggles meaningfully navigate the
1 Introduction
There has been surge of recent progress in using large language models (LLMs) for autonomous
scientific discovery (ASD) MMjjumder et al. 2024b Wang al. 2024 Lu et 2024 Skarlinski
et al. 2024 Majumder et al. 2025 Gottweis et al. 2025 Huang et al. 22225. J. Most prior works
operate within "ggoal-drven' setting: given some data the user is required provide a research
collection of datasets (data-driven discovery) related papers (litcrature-driven discovery).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Experiment and Hypethesis
LLM Prior
BCC
Budget=6
4/6
Dataset: Hurricanes their names and fatalities Noture Bosnix 2003
Experiment
4/5
Programmer/
Diecunor/
- Hernifa?
Betales,
0/1
LLM Posterior
Disovery
H.W D
Hypothesis
False
1/1
2/3
Maximumn hunikare the domoge positielly corretind.
Bayesian Surprise!
X P Hemody)
Budgetrd
Buitgetnó
Buxiget
Busgettn
1/1
1/1
Theforkkhily Kone's sanijiann prokitoons the property domsgge
2012.
Repeated Sampling ASulpesal-11
Linear Surppisaa-
Greedy SSurpisal=2
Beam
Search Progressive Widening #Surprisam
#Surprsal=2
Overview of AUTODISCOVERY A method for open-ended ASD that is guided by Bayesian surprise. We elicit LLM prior and posterior beliefs about hypotheses sampling, and use surprisal reward function within an MCTS procedure to hypotheses by trading-off exploration and exploitation the hypothesis space in search for surprising discoveries.
Figure I:
question; then LLM is prompted (1) generate hypothesis (i.e., assertion about the true state of the world) that is relevant to the research question, (2) propose an experiment to test the hypothesis, (3) generate and execute code to perform the experiment, and (4) analyze the experiment results to derive a conclusion.
On the other hand, while there has been some work on ideating research ideas from the literature (Sanngler et al. 2014 Back et al. 2025), there has been limited investigation of the full "open-ended" setting, where the ASD system itself explores more broadly by generating hypotheses according to its own measures research promise, executing the aforementioned steps, and then using results to propose new hypotheses in never-ending process (akin to the workflow of human scientist). A key challenge, then, is that search-which hypotheses should be investigated next that will lead to novel, impactful scientific discoveries?
Prior efforts open-ended automated discovery have used search strategies such as rejection sampling and evolutionary algorithms with either a human in the loop Yamada et al. 2025 Gottweis et al. 2025 Jansen al. 2025) to filter the generated hypotheses (thus, not fully autonomous) or using automatic rewards such diversity and LLM-as-judge proxies for human interestingness, novelty, or utility [Zhang et al. 2023a Lu et al.) 2024 Diversity alone, however, is insufficient due to massive search space of hypotheses in real-world scientific domains, where not all regions may equally likely to lead to discoveries. Moreover, the ability to sample diverse sequences from an LLM has itself been shown to be challenging [Lanchantin al. [025 Krishnamurthy et al. [024]. Human proxy metrics, c.g., interestingness, are not suitable either due to their subjective nature, with human scientists demonstrating high degree of disagreement [Ceci and Peters 1982, Cicchetti
1991 Rothwell and Martyn, 2000 Weller 2001 Pier et al. 2018), making their automated proxies unreliable. Therefore, it remains unclear: how diverse hypotheses be explored at scale and what automatic metrics may guide scientific discovery?
In this work, we address these questions and propose AUTODISCOVERY method for open-ended ASD that guided by Bayesian surprise [Itti and Baldi 2005), which quantifies how data affects natural or artificial observer by measuring the distance between its posterior and prior belief distributions (see Fig. D. Our choice motivated by recent findings from Shi and Evans 2023), which show that the improbability surprisal of hypothesis is often a strong predictor of scientific impact. To automate the computation of surprisal, we use an LLM model itself as the Bayesian observer. In doing, make simplifying assumption and focus developing procedure to expand the knowledge frontier the model itself We mechanize this frontier by deriving prior and
We anticipate that this knowledge frontier will rapidly approach that of humans as models, especially setieval-augmenter ones, continue to advance.

---

posterior distributions about an LLM's belief about hypotheses, without and with conditioning on an empirical evaluation of the hypotheses given data, respectively.
To sample hypotheses with high surprisal, we propose a Monte-Carle tree search (MCTS) [Coulom
2006) procedure with progressive widening [Couêtoux et al., 201T], which provides principled
mechanism to balance exploration and exploitation of the vast hypothesis search space. In Figure [ we show how MCTS is able to navigate the search space and find the most number of surprising hypotheses under a fixed budget, where surprisals may even be discovered from non-surprising nodes.
We evaluate AUTODISCOVERY in data-driven discovery (DDD) [Majumder et al., 2024b 2025 where the imput to a discovery task is a collection of datasets and a programming environment making it suitable for evaluating fully autonomous discovery. In experiments over 21 real-worke datasets across behavioral science, economics, biology, and finance, we find that AUTODISCOVERY finds 5-29% more number of hypotheses surprising to the LLM agent compared to strong search baselines. In a human study, we further find that two-thirds of surprising discoveries found by AUTODiScOVERE correlate with human surprisal, indicating that optimizing for Bayesian surprise may be an effective proxy to guide real-world open-ended ASD.
In summary, contributions are as follows:
We provide the first formal definition of surprise within the context of autonomous scientific discovery, inspired by prior work on Bayesian surprise,
We present novel method, AutoDS, that combines this notion of surprise with MCTS to perform hypothesis search in an open-ended setting (no goal specified).
We show that AUTODISCOVERY outperforms competitors by 5-29% at finding discoveries that are surprising to the LLM in extensive data-driven discovery experiments, spanning 21 real-world datasets. In human study with 500+ hypotheses, find that 67% of the surprising discoveries made by DISODESC are surprising to domain experts, showing promise for real-world open-ended ASD.
2 Preliminaries
DDD formalization. Following Majumder al. [2025], we define a data-driven hypothesis H in H (the space of such hypotheses) as a natural language statement that defines relationships (r) between set of variables (v) under contexts (c). Further, given a dataset D, the truth value for H may be inferred using verification procedure Vp H. + (supported, unsupported), where the space of valid Vo is potentially any executable Python program.
Discovery agent. We call any agent capable of generating and verifying a data-driven hypothesis given dataset and a programming environment discovery agent. In this work, we use multi-agent architecture [Majumder et al.) (2024a], composed of LLMs that collaboratively propose experiment plans, write and execute Python code conduct those experiments, critique and fix mistakes, and analyze their results. Figure [ shows the input/output flow for our discovery agent. Please see the appendix for the complete details.
Open-ended DDD. Unlike the goal-driven setting, where the task is to search for a verifiable hypothesis that may answer research question provided explicitly as input, the open-ended setting requires a discovery agent to iteratively generate and verify hypotheses given only the dataset to make discoveries. To improve search efficiency, an exploration strategy may be used that repeatedly invokes the discovery agent in its inner loop and typically terminates after a predefined budget is exhausted.
AUTODISCOVERY. Autonomous Discovery via Surprisal
We present AUTODISCOVERS (Fig. []. method for open-ended autonomous scientific discovery that leverages LLMs to identify and prioritize hypotheses based on surprisal-as principled measure of belief shift induced by experimental evidence. To this end, we formalize surprisal using Bayesian framework, introduce a practical method for belief elicitation via LLM sampling, and describe how surprisal can guide efficient exploration of the hypothesis space via tree-based search.
No wet lab experiments.
3

---

3.1 Measuring Surprisal
Consider a dataset D. a data-driven hypothesis H H. and its verification procedure VD. For a given agent, let 0ي E [0, 1] denote its belief about the support for H in D, i.c., probability that H may be verifiable by some VD. We assume the agent uncertain about the value of f and we model this uncertainly using Beta distribution, i.e., Ofr ~ Beta (a, .). In particular, let P(O,)) denote the agent's s prior (Beta) distribution for the value of 0Hr given only the hypothesis, and P(O VD) its posterior (Beta) distribution after observing results from the verification procedure VD.
We first discuss how to estimate these two distributions by querying the agent, and then describe how to use these estimated distributions to compute the agent's surprisal.
Belief Elicitation via Sampling, In order to empirically estimate LLM agent's 's prior and posterior distributions of 8g, we use the boolean responses from the LLM about the truth value for H, before and after revealing results from (H) in the prompt. The empirical frequencies of "true" " responses (Kpriir and kpos)) are treated as Bernoulli outcomes, which are used to make Bayesian update to estimate the prior and posterior distributions of Hz denoted Pesa (fxx and ( VD)), respectively, as follows:
conjugacy to propose a simple procedure that samples
(0s(0)) = Bet 1 + kppioo: 1 + n kprior): ): and (0 Vn) := Beta(On + kiprior Appsat + (n kprior) + (n - Kposs))
(2)
We will henceforth use (0n( and (0n VD) as empirical estimates of P(OH) and P(0μ VD).
Bayesian surprise. Inspired by [tti and Baldi 2005), we propose the use of Bayesian surprise, a distance measure between the prior and posterior beliefs, to quantify the magnitude of change in beliefs that occurs when a discovery agent observes results from (D(H). Specifically, we define
BH, Vp) == DKL(((0n) (D)) || P(0#)).
(3)
Surprisal. We now formalize the intuition that surprisal arises when beliefs update in a directionally significant way. We say that a surprisal has occurred if there is a shift in the agent's expected belief about H, i.e., change in leaning about its beliefs (e.g., from supported to unsupported), given evidence from VD: specifically if Ep [0H] lies on different side of decision threshold (typically, we set 8 = 0.5) than Ep(ax) (0)). To capture both the surprise due directional change and its informational significance, we define Bayesian surprise under belief shif as
BS(H VD), if (Er(onlVo) A Ep(onlo, [0H] A (P((,)) [0H]
(001) [0H] 6)( [0H] 6) S 0
BS ( (H, Vp)
0,
otherwise.
We can then formally define surprisal as an indicator function
S(H, Vp) == 1[S (H, Vp) > 0j.
(5)
which captures whether belief shift about hypothesis has occurred on observing new evidence.
3.2 Search using Surprisal-driven MCTS
Our goal is expand the knowledge frontier of the LLM by proposing hypotheses that yield surprisal under verification. However, naïve strategies, such as repeated independent sampling and greedy search by LLM, (a) struggle to reliably generate diverse hypotheses, and (b) do not optimally balance exploration and exploitation of the vast hypothesis search space H.
To address both problems, AUTODISCOVERY uses Monte Carlo tree search (MCTS) [Coulom 2006] guided by surprisal as the reward function. In particular, we build a hicrarchy of diverse hypotheses by iteratively conditioning the LLM on branch of hypothesis sequences composed of prior discoveries to sample & new experiments to investigate further. To prioritize the expansion of nodes that may be more likely to yield surprisal, use upper- upper-confidence bound on trees (UCT)
Kocsis and Szepesvári] 2006) as a principled method to trade off exploration and exploitation, a
strategy commonly applied in large, combinatorial spaces (e.g., game playing [Gelly et al., 2012) and program synthesis [Lim and Yoo 2016)).
We assume an uninformed prior Beta(1, 1) when no hypothesis provided.
'The first conditional clause ensures that the expected prior and posterior beliefs lie on different sides of 6, while the second ensures that they not both equal to 8.

---

Procedure. We build a search tree where each node represents a hypothesis H E H. and edges correspond to sampling steps ex- ecuted by discovery agent to generate new hypotheses. The algorithm proceeds in four phases in each iteration:
Algorithm MCTS with Progressive Widening
Require: E Ryu; [0,
1: procedure EXPAND(H H
2:
if [children( H parem) |
< kN (Hprrc)) then
D- Progressive Widening
3:
H ~ LLM(- {h E path(H root)))
4:
children( (H parem))
5:
1. Selection: Starting from the root, the tree is traversed by selecting node H pareeu for expansion that represents region with high potential for surprisal. particular, we use the UCT acquisition function as described in Eq. c where (H) the number of visits to any node in the sub- tree rooted at H. i.e, subtrce H), and C is a tunable constant that controls the strength of exploratory behavior. The first term computes the average surprisal from node and encourages exploitation of known good regions, while the second term encourages exploration of new nodes.
return H
6:
else
7:
H arg m (hapeereeedddee)ee)))) (CT(N)
8:
return EXPAND(H)
9: end
end procedure
2. Expansion: We then sample a child hypothesis H by conditioning on all prior experiments and results the branch from H parent to the root (see Algorithm| T Since is intractable to sample all possible children at node, employ progressive widening (Couêtoux et al. 2011]. which dynamically increases the number of children node must have based on its visitation count. Importantly, this encourages search to revisit multiple promising regions within the search space before expanding any one them in an unbalanced manner.
3. Execution The sampled hypothesis H from expansion is evaluated by executing its correspond- ing Vn and estimating its surprisal S(H, using the belief elicitation procedure ($ 3.1.
4. Backpropagation: The estimated surprisal is propagated back through the tree from H the root, updating surprisal and visitation statistics for each node in the path.
E Ehesubtreo(1)) S(h, Vith)
2 log N (HHparem))
UCT(H)
+C.
(6)
N(H)
N(H)
Exploit
Explore
3.3 Deduplication via LLM-based HAC
Despite incorporating a search strategy to guide discovery, hypothesis generation in AUTODIScOV- ERY may sample semantic duplicates. To identify these, we propose an LLM-based hierarchical agglomerative clustering (HAC) procedure (inspired by Zhang et al. [2023b) that combines similarity within a textual embedding space with LLM reasoning to identify semantically equivalent hypotheses. We run this procedure once after the search budget is exhausted.
We start by constructing an HAC tree using text embeddings of hypotheses. For every merge decision between pair of clusters identified in the HAC linkage matrix, two representative hypotheses- each with its structured breakdown context, variables, and relationships- passed to LLM (GPT-4o, in our experiments) to determine whether they are semantically equivalent. Specifically, we sample boolean response from the LLM about whether the structured hypotheses are equivalent. If the pro- portion of "true" responses exceeds 0.7, we merge the clusters and propagate the updated assignment before proceeding with the next linkage step. If it does not, the clusters remain independent. The iteration proceeds until no further merges remain to be evaluated, either because all candidate pairs involve clusters whose descendants have already been labeled non-duplicates or because the LLM has reviewed every remaining cluster pair.
'Note that our departure from the "simulation" step is motivated by the fact that execution of (H) (a) is inexpensive and (b) does not alter the state for subsequent actions in our setting (unlike, e.g., in game playing).
5

---

Experiments
Our empirical evaluation assesses the effectiveness various methods for the task of open-ended DDD. The input for the task is dataset D, its associated metadata, and budget (which we set to 500) specifying the total number of hypotheses the agent is allowed to explore and verify. The goal of the agent is to discover many surprising, but verifiable (over D), hypotheses possible. We assess performance on this based on (a) the number of unique hypotheses generated, and (b) the number of surprisals they produce under the fixed experiment budget.
4.1 Datasets
We utilize total of 21 datasets (D) from the following benchmark sources spanning areas such as biology, economies, finance, and behavioral science. We selected the range of datasets to maximize data-shape heterogeneity, scientific salience (associated with top-tier publications), and breadth of domains in our evaluation.
DiscoveryBench MMajumder et al. 2025], a comprehensive benchmark designed to assess the ability of large language models to autonomously search for and verify hypotheses using asso- ciated datasets. DiscoveryBench comprises 264 real-world discovery tasks sourced from pub- lished papers across six domains (e.g., sociology, engineering) spanning across 14 scientific datasets. We selected the following five datasets and associated metadata from DiscoveryBench representative sample: freshwater-fise nls-bmi, nls-ses, als-incarceration, and requirement- gnnuurroment-ennnnneinnn
BLADE [Gu et al.] [024] is benchmark evaluating language agents justifiable sci- entific data analysis using real-world datasets and expert-defined analysis decisions. We all 15 datasets from BLADE in our work: affairs, amtl, boxes, caschools, conversation, crofoot, fertility, fish, hurricane, mortgage, panda_nuts, reading. soccer, teachingratings. and toy.
SEA-AD (Hawrylycz et al. [2024] is multimodal cellular atlas of the human brain across the Alzheimer's s disease spectrum, developed by the Allen Institute. We utilize the donor-level metadata, including demographic details, cognitive status, and neuropathological assessments.
4.2 Baselines
We rigorously evaluate our method against common repeated sampling baselines as well as tree-based search methods. To ensure fair comparison, we keep the following constant across methods the reward function (here, surprisal) and the exploration budget (500 in all experiments). All baselines and AUTODISCOVERY to a Python environment with available statistical and data analysis packages (e.g., sklearn), and can generate Python code to run in execution environment to verify a hypothesis. The agent can also install additional Python packages to successfully execute proposed verification experiment.
the same discovery agent with GPT-4o. The discovery agent has access
Repeated (independent) sampling generates hypotheses in parallel, context-free manner, i.c. without knowledge of other experimental results within the same run, using ancestral sampling. (schematic in "Repeated Sampling"; Figure ID.
Last-K (linear) sampling a context-aware version of repeated sampling with a strictly sequential exploration strategy in which each new experiment directly follows from the most recent one, forming a single, linear path of reasoning. To accommodate the context length limitation of LLMs, we retain only the last K 100 parent nodes context during hypothesis generation. See search schematic in "Linear"; Figure [
Greedy tree search is one of two tree-based search baselines we evaluate. It focuses on exploitation by always selecting the highest-value node at each step to condition on for hypothesis generation, resulting in a narrow, semi-linear search path (schematic in "Greedy"; Figure T.) This translates to an MCTS variant with the exploration constant C' 0.
Beam search is a tree-based exploration strategy (inspired by [Li et al. 2025)) that restricts search at each level by retaining only the top-b candidate states (i.e., beam width 6), ranked by surprisal and visitation statistics. We set both the branching factor and beam width to 8. At every level of
the tree, 64 candidate states are generated, and the top 8 are retained for expansion at the next level. Unlike MCTS, beam search performs breadth-firs expansion and aggresively prunes the search tree, making it more sensitive to early ranking errors (schematic in "Beam"; Figure

---

MCTS
(b)
126
Beam
Greedy
Seardh gil pillgeerg Crudtegt
125
RC13
Beam
Greedy
Rep. Samp
Unear
26 Hypotheses Evaluated
166
Hypothoss Evalusted
MCTS
Beam
Gready
Rep. Samp.
Linear
% Surprisals
30-
20
10
boxes
crofoot
reading
loy
amd
500
bm
incarceration
fis
evolution
mortgage
conversation
teachingratings
parda
saccer
affairs
seg
caschoels
requirements
fertilty
humicane
Figure 2: Search Performance. (a) Cumulative number of surprisals discovered across timesteps within budget of 500 evaluations, averaged over 21 datasets. (b) Search efficiency gradient computed using sliding window of 10 iterations. (c) Number surprisals discovered per dataset. Takeaway: AutODiscoveRy outperforms baselines, including other tree-search methods, in both search efficiency and number of surprisals discovered.
5 Results and Discussion
We first demonstrate via human studies the correlation between LLM and human surprisal and showcase also a correlation with human interestingness and utility. We then show a comparison between proposed Bayesian surprisal and alternative automatic proxies reward functions to guide search using MCTS. We then compare AUTODISCOVERY with strong search baselines on the ability optimize for surprisal. Finally, manually validate the accuracy of various aspects within the discovery agent workflow. Complete experimental details are provided in the appendix.
5.1 Bayesian Surprise versus other Automatic Rewards
We run an analysis of AUtODiscOVERY swapping out the Bayesian surprise reward for other automatic metrics that are commonly used to assess hypothesis quality in ASD--LAN interestingness and LLM utility. We also include direct LLM surprisal provide a comparison with an alternative mechanism for surprisal elicitation. Each of these metrics is operationalized by prompting the LLM provide boolean responses to their respective questions, e.g., "is this hypothesis interesting to you?". then using the average number of "yes" responses as reward in MCTS.
Bayesian surprise correlates with expert sur- prisal. We assess performance by evaluat- ing how often the hypotheses generated ing each automatic reward result in human sur- prisal. Each hypothesis annotated by 3 STEM MS/PhDs with total of 1,620 LLM surprisal hypotheses pooled from 4 datasets across MCTS runs with four automatic rewards. Our results in Table show that across annotator familiarity, AUTODISCOVERY with Bayesian surprise finds hypotheses that have a much higher correlation (67%) with human surprisal than any of the other automatic rewards, with a 95% confidence interval of [0.63, 0.71] computed using bootstrapping with 10,000 re-samples.
Reward
H. Interesting
H. Utility
Bayesian Surprise
0.73
0.79
L.I. Surprisal
0.76
0.80
I.I Interestingness
0.74
0.78
LLM Utility
0.73
0.78
Table 1: Human Interestingness and Utility Scores Across Automatic Rewards. Average human ratings for interestingness and utility for four automatic rewards used in MCTS.
7

---

Familiarity
Bayesian Surprise
LLM Surprisal
LLM Interesting
LLM Utility
IAA
Low
0.64
0.13
0.18
0.22
0.61
Medium
0.66
0.12
0.13
0.24
0.51
High
0.62
0.08
0.14
0.19
0.71
Overall
0.67
0.11
0.15
0.21
Table Human Surprisal Across Automatic Rewards. Average human surprisal ratings across different annotator familiarity levels for four automatic rewards used in MCTS: Bayesian surprise, direct surprisal, interesting(ness), and utility with inter-annotator agreement (IAA). Takeaway: Bayesian surprisc results in the highest number of human surprisals, with the next best reward showing average score lower by 0.46 points.
Interestingness and utility lack clear semantics. We ran another study to assess whether these automatic metrics correlate with other human notions such interestingness and utility, Our results Table show that though Bayesian surprise clearly does, so do all the other metries. This suggests that eliciting human interestingness and utility may be difficult due to their subjective nature. A corollary of this is that deriving automatic versions of such metrics without clear semantics is likely not useful for guiding open-ended ASD.
Optimizing for Bayesian Surprisal
MCTS outperforms other search strategies. In Fig. 2 2a), we show the cumulative number of surprisals discovered across timesteps, averaged over all datasets. We find that all tree search base- lines outperform repeated sampling and linear search, with MCTS AutoDIscovERy show- ing the highest efficiency for discovery as well as the highest number of surprisals collected. Notably, as shown in Fig. b)). MCTS shows minimal reduction in search efficiency across time, unlike other baselines, including greedy tree search and beam search. This indicates room for scaling AUTODISCOVERY with higher budget continue collecting surprising discoveries. In Fig. [cc.. we show that the aggre- gate trend in search performance holds across the evaluated datasets, with AUTODISCOV ERY (MCTS) showing the best performance in 17 out of 21 datasets.
tedids
unvernatton
eeclibor
cande
unt
Exschook
ahork
culud
LLM beliefs shift differently across domains. In Fig. 3) we plot B- i.c. the KL diver- gence between the prior and posterior beliefs of surprisals, for the hypotheses found using AUTODISCOVERY along with the direction of the belief shift. Our analysis reveals different directional tendencies across domains, with
200
30
50
100
Figure 3: Belief shift across datasets. Bayesian sur- prise under belief shift for surprisals discovered using AUToDISCOVERY, grouped by domain and direction of shift.
higher proportion of surprisals shifting from supported to unsupported. We also observe lower KL divergence when the model's s beliefs shift towards supporting the hypothesis, possibly indicating the need for greater evidence in confirmatory studies than in falsification ones [Huang et al.) 2025
5.3 Validating the Discovery Agent Framework
To evaluate the faithfulness of the verification in AU- ToDISCOVERY, we evaluate two critical pieces: (1) whether the experiment is valid, i.e., whether it can be implemented with the available data and helps to confirm hypothesis, and (2) whether the experiment was faith- fully implemented. We sampled 175 nodes for the MCTS tree run on nls_ bmi from Discovery with associ- ated experiments, code, and analysis, and asked two annotators to annotate the artifacts for experiment
Artifact
% Validity
TAA
Experiment Implementation
98.58
0.97
98.01
0.98
Deduplication
90.76
0.75
Table 3: Discovery agent validity

---

and implementation validity. Table 3 demonstrates very high validity (>95%) for both experiment and implementation artifacts, also with a very high inter-annotator agreement (Gwet's AC1 as 0.95).
To evaluate the deduplication via L.LM-based HAC pipeline, we subsampled 120 pairs of hypotheses from MCTS trees on all five BiscoveryBench datasets. Two hypotheses in pair are sampled from the same HAC clusters. We asked a range of 1-3 annotators if the two hypotheses from a pair are structurally equivalent or not. Table 3 shows clusters found by our LLM- HAC method indeed group duplicate hypotheses with 91 % validity, with good annotator agreement (Gwet's 's ACI as 0.75).
Additional Related Work
As noted carlier, there has been an explosion of interest in A--asisted/automaaee discovery in the last few years, e.g., AlScientist (Yamada et al. 2025]. CodeScientist [Jansen et al. 2025], J, AgentLab
[Schmidgall et al. 2025). Popper [Huang et al. 2025]. HypoBench [Liu et al. 2025 However, these
systems are mainly goal-driven, performing start-to-finish experimentation given clear research goal, rather than iterative, open-ended, goal-free exploration (our context). While there has been some work initial hypothesis generation, in particular from the literature (Baek et al. 2025
Spangler
et al. 2014). our goal is iterative generation and search over a large space.
While our framework is general, we have evaluated it in the context of data-driven discovery, rich context for science [Majumder et al.) 2024b 2025 Gu et al. (2024.. While some commercial tools, c.g., [Bailis et al. 2017]. J. ofter programmatic ways to exhaustively sweep size-bounded hypothesis space, our goal is different, namely, to heuristically search a much larger space using LLM.
More generally, surprise (or equivalently, failed expectations) have played historically important role in AI, leading to work on encoding expectations and failure-driven learning [Riesbeck] 1981 Schank and Abelson 1988 [Shaank 1983) (indeed, almost all learning can be viewed as responding to failed expectations). We adopt formal notion of surprise here and show how can be successfully applied for guiding open-ended exploration. We also note the close connection between Bayesian surprise and other metrics from information theory, e.g., mutual information [Gao et al. 2024].
7 Conclusion
We introduce a formal framework for Bayesian surprise autonomous scientific discovery and propose AUTODISCOVERY, a method for open-ended scientific discovery that uses this frame- work alongside MCTS to find hypotheses that can expand an L.M''s knowledge frontier. Through evaluations across 21 real-world datasets and comprehensive human study, we demonstrate that AUTODISCOVERY not only outperforms strong baselines in making surprising discoveries but also aligns well with human judgements of surprise. While we remain cautious about open-ended AI systems for scientific discovery without sufficient guardrails, academic skepticism, and peer review, our results underscore the potential benefit system such as AUTODISCOVERY may provide in accelerating science
References
J. Back, S. K. Jauhar, S. Cucerzan, and S. J. Hwang. Researchagent: Iterative research idea generation scientific literature with large language models, 2025. htll:tt//://aart org/ abs/ 2404.07733
P. Bailis, E. Gan, S. Madden, D. Narayanan, K. Rong. and S. Suri. Macrobase: Prioritizing attention in fast data. In Proceedings of the 2017 ACM International Conference on Management of Data pages 541-556, 2017.
S. J. Ceci and D. Peters. Peer review: A study of reliability. Change: The Magazine of Higher Learning, 14(6):44-48 1982.
D. V. Cicchetti. The reliability of peer review for manuscript and grant submissions: cross- disciplinary investigation. Behavioral and brain sciences, 14(1):119-135, 1991.

---

A. Couêtoux, J.-B. Hoock, N. Sokolovska, O. Teytaud, and N. Bonnard. Continuous upper confidence trees. Learning and Intelligent Optimization: 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Pupers 5, pages 433-445 Springer, 2011.
R. Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pages 72-83. Springer, 2006.
Y. Gao, N. Gu, J. Lam, Henderson, and R. Hahnloser. Evaluating unsupervised argument aligners via generation of conclusions of structured scientific abstracts. In Y. Graham and M. Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 151-160, St. Julian's ss Malta, Mar. 2024. Association for Computational Linguistics. doi: 1/205/5//1122222aalllaaoao/ 14. URL https://acllntthllee wwwp://o.ro/22.111..111... wac. 14/
S. Gelly, L. Kocsis, M. Schoenauer, M. Sebag, D. Silver, C. Szepesvári, and O. Teytaud. The grand challenge of computer Go: Monte carlo tree search and extensions. Commun. ACM, 55(3):006-133 113, Mar. 2012. ISSN 0001-0782. doi: 10.1 00.1145/003348.20993337
https
//doi. 11411119112244 2093574
J. Gottweis, W.-H. Weng, A. Daryin, T. Tu, Palepu, P. Sirkovic, A. Myaskovsky, F. Weissenberger, K. Rong, R. Tanno, K. Saab, D. Popovici, J. Blum, F. Zhang, K. Chou, A. Hassidim, B. Gokturk, A. Vahdat, Kohli, Y. Matias, A. Carroll, K. Kulkarni. N. Tomasev, Y. Guan, V. Dhillon, D. Vaishnav, B. Lee, T. R. D. Costa, J. R. Penadés, G. Peltz, Y. Xu, A. Pawlosky, A. Karthikesalingam, and Natarajan. Towards an AI co-scientist, Feb. 2025. UtL/ttt::/ arxiv org abs/ 18864 arXiv:2502. 18864 [cs].
K. Gu, R. Shang. R. Jiang, K. Kuang, R.-J. Lin, Lyu, Y. Mao, Y. Pan, T. Wu, Yu, Y. Zhang, T. M. Zhang, L. Zhu, M. A. Merrill, J. Heer, and T. Althoff. Blade: Benchmarking language model agents for data-driven science. arXiv, 2024. URL [ott[ntts//arrae www.:/orrroo..o 22t.//408.0800
M. Hawrylycz, E. S. Kaplan, K. J. Travaglini, et al. SEA-AD is a multimodal cellular atlas and resouree for Alzheimer's disease. Nature Aging, 4:1331-1334, 2024. doi: 10.1038/ -43587-024-00711 00719-8. URL https://///o....... org/ 10.
K. Huang, Y. Jin, R. Li, M. Y. Li, E. Candès, and J. Leskovec. Automated Hypothesis Validation with Agentic Sequential Falsifications, Feb. 2025. hattp://aaxxin org/ abs/ arXiv:2502.09858 [es].
aataaabb//222222// 2502.09858
L. Itti and P. Baldi. Bayesian surprise attracts human attention. Advances in neural information processing systems, 18, 2005.
P. Jansen, O. Tafjord, M. Radensky, P. Siangliulue, T. Hope, B. D. Mishra, B. P. Majumder, D. S. Weld, and P. Clark. Codescientist: End-to-end semi-automated scientific discovery with code-based experimentation. arXiv preprint arXiv:2503..27008 2025.
L. Kocsis and C. Szepesvári. Bandit based monte-carlo planning. In European conference machine learning, pages 282-293. Springer, 2006.
A. Krishnamurthy, K. Harris, D. J. Foster, C. Zhang, and A. Slivkins. Can large language models explore in-context?, Mar. 2024. URL http:://rxiv org httpg/bb/240...15550 arXiv:2403.15371 [es].
J. Lanchantin, A. Chen, S. Dhuliawala, Yu, J. Weston, S. Sukhbuatar, and Kulikov. Diverse preference optimization. arXiv preprint 2rXiv:2501.181011 2025.
X. Li, F. Kaiyom, E. Z. Liu, Y. Mai, Liang, and Hashimoto. Autobencher: Towards declarative benchmark construction. In The Thirteenth International Conference on Leurning Representations, 2025.
J. Lim and S. Yoo. Field report: Applying monte carlo tree search for program synthesis. In Search Based Software Engineering: 8th International Symposium, SSBSE 2016, Raleigh, NC, USA, October 8-10, 2016, Proceedings 8, pages 304-310. Springer, 2016.
10

---

H. Liu, S. Huang, J. Hu, Y. Zhou, and C. Tan. Hypobench: Towards systematic and principled benchmarking for hypothesis generation. arXiv preprint arXiv:2504.11524, 2025.
C. Lu, C. Lu, R. T. Lange, Foerster, J. Clune, and D. Ha. The Al Scientist: Towards Fully Automated Open-Ended Scientific Discovery, Sept. 2024. URL htttt://aaxiii... owwooogg/aaao. 2408 .62922 arXiv:2408.06292.
B. P. Majumder, H. Surana, D. Agarwal, S. Hazra, A. Sabharwal, and P. Clark. Data-driven discovery with large generative models. arXiv, 2024a. ltttss//t// www..orggsss......... htt//bbs/442... 2402. 13610
B. P. Majumder, H. Surana, D. Agarwal, S. Hazra, A. Sabharwal, and P. Clark. Data-driven discovery with large generative models. ICML, 2024b.
B. Majumder, H. Surana, D. Agarwal, B. D. Mishra, Meena, Prakhar, Vora, T. Khot, A. Sabharwal, and P. Clark. Discoverybench: Towards data-driven discovery with large language models. In The Thirteenthh International Conference on Learning Representations, 2025. URL https://openreviert met//ooruiid-vyyla.e...
E. L. Pier, M. Brauer, A. Filut, A. Kaatz, J. Raclaw, M. J. Nathan, C. E. Ford, and M. Carnes. Low agreement among reviewers evaluating the same nih grant applications. Proceedings of the National Academy of Sciences, 115(12):952-22957 2018.
C. Riesbeck. Failure-driven reminding for incremental learning, IJCAI, 1981.
M. Rothwell and C. N. Martyn. Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone? Brain, 123(9): 1964-1969, 2000.
R. Schank. Dynamic Memory: A Theory of Reminding and Learning in Computers and People. Cambridge Univ Press, 1983.
R. C. Schank and R. Abelson. Scripts, Plans, Goals, and Understanding. Routledge, 1988.
S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, Z. Liu, and E. Barsoum. Agent laboratory: Using Ilm agents as research assistants. ArXiv, 2501.04227, 2025.
Shi and J. Evans. Surprising combinations of research contents and contexts are related to impact and emerge with scientific outsiders from distant disciplines. Nature Communications,
14(1):6641, Mar. 2023. ISSN 2041-1723, doi: 10. 00.10088/114477003-37741111 URL https
7/www. nature. comm
M. D. Skarlinski, S. Cox, J. M. Laurent, J. D. Braza, M. Hinks, M. Hammerling, M. Ponnapati, S. G. Rodriques, and A. D. White. Language agents achieve superhuman synthesis of scientific knowledge. arXiv preprint arXiv:2409. 13740, 2024.
S. Spangler, A. D. Wilkins, B. Bachman, M. Nagarajan, T. Dayaram, Haas, S. Regenbogen, C. Pickering, A. Comer, J. Myers, I. Stanoi, L. Kato, A. Lelescu, J. Labrie, N. Parikh, A. M. Lisewski, L. Donehower, Y. Chen, and O. Lichtarge. Automated hypothesis generation based on mining scientific literature. Proceedings of the 20th ACM SIGKDD international conference Knowledge discovery and data mining, 2014. URL http: acm. org/ citatiionn.../// .efiaiiiiin.comiiii...... cfm?id= 2623667
Q Wang, D. Downey, H. Ji, and T. Hope. SciMON: Scientific Inspiration Machines Optimized for Novelty, June 2024. URL http://arxix/ http///xivv..//////////a////.avl http///xxvv.oog/bb/.///// wwwg///////aa. 14259 arXiv:2305.1422
A. C. Weller. Editorial peer review: Its strengths and weaknesses. Information Today, Inc., 2001.
Y. Yamada, R. T. Lange, C. Lu, S. Hu, C. Lu, J. Foerster, J. Clune, and D. Ha. The ai scientist-
v2: Workshop-level automated scientific discovery via agentic tree search, 2025. URL https org/ abs 2504. 08066
Zhang, J. Lehman, K. Stanley, and Clune. OMNI: Open-endedness via models of human notions of interestingness. 2.Xiv23366001111.. 2023a.
Y. Zhang, Z. Wang, and J. Shang. Clusterlim: Large language models as guide for text clustering, 2023b. URL https: wttp:/iww.rr//////25555/55/ 14871

---

Appendix
8
Discovery Agents
8.1 Finite State Machine
When evaluating a hypothesis, agents work collaboratively within a shared context. Speaker selection determined by finite state machine where transitions are determined based on the prior speaker and the content of the last messsage. Algorithm 2 provides pseudocode for the speaker selection transitions.
Algorithm 2 Speaker Selection Algorithm
procedure SELECT NEXT SPEAKER speaker, last_ response) if' mnst_speaker-= = "user puser_proxy" then
2:
return
4: else if last_speaker = "bypothesissgeeeerator" then
5: return exper_mment_prograamer
6:
else if hist_speaker> "exper_imen__prorramee then
7:
return cwee_ooeeeeeeeeeeeeeeeeeeeeeeeee executor
8:
else if last_speaker "code_executor" then
9:
return
10:
else if last_speakert "experiment, analyst" then
11:
last rest_responseeeree = True and code failure_coontt count < then
12:
code fitee_faluree_eeeeeee count +- self.code failure_count- count 1
13:
return experiment_programee
14:
else
15:
self.code caff.codc_ailluee___ec coarerecoocttteeeeeeeeeeeeeeeeee +- 0
16:
return experiment_revieeer
17:
end
18: else if last_speaker "esperimeent_revieeeer""" then
19:
last uist_response.ervv = True and self. welf.eexerimentteeviiieee count < then
20:
self cexperiment_revisiionn count
eesppermeent_reviiinnnommm
21:
return experiment_revice
else
23:
self. wetfeeeeeeetteeeviiiiee certittreviiiioioeeeeeee count + 0
return experiment-generator
25:
end if end if
27:
if last_ set__ppeaer--e = "wxperimenttreeveeeeeeeeee reviser" then
28:
return experiment, parrrttttrrrrrmrerrrr
29:
else if last__ppade speakereaerr-
then
return None
31:
end if
32:
end procedure
8.2
LLM Agents
AUTODISCOVERY defines the following agents:
Experiment Generator: Produces experiment plans for evaluating a hypothesis
Hypothesis Generator: Proposes hypothesis which predicts the outcome an experiment.
Experiment Programmer: Writes code to implement an experiment plan. The programmer is allowed 6 attempts to correctly implement the experiment based on feedback from the Experiment Analyst.
Executes code produced by the Experiment Programmer; returns the exit code and Standard Output. (Not LLM- LLM-based).
Code Executor:
12

---

Experiment Hypothesis (from parent)
Discovery Agent
Experiment Programmer
Experiment Revisor
Experiment+ Hypothesis Generator
Experiment Analyst
Experiment Reviewer
Experiment
+ Hypothesis
Figure 4: Finite state machine for the discovery agent.
Experiment Analyst:
Analyzes code execution output. Provides feedback to Experiment Programmer if code needs adjustments.
Reviews experiment results for alignment with original experiment plan and ability to support validation of the hypothesis.
Experiment Reviewer:
Experiment Reviser: Produces a revised experiment should the experiment or its imple- mentation fail to be successfully implemented allow validation of the hypothesis.
Hypothesis Belief: Evaluates whether the hypothesis is believed true based on only the hypothesis statement (prior) the hypothesis statement and associated experimental results (posterior). A temperature of 0.7 was used for the experiments underlying the results presented.
Image Analyst: Provides textual description of images produced by code from Experiment Programmer.
13

---

8.2.1 System Prompts
The system prompts for each agent are listed below:
Experiment Generator
You are a curious researcher who is interested in open-ended research based on the provided dataset. Think of a creative and interesting new axpeiment/anallyet to conduct. Do not provide the code yourself but explain in natural language what the experiment should research, so not hesitate to design experiments/ analyses that do not directly relate to the previous one.
programmer. Remember, you are interested in open-ended
Here are a few instructions that you must follow: 1. Strictly use only the dataset provided and do not create synthetic data columns that cannot be derived from the given columns. contained Check prior experiments, especially look through any recommenaatione made to improve the richness of the hypothesis and consider this information while proposing new experiments. Houever, do not repeat the same experiment plan.Here Here are experiments: 1. You
Each experiment should creative, independent, and self
few suggestions that will help you create creative encouraged to create composite attributes derived from the given columns. This is formally known as feature engineering in the machine learning literature. 2. You are encouraged to propose experiments involving complex statistical tests.
install arbitrary python packages which would allow it write code for complex statistical analysis. tests involving categorical variables. 3. are encouraged to propose experiments iinvlwwin complex predictive or causal models. For example: propose non linear predictive models such as gradient boost trees SVM as appropriate to model encouraged
Remember, your programmer
example: propose appropriate
multivariate relationships. 4. You focus on a subset the given dataset. This will help create unique interesting context to validate a hypothesis. For example: if dataset has multiple categorical variable, you could split the data based on specific values of such variable which would then allow you to validate hypothesis in that specific context. Generally,
propose experiments that only
typical data-driven discovery workflow, you may need to explore and visualize the data for possible high-level insights, clean, transform, or derive new variables from the dataset to be suited for the investigation, deep dive into specific parts of the data for fine-grained analysis, perform data modeling and statistical tests.
Experiment Programmer
will generate code based on an experiment description. State is not preserved between code blocks. Your code will be included in a python file that is executed. You must explicitly print any relevant results to standard out appropriately.
Anything that you vant displayed must be printed standard plt. showMake sure you return code the proper format execute, i.e. python code. Ensure your code is clean and concise, and include debug statements only when they are absolutely necessary. Use only the dataset given and do not assume any Always attempt
presented using
other files are available. Import any libraries you need import library first in case it already installed. You may install libraries if they are not already available. If you need to install library, use the following code example:
smmport subprocess importt sys
def install (package):
subprocess. check_call ([ays. executable, "-m", "pip", "install", "--quiet", packagel)
14

---

When installing python packages the --quiet option to minimize unnecessary outputPrefer using installed libraries possible. If possible, instead of downgrading library versions, try to adapt your code to work with more updated version that is already installed. Never attempt to create new environment. Always use the current environment. If the code requires generating plots, whole data structure to the conaole directly if it is large; instead, print concise results which directly address the experimentYou are allowed 6 total attempts to run the code, including debugging attempta. Debugging Instructions: 1. Only debug if you are either unsure about the executability of the code the validity of the code satisfying the proposed experiment.2. If the code you writing is intended for debugging purposes, you MUST clearly tag using comment line that contains only " "[debug]".2 .3. DO NOT use "[debug] anyuhere in your code when you are sure about your implementation. 4. DO NOT combine debug code and actual implementation of the experiment, keep them seperate. 5. For each experiment, you are allowed debug each 3 times.6. It is still good to minimise the number of debugging steps.
installing libraries whenever
sho, (not savefig) Avoid printing the
Experiment Analyst
You are responsible for analyzing the execution output generated by the programmer If no code was executed, indicate that there was error. If the code includes line # [debug] i.e " "[debug]" as comment, strictly treat this debugging experiment. provide information that it was a debug code execution, take feedback and request the experiment to be retried with the new information. Otherwise, you should analyze the results and provide short summary of the findings from the current experiment.
failed or
such cases strictly return error status as **true**,
Experiment Reviewer
You are responsible to holitically review the generated code, the output, and the analysis w.r.t the original experiment Alan.Assess whether the experiment faithfully implemented. The implementation follows the experiment plan without any significant deviations.A A successful experiment should have clear results that can be interpreted irrespective of the fact that it supports rejects the initial hypothesis. If there were issues, provide feedback on what went wrong.
Hypothesis Generator
Propose hypothesis which predicts the outcome of the experiment. The hypothesis should be statement that be tested by the experiment. Provide the context, variablea, and relationships that be a set boundary conditions for the hypothesis. The variables should be the concepts that interact in a meaningful way under the context to produce the hypothesis. The relationships should be the interactions between the variables under the context that produces the hypothesis Keep relationships concise. e.g., "inversely proportional", "positive quadratic", "significant predictor", "causally mediating", to name few.
relevant the hypothesis. The context should
Experiment Reviser
You are curious researcher revisiting the most recent hypothesis that could not be validated effectively in the previous experiment which eventually failed as indicated by the experiment_reviesee experiment by addressing the veaknesses/iimitttiion experimenttrrvieeerr The revised experiment should still recent hypothesis. Do not provide the code yourself but explain in natural language what the experiment should do for a programmer.
Your goal is to revise thismost recent failed
given by the validate the
15

---

Here few instructions that you must follow: 1. Strictly use only the dataset provided and do not create aynthetic data columns that cannot be derived from the
the given columns. 2. You must consider the most recent failed experiment feedback and revise accordingly so that it is effective to validate the most recent hypothesis. Here are a few generic suggestions that will help you to revise the experiment along with the feedback you received from experimment_reveeeer: 1. You encouraged revise experiments include focused analysis dataset, using feature engineering techniques where appropriate. 2. You are encouraged to revise experiments to retain complex statistical analyses, leveraging external Python packages if needed to support sophisticated methods. example: revise appropriate tests involving categorical variables. 3. You
subset of the
encouraged revise experiments invovling complex predictive or causal models. For example: revise non linear predictive models such as gradient boost trees or SVM as appropriate to model multivariate relationships. 4. You are encouraged to revise experiments that only focus unique interesting context validate hypothesis. For example: if dataset has multiple categorical variable, you could split the data based on specific values of such variable which would then allow you to validate a hypothesis in that specific context.
subset of the given dataset. This will help create
Generally, In typical data-driven discovery workflow, you may need to explore and visualize the data for possible high-level insights, clean, transform, or derive new variables from the dataset to be suited for the investigation, deep dive into specific parts of the data for fine-grained analysis, perform data modeling and statistical tests.
Hypothesis Belief'
You are belief distribution agent that evaluates the latest hypothesis. Based whether the hypothesis is
the available evidence from prior experiments, true or false. Respond with a JSON object: "heelieveshyppoheeisss true} or sealievvs__hpotheeiie false}.
Hypothesis: (hypothesis)
Carefully consider the evidence and reasoning before making your Be critical your evaluation, but fair the evidence presented.
Image Analyst
Analyze the given plot image and provide the following:
1. Plot Type: Identify the type of plot (e.g., heatmap, bar plot, scatter plot) and its purpose.
2. Axes:
Titles and labels, including units.
Value ranges for both axes.
3. Data Trends:
For scatter plots: note trends, clusters, or outliers.
For bar plots: highlight the tallest and shortest bars and patterns.
For heatmaps: identify areas of high and values.
4. Statistical Insights: Mention any relevant statistics if applicable.
5. Annotations and Legends: Describe key annotations legends.
Overall Impression: Summarize insights and implications for further analysis.
7. Interpretation: Provide insights or perspectives based on the data presented.
What conclusions can be drawn?
8. User Objective: If applicable, address the user's question objective related
to the image.
9. Limitations: Discuss any limitations or biases in the data that could affect conclusions.
8.3 Deduplication (Clustering)
Prompt
16

---

are given two sets of hypotheses. Each set describes a context, the variables innolved, and the statistical relationahips between them. Your taak is to determine if both sets indicate the same statistical behavior. Consider the following:
Context: The conditions boundaries under which the relationship holds. Both must have identical contexts.
Variables: All variables must match. Even if their names differ, they must refer to the same concept.
Relationships: Each hypothesis may include
pairs of explanatory and response variables. The statistical relationship between these variables must equivalent, regardless of how is described. Your ansver should be either "Yes" or "No" with no additional explanation. Hypothesis Set 1:
(hypothesis_1)
Hypothesis Set 2:
(hypothesis_2] Answer:
8.4 Agent Hyperparameters
Parameter
Value/Setting
Image Analyst
GPT-4o
Model
otherwise
{GPT-40.
Image Analyst
1.0
Hypothesis Belief
0.7
Temperature
o4-mini
NA
otherwise
0
Timeout
600 seconds
Max Network Retries
Response Caching
Disabled
GPT-4o
30
Number of Belief Samples
04-mini
Maximum Context Tokens
128,000
Maximum Message Tokens
4,096
Number Revisal Attempts
Number of Code Attempts
Table 4: System Configuration Parameters
9 Baselines: Search Algorithms
9.1 Repeated Sampling
Repeated sampling is achieved by deriving experiments independently from the root of the tree, ie., all nodes have only single ancestor node. Repeated sampling can be seen as a special case of MCTS in AUTODISCOVERY, by either disabling progressive widening, or using progressive widening with sufficiently large constants, e.g. k 2 sampling budget.
9.2 Linear Search
Linear search conditions subsequent experiments prior hypotheses in single experimental trajectory. MCTS can configured to enable this type of search by setting appropriate constants, c.g., k 0.5, 0, which constrains each node to have more than single child.
17

---

9.3 Greedy Tree Search
Greedy tree search focuses on exploitation by always selecting the highest-value node each step to condition on for hypothesis generation, resulting in a narrow, semi-linear search path. This translates to an MCTS variant with the exploration constant C' ===0 0.
9.4 Beam Search
The experimental results presented utilize beam width / 8 and branching factor 1: 8.
Algorithm 3 Beam Search
Require: beam width 3, branching factor k procedure SAMPLE(8) for all E beam do exps <- untried for all exps do
2:
4:
~ (s. children(s] [alldren[s].add(:5) end for 8: children(s| < sort schilciidreen(s) by
6:
w
n
beam + children [s] [1:3]]
10: end for end procedure
9.5 Programmatic Search: alternative to LLM agents?
bottleneck of LLM-driven hypothesis search and verification is latency due to API calls used for both hypothesis and code generation/debugginn for verification. Average time node/hypothesis 75 seconds, with a maximum of 600 seconds in some of the AUTODISCOVERY trees. However, particularly for data-driven discovery, [Biliss et al. [2017] developed heuristic-based programmatic system with efficient sampling that can ingest up to 2M data events per second. Inspired by this and to consider an alternative to AUTODISCOVERY in the setting of DDD, we developed a deterministic programmatic search baseline devoid of LLM calls. The system exhaustively enumerates up to million contexts (with enough data coverage) and performs pre-written (often shallow, e.g., correlation analysis) statistical analyses under ten minutes.
To compute Bayesian surprise over these programmatically generated insights, we use an LLM to translate them into a hypothesis statement. Despite being mathematically unique, many insights are semantically similar, especially hypotheses with the same interacting variables and relationship. computed across exploded contexts, but essentially encode the same generalized insight. After an LLM-based deduplication, the programmatic search generated an average of 109 unique surprisals, when computed post-hoc. While limited due to dataset-specific human interventions and shallow insights, programmatic search can empower the initial explorations done by AUTODISCOVERY with x0x speed while L.LM-driven hypothesis generation (and verification) can focus on complex hypotheses beyond the scope of the programmatic search leave this as future work.
18

---

Algorithm 4 Programmatic Search Baseline
Require: Dataset DD target categorical set C, numeric set N,, max depth dmax, coverage threshold
T, significance level
Ensure: Ranked insight list I
Coin + QUANTILEBIN(A7)
bin numerics
2: F C
3: M M {
valid context masks 4: for k 4- 1 to dmax do
5:
for all feature subsets S F with |8| = k do for all level vectors of do m < (CL.OE(E = 0) 2 then
6:
7:
8:
if REREATIVVEREROO M MU {m)
9:
10:
end if end for
12: end for
end for
14: M < MU (full-data mask]
D- First-pass statistics
15:
R1 + 0
16:
for all E MM do
17:
Dm + -D(m]
18:
R1 R1 U SINGLEFACTOR m: y)
19:
R1 + R1 U CORRELATION y)
20:
R1 R, SIGNHECANTDRIVERSSDD
21:
end for
22: R1
FILTERSORT(R1,
Second-pass pattern mining
23: R₂ + DETTCCTFUPSEMEREENNCC..
24: R₂ FILTERSORT(R2,,,77 a,
Symbolic rendering RENDERMATHSTTRINSSSRR U R2) return I
25: I +
10
Limitations
Pitfalls with agentic frameworks. Providing LLM agents agency over experimental direction and implementation lead unexpected failures, e.g., corruption of the execution environment by installing improper libraries.
Context window corruption/limits. Particularly in setting of open-ended discovery, the discovery agent may show aberrant behavior if the context either becomes exceedingly long or corrupted due to, e.g., unexpected code execution output. Nodes with many ancestors (e.g. 150) can also suffer from context-overflowinn and show a similar failure mode.
Ungrounded generations. LLMs occasionally introduce attributes not grounded in the dataset, with prompt-based based constraints insufficient to address the issue completely. Without human oversight, this may result in the generation of scientifically spurious findings, which is important to guard against.
LLM surprisal vs. human surprisal. Despite the 67% agreement between Bayesian surprise and our human evaluation T). surprising the L.M is only proxy for novelty to humans. To reduce this gap, future work may consider literature- literaure-grounnee extensions that explicitly incorporate information such citations into the LLM surprisal computation.
19

---

11 Human Annotations
11.1 Qualtrics/Prolific Annotation Flow
We conducted our human annotation study using the Qualtries platform with annotator recruitment and access managed via Prolific (Fig. [5) Each participant was assigned to specific dataset through a unique Prolific link. For every dataset, we created a distinct Qualtrics survey set, comprising 20 hypotheses, and three independent annotators were assigned to evaluate each set.
The anmotation flow within Qualtrics was structured as follows:
Prolific ID Capture: At the beginning of the survey, participants were asked to enter their unique Prolific ID to ensure traceability and consistency of responses.
2. Dataset Exposure and Familiarity Assessment: Before any hypothesis was shown, partic- ipants were introduced the dataset context and structure. They were then asked indicate their familiarity level with the dataset and its domain using the following prompt:
How familiar are you with the dutaset and its domain?
Not familiar at all
Slightly familiar
Moderately familiar
Very familiar
Extremely familiar
Following this, participants rated their confidence their familiarity score on slider
ranging from (no confidence) to 100 (complete confidence):
What is your confidence in the familiarity score you provided above? (Slider: (-100)
3. Hypothesis Evaluation: Participants were then presented with a sequence of 20 hypotheses, shown one at time. For each hypothesis, they responded the following four questions using sliders ranging from to 1. Here, O indicates False, indicates True, and 0.5 represents Unsure:
(a) What is your belief about this hypothesis? (Slider: = False, 0.5 = Unsure, 1= True)
(b) Could knowing the truth value for this hypothesis be useful to you? (Slider: 0 == False,
0.5 = Unsure, = True)
(c) Could knowing the truth value for this hypothesis be useful the scientific community? (Slider: 0 = False, 0.5 Unsure, = True)
(d) Do you think this hypothesis is interesting? (Slider: 0 = False, 0.5 = Unsure, = True)
All slider responses were recorded as real-valued scores in the range [0, 1] for subsequent quantitative analysis.
To manage participant recruitment and validate response completeness, we used the Prolific platform. Each annotator accessed specific Qualtrics survey set through unique Prolific link tied to the hypothesis sets.
At the end of the survey, participants were shown a completion code, which they were instructed to submit on Prolific confirm they had fully completed the task. This code allowed us to:
Verify that only participants who completed the entire annotation process were compensated,
Match survey responses with Prolific IDs for traceability,
Filter out any incomplete or prematurely exited surveys ensure data quality.
Only responses associated with valid completion code were included in our final analysis.
Annotator population and payment Evaluating scientific hypotheses is a knowledge-intensive task. To obtain high-quality, meaningful annotations on the hypotheses, we screen participants and only allow those with Master's/PhD degree in Mathematics and statistics, Information and Communication Technologies, Engineering, manufacturing and construction, or Natural sciences, to
20

---

Rypsthatt: The ontl mut Bor dereroo coroooattnnnnn www.annnggggrooo aarthurttty of so ii to bodloco licellul Choorng the mapi as ofc moderusr both zgp and culurr coEMs.
Codd
provided abroer.
We going show 20 hypotheses abrutt the mentiond them.
dataset, We are going to ask your belief about them.
Figure 5: Annotation interface with Qualtrics/Prolific
How familiar you dataset domain? Refer to the following definitioss to select the level that best describo What confiddnce your familarity.
orldane 20 100
new conerction nnth AN dhrat or ander deraum deramn ruppir thot You seserhh thrraaed aeeeeeeee rerieet gually cenccioooo and and Ait
ral caterppcoe
No lover cooa the obcemnic dinda 8OX eor 201y perigggne fhe any
Mederastt too menure wich socrel AnD on opereett bghe No undbosaa mot peveo and
thu heve uour ees ied oot thi deaee daaks masgggggte
Edts Sdtermb//to (emilar hor Bugn puulcs rocrcs dn can No snsercoi cng ogeenne
No Can for ucul
Sighicy Lan Buy
very fomittr
chidheets socal harming pnngie dree hnesigitdd age cutural conoos: Socia foaning Konking from from other is Woy aepect of hummn orgritinn and culter crsnmmacionn Dhitnnn may en socal CutS to ffferee enters, ofta showng proferena for maporty minorry
Soubt thi, chtttenn apod from soee diffrennt socedtes (tratt Canalla Chara, Clensacc Jacin ham ta Genmuryy and Aopert wostt ad video viouidhowt Sooii three dono donorisiatt aittou anottttsssssttttiooooooooo doooiing one option and one samontraatr
Chidennone then unkerd to choose
calture: cuttur where dosted tungging from
indertanninng development
Yuturam ventalee indicatin ciid chotos Valaae
majority mestriy.fiatt firat dhuuto wlaliee the mmponny tostor was
gender: Genter the rrsel git tay
on
Descrptttnn study,
majortyy ceroag
informatoo eac mrat.
age bge the paaar
demonstrone firs
doverommnnt
Colune::
21

---

take part in the evaluation task. Other pre- pre-screen qualifications were: Residence to be US, UK, or Canada, and they should be fluent English. We set an hourly pay rate of $16.68, and we anticipate taking on average of 20 minutes (which turned out to be the median completion time, when computed post-experiment) to complete the survey. For each unique Qualtries link (each having 20 hypotheses), we require each participant to be unique too, however, the same participant can take part in tWO or more unique Qualtrics links.
11 Infernal Annotation
Structured Annotation Interface
cooe Sateed restiber Sare Marth unbeln
repecers unapg ant
Ree parpancion poreeets be clastored inta distinct greape based on iiccne and casinge betastor, revealing patterns in Finueciat hakite and ecanonsc
Dedupticaton
typthers dipitater cament typtheas aboue?
Respandert clasteed grups besed sevings betarlo wealing pahoru frarde habls skality Eastnest
Node
Figure 6: Internal annotation tool for discovery agent and deduplication verification.
To verify the faithfulness of the different components within our discovery pipeline, we additionally built internal annotation tool. Specifically, we verify three key pieces: (1) whether the experiment proposed by the LLM is valid, i.c. its feasibility to implemented using the available data and its ability to confirm the hypothesis, (2) whether the Python code implementation is faithful to the proposed experiment, and (3) whether the deduplication procedure is valid. Each author annotator was shown the experiment plan, experiment analysis, and review summary for hypotheses (Fig. (6) Based on this information, they were asked the following questions:
1. Is the experiment valid? Options: Yes / No Unsure
2. Was the experiment faithfully implemented? Options: Yes / No Unsure
3. Are the following hypotheses duplicates of the current hypothesis (see above)? Compare the listed hypotheses to the target hypothesis and decide whether they are exact or near-duplicates meaning. Options for each listed hypothesis: Duplicate Not duplicate/ / Unsure
22

---

12 Sampling Unique Hypotheses across Methods
In Fig. 7 aim disentangle the efficiency of generating hypotheses that unique (i.e., evaluating diversity) vs. our joint objective of unique surprisals (as described by our main experiments), Our results indicate that the diversity-only trend (Fig. [ closely aligns with the unique surprisal trend across methods. In particular, MCTS results in very few duplicate hypotheses on average across datasets (low standard deviation). We conjecture that the ability to dynamically sample any node across the tree in each iteration, allows MCTS to leverage unique branches as context while prioritizing regions in the search space that are likely to result in high surprisal and diversity. Other tree search methods, such as greedy, follow instead sequential root-to-leat sampling procedure, which does not allow for dynamic sampling.
Cumulative Hypotheses Count
MCTS
Beam
Greedy
Rep. Samp.
Linear
0
100
200
300
400
500
Hypotheses Evaluated
Figure 7: Uniqueness Efficiency with GPT-40. Cumulative number of unique hypotheses discovered across timesteps within a hudget of 500 evaluations, averaged over 21 datasets.
13 Results with 04-mini ("reasoning" models)
This section evaluates how search behavior changes when replace GPT-4o in AUTODISCOVERY with a "reasoning" model, o4-mini, which is trained to output multiple (and longer) chains th-tuuh.. We repeat our main experiments with AUTODISCOVERY as well as each baseline.
Search performance. Overall, we find a similar trend in our main experiments (as shown in Fig. (() when comparing search performance across methods, with MCTS and greedy tree search outperforming other algorithms. However, we now observe difference in performance between MCTS and greedy under our sampling budget of 500 hypotheses" We conjecture that this due to the ability of reasoning models in proposing fewer duplicate experiments. Further, Fig. (bb shows that, ullike with non-reasoning models, the search efficiency gradient does not measurably decline for any method but shows variable absolute values, pointing differences in ability find surprisals for different methods irrespective of ability to find unique hypotheses.
04-mini vs. GPT-4o. Fig. (b), we plot how AUTODISCOVERY performance changes when using reasoning model (o4-mini) versus non-reasoning model (GPT-4o). Our results show that modest, but steady, gains can be seen in terms of cumulative surprisal counts with reasoning models. Furthermore. qualitatively, mini higher than GPT-4o. E.g.. the following is a level node found from the Freshwater Fish dataset DiscoverrBenchh "Within South American freshwater-fiie sub-basins, evolutionary rates (diversification and morphological evolution) exhibit significant positive spatial autocorrelation, such that geographically proximate basins have similar rates than distant ones.
find that the complexity of hypotheses generated by 04-
'I is likely that we would observe a drop-off off (similar to the one from the GPT-4o experiments) using greedy with larger budget.
23

---

(8]
0-5
(b)
Beam
Groedy
curuuletvee Surgrtes
Rep. Samp.
62
MC15
Beam
10
Groody
Rep.
520
300
Hypotheses Evalusted
Hypotheses Evolusted
50-
MCTS
Beam
Greedy
Rep. Samp.
Lincar
(c)
Surprisafs
40
30
20
10
crofent
5e3
incarceration
fish
affairs
requiements
amti
reading
evalution
caschools
mortpage
teachingratings
Doxes
bm
toy
soccer
conversation
ferilliy
humricane
Figure 8: Search Performance with 04-mini. (a) Cumulative number of surprisals discovered across timesteps within budget of 500 evaluations, averaged over 21 datasets. (b) Search efficiency gradient computed using sliding window iterations. (c) Number surprisals discovered per dataset.
bmong
24-mini
Cumulative Surprisal Count
GPT-4o
(b)
300
400
Porcntaae of Teadoa
Mypotheses
Figure 9: (a) o4-mini: Belief shift across datasets. Bayesian surprise under belief shift for surprisals discovered using AuToDIscoVERy, grouped by domain and direction of shift. (b) Comparison between o4-mini and GPT-40. Cumulative surprisal counts using reasoning and reasrsnnnnng models with AUTODISCOVERY (MCTS).
14
Example Outputs
Code Execution:
Random Forest with Feature Importance
exitcode:
(execution succeeded)
Code output: Total samples after cleaning: 84
Best parameters: {'max_des ''max_dephh': 5, 'min_samples_leat 2, 'n_estimators' 2003
AUC: mean=0. 716, atd=0. 100
Sensitivity: mean-0.2 413, std-0. 230 Specificity: team=0.794, std=0. 162
Feature Importances:
Age at Death
0. 08189995
cerad_num
0. 0.176595
braak_num
0.14229
24

---

thal num
0. 140306
Years of education
135508
duscular_burdes
0. 126202
apoe4_ccuntt
0. 0.051388
sin_bin
0.040713
dtype: float64
Plot Analysis (1)
1. **Plot Type**:
This is a bar plot. Its purpose is to display the feature importances from Random Forest model.
2. ** Axes**
**x-Axis*=;
Title/Labels: Feature names such as "Age at Death, " "cerad_num, etc.
Value Range: Categorical (not numeric).
**Y-Azis*=;
Title: Importance
Value Range: 0.000 to 0.200
3. ++Data Trends**:
+*Bar Plot**:
The tallest bars indicate "Age Death" and "cerad_num" the most
important features.
The shorteat bar is "sex_bin," indicating it is the least important feature.
**Statistical Insights**:
The featurea have varying importance, with certain features contributing significantly more to the model's 's predictions.
**Annotations Legends**:
Title: "Random Forest Feature Importances"
additional legends annotations present.
6. +veeral Impression**:
The chart clearly indicates which features Forest model. This can guide further data analysis or model tuning by focusing on the most important features.
influential in the Random
**Interpretation**:
"Age at Death" and "cerad_num' are critical to the model's 's decision-making process. Understanding these features' importance can provide insight into the modeled outcome, possibly emphasizing their relevance in predicting or classifying the target variable.
8. ++es Objective**:
the user's objective is understand which features prioritize in data collection or analysis, this plot effectively highlights the priority features based on their influence in the model.
#Limitations**:
Feature importances are specific to the Random Forest model and may not gemeralize across different models.
This plot does not provide information on feature interactions potential multicollinearity.
The analysis doesn't take into account external factors that could influence the model's 's outcomes.
Regression
Code output: Interaction effects (Pathology APOE4) Last MMSE:
25

---

path_var
coef
pval
pval_fdr
significant
thal_min num
80 ...991251
0. 00.08886
0.026687
True
1
micro-grose
80 0. 0.112921
0. 768721
0. 0.816070
False
2
micro_screen
80 -0.130590
0. 0.816070
0.816070
False
Plot Analysis (1)
1. **Plot Type**:
The plot scatter plot with regression lines. used visualize the relationship between two variables: thal_num and Last MMSE Score, stratified by APOE4 carrier atatus.
*xxe***
**X-Axi=*:;
Title/Label: thal_nur
Value Range:
**Y +*Y-Axis==;
Title/Label: Last MMSE Score
Value Range: 35
3. Data Trends**:
The scatter plot shows different linear trends for APOE4 carriers (1) and non carriers (0).
Carriers (blue) show a steeper declining trend compared to non-cas carriers (red).
There is overlap between the data, but darker regions indicate with more points.
4. #*Statistical Insights**:
The regression lines suggest that APOE4 carriers have lower scores as thal_nus num increases, more so than non-carriers.
5. Annotations and Legends**:
The legend indicates two groups: AP0E4 carriers (1) non-carriers (0), represented by different colors (blue for carriers, red for non-carriers).
#vverall Impression**:
The analysis indicates that as the thal_num increases, the MNSE score tends to decrease, especially for APOE4 carriers. This suggests potential impact of thal_num cognitive performance, moderated by APOE4 carrier status.
*#Interpretation**:
The data implies that individuals carrying the AP0E4 gene may experience more pronounced decline cognitive performance (as measured by the MMSE score) associated with increasing thal_num.
8. +seer Objective**:
the user's s objective is assess cognitive decline relation to thal_num and genetic factors, the plot provides a clear visual representation of these relationships.
##Limitation=*:;
The data does not indicate causality.
There might be confounding factors accounted for in the analysis.
The sample size at each thal_num level the variability are not specified, which may affect the reliability of the trend lines.
Ordinary Least Squares Regression
Regression Summary:
Regression Results
Dep. Variable:
last_mmse mmse
R squuree::
0.241
26

---

Model:
OLS
Adj. R-squared:
0.189
Method:
Least Squares
F-statistic:
4.690
Date:
Tue, 20 May 2025
Prob (F-statistic):
0. 000895
Time:
12: 10:18
Log-Likelihodd:
-211 68
Observations:
80
AIC:
455 4
of Residuals:
74
BIC:
469.7 7
Of Model:
5
Covariance Type:
nonrobust
coef 0.975]
std
P>t
[0.025
Intercept
20.4041
14.935
1.366
0.176
-9.354
50.162
C(sex) Male]
0. 1244
0.961
0. 129
0.897
-1.791
2.039
years_edu
-0.0143
0.797
-0.018
0.986
-1 602
1.573
braak_num
-. 1.5258
2.725
-0.560
0.577
-6.956
3.905
_ears_edu: braak_num num
-.0050
0.166
-0.030
0.976
-0.336
0.326
age_at_deate
0. 1295
0.063
2.
0.045
0.003
0.256
Omnibus
8.277
Durbin-Waten Watson:
2.013
Prob (Dmmiibus):
0.016
Jarque-Bers Bera (JB)
8.367
Skew:
-0.590
Prob(JB)
0.0152
Kurtosis:
4.058
Cond. No.
95e+03
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2) The condition number is large, 3. 3.95e+03. This might indicate that there strong miticollinearity or other numerical problems.
Interaction term 'years_ee wwweeduuuaaaaaeee num' coefficient -0.0050, p-value 0.9759
Clustering
Code output:
Plot Analysis (1)
**Plot Type**:
This is a dendrogram, used in hierarchical clustering to show the arrangement
of clusters formed the algorithm.
2. **Axes**:
**X-axis*=: Labeled as "Donors" with specific units indicated. It
represents the data points (or samples) being clustered. **Y-axis*:; Labeled as "Distance," " This axis shows the distance or dissimilarity between clusters. The value range is from 0.0 to 20.0.
**aaa Trends**:
The dendrogram shows the hierarchical relationship of the clusters.
Longer vertical lines at the top indicate greater dissimilarity between clusters.
Shorter lines at the bottom represent more closely related clusters.
4. staatistical Insights**:
27

---

The height at which two clusters join indicates the dissimilarity. Higher joins dissimilar clusters.
5. ** Annotations and Legends**:
No specific annotations legends present. The branching structure itself provides the clustering information.
6. **Overall Impression**:
The data points grouped hierarchically, revealing patterns of similarity. The clusters closer to the bottom are more similar, whereas those joining higher have more dissimilarity.
**Interpretation**:
The dendrogram allows visualization of the data's 's natural groupings, which can be useful for determining the optimal number of clusters by cutting the dendrogram at different heights.
8. Usee Objective**:
Likely to identify the grouping structure of the donors based certain attributes metrics of eimilarity/distance..
9. * #imitattonss*:
Dendrograms become cluttered with large datasets, making them harder interpret.
The method's s sensitivity to distance metrics and linkage methods can affect conclusions significantly.
Plot Analysis (1)
1. **Plot Type:**
**Type:** Heatmap
**Puposee:** To visually represent the normalized values of different neuropathological features, allowing for easy identification of patterns, variations, and extreme values various categories.
2. **es:***
**iitees and Labels:**
=* Neuropathological features such as Thal, Braak, etc.
Overall_AD_ Latrall_AD_neuurope www.oopttohoooiccal.coomm
**Y-Axis:* Sample observation index (not labeled explicitly, appears categorical).
**Value Ranges:**
-**--Aiis:** Categorical features.
>*+--Axis:** Categorical index.
3. ** Data Trends:**
**Hggh Values (Red) Notable in columns such "LATE" and "Arteriosclerosis."
**Low Values (Blue):** :** Prominent in "Overall Baiill__D___e_e neuroppthhollogccal Change" and
MicoaallMccossaarrttss...
**Patterns:** Stripes of consistent color indicate similarities in features samples.
4. Staatistical Insights:**
**Normalized Values:** Range from -2.0 (low) to (high), enabling
comparison across features. *#Distribution:* Variation suggests differences in presence and severity of the conditions measured.
5. Annotations and Legends:**
**Legend:** Color bar on the right labeling the normalized values from -2.0 (blue) to 2.0 (red).
28

---

6. ** #verall Impression:**
The heatmap effectively shows diverse variability neuropathological measures. Features such as "Arteriosclerosis" and "LATE" display higher normalized values, suggesting greater severity in those areas for certain individual observations.
**Interpretation::*
**Conclusions:** Some features, like "Overall_Ar Abt Chaggggaaaaaaaaeeeeapaa often exhibit low values across samples, indicating potential pattern of less severity or frequency in this dataset compared to features like "Arteriosclerosis.
8. * User Objective:**
**0bjective:** Identify patterns in neuropathological data that may correspond to different pathological states or severities across samples. Recognize high and low prevalence of pathological features.
**Limiations::**
**Data Bias:** The heatmap portrays normalized data; without raw data, assessing actual severity is difficult. **Interpretation Bias:** Over-reliance color may overlook nuanced details. **Sample Size:** Not visible; the sample size could affect the robustness of observed patterns.
heatmap provides comprehensive visualization of various neuropathological features, offering insights into potential underlying patterns across samples. For further analysis, exploring how these features correlate with clinical outcomes demographic data could be valuable.
Cluster (n=14):
Mean Age Death: 88.21
Mean RIN: 8.58
Sex distribution: ('Female': 0.5714285714285714 'Male': 0. 428571428571428555)
Cognitive Status distribution: ('No dementia'; 0.857142857 1428571, -112277142887142885
'Dementia'
Cluster 2 (n-17):
Mean Age Death: 90.00
Mean RIN: 8.40
Sex distribution: ('Female': 0. 0.529411764758824,,
'Male': 0. (.47658823229411744)
Cognitive Status distribution: No dementia': 1.6470588235294188, 'Dementia':
0. .352941178470582206
Cluster (n=5) Age Death: 90.40
Mean RIN: 8.47
Sex distribution: ('Female': 0.6, 'Kale'; 0.4)
Cognitive Status distribution: ('Dementia:; 0.6, 'No dementia': 0.4)
Cluster (n=19):
Mean Age Death: 89.47
Mean RIN: 8.65
Sex distribution: ('Female': 6.33577947368421. 'Male': 0. 0.368421062631577))
Cognitive Status distribution: ('Dementia'; 0.t.eettii.......6....... 0.5263157894736882, 'No dementia': 7.4736882105263157666
Cluster (n=29):
Mean Age at Death: 87.45
Mean RIN: 7.51
Sex distribution: {'Female': 16651724137991104,, 'Male': 0.344827862268966)
Cognitive Status distribution: { ('Dementia': 0. 77721379310344828,, 0. 2776662266866557724
'No dementia':
29

---

15 Example Errors
Excessive Code Execution Output. Automatic code generation is susceptible to unexpected errors. e.g., due to malformed arguments. Such instances, therefore, may result in uncaught exceptions and repetitions. For example, the following example shows the same log message being generated >66,000 times for single hypothesis node.
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf
Network Error. Our current experiments use OpenAl API calls and are, thus, reliant on the stability of their hosted service. Future work may look into using local models.
Traceback (most recent call last):
File http://www.....tettte/es///////a _create_ _rrate_or_parss_ _r_asse return self._oai www.et.ceettteeeeeeeeee oel..aa...ccca...cc.c.....e.eeee (kaage)))
line 466, in
File "/jib/python3.111 http://www....aaccacooe////aaaaa
line 287, in wrapper
return func (*arga, **kwargs)
File
"/lib/python3. 11/site http://leeeeaaaeeeeiiieeeeeee line 925, create return selt._post(
File "/1ib/pythen3. h/a.p///.....ttteeeccacae//pppaa wwa.aaaaeeeee...a.eeaaaaaeeaaeee clwcnt......mc pilent.py", line 1239, poat return cast (ResponseT, self. self. request opts, strean-stream, stream_cls-stream
File "/lib/python3. htsp://aeeeeeeccaggeaappaaall basse_clent .er...... line 1001, request raise APIComeectioBeroor (requess from err
Flagged Prompts. a few instances, we observe that API calls are rejected as violating usage policy when we're 'e processing datasets that involve race gender, likely the effect of aggressive safety tuning.
=== Plot Analysis (1) ===
**Plot Type**;
This is a heatmap.
Purpose: To visualize the relationship between age, culture, and conformity to majority proportions.
2. **xx=***;
**x-axis*:: Labeled "Culture" with values 8.
**Yaxxi==:; Labeled "Age" with values to 14.
No specific units are provided for these axes.
3. ++aata Trends**:
Areas of high values (yellow) indicate high proportions of
cajjoiity-confomitty such age 12 and culture 1, and ages 13-14 with cultures and 8.
Areas of low values (dark blue) indicate low conformity, noticeable at certain combinations like ages 9 and 11 with culture 8.
30

---

4. Statistical Insights**:
Values range from 1.0, representing proportions.
High conformity (values of 1.0) is clearly marked, suggesting strong tendencies toward cojority in specific groups.
**Annotations L
The heatmap includes a color bar on the right side to indicate the proportion range (0.0 to 1.0).
Numerical values within the heatmap provide specific data points for conformity proportions.
6. +veeral Impression**:
There are distinct clusters of high conformity at specific age and culture intersections.
There is variability across both axes.
7. "<Interpretation=:
Certain groups show strong conformity in specific cultures, which could indicate social cultural influences.
The patterns suggest that age and cultural background significantly impact conformity behaviors.
8. +User Objective**:
the objective to understand how culture and age influence conformity, this heatmap effectively highlights potential areas for further sociological or psychological amalysis.
#Lmitatiinns=:
The heatmap does not indicate causality.
Missing data points could lead incomplete conclusions. Cultural categories and age groupinga might be subjective or oversimplified, impacting data interpretation.
ooeeaa..aaeeeeeeeeeeeeeeeeeeee BadRequestEEror: Error code: 400 ('error': ('message': 'Invalid prompt: your prompt was flagged potentially violating our usage policy. Please try again with a different prompt:
attps://platform www.m.ppeeaiicc...
"iivaliddrequee hwtpaaaa.cccccoaaeeaaaaaaaaaaaaa
'type':
seaiiddrequust_eerrrrr 'param', No
ne, 'code': 'invaliddproomt"
Timeout.
The discovery agent, in cases, may result in timeout emors. This is particularly seen when the generated experiment program requires a long-running operation or the amount of data being processed is large.
("code": :"oope""""mmort subprocess, sys\n# Install SALib for sensitivity analysis Andef (pksa)l((kgg: subprocess... check_ call (aaal((aaaexettaaalllll executable '-m' 'pip', 'install', "--quiet', pkg])\ninstall
pandas as pd'nimport numpy (aatab'"\nnnimpooo
up ap''immortt matplotlib. hypllot as plt\mnrrom sklearn. prerrrpppppoocssinggg import Stan aklearn.gaussian_ppp.... gaussian__rooess import GauesianproceeRRgressoo Anfrom dardScaler\nfron sklearn.
bereesss.keeneee kernels import RBF, WaiteKermellnfrow SALib, sample import ssateellllffroo SALib. analyze import sobol \nimport
sklearn.
waarningsnwwarnnngg.. wwwerrartttteeteneeeeeeee ('igaainnnngigooeeee 'ignore' \n'n 1. Data Preparation ind
ced.read_csv'nn133
mi.pproceseddec cataaaeddddddvv)) An# Compute raw saving direction and age
squared (ndff Indf ('SAVING 3IR_rav') = np where (df DISSAVED ]==1, -1, \n
mp. where (df [[ 0, 1)) Andf [" AGE2' ] df AGE # Standardize continuous variablea [' A'AGE 'INCOME' 'BMI' "SAVING_DIE B_AIIGGGDDR__WWVV_ Ansc
31

---

aler = StandardScaler (aaaaarrrrrraa ()nnff) =
(df [cont] Infor i, col in
enumerate :cont): df [f [col]_zo''''' _z'] df_z[: \n# Ensure binary
covariates und ['GENDER_MALE" M ['GENDER == '= astype (int) \ndf ('BLAKK]
of ['BLACKK]] astype (int) \ndf HISPPNIC'' df ('HISPANIC' .astyp (int) An# Predictor
nam
es ea'nneements
['AGE_z', 'INCOME_2' 'BMI_z', 'SAAVIN_DDRR, DWVIINGDIRR_av... raw_'', GENDER, MALE' 'BLACK' 'HISPANIC') \nX Ann, P X. shape \n'n# 2. Fit Gaussian Process surrogate on full data/ngg CauueiiafrroeesRegressss (kernel=RBF (1ength_ac
df [elements] values\nY
ale=np. ones (p)ooo)))))))eeoooooe))e))) +WhiteKeeenn \n
normalize_y=True random sando__stat=0)) ffit(X, Y) \n'nn
3. Sebol sampling problem definitioonnndecrrittor (\n 'num_ vars's p, \n "names'; elements, \n 'bounds', [[ol'' [co[[cco='']]_== .min(), df [f _2'] max for col in cont] \nHnnn Use smaller sample for speed'nnN' 200 200'mmpram_ lues saltelli. sat.li...mmpleeee. (descriptor, N, aurrogate and compute Sobol indices'ny_sobol adaaaaaanayyedessrpppttooo
ln'nt In'n# 4. Evaluate
gp- (param. \nsi
anbll. (descriptor, y_sobol, sall__secnddordee-=aalas... print_ print_to_comsole-Fa esti
order=False,
in$1 si si[' ST si''ST']\n's \n'n# 5. Bootstrap uncertainty
mation (B=100) 100\nboott_ Anfor b in range :An idx mp. random choice n, replace=True)<s x[idx], Y[ gP_b Baussiannrocesseegreen (kereeee *WhitWKerneeeleeeeeeeeeeeeeeeeet (eoie (p))
up. zep.zeroa(() p)) \nboot_sT up. up. zBr
Xb, Yb
(kernel=RBF (Length_
normalize_.--rrue random_state-0 (Xb, Yb) \n ys
op..ppredict(paraa__alllll
sobol. (descritttr ys, srr_second__rreerFaaaee print_to. tto_coosoll=aaase) In boot [b,_]]
sib['S1 boot_ST[b,:: = sib cib['sT']\nci,_ \nci_S1 = percentile (boot_
[2.5,97.5], axis=0)\nci_ST np
percentillebboot.51 [2.5,97.5], axis=0) In'n# Visualization of Sobol inniiees/mplt..te www.ic/eaaalt..iiuueffiiieaeee (8)44)
np. (prage((p)rie Anwidth 1.4mmptt.bar (indicess width/2, S1, width, (err-$Si-i__$100 [i]S1(
capsize=5, label= bappt.barr(tt (indicess + width/2, ST, width, (ST-ciSSS-_
ci__ST()-STT) capsize=5, label='Total-order (mmppto xtpplt.xtickstii (indices, elements,
protatiionn5: "i ylabel ('Sobll ('Sobol index ) Amplt.title( title Sobol Sensitivity Indices for BMI GP Surrogate Anplt. loglt.e () /mplt/ptt.... www.tttgghllooooeee () Wtp./....ooo./o. \n'n#
7, Reporting results\uprint, "Sobol Sensitivity Indices with 95%
CIs:\") \nfor i, var enumerate (elementt)))))))) [Cci_Si S1 [1, :.)f), [ci_si[1,1]:.... :.3f]] \", {ci_ST[1, i3) 3f]]]
print "(var): (S1 :.)
f\"ST [
Next speaker: code_execute
>>>>>>>> EXECUTING CODE BLOCK (inferred language is python)...
code executor (to chat maat_manager):
exitcode: 124 (execution failed)
Code output:
Timeout
Next speaker: sxperimen__analyee
("error" trree, "amalysis": "The Sobol sensitivity analysis timed out during bootstrap computation (B=100) The high computational cost of refitting the Gaussian--rocess surrogate and recalculating Sobol indices for each replicate exceeded the time limit. Consider reducing the sample size for Sobol sampling (N) and the number of bootstrap replicates (B), or implementing parallelization to complete the analysis."]
32

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide empirical results to support our claims. See Results section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We talked about an alternative approach, programmatic search, to overcome the latency limitation of AUTODISCOVERY. We also did error analysis.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes. We provide all details in the paper and/or supplementary. Code will be shared upon publication.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification:
We share through an anonymous repository as detailed in the supplementary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com htte:/ublllllliiiee/e//eeeuuaasl ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We do this as necessary. Details are the supplementary as applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Plots include error bars/bands needed. Tables have highlights for statistically significant results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
35

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We do this as necessary. Details are in the supplementary as applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pwwp.aaaa......o/opo//oee
Answer: [Yes]
Justification:
We follow the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential impact of our method on making real scientific discoveries. Similarly, for being LLM-driven, it runs with the risks of producing false positive discoveries.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the sources appropriate.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We do not share new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not perform crowdsourcing experiments. Details of the human evalua- tion study are included in the supplementary.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do not perform research involving humans.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
38

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We did not use LLM to perform any son-standard component of our paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: for what should or should not be described.
39


Reviews:
[{"review_id": "npwtRQOjVv", "reviewer": "Reviewer_WY1X", "summary": "This paper introduces AutoDS, a Bayesian framework that emulates the process of scientific discovery by modeling posterior surprisal through Monte Carlo Tree Search. The methodology provides a formal definition of surprise in the context of automated scientific discovery (ASD), characterizing it as the Kullback–Leibler divergence (DKL) between prior and posterior beliefs. To further capture the directional and informational aspects of surprise, the concept is refined as Bayesian surprise under belief shift. Experiments on 21 real-world datasets demonstrate that AutoDS achieves 5–29% more discoveries deemed surprising by a large language model (LLM), outperforming strong search-based baselines.", "questions": "1. Is any scientific discovery process truly binary? The assumption that the verification procedure must yield a binary outcome—i.e., supported vs. unsupported—feels overly restrictive. In NLP, for instance [2], researchers have explored the use of p-values reported in papers as a more continuous measure of how strongly evidence supports a hypothesis.\n\n2. What happens if an LLM produces different Boolean responses when given the same prompt multiple times? What temperature did you use when calling the APIs? Would such inconsistency undermine the reliability of the empirical frequency estimates?\n\n3. What $n$ was used for belief elicitation? \n\n[2] Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020, November). Fact or Fiction: Verifying Scientific Claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 7534-7550).", "limitations": "It would be good to include results from LLMs other than GPT-4o.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "I believe the authors address a highly relevant and timely question—shifting the focus from merely discovering novel or diverse scientific hypotheses to leveraging the emergent capabilities of LLMs to guide their own discovery processes. Given the rapid advancement of foundation models, I anticipate that this approach will gain broader traction within the AI4Science community. The paper is clearly written, and the experiments are thoughtfully and comprehensively designed.\n\nOne area for improvement would be a more explicit comparison with alternative formulations of uncertainty. For instance, in addition to using the Kullback–Leibler divergence (DKL) between prior and posterior distributions, it would be interesting to consider mutual information as a measure—evaluating how much information is gained after observing the data. (Although these formulations may be mathematically related, their interpretations can differ depending on the application.) I would point to Gao et al. [1] as an example, where mutual information is used to align conclusions with their most probable premises in scientific texts.\n\nThat said, the authors provide an in-depth treatment of DKL, which is sufficient for supporting the central idea and validating the proposed methodology.\n\n[1] Gao, Y., Gu, N., Lam, J., Henderson, J., & Hahnloser, R. (2024, March). Evaluating Unsupervised Argument Aligners via Generation of Conclusions of Structured Scientific Abstracts. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 151–160.", "was_revised": true, "final_justification": "While some of my concerns were asked by other reviewers and addressed by the authors, I remain my original scores, as I believe my rating was fair.", "rebuttal": "Thank you for taking the time to review our work! We address each of your comments and questions below.\n\n---\n\n> `W1: …interesting to consider mutual information as a measure—evaluating how much information is gained after observing the data…Gao et al. [1]...That said, the authors provide an in-depth treatment of DKL, which is sufficient for supporting the central idea and validating the proposed methodology.`\n\nThank you for this suggestion and the reference! Mutual information is indeed a related information theoretic concept to surprisal. As you note, there is a mathematical equivalence as well. We can surely add this as a discussion in our related works in the final version of the paper.\n\n---\n\n> `Q1: Is any scientific discovery process truly binary?...researchers have explored the use of p-values reported in papers as a more continuous measure…`\n\n> `Q2: …What temperature did you use when calling the APIs?...`\n\nWe clarify that our belief elicitation mechanism does indeed produce graded correctness. Note that while the individual Bernoulli samples from the LLM provide a boolean response, we generate n=30 samples from the LLM using a temperature of 0.7. These Bernoulli samples allow us to compute a Beta distribution through the Beta-Bernoulli conjugacy resulting in mean belief probabilities that lie anywhere between 0 and 1. To show that this also happens in practice, here is the histogram (bin count=10; bin width=0.1) of mean prior and mean posterior belief probabilities elicited from the LLM across ~600 hypotheses from runs on 4 datasets:\n\n**Prior Beliefs:**  \nHistogram: (array([ 80,  25,  20,  20,  21,  33,  35,  37,  58, 280]), array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]))  \nMean & std.: 0.6818 ± 0.3396\n\n**Posterior Beliefs (after Bayesian updates):**  \nHistogram: (array([  0,   0,  13,  73, 150, 308,  40,  25,   0,   0]), array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]))  \nMean & std.: 0.4913 ± 0.0929\n\nAs shown above, our procedure is indeed able to yield graded beliefs - not 0s and 1s only. Below, we also show the **human prior beliefs** for the same set of hypotheses for comparison:  \nHistogram: (array([  1,   0,   5,   7,  28, 102, 129, 176, 115,  46]), array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]))  \nMean & std.: 0.7067 ± 0.1411\n\nFurthermore, our hypothesis verification procedures do produce p-values. However, instead of hard-coding the significance value that should be used to reject the null hypothesis, we instead sample LLM posterior beliefs given the evidence from the result of the statistical tests (including the p-values) to determine the change in belief for the hypothesis.\n\n---\n\n> `Q3: What n was used for belief elicitation?`\n\nWe use 30 for GPT4o, and 8 for o4-mini (we had to reduce the samples for o4-mini because 8 is the maximum number of responses we can generate in a single call). Note that the number of samples can be scaled up to any arbitrary number, though we do not see enough variance when we increase it from our current number for these models. Please see below a sensitivity analysis of the choice of number of samples on the proportion of surprisals found conducted using 100 randomly sampled hypotheses generated from the SEA-AD dataset:\n\n| Samples | Surprisal Proportion |\n|---------|----------------------|\n| 1       | 0.9531                 |\n| 5       | 0.9219                 |\n| 10      | 0.8906                 |\n| 20      | 0.9219                 |\n| 30      | 0.8906                 |\n| 40      | 0.8906                 |\n| **Aggregate** | **0.9115 ± 0.0233 (std.)**         |\n\n---\n\n> `Limitations: It would be good to include results from LLMs other than GPT-4o.`\n\nWe do provide additional results comparing all baselines with another model - o4-mini (a “reasoning” model) - in Section 6 of our supplementary material. We see modest but steady gains in terms of cumulative surprisal counts with reasoning models (o4-mini vs. GPT-4o). Furthermore, qualitatively, we find that the complexity of hypotheses generated by o4-mini is higher than GPT-4o."}, {"review_id": "ywD53PNVfZ", "reviewer": "Reviewer_wYSZ", "summary": "This paper addresses the critical challenge of open-ended autonomous scientific discovery (ASD), where an AI system must decide which hypotheses to investigate without human guidance. The authors propose AUTODS, a novel framework that moves beyond ill-defined heuristics like \"interestingness\" and instead uses the principled metric of Bayesian surprise to guide exploration. Bayesian surprise is formalized as the KL-divergence between an LLM's prior and posterior beliefs about a hypothesis, before and after observing experimental evidence from an auto-generated verification procedure. To efficiently navigate the vast hypothesis space, AUTODS employs a Monte Carlo Tree Search (MCTS) algorithm that uses this surprisal metric as its reward function. The method is rigorously evaluated on 21 real-world datasets, demonstrating superior performance in discovering surprising findings compared to strong search baselines. Crucially, a human study validates that the model's \"surprise\" correlates strongly with that of human experts, marking a significant step towards truly autonomous scientific exploration.", "questions": "The framework equates surprising the LLM with scientific discovery. How does the system defend against simply rediscovering facts that are known to human experts but were absent in the model's training data? Could the surprisal metric be augmented with a \"novelty to humanity\" check (e.g., via targeted literature or web search) to better align the model's discovery frontier with the human scientific frontier?\n\nThe Beta-Bernoulli model for belief elicitation is a key component. How sensitive are the resulting surprisal scores and the MCTS search performance to the choice of both the elicitation model (e.g., GPT-3.5 vs. GPT-4o) and the number of samples (n) used for estimation? Please provide analysis on the stability and variance under these differing conditions.\n\nMCTS is well-motivated, but could the surprisal reward incentivize the agent to find \"gotchas\" or quirky statistical outliers rather than pursuing a coherent and systematically surprising research direction? How does the system ensure a high surprisal on a single node translates to a fruitful research branch, rather than a dead end?\n\nFor reproducibility, could you please report the key MCTS hyperparameters used in your experiments (specifically, the exploration constant C, and the progressive widening parameters k and alpha) and provide brief justification for their selection?", "limitations": "The authors provide a good error analysis in the appendix and discuss the latency of their approach. For a more complete discussion, they should also explicitly address the risks of LLM hallucinations in the code generation and analysis steps, and the broader ethical implications of deploying autonomous agents that could generate and promote potentially spurious scientific hypotheses without sufficient human oversight.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Principled Guiding Metric\nThe paper's primary strength is its conceptual shift from ambiguous proxies like \"interestingness\" to the formally defined and theoretically grounded metric of Bayesian surprise. This provides a robust and replicable foundation for guiding open-ended discovery.\n\n2. Novel and Effective Synthesis\nThe authors impressively synthesize three complex components into a coherent system (1) a practical method for eliciting Bayesian belief distributions from LLMs, (2) the use of KL-divergence as a reward signal, and (3) a sophisticated MCTS with progressive widening to effectively balance exploration and exploitation. This combination is both novel and demonstrably effective.\n\n3. Exceptional Empirical Validation\nThe evaluation is comprehensive and convincing. The system's outperformance of strong baselines across 21 diverse datasets is significant. The inclusion of a human evaluation study is a critical and standout feature, providing strong evidence that optimizing for the LLM's surprisal is a valid proxy for discovering findings that are surprising to human domain experts. The careful validation of the agent's internal components, such as experiment correctness and deduplication, further strengthens the work's quality.\n\nWeaknesses\n\n1. The \"Observer\" Problem\nThe framework's goal is defined as expanding the LLM's own knowledge frontier. This pragmatically conflates the LLM's surprise—an artifact of its training data's scope—with genuine scientific surprise. The system may therefore be surprised by a fact that is well-established within a niche scientific field but absent from its general training corpus, potentially leading it to \"rediscover\" known science.\n\n2. Fidelity and Robustness of Belief Elicitation\nThe belief elicitation process models the LLM's complex epistemic state with a Beta distribution estimated from a small number of boolean samples from a single proprietary model (GPT-4o). This elegant simplification may not fully capture the nuances of a model's confidence, and its robustness to smaller, open, or domain-specific models is untested.\n\n3. Computational Intensity\nThe search process is inherently computationally expensive, with each node in the MCTS tree requiring multiple LLM calls for hypothesis generation, verification, and belief elicitation (reported as 75 seconds/node on average). While the method is more efficient than baselines, this high absolute cost may limit its scalability for truly massive discovery campaigns.", "was_revised": true, "final_justification": "I re-read the paper and rebuttal and keep my score at 5 (Accept), confidence 4. The authors addressed my main questions: (i) the “binary” belief concern—belief elicitation is graded via Beta-Bernoulli with n=30 (GPT-4o) / n=8 (o4-mini) at T=0.7, and their sensitivity analysis shows low variance in surprisal rates as n varies; (ii) reproducibility—MCTS hyperparameters are now reported (C=1, progressive-widening k=1, α=0.5) with rationale; (iii) robustness—results with a second model (o4-mini) preserve the trends and yield more complex hypotheses; and (iv) reward hacking—the surprisal signal sits in the outer loop while hypothesis generation remains decoupled, which mitigates “gotcha” optimization in practice. The human study is also strengthened with a 95% CI for LLM–human surprisal agreement (~0.67, CI [0.63, 0.71]).\n\nI still view two limitations as worth highlighting for the camera-ready: (1) Compute/latency (~75s per node) and heavy multi-call dependence, even if the throughput compares favorably to manual workflows; and (2) the “observer” problem—surprising the LLM is not identical to novelty-to-humanity. The paper already sketches literature-grounded extensions; I encourage adding a lightweight “novelty-to-human” check (targeted retrieval or citation search) and a brief note on dataset-selection/open-endedness. A larger multi-model sweep and a broader human study would further solidify the claims but are not blockers. Overall, the work is well-motivated, carefully engineered, and empirically convincing; I recommend accept.", "rebuttal": "Thank you for the time you have taken to review our work! Below, we address each of your concerns and questions.\n\n---\n\n> `W1: The \"Observer\" Problem…to \"rediscover\" known science.`\n\n> `Q1: Could the surprisal metric be augmented…via targeted literature or web search…with the human scientific frontier?`\n\nIndeed, the performance of AutoDS depends on the quality of priors that are encoded in the LLM with respect to the domain of interest. However, the main contribution of our work is in proposing a computational framework that is useful for scientific discovery. Our vision is that practitioners will use AutoDS with custom models—pre-trained or fine-tuned on their domains of interest—to align discovery with their research goals. Furthermore, AutoDS is not only applicable to the data-driven discovery setting, but has a natural extension to literature-driven discovery (as also suggested in your question). This offers another avenue to prevent the re-discovery problem by grounding both hypothesis generation and belief elicitation in a retrieved set of relevant papers, which would provide additional domain knowledge that goes beyond model pre-training.\n\n---\n\n> `W2: Fidelity and Robustness of Belief Elicitation…may not fully capture the nuances of a model's confidence`\n\nDespite the simplicity of the Bernoulli-Beta belief formulation we use in AutoDS, we do find that it is sufficiently robust in providing alignment with human beliefs. As described in Section 5.2 (and Section 4 in the supplementary material), we test this by conducting a human study across 4 datasets. Our analysis finds that human surprisal has a correlation with LLM surprisals found via the Beta-Bernoulli belief elicitation in 67% of the hypotheses. Additionally, the 95% confidence interval is [0.63, 0.71], computed using bootstrapping with 10,000 re-samples (we will include this detail in the final version of the paper).\n\n---\n\n> `Q2: How sensitive are the resulting surprisal scores and the MCTS search performance to the choice of both the elicitation model (e.g., GPT-3.5 vs. GPT-4o) and the number of samples (n) used for estimation?...`\n\nWe have included a comparison between GPT-4o and o4-mini in our supplementary material (Section 6 “Results with o4-mini (“reasoning” models)” and Figures 5 and 6). In particular, we find that the same trend across search methods holds for both models. With o4-mini, we additionally observe that the gap between MCTS and greedy reduces; quantitatively, we find more number of surprisals than with GPT-4o; qualitatively, we find the hypotheses proposed by o4-mini to involve more complex statistical analyses.\n\nOn your suggestion, we have now run a sensitivity analysis of the number of samples used for belief elicitation on the downstream cumulative surprisal count. To do this analysis, we sample 100 hypotheses at random from the MCTS run on the Seattle-Alzheimer’s (SEA-AD) dataset. We then regenerate prior and posterior beliefs for these hypotheses using different numbers of samples and report the proportion of hypotheses found surprising in each case. Our results are as follows:\n\n| Samples | Surprisal Proportion |\n|---------|----------------------|\n| 1       | 0.9531                 |\n| 5       | 0.9219                 |\n| 10      | 0.8906                 |\n| 20      | 0.9219                 |\n| 30      | 0.8906                 |\n| 40      | 0.8906                 |\n| **Aggregate** | **0.9115 ± 0.0233 (std.)**         |\n\nAs indicated by the low standard deviation, we see only modest variation in the final surprisal proportion across settings of the number of samples used for belief elicitation.\n\n---\n\n> `W3: …While the method is more efficient than baselines, this high absolute cost may limit its scalability for truly massive discovery campaigns.`\n\nIn section 5.4, while we highlight that the average processing time per node (hypothesis generation to verification) of 75 seconds may be improved by an initial programmatic search, we emphasize that AutoDS, nonetheless, provides a massive improvement over the average time a human researcher takes to perform all steps in a node. Gu et al. (2024) highlight that data analysts often take ~20 minutes to even detail their analysis plan before implementing standard data analysis tasks. Further, Majumder et al. (2025) show that it can take 90 human hours to reasonably replicate complex data-driven scientific studies from documentation and open-sourced code. Moreover, we expect that the gap between the time taken by a human vs. AutoDS to generate and verify a new hypothesis and experiment increases with complexity. We argue, therefore, that 75 seconds per node represents very high throughput for autonomously generating and verifying scientific hypotheses in data-driven discovery compared to a completely manual undertaking, and does not, therefore, present a barrier to larger discovery campaigns. We believe that the barrier comes from being able to consistently generate useful hypotheses, which we aim to address in our paper.\n\n**References:**  \n*How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. Gu et al., 2024.  \nDiscoveryBench: Towards Data-Driven Discovery with Large Language Models. Majumder et al., 2024.*\n\n---\n\n> `Q3: …could the surprisal reward incentivize the agent to find \"gotchas\" or quirky statistical outliers... How does the system ensure a high surprisal on a single node translates to a fruitful research branch, rather than a dead end?`\n\nReward hacking is indeed important to guard against in a system like AutoDS. We argue that the modular agent structure in our system provides a natural safeguard against this behavior. That is, in AutoDS, the MCTS search mechanism, which utilizes surprisal-based rewards, is what is used to prioritize which nodes amongst previously generated ones should be selected for further expansion (new hypothesis generation). However, the hypothesis generation agent, itself, operates independent of the knowledge of this surprisal reward. The diversity in new hypotheses thus comes from conditioning on diverse branches of the tree, and not a direct signal to optimize for the surprisal reward, which is only used by the outer-loop.\n\nIt is worth mentioning, additionally, that AutoDS, too, cannot ensure that a node translates to a fruitful research branch. However, using the hypothesis that nodes that are surprising are likely to lead to further surprisals, we use MCTS to prioritize surprisal nodes for expansion. As shown in our main results, this indeed leads to the most number of cumulative surprises across 21 datasets vs. using random sampling or a greedy strategy.\n\n---\n\n> `Q4: For reproducibility, could you please report the key MCTS hyperparameters used in your experiments (specifically, the exploration constant C, and the progressive widening parameters k and alpha) and provide brief justification for their selection?`\n\nWe set exploration constant C=1, as a default setting across datasets to have exploration and exploitation contribute equally. Note that given either prior knowledge about the domain or additional budget to test different settings, C may be tuned to obtain better search performance. For progressive widening, we set 𝛼 = 0.5 and k = 1 as the constant setting across domains. Again, we do not tune this, but could if the budget allows. Our selection aims to get trees of balanced depth and width. We will include these details in the main paper. Thank you!\n\n---\n\n> `Limitations: …explicitly address the risks of LLM hallucinations in the code generation and analysis steps, and the broader ethical implications of deploying autonomous agents that could generate and promote potentially spurious scientific hypotheses without sufficient human oversight.`\n\nWe are committed to responsibly discussing our results and its implications. We address a few of these concerns in Appendix B Error Analysis as well as take a cautionary tone in our conclusion. We will additionally provide an expanded discussion on the risks of automated discovery in our final version of the paper."}, {"review_id": "gaSX2Mu59g", "reviewer": "Reviewer_8GRz", "summary": "This paper introduces AUTODS, a novel framework for open-ended autonomous scientific discovery (ASD) that leverages Bayesian surprise as a guiding principle to explore hypotheses without predefined goals. While prior work in ASD often requires human-specified questions, AUTODS allows a language model to autonomously propose, test, and evaluate hypotheses based solely on data, thereby simulating the behavior of an independent scientific agent.The core idea is to quantify epistemic shifts—changes in the model’s belief before and after observing experimental results—using Bayesian surprise. The method formalizes belief elicitation via sampling, applies a Beta-Bernoulli model for prior/posterior estimation, and defines a surprisal score to capture meaningful belief updates. To efficiently explore the hypothesis space, AUTODS integrates Monte Carlo Tree Search (MCTS) with progressive widening, using surprisal as the reward signal to balance exploration and exploitation.Experiments across 21 real-world datasets from domains including biology, economics, and behavioral science show that AUTODS outperforms strong baselines—producing 5–29% more discoveries judged surprising by the LLM. A human study found that 67% of these LLM-surprising discoveries also surprised human experts, confirming the alignment between Bayesian surprise and human intuition. The authors also compare Bayesian surprise with other automatic reward metrics (e.g., LLM interestingness/utility) and validate AUTODS’s internal experiment execution and deduplication mechanisms.In conclusion, the paper presents a promising and scalable approach to fully autonomous, open-ended scientific discovery by marrying principled Bayesian reasoning with powerful language models.", "questions": "1. How does the system respond to misleading or incorrect data? Is it susceptible to being misled by \"spurious surprise\"?\n\n2. How can the model's ability to transition from \"discovering correlations\" to \"inferring causality\" be improved?\n\n3. Can this method be extended to literature-driven open discovery, rather than being limited to data-driven discovery?", "limitations": "Despite proposing an innovative approach in the field of open-ended scientific discovery, this paper still has some limitations. First, its \"surprise\" metric relies on the belief changes of the language model itself, which is subjective and may not always represent truly valuable scientific discoveries. Second, due to the limited context window of the language model, its reasoning ability is restricted as the exploration deepens. In addition, the current verification process mainly depends on simple statistical analysis and is not yet capable of handling complex scientific mechanisms or causal reasoning problems. The method is computationally expensive, with each hypothesis node taking a long time on average, which limits the efficiency for large-scale applications. Finally, the system is sensitive to data quality and tends to discover \"surprising\" positive conclusions, potentially overlooking negative findings or the closed loop of scientific validation.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. **Well-Designed Methodology**: Effectively balances exploration and exploitation by integrating Monte Carlo Tree Search (MCTS) with Bayesian surprise.\n\n2. **Outstanding Empirical Results**: Significantly outperforms existing methods on 21 real-world datasets.\n\n**Weaknesses / Limitations**\n\n1. **Dependent on Subjective Beliefs Expressed by LLMs**: The model's \"surprise\" does not equate to objective \"novelty\" or \"importance,\" and may be biased.\n\n2. **Limited Depth of Reasoning**: Currently focuses on shallow statistical hypotheses and has not demonstrated the ability to model complex scientific mechanisms.\n\n3. **High Computational Overhead**: Each node requires an average of 75 seconds for processing, resulting in a relatively slow overall search speed.", "was_revised": true, "final_justification": "Primary concerns have been resolved", "rebuttal": "Thank you for the time you have taken to review our work! Please see our responses below to each of your comments and questions.\n\n---\n\n> `W1: The model's \"surprise\" does not equate to objective \"novelty\" or \"importance\"`\n\nIndeed, “surprising” hypotheses need not always be novel or important. We, therefore, conducted an extensive human study to assess how many surprisals from our system are indeed “interesting” and of “high utility”. We find that, when optimized for Bayesian Surprise, 73% of hypotheses deemed surprising were also attributed as “interesting” and 79% of the surprising hypotheses were of “high utility” according to STEM MS/PhDs. We detail these experimental results in Section 5.2 from Line 284 and Table 2.\n\n---\n\n> `W2: Shallow statistical hypotheses`\n\nWe find that the richness of statistical hypotheses is a function of the underlying LLM. For example, as described in Section 6 of our supplementary material (L176), we find, qualitatively, that the complexity of hypotheses generated by o4-mini is significantly higher than GPT-4o. \n\nE.g., the following is a level 5 node found from the Freshwater Fish dataset (DiscoveryBench):  \n*“Within South American freshwater-fish sub-basins, evolutionary rates (diversification and morphological evolution) exhibit significant positive spatial autocorrelation, such that geographically proximate basins have more similar rates than distant ones.”*\n\nThis hypothesis was verified by the following experimental plan:  \n1. Compute global and local spatial autocorrelation (Moran's I, LISA) for the key response variables: diversification rate (DR, BAMM_NetDiv) and a morphological composite (e.g., the PC1 from PCA on BEL_evol–RML_evol).  \n2. Construct a spatial weights matrix (e.g., k‐nearest neighbors or distance‐based) from the lat/long coordinates using PySAL.  \n3. Report global Moran's I and its significance for each variable, then map local Moran's I clusters to identify hotspots and coldspots of high/low evolutionary activity across basins.\n\n---\n\n> `W3: High computational overhead`\n\nIn section 5.4, while we highlight that the average processing time per node (hypothesis generation to verification) of 75 seconds may be improved by an initial programmatic search, we emphasize that AutoDS, nonetheless, provides a massive improvement over the average time a human researcher takes to perform all steps in a node. Gu et al. (2024) highlight that data analysts often take ~20 minutes to even detail their analysis plan before implementing standard data analysis tasks. Further, Majumder et al. (2025) show that it can take 90 human hours to reasonably replicate complex data-driven scientific studies from documentation and open-sourced code. Moreover, we expect that the gap between the time taken by a human vs. AutoDS to generate and verify a new hypothesis and experiment increases with complexity. We argue, therefore, that 75 seconds per node represents very high throughput for autonomously generating and verifying scientific hypotheses in data-driven discovery compared to a completely manual undertaking.\n\n**References:**  \n*How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. Gu et al., 2024.  \nDiscoveryBench: Towards Data-Driven Discovery with Large Language Models. Majumder et al., 2024.*\n\n---\n\n> `Q1: How does the system respond to misleading or incorrect data? Is it susceptible to being misled by \"spurious surprise\"?`\n\nWe argue that any research undertaking (human or machine) is only as good as the quality of the data sources used in the research. AutoDS is not an exception to this—if the input dataset is noisy, each of hypothesis generation, verification, and analysis will accumulate errors, leading to failure. Regarding spurious surprise, we find that priors encoded within the LLM from pre-training as well as instruction-following skills imbued at post-training lend significant robustness to this error mode. In our experiments, we do not encounter instances where the model attempts to “reward hack” to generate contrived hypotheses that would always lead to surprisal.\n\n---\n\n> `Q2: How can the model's ability to transition from \"discovering correlations\" to \"inferring causality\" be improved?`\n\nThis is a great question! We believe there are several ways this transition may be made. First, if the input dataset itself is large enough, predictive models to verify hypotheses may be trained on a train subset of the dataset and then further evaluated on an unseen test split. Second, verification may be conducted on additional datasets (our implementation already supports taking in multiple data sources). Third, equipping the model with knowledge of causal modelling via existing programmatic constructs may be done explicitly. For e.g., this may be provided as additional instructions to the programmer agent in the inner-loop agentic framework.\n\n---\n\n> `Q3: Can this method be extended to literature-driven open discovery, rather than being limited to data-driven discovery?`\n\nIndeed! Our framework provides natural extensions to incorporating literature in two places. During hypothesis generation, relevant papers to the node selected by MCTS may be retrieved to inform subsequent generation. During belief elicitation, papers relevant to the hypothesis being evaluated can be retrieved as additional evidence and injected in-context before generating Bernoulli belief samples.\n\n---\n\n> `Limitations: …tends to discover \"surprising\" positive conclusions, potentially overlooking negative findings or the closed loop of scientific validation.`\n\nWe disagree with this conclusion. Our experiments show that AutoDS is able to surface positive and negative findings in equal measure. Please see the following generations as examples of negative findings that AutoDS discovered:  \n*“There is no significant difference between the property damage estimates adjusted to 2013 and 2015 monetary values”*  \n*“There is no significant difference in the total number of deaths between hurricanes with male and female names.”*  \n*“There is no significant correlation between the last MOCA score and brain pH levels in the dataset.”*"}, {"review_id": "d5AMywuTSB", "reviewer": "Reviewer_7cwK", "summary": "This paper is about LLM-based autonomous scientific discovery. It focuses on open-ended hypotheses selection and proposes to use bayesian surprise over diversity and human interestingness as an objective to be optimized. Bayesian surprise is calculated by assuming that a hypothesis is either true of false (no graded correctness) with respect to data and a verifier, based on a given data, can be synthesized and executed. To optimize this quantity, the paper uses MCTS to search for hypotheses with high surprisal (which is Bayesian Surprise under belief shift). Experiments are done on a subset of DiscoveryBench, BLADE, and SEA-AD, totaling 21 datasets. Results show that MCTS outperforms other methods in optimizing this bayesian surprisal objective and demonstrate on 4 datasets that humans (3 MS/PhDs) hypotheses chosen by the proposed method more surprising than MCTS optimizing other objectives.", "questions": "- How would the proposed method work with a hypothesis like \"the coin is unbiased\" and observe coin flips? \n- Displaying more qualitative results on how the proposed method is doing compared to other baselines / ablated methods would be good for the paper.\n- Human evaluation is done on the hypotheses generated on only 4 out of 21 datasets. What are these 4 datasets? What do they look like?\n- What do the error bars look like for table 1, given that there are only 3 human evaluators? How was inter-annotator agreement calculated? Are the human evaluators the authors themselves? Did the evaluators know which method was used to generate each hypothesis? I think the paper can benefit greatly from a full, proper human study", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n- The proposed method is well motivated (surprise, progressive widening, deduplication)\n- The paper is well written, easy to read and follow\n- The paper shows that humans do find hypotheses generated by the proposed method surprising, and I like how the paper completes this part of the story by showing that interesting ness and utility lack clear semantics, as judged by humans.\n\nWeakness:\n- I find the results to be very weak in supporting the main message of the paper:\nThe paper is proposing a new objective to optimize ASD, and I do not see how section 5.1 (whose main message is MCTS outperforms other search strategies) contribute to the main story: given an objective, it's not surprising or novel that MCTS performs better than beam search, greedy search etc. In section 5.2, only 3 human participants participate in evaluation. No error bars are provided. And given how LLM-heavy the method is, I expect evaluations on more LLMs, not just a single one (gpt-4o).\n- The assumptions made, in order to calculate bayesian surprise, is very limiting: a hypothesis is either true of false (no graded correctness) with respect to data. I think when we do science, we interpret data subjectively, and many times we see data that is inconclusive / confusing with respect to a hypothesis.", "was_revised": true, "final_justification": "All of my concerns are addressed.\n\nOther things that prevent me from giving even higher scores include the complexity of the domains used (also related to w1a) (they are not simple, and they are a good start, but they are not particularly impressive).", "rebuttal": "We sincerely appreciate the time you have taken to review our work! Below, we address each of your concerns and questions.\n\n---\n\n> `W1a: …results to be very weak in supporting the main message of the paper…I do not see how section 5.1…contribute to the main story…`\n\nOur main contribution is in developing a feasible computational framework to repeatedly sample hypotheses with LLMs that lead to scientific discoveries. Thus, we argue that the choice of which search algorithm drives our framework is indeed a key consideration, besides what reward function guides search (which we evaluate in Section 5.2). Amongst the choices lie various methods that have been shown to be strong candidates in several NLP tasks. For e.g., temperature-based ancestral sampling is often preferred when sampling textual sequences from LLMs despite the existence of more sophisticated techniques such as beam search and MCTS. Therefore, we contend that it remains a non-trivial result that MCTS in our setting of hypothesis generation is the best performer, and, therefore, does contribute meaningfully to our message.\n\n---\n\n> `W1b: …In section 5.2, only 3 human participants participate in evaluation. No error bars are provided…`\n\n> `Q4: What do the error bars look like for table 1, given that there are only 3 human evaluators? How was inter-annotator agreement calculated? Are the human evaluators the authors themselves? Did the evaluators know which method was used to generate each hypothesis?...`\n\nWe clarify that the human study described in section 5.2 is conducted not with 3 humans participants in total, but 3 participants per hypothesis. We provide a longer description of our human study in Section 4 Human Annotations in our supplementary material. The annotators were sourced from the Prolific platform, with a qualification of an MS/PhD STEM degree in either Mathematics and statistics, Information and Communication Technologies, Engineering, manufacturing and construction, or Natural sciences. We also required the residence of annotators to be US, UK, or Canada, and fluency in English. The annotators were not made aware of what method was used to generate the hypotheses.\n\nThank you for pointing out the lack of error bars in our results. We have now computed 95% confidence intervals using bootstrapping with 10,000 re-samples, resulting in a mean LLM agreement of 0.67 with 95% CI intervals [0.63, 0.71]. We will add these in the final version of the paper.\n\nFor inter-annotator agreement, we compute intraclass correlation (ICC) per familiarity class (low/medium/high) using the pingouin python package, as reported in Table 1.\n\n---\n\n> `W1c: …And given how LLM-heavy the method is, I expect evaluations on more LLMs, not just a single one (gpt-4o).`\n\nWe provide analyses with one additional model - o4-mini - in our supplementary material. We show its comparison with GPT-4o in Section 6 “Results with o4-mini (“reasoning” models)” and Figures 5 and 6. In particular, we find that the same trend across search methods holds for both models. With o4-mini, we additionally observe that the gap between MCTS and greedy reduces; quantitatively, we find more number of surprisals than with GPT-4o; qualitatively, we find the hypotheses proposed by o4-mini to involve more complex statistical analyses. \n\nFor e.g., the following is a level 5 node found from the Freshwater Fish dataset (DiscoveryBench):  \n*“Within South American freshwater-fish sub-basins, evolutionary rates (diversification and morphological evolution) exhibit significant positive spatial autocorrelation, such that geographically proximate basins have more similar rates than distant ones.”* \n\nDue to budget constraints, we are unable to provide analyses with any other models.\n\n---\n\n> `W2: The assumptions made, in order to calculate bayesian surprise, is very limiting: a hypothesis is either true of false (no graded correctness) with respect to data. I think when we do science, we interpret data subjectively, and many times we see data that is inconclusive / confusing with respect to a hypothesis.`\n\nWe clarify that our belief elicitation mechanism does indeed produce graded correctness. Note that while the individual Bernoulli samples from the LLM provide a boolean response, we generate n=30 samples from the LLM. These Bernoulli samples allow us to compute a Beta distribution through the Beta-Bernoulli conjugacy resulting in mean belief probabilities that lie anywhere between 0 and 1. To show that this also happens in practice, here is the histogram (bin count=10; bin width=0.1) of mean prior and mean posterior belief probabilities elicited from the LLM across ~600 hypotheses from runs on 4 datasets:\n\n**Prior Beliefs:**  \nHistogram: (array([ 80,  25,  20,  20,  21,  33,  35,  37,  58, 280]), array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]))  \nMean & std.: 0.6818 ± 0.3396\n\n**Posterior Beliefs (after Bayesian updates):**  \nHistogram: (array([  0,   0,  13,  73, 150, 308,  40,  25,   0,   0]), array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]))  \nMean & std.: 0.4913 ± 0.0929\n\nAs shown, our procedure is able to yield graded beliefs, and not 0s and 1s only. Below, we also show the **human prior beliefs** for the same set of hypotheses for comparison:  \nHistogram: (array([  1,   0,   5,   7,  28, 102, 129, 176, 115,  46]), array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]))  \nMean & std.: 0.7067 ± 0.1411\n\n---\n\n> `Q1: How would the proposed method work with a hypothesis like \"the coin is unbiased\" and observe coin flips?`\n\nAutoDS is well-suited to handle such a hypothesis. In particular, let’s assume that we have a dataset of coin flips available with a column indicating whether the coin came up heads or tails and each row corresponding to a single coin flip. Now, given the hypothesis “the coin is unbiased”, AutoDS might propose a verification experiment that uses the binomial test (using scipy.stats.binomtest) to check the null hypothesis that the probability of heads is 0.5. This test will then output a p-value, on the basis of which AutoDS will decide whether the hypothesis is supported or not, proceeding to elicit a posterior belief given the hypothesis and evidence from the verification experiment.\n\n---\n\n> `Q2: Displaying more qualitative results on how the proposed method is doing compared to other baselines / ablated methods would be good for the paper.`\n\nWe clarify that our experiments do not reveal significant qualitative variations between different choices of search algorithms (e.g., MCTS, greedy, repeated sampling, etc.) used within AutoDS. This is due to the fact that the same agentic framework is used by all methods as the inner-loop in AutoDS. However, the key difference between different search methods is in the ability to repeatedly find new, surprising hypotheses as described in Figure 2. It is worth noting that where we do see qualitative differences are when the underlying LLM is changed. In Section 6 Results with o4-mini (“reasoning” models) within our supplementary material, we describe how the complexity of statistical analyses increases when using o4-mini, which is a reasoning model from OpenAI, versus using GPT-4o. \n\n---\n\n> `Q3: Human evaluation is done on the hypotheses generated on only 4 out of 21 datasets. What are these 4 datasets? What do they look like?`\n\nWe use SEA-AD, BLADE-Hurricane, BLADE-Boxes, and DiscoveryBench-FreshwaterFish. Our choice was motivated by covering each of the data sources used in our study, while balancing costs for the human study (thus limiting to 4). Here are the natural language descriptions and example columns from each dataset:\n\n**SEA-AD:**  \n**Description:** Dataset containing information on donors including demographic details, cognitive status, and neuropathological assessments related to Alzheimer's disease and other conditions.  \n**Example columns:** APOE Genotype (a genetic risk factor for Alzheimer's disease), Age at Death, Fresh Brain Weight\n\n**BLADE-Hurricane:**  \n**Description:** The dataset is from the Simonsohn et al's “Specification curve analysis” paper published in Nature. It includes archival data on fatalities caused by hurricanes in the United States (1950–2012). Ninety-four Atlantic hurricanes made landfall in the United States during this period.  \n**Example columns:** alldeaths (Total number of deaths caused by the hurricane), gender_mf (Binary gender indicator of the hurricane name (0 for male, 1 for female)), wind (Maximum wind speed of the hurricane at the time of landfall in the United States from the NOAA)\n\n**BLADE-Boxes:**  \n**Description:** In this study, researchers investigated how children's social learning strategies develop across age in diverse cultural contexts. Social learning, or learning from others, is a key aspect of human cognition and cultural transmission. Children can rely on social information to varying degrees, and they may preferentially learn from majority demonstrations over minority ones. Understanding how these social learning biases emerge and change with age, and how they might be influenced by cultural context, is important for shedding light on the development of human cognition and the emergence of cultural diversity.  \n**Example columns:** majority_first (whether the majority option was demonstrated first), gender, age\n\n**DiscoveryBench-FreshwaterFish:**  \n**Description:** This dataset contains the drivers of speciation rates in South American freshwater fishes, employing an integrative approach that considers multiple biotic and abiotic factors.  \n**Example columns:** soil_div (measures the diversity of soil types or conditions within each sub-basin studied), RML_evol (Rate of Relative Maxillary Length evolution), BEL_evol (Rates of Body Elongation evolution)"}, {"review_id": "wIKK6jEkkz", "reviewer": "Reviewer_TsGq", "summary": "The authors observed that much of the prior work on Autonomous Scientific Discovery (ASD) has been goal-driven. In contrast, they aimed to propose a hypothesis generation method for a more open-ended setting, where the ASD system explores more broadly by generating hypotheses based on its own measures of research promise, executing the aforementioned steps, and then using the results to propose new hypotheses in a never-ending process.\n\nTo achieve this, the authors proposed a method that uses Monte Carlo Tree Search (MCTS) to explore hypotheses that yield high Bayesian surprise. Specifically, they conceptualized a tree where each node represents a hypothesis. For each node, they defined a hypothesis as surprising if it significantly changes the beliefs of a large language model (LLM) before and after observing the experimental result. Based on this formulation, they used MCTS to search for such hypotheses.\n\nThey evaluated their method on several datasets and demonstrated that it could discover more surprising hypotheses than baseline methods. Moreover, the surprises identified by their system were shown to correlate with the surprises perceived by human researchers.", "questions": "- Let me confirm the definition of a hypothesis. How is ‘hypothesis’ defined here? Are you referring to experimental ideas as hypotheses?\n- Is there any qualitative evaluation or explanation of why humans made such judgments? If so, I would like to refer to it.", "limitations": "Since the study aims for a more autonomous framework, I believe the practical quality of the generated output has declined, which I consider to be a limitation.\n\nAlso, the assumption of the existence of a dataset can be seen as a limitation in light of the ultimate goal this research is aiming to achieve.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Improving autonomy and open-endedness in ASD (Autonomous Scientific Discovery) is a central research challenge. The authors’ work is notable in this context, as they explore methodologies for autonomously evaluating hypotheses and proposing subsequent ones.\n\nWhile the idea of using Bayesian beliefs for hypothesis evaluation has been long-standing, the authors provide concrete computational formulations. Moreover, their method is carefully designed—not only to maximize Bayesian surprise, but also to ensure that such surprise leads to meaningful shifts, demonstrating thoughtful engineering.\n\nThe experiments are well-validated using multiple datasets and include human evaluations, adding robustness to the claims. Additionally, the authors conduct a variety of analytical experiments to examine their method from different perspectives, which further broadens the implications of this research. The paper documents the experimental details—including prompts, annotation processes, and algorithmic explanations—in both the main text and appendices, ensuring transparency in the verification process.\n\nThe paper also appropriately situates itself within the context of prior work. Not only does it reference background research relevant to this study, but it also draws upon a wide range of earlier findings in developing its methodology, making it a fruitful reference for future researchers developing novel approaches in this field.\n\nThat said, there are several concerns that warrant discussion.\n\nFirst is the practical significance of improving open-endedness, which this work aims to address. As I understand it, the authors recognize that many previous efforts in ASD have assumed the availability of data and goals (as well as researcher questions), and seek to move beyond goal-driven constraints by allowing the system to autonomously determine its own goals—thus achieving a more open-ended hypothesis generation.\n\nI agree that enabling AI to autonomously define its own goals is essential for autonomous scientific discovery. Of course, no goal can be defined from a blank slate—some prior assumption is always necessary. The authors assume that only data (and metadata) are given, and investigate how a system can autonomously discover goals from that. However, the importance of this “data-only” assumption compared to other initializations remains debatable. For example, instead of specifying a concrete research goal (which may be overly constraining), one could provide only an abstract goal (e.g., “develop safe AI”) and allow the system to autonomously derive more concrete research problems. This may still be more open-ended than traditional goal-driven systems.\n\nIn this alternative framework, the choice of dataset is not predetermined; in fact, the AI could autonomously select or even create its own datasets from the web. This arguably introduces greater open-endedness than assuming a fixed dataset. For instance, systems such as MLR-Copilot (Li et al., 2024) and The AI Scientist-v2 (Yamada et al., 2025), while not fully automating dataset acquisition, allow autonomous selection among diverse datasets by leveraging collections like Huggingface—similar to Prompt2Model (Viswanathan et al., 2023).\n\nMoreover, as the authors themselves mention, one could envision a framework where research problems are selected based on a literature survey, even in the absence of explicit goals. While the authors argue that their approach allows exploration of a broader space compared to literature-based methods, it’s worth noting that assuming only a literature survey grants flexibility in dataset selection—something the present approach lacks.\n\nConsidering these perspectives, it is not immediately obvious which framework—assuming only an abstract goal, a literature survey, or a dataset—leads to greater open-endedness in terms of goal selection. In real-world research scenarios, starting from a high-level objective and designing a study to achieve it may have more practical impact than starting from data alone. I acknowledge that this may reflect my own bias, and I would be very interested in hearing the authors’ reasoning on why their “data-only” approach is particularly important for advancing open-endedness.\n\nSecond, regarding related work: one closely relevant study is data-to-paper by Ifargan et al. (2024), which demonstrates fully autonomous LLM-driven research from raw data to publishable papers. While this study differs significantly in its methodology and does not diminish the novelty of the present work, it should still be acknowledged and discussed as a related approach. Also, this research seems to be related to the automation of data analysis. Since there are many previous studies on this topic as well, it would be good to refer to them appropriately.\n\nNext, regarding the experiments: Table 1 shows that the system can indeed generate hypotheses that are surprising to humans. This seems sufficient for validating the main claim. However, given the existence of many previous works on hypothesis generation, it would strengthen the case if the proposed method were compared—even partially—to some of these prior approaches. Although perfectly fair comparisons may be difficult due to differing assumptions, such a comparison would help substantiate the practical value of this method. Looking at the generated samples, it appears that the code produced involves relatively simple data analysis. \n\nFinally, let me discuss the output from the system. At first glance, it seems challenging to derive hypotheses with academic value from such output. I was curious about the qualitative evaluation—specifically, in what sense the generated hypotheses were perceived as “surprising.” While these hypotheses might indeed be more surprising compared to a baseline, I also wondered how they qualitatively compare to hypotheses generated by other hypothesis-generation methods. For instance, if we naively sampled hypotheses at random from both groups and asked an LLM-as-Judge to evaluate which hypothesis seemed better, I would be interested in knowing to what extent one group would be judged statistically superior.\n\nThere are some concerns, but overall, I believe the strengths of this work outweigh the weaknesses. I recommend acceptance of this paper.", "was_revised": true, "final_justification": "In my initial evaluation, I was concerned that although this study conducts a certain level of sound validation, the comparison with existing hypothesis generation and exploration methods is limited. However, after receiving the authors’ response, I came to interpret that the main contributions—the proposed exploration reward and framework—are supported by experiments. Additionally, the authors conducted an analysis showing that the exploration framework remains effective to some extent even when the internal hypothesis generation method is varied. This helped me judge that the experiments appropriately demonstrate the effectiveness of the exploration method.", "rebuttal": "Thank you for taking the time to review our work! We address your concerns and questions below.\n\n---\n\n> `W1: …it is not immediately obvious which framework—assuming only an abstract goal, a literature survey, or a dataset—leads to greater open-endedness…I would be very interested in hearing the authors’ reasoning on why their “data-only” approach is particularly important for advancing open-endedness.`\n\nWe emphasize that our main contribution is in presenting an efficient computational framework for repeatedly sampling hypotheses that are likely to lead to scientific discoveries (AutoDS). Our specific choice of using data-driven discovery as a test-bed was motivated by the feasibility of data-driven hypotheses to be evaluated via statistical tests using python programs. In particular, this paradigm does not need to access external data or perform physical experiments to verify hypotheses. Therefore, indeed, the search framework of AutoDS can be utilized with just an abstract goal or with literature. However, the mechanism for hypothesis verification would need to be specified.\n\n---\n\n> `W2: regarding related work: one closely relevant study is data-to-paper by Ifargan et al. (2024), which demonstrates fully autonomous LLM-driven research from raw data to publishable papers. While this study differs significantly…it should still be acknowledged and discussed as a related approach…`\n\nThank you for the reference! We will surely add it to our related works discussion in the final paper.\n\n---\n\n> `W3: …Table 1 shows that the system can indeed generate hypotheses that are surprising to humans…it would strengthen the case if the proposed method were compared—even partially—to some of these prior approaches…Looking at the generated samples, it appears that the code produced involves relatively simple data analysis.`\n\n> `W4: …I also wondered how they qualitatively compare to hypotheses generated by other hypothesis-generation methods…`\n\nAs mentioned above, our main contribution is in presenting a search framework. The inner-loop of hypothesis generation used in AutoDS can easily be swapped out for a different method. For the instantiation in our experiments, we use DataVoyager (Majumder et al., 2024) for the inner-loop, but could have instead used a literature-based method like Scideator (Radensky et al., 2024) as well. Furthermore, as described in our supplementary material L177-180, the statistical analyses deeper in the tree, especially with o4-mini, show increased complexity. E.g., *“Within South American freshwater-fish sub-basins, evolutionary rates (diversification and morphological evolution) exhibit significant positive spatial autocorrelation, such that geographically proximate basins have more similar rates than distant ones.”*\n\n**References:**  \n*Data-driven Discovery with Large Generative Models. Majumder et al., 2024.  \nScideator: Human-LLM Scientific Idea Generation Grounded in Research-Paper Facet Recombination. Radensky et al., 2024.*\n\n---\n\n> `Q1: Let me confirm the definition of a hypothesis. How is ‘hypothesis’ defined here? Are you referring to experimental ideas as hypotheses?`\n\nPlease see our formal definition in Section 2 Preliminaries L85-89, which follows from Majumder et al., 2025. We clarify that we use an “experiment” as a verification procedure to determine whether the hypothesis is supported or not given a dataset (in the data-driven discovery paradigm).\n\n**References:**  \n*DiscoveryBench: Towards Data-Driven Discovery with Large Language Models. Majumder et al., 2025.*\n\n---\n\n> `Q2: Is there any qualitative evaluation or explanation of why humans made such judgments? If so, I would like to refer to it.`\n\nWe do not collect reasoning from annotators about their beliefs and, in fact, note that it may be hard to verbalize the reason when prompted to provide a prior belief given a hypothesis (Rozenblit & Keil, 2002). We do note, however, that our annotations are averaged over 3 annotator responses, and, therefore, aim to capture variability in responses.\n\n**References:**  \n*The misunderstood limits of folk science: an illusion of explanatory depth. Rozenblit & Keil, 2002.*\n\n---\n\n> `Limitations: Since the study aims for a more autonomous framework, I believe the practical quality of the generated output has declined, which I consider to be a limitation. Also, the assumption of the existence of a dataset can be seen as a limitation in light of the ultimate goal this research is aiming to achieve.`\n\nWe re-emphasize that the main contribution of our work is in providing a computational search framework for scientific discovery. The quality of the inner-loop, therefore, is a related but distinct question. In order to improve the robustness of our results, one may, for example, use a mechanism such as POPPER to conduct several rounds of falsification experiments before concluding in order to build confidence in the verification of a proposed hypothesis. Similarly, one may choose to augment an LLM with relevant literature or to use models fine-tuned on domain-specific knowledge to inform both hypothesis generation and belief elicitation. AutoDS can seamlessly integrate each of these modifications."}]

Author rebuttal:

Meta-review: The paper proposes an approach for autonomous scientific discovery using the concept of Bayesian surprise in LLMs. Bayesian surprise is defined as the shift from prior beliefs  about a hypothesis to posterior beliefs after observing data, and is quantified using as the KL divergence between the LLM’s prior and posterior. This  is used as the reward function to drive MCTS. Empirical results show significant gains compared to alternatives under fixed budget conditions. Evaluation confirming 2/3 alignment with human judgements further supports the credibility of the proposed approach, specifically 73% of the hypotheses deemed surprising were also attributed as interesting and 79% were judged to be high quality.  

Given the ever increasing capabilities of LLMs, autonomous scientific discovery is highly  relevant and timely as noted by the reviewers. The proposed approach is well motivated,  clearly explained with sufficient  detail. The question about using “surprise” as a proxy to “scientific novelty” is addressed to some extent by the human study to assess how many surprisals suggested by the system are “interesting” or “high quality”. Authors’ rebuttal and discussion that followed helped clarify some points, especially regarding the belief elicitation mechanism about which multiple reviewers were confused. It would be good to incorporate some of the discussion points that helped clarify the approach in the camera ready version.
Decision: Accept (poster)