Title: AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise
Venue: NeurIPS 2025
Accepted: yes
Keywords: scientific discovery

LLM

open-ended

Bayesian

surprise

data-driven discovery

autonomous discovery
Abstract: The promise of autonomous scientific discovery (ASD) hinges not only on answering questions, but also on knowing which questions to ask. Most recent works in ASD explore the use of large language models (LLMs) in goal-driven settings, relying on human-specified research questions to guide hypothesis generation. However, scientific discovery may be accelerated further by allowing the AI system to drive exploration by its own criteria. The few existing approaches in open-ended ASD select hypotheses based on diversity heuristics or subjective proxies for human interestingness, but the former struggles to meaningfully navigate the typically vast hypothesis space, and the latter suffers from imprecise definitions. This paper presents AutoDiscovery—a method for open-ended ASD that instead drives scientific exploration using Bayesian surprise. Here, we quantify the epistemic shift from the LLM’s prior beliefs about a hypothesis to its posterior beliefs after gathering experimental results. To efficiently explore the space of nested hypotheses, our method employs a Monte Carlo tree search (MCTS) strategy with progressive widening using surprisal as the reward function. We evaluate AutoDiscovery in the setting of data-driven discovery across 21 real-world datasets spanning domains such as biology, economics, finance, and behavioral science. Our results demonstrate that under a fixed budget, AutoDiscovery substantially outperforms competitors by producing 5-29% more discoveries deemed surprising by the LLM. Our human evaluation further reveals that two-thirds of discoveries made by our system are surprising to domain experts as well, suggesting this is an important step towards building open-ended ASD systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors observed that much of the prior work on Autonomous Scientific Discovery (ASD) has been goal-driven. In contrast, they aimed to propose a hypothesis generation method for a more open-ended setting, where the ASD system explores more broadly by generating hypotheses based on its own measures of research promise, executing the aforementioned steps, and then using the results to propose new hypotheses in a never-ending process.\n\nTo achieve this, the authors proposed a method that uses Monte Carlo Tree Search (MCTS) to explore hypotheses that yield high Bayesian surprise. Specifically, they conceptualized a tree where each node represents a hypothesis. For each node, they defined a hypothesis as surprising if it significantly changes the beliefs of a large language model (LLM) before and after observing the experimental result. Based on this formulation, they used MCTS to search for such hypotheses.\n\nThey evaluated their method on several datasets and demonstrated that it could discover more surprising hypotheses than baseline methods. Moreover, the surprises identified by their system were shown to correlate with the surprises perceived by human researchers.\n\nSTRENGTHS AND WEAKNESSES:\nImproving autonomy and open-endedness in ASD (Autonomous Scientific Discovery) is a central research challenge. The authors’ work is notable in this context, as they explore methodologies for autonomously evaluating hypotheses and proposing subsequent ones.\n\nWhile the idea of using Bayesian beliefs for hypothesis evaluation has been long-standing, the authors provide concrete computational formulations. Moreover, their method is carefully designed—not only to maximize Bayesian surprise, but also to ensure that such surprise leads to meaningful shifts, demonstrating thoughtful engineering.\n\nThe experiments are well-validated using multiple datasets and include human evaluations, adding robustness to the claims. Additionally, the authors conduct a variety of analytical experiments to examine their method from different perspectives, which further broadens the implications of this research. The paper documents the experimental details—including prompts, annotation processes, and algorithmic explanations—in both the main text and appendices, ensuring transparency in the verification process.\n\nThe paper also appropriately situates itself within the context of prior work. Not only does it reference background research relevant to this study, but it also draws upon a wide range of earlier findings in developing its methodology, making it a fruitful reference for future researchers developing novel approaches in this field.\n\nThat said, there are several concerns that warrant discussion.\n\nFirst is the practical significance of improving open-endedness, which this work aims to address. As I understand it, the authors recognize that many previous efforts in ASD have assumed the availability of data and goals (as well as researcher questions), and seek to move beyond goal-driven constraints by allowing the system to autonomously determine its own goals—thus achieving a more open-ended hypothesis generation.\n\nI agree that enabling AI to autonomously define its own goals is essential for autonomous scientific discovery. Of course, no goal can be defined from a blank slate—some prior assumption is always necessary. The authors assume that only data (and metadata) are given, and investigate how a system can autonomously discover goals from that. However, the importance of this “data-only” assumption compared to other initializations remains debatable. For example, instead of specifying a concrete research goal (which may be overly constraining), one could provide only an abstract goal (e.g., “develop safe AI”) and allow the system to autonomously derive more concrete research problems. This may still be more open-ended than traditional goal-driven systems.\n\nIn this alternative framework, the choice of dataset is not predetermined; in fact, the AI could autonomously select or even create its own datasets from the web. This arguably introduces greater open-endedness than assuming a fixed dataset. For instance, systems such as MLR-Copilot (Li et al., 2024) and The AI Scientist-v2 (Yamada et al., 2025), while not fully automating dataset acquisition, allow autonomous selection among diverse datasets by leveraging collections like Huggingface—similar to Prompt2Model (Viswanathan et al., 2023).\n\nMoreover, as the authors themselves mention, one could envision a framework where research problems are selected based on a literature survey, even in the absence of explicit goals. While the authors argue that their approach allows exploration of a broader space compared to literature-based methods, it’s worth noting that assuming only a literature survey grants flexibility in dataset selection—something the present approach lacks.\n\nConsidering these perspectives, it is not immediately obvious which framework—assuming only an abstract goal, a literature survey, or a dataset—leads to greater open-endedness in terms of goal selection. In real-world research scenarios, starting from a high-level objective and designing a study to achieve it may have more practical impact than starting from data alone. I acknowledge that this may reflect my own bias, and I would be very interested in hearing the authors’ reasoning on why their “data-only” approach is particularly important for advancing open-endedness.\n\nSecond, regarding related work: one closely relevant study is data-to-paper by Ifargan et al. (2024), which demonstrates fully autonomous LLM-driven research from raw data to publishable papers. While this study differs significantly in its methodology and does not diminish the novelty of the present work, it should still be acknowledged and discussed as a related approach. Also, this research seems to be related to the automation of data analysis. Since there are many previous studies on this topic as well, it would be good to refer to them appropriately.\n\nNext, regarding the experiments: Table 1 shows that the system can indeed generate hypotheses that are surprising to humans. This seems sufficient for validating the main claim. However, given the existence of many previous works on hypothesis generation, it would strengthen the case if the proposed method were compared—even partially—to some of these prior approaches. Although perfectly fair comparisons may be difficult due to differing assumptions, such a comparison would help substantiate the practical value of this method. Looking at the generated samples, it appears that the code produced involves relatively simple data analysis. \n\nFinally, let me discuss the output from the system. At first glance, it seems challenging to derive hypotheses with academic value from such output. I was curious about the qualitative evaluation—specifically, in what sense the generated hypotheses were perceived as “surprising.” While these hypotheses might indeed be more surprising compared to a baseline, I also wondered how they qualitatively compare to hypotheses generated by other hypothesis-generation methods. For instance, if we naively sampled hypotheses at random from both groups and asked an LLM-as-Judge to evaluate which hypothesis seemed better, I would be interested in knowing to what extent one group would be judged statistically superior.\n\nThere are some concerns, but overall, I believe the strengths of this work outweigh the weaknesses. I recommend acceptance of this paper.\n\nQUESTIONS:\n- Let me confirm the definition of a hypothesis. How is ‘hypothesis’ defined here? Are you referring to experimental ideas as hypotheses?\n- Is there any qualitative evaluation or explanation of why humans made such judgments? If so, I would like to refer to it.\n\nLIMITATIONS:\nSince the study aims for a more autonomous framework, I believe the practical quality of the generated output has declined, which I consider to be a limitation.\n\nAlso, the assumption of the existence of a dataset can be seen as a limitation in light of the ultimate goal this research is aiming to achieve.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about LLM-based autonomous scientific discovery. It focuses on open-ended hypotheses selection and proposes to use bayesian surprise over diversity and human interestingness as an objective to be optimized. Bayesian surprise is calculated by assuming that a hypothesis is either true of false (no graded correctness) with respect to data and a verifier, based on a given data, can be synthesized and executed. To optimize this quantity, the paper uses MCTS to search for hypotheses with high surprisal (which is Bayesian Surprise under belief shift). Experiments are done on a subset of DiscoveryBench, BLADE, and SEA-AD, totaling 21 datasets. Results show that MCTS outperforms other methods in optimizing this bayesian surprisal objective and demonstrate on 4 datasets that humans (3 MS/PhDs) hypotheses chosen by the proposed method more surprising than MCTS optimizing other objectives.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The proposed method is well motivated (surprise, progressive widening, deduplication)\n- The paper is well written, easy to read and follow\n- The paper shows that humans do find hypotheses generated by the proposed method surprising, and I like how the paper completes this part of the story by showing that interesting ness and utility lack clear semantics, as judged by humans.\n\nWeakness:\n- I find the results to be very weak in supporting the main message of the paper:\nThe paper is proposing a new objective to optimize ASD, and I do not see how section 5.1 (whose main message is MCTS outperforms other search strategies) contribute to the main story: given an objective, it's not surprising or novel that MCTS performs better than beam search, greedy search etc. In section 5.2, only 3 human participants participate in evaluation. No error bars are provided. And given how LLM-heavy the method is, I expect evaluations on more LLMs, not just a single one (gpt-4o).\n- The assumptions made, in order to calculate bayesian surprise, is very limiting: a hypothesis is either true of false (no graded correctness) with respect to data. I think when we do science, we interpret data subjectively, and many times we see data that is inconclusive / confusing with respect to a hypothesis.\n\nQUESTIONS:\n- How would the proposed method work with a hypothesis like \"the coin is unbiased\" and observe coin flips? \n- Displaying more qualitative results on how the proposed method is doing compared to other baselines / ablated methods would be good for the paper.\n- Human evaluation is done on the hypotheses generated on only 4 out of 21 datasets. What are these 4 datasets? What do they look like?\n- What do the error bars look like for table 1, given that there are only 3 human evaluators? How was inter-annotator agreement calculated? Are the human evaluators the authors themselves? Did the evaluators know which method was used to generate each hypothesis? I think the paper can benefit greatly from a full, proper human study\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AUTODS, a novel framework for open-ended autonomous scientific discovery (ASD) that leverages Bayesian surprise as a guiding principle to explore hypotheses without predefined goals. While prior work in ASD often requires human-specified questions, AUTODS allows a language model to autonomously propose, test, and evaluate hypotheses based solely on data, thereby simulating the behavior of an independent scientific agent.The core idea is to quantify epistemic shifts—changes in the model’s belief before and after observing experimental results—using Bayesian surprise. The method formalizes belief elicitation via sampling, applies a Beta-Bernoulli model for prior/posterior estimation, and defines a surprisal score to capture meaningful belief updates. To efficiently explore the hypothesis space, AUTODS integrates Monte Carlo Tree Search (MCTS) with progressive widening, using surprisal as the reward signal to balance exploration and exploitation.Experiments across 21 real-world datasets from domains including biology, economics, and behavioral science show that AUTODS outperforms strong baselines—producing 5–29% more discoveries judged surprising by the LLM. A human study found that 67% of these LLM-surprising discoveries also surprised human experts, confirming the alignment between Bayesian surprise and human intuition. The authors also compare Bayesian surprise with other automatic reward metrics (e.g., LLM interestingness/utility) and validate AUTODS’s internal experiment execution and deduplication mechanisms.In conclusion, the paper presents a promising and scalable approach to fully autonomous, open-ended scientific discovery by marrying principled Bayesian reasoning with powerful language models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Well-Designed Methodology**: Effectively balances exploration and exploitation by integrating Monte Carlo Tree Search (MCTS) with Bayesian surprise.\n\n2. **Outstanding Empirical Results**: Significantly outperforms existing methods on 21 real-world datasets.\n\n**Weaknesses / Limitations**\n\n1. **Dependent on Subjective Beliefs Expressed by LLMs**: The model's \"surprise\" does not equate to objective \"novelty\" or \"importance,\" and may be biased.\n\n2. **Limited Depth of Reasoning**: Currently focuses on shallow statistical hypotheses and has not demonstrated the ability to model complex scientific mechanisms.\n\n3. **High Computational Overhead**: Each node requires an average of 75 seconds for processing, resulting in a relatively slow overall search speed.\n\nQUESTIONS:\n1. How does the system respond to misleading or incorrect data? Is it susceptible to being misled by \"spurious surprise\"?\n\n2. How can the model's ability to transition from \"discovering correlations\" to \"inferring causality\" be improved?\n\n3. Can this method be extended to literature-driven open discovery, rather than being limited to data-driven discovery?\n\nLIMITATIONS:\nDespite proposing an innovative approach in the field of open-ended scientific discovery, this paper still has some limitations. First, its \"surprise\" metric relies on the belief changes of the language model itself, which is subjective and may not always represent truly valuable scientific discoveries. Second, due to the limited context window of the language model, its reasoning ability is restricted as the exploration deepens. In addition, the current verification process mainly depends on simple statistical analysis and is not yet capable of handling complex scientific mechanisms or causal reasoning problems. The method is computationally expensive, with each hypothesis node taking a long time on average, which limits the efficiency for large-scale applications. Finally, the system is sensitive to data quality and tends to discover \"surprising\" positive conclusions, potentially overlooking negative findings or the closed loop of scientific validation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the critical challenge of open-ended autonomous scientific discovery (ASD), where an AI system must decide which hypotheses to investigate without human guidance. The authors propose AUTODS, a novel framework that moves beyond ill-defined heuristics like \"interestingness\" and instead uses the principled metric of Bayesian surprise to guide exploration. Bayesian surprise is formalized as the KL-divergence between an LLM's prior and posterior beliefs about a hypothesis, before and after observing experimental evidence from an auto-generated verification procedure. To efficiently navigate the vast hypothesis space, AUTODS employs a Monte Carlo Tree Search (MCTS) algorithm that uses this surprisal metric as its reward function. The method is rigorously evaluated on 21 real-world datasets, demonstrating superior performance in discovering surprising findings compared to strong search baselines. Crucially, a human study validates that the model's \"surprise\" correlates strongly with that of human experts, marking a significant step towards truly autonomous scientific exploration.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Principled Guiding Metric\nThe paper's primary strength is its conceptual shift from ambiguous proxies like \"interestingness\" to the formally defined and theoretically grounded metric of Bayesian surprise. This provides a robust and replicable foundation for guiding open-ended discovery.\n\n2. Novel and Effective Synthesis\nThe authors impressively synthesize three complex components into a coherent system (1) a practical method for eliciting Bayesian belief distributions from LLMs, (2) the use of KL-divergence as a reward signal, and (3) a sophisticated MCTS with progressive widening to effectively balance exploration and exploitation. This combination is both novel and demonstrably effective.\n\n3. Exceptional Empirical Validation\nThe evaluation is comprehensive and convincing. The system's outperformance of strong baselines across 21 diverse datasets is significant. The inclusion of a human evaluation study is a critical and standout feature, providing strong evidence that optimizing for the LLM's surprisal is a valid proxy for discovering findings that are surprising to human domain experts. The careful validation of the agent's internal components, such as experiment correctness and deduplication, further strengthens the work's quality.\n\nWeaknesses\n\n1. The \"Observer\" Problem\nThe framework's goal is defined as expanding the LLM's own knowledge frontier. This pragmatically conflates the LLM's surprise—an artifact of its training data's scope—with genuine scientific surprise. The system may therefore be surprised by a fact that is well-established within a niche scientific field but absent from its general training corpus, potentially leading it to \"rediscover\" known science.\n\n2. Fidelity and Robustness of Belief Elicitation\nThe belief elicitation process models the LLM's complex epistemic state with a Beta distribution estimated from a small number of boolean samples from a single proprietary model (GPT-4o). This elegant simplification may not fully capture the nuances of a model's confidence, and its robustness to smaller, open, or domain-specific models is untested.\n\n3. Computational Intensity\nThe search process is inherently computationally expensive, with each node in the MCTS tree requiring multiple LLM calls for hypothesis generation, verification, and belief elicitation (reported as 75 seconds/node on average). While the method is more efficient than baselines, this high absolute cost may limit its scalability for truly massive discovery campaigns.\n\nQUESTIONS:\nThe framework equates surprising the LLM with scientific discovery. How does the system defend against simply rediscovering facts that are known to human experts but were absent in the model's training data? Could the surprisal metric be augmented with a \"novelty to humanity\" check (e.g., via targeted literature or web search) to better align the model's discovery frontier with the human scientific frontier?\n\nThe Beta-Bernoulli model for belief elicitation is a key component. How sensitive are the resulting surprisal scores and the MCTS search performance to the choice of both the elicitation model (e.g., GPT-3.5 vs. GPT-4o) and the number of samples (n) used for estimation? Please provide analysis on the stability and variance under these differing conditions.\n\nMCTS is well-motivated, but could the surprisal reward incentivize the agent to find \"gotchas\" or quirky statistical outliers rather than pursuing a coherent and systematically surprising research direction? How does the system ensure a high surprisal on a single node translates to a fruitful research branch, rather than a dead end?\n\nFor reproducibility, could you please report the key MCTS hyperparameters used in your experiments (specifically, the exploration constant C, and the progressive widening parameters k and alpha) and provide brief justification for their selection?\n\nLIMITATIONS:\nThe authors provide a good error analysis in the appendix and discuss the latency of their approach. For a more complete discussion, they should also explicitly address the risks of LLM hallucinations in the code generation and analysis steps, and the broader ethical implications of deploying autonomous agents that could generate and promote potentially spurious scientific hypotheses without sufficient human oversight.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AutoDS, a Bayesian framework that emulates the process of scientific discovery by modeling posterior surprisal through Monte Carlo Tree Search. The methodology provides a formal definition of surprise in the context of automated scientific discovery (ASD), characterizing it as the Kullback–Leibler divergence (DKL) between prior and posterior beliefs. To further capture the directional and informational aspects of surprise, the concept is refined as Bayesian surprise under belief shift. Experiments on 21 real-world datasets demonstrate that AutoDS achieves 5–29% more discoveries deemed surprising by a large language model (LLM), outperforming strong search-based baselines.\n\nSTRENGTHS AND WEAKNESSES:\nI believe the authors address a highly relevant and timely question—shifting the focus from merely discovering novel or diverse scientific hypotheses to leveraging the emergent capabilities of LLMs to guide their own discovery processes. Given the rapid advancement of foundation models, I anticipate that this approach will gain broader traction within the AI4Science community. The paper is clearly written, and the experiments are thoughtfully and comprehensively designed.\n\nOne area for improvement would be a more explicit comparison with alternative formulations of uncertainty. For instance, in addition to using the Kullback–Leibler divergence (DKL) between prior and posterior distributions, it would be interesting to consider mutual information as a measure—evaluating how much information is gained after observing the data. (Although these formulations may be mathematically related, their interpretations can differ depending on the application.) I would point to Gao et al. [1] as an example, where mutual information is used to align conclusions with their most probable premises in scientific texts.\n\nThat said, the authors provide an in-depth treatment of DKL, which is sufficient for supporting the central idea and validating the proposed methodology.\n\n[1] Gao, Y., Gu, N., Lam, J., Henderson, J., & Hahnloser, R. (2024, March). Evaluating Unsupervised Argument Aligners via Generation of Conclusions of Structured Scientific Abstracts. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 151–160.\n\nQUESTIONS:\n1. Is any scientific discovery process truly binary? The assumption that the verification procedure must yield a binary outcome—i.e., supported vs. unsupported—feels overly restrictive. In NLP, for instance [2], researchers have explored the use of p-values reported in papers as a more continuous measure of how strongly evidence supports a hypothesis.\n\n2. What happens if an LLM produces different Boolean responses when given the same prompt multiple times? What temperature did you use when calling the APIs? Would such inconsistency undermine the reliability of the empirical frequency estimates?\n\n3. What $n$ was used for belief elicitation? \n\n[2] Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020, November). Fact or Fiction: Verifying Scientific Claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 7534-7550).\n\nLIMITATIONS:\nIt would be good to include results from LLMs other than GPT-4o.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-06"}