Title: MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search
Venue: NeurIPS 2025
Accepted: yes
Keywords: scientific discovery
Abstract: Large language models (LLMs) have shown promise in automating scientific hypothesis generation, yet existing approaches primarily yield coarse-grained hypotheses lacking critical methodological and experimental details. We introduce and formally define the new task of fine-grained scientific hypothesis discovery, which entails generating detailed, experimentally actionable hypotheses from coarse initial research directions. We frame this as a combinatorial optimization problem and investigate the upper limits of LLMs' capacity to solve it when maximally leveraged. Specifically, we explore four foundational questions: (1) how to best harness an LLM's internal heuristics to formulate the fine-grained hypothesis it itself would judge as the most promising among all the possible hypotheses it might generate, based on its own internal scoring-thus defining a latent reward landscape over the hypothesis space; (2) whether such LLM-judged better hypotheses exhibit stronger alignment with ground-truth hypotheses; (3) whether shaping the reward landscape using an ensemble of diverse LLMs of similar capacity yields better outcomes than defining it with repeated instances of the strongest LLM among them; and (4) whether an ensemble of identical LLMs provides a more reliable reward landscape than a single LLM. To address these questions, we propose a hierarchical search method that incrementally proposes and integrates details into the hypothesis, progressing from general concepts to specific experimental configurations. We show that this hierarchical process smooths the reward landscape and enables more effective optimization. Empirical evaluations on a new benchmark of expert-annotated fine-grained hypotheses from recent literature show that our method consistently outperforms strong baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study reports a method for scientific hypothesis generation that refines hypotheses in stages, from abstract to concrete, thereby enabling the generation of hypotheses that go beyond coarse-grained ones to include methodological and experimental details. The authors empirically validated that this hypothesis generation approach can produce better hypotheses and more closely align with ground-truth hypotheses. Furthermore, they found that the iterative application of a powerful LLM yields better results than using an ensemble of diverse LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nThe use of large language models (LLMs) for hypothesis generation in scientific research represents a highly significant and timely challenge. While substantial breakthroughs are expected in the near future, there remain numerous unresolved issues at present. As the authors of the paper under discussion rightly point out, one such issue is that naively prompting LLMs to generate hypotheses often results in overly abstract ideas. Bridging the gap between such abstract hypotheses and their concrete implementation remains an open research problem. The authors’ attempt to address this critical issue of “hypothesis concretization” is a commendable aspect of the work.\n\nThroughout the paper, the authors strive to enhance the reader’s understanding by associating key concepts with mathematical formulations and by employing various analogies. Their approach of interpreting LLM-based generation as a process akin to search and building their methodology around this analogy is both novel and intellectually stimulating.\n\nIn terms of evaluation, the authors go beyond relying solely on automated LLM assessment by incorporating human evaluations, which strengthens the credibility of their findings. Moreover, their focus on evaluating hypotheses in terms of their “detailedness” is an insightful and original perspective that adds value to the study. Finally, the proposed method consistently outperforms baseline approaches, demonstrating its practical effectiveness and further suggesting the potential significance of this contribution.\n\nWhile this is an excellent piece of work that contributes meaningfully to the field of AI-driven hypothesis generation, I believe that further improvements are necessary for it to meet the high standards required for acceptance at NeurIPS. Below, I outline the key reasons for this assessment.\n\nIf my understanding is correct, what this paper essentially proposes is the following:\n\t1.\tIteratively generate hypotheses while evaluating their quality using LLMs as judges,\n\t2.\tRepeat this process multiple times and treat the combined outputs as the hypothesis at a given level of abstraction,\n\t3.\tUse the high-level hypotheses obtained so far to move on to generate more concrete hypotheses (i.e., return to step 1).\n\nTo my knowledge, iteratively improving LLM outputs, generating a final output from multiple candidates, and progressively refining outputs into more concrete forms are not particularly novel ideas in and of themselves. In that sense, I felt that the method proposed in this study may not be sufficiently innovative or unique to be considered a significant advancement. The annotation of fine-grained hypotheses added to the dataset is indeed a contribution, but since it is based on an existing dataset, it is not an entirely new dataset.\n\nThe authors do not propose a new training method or architecture for LLMs but rather use existing models via APIs. Therefore, one would generally expect a carefully designed method for combining these models or crafting prompts; however, I found these aspects to be somewhat lacking. For these reasons, I believe there is still room for improvement in terms of originality.\n\nThis paper reports that the proposed method consistently outperforms baseline approaches, which suggests that the direction of the method could be practically valuable. However, the baselines used for comparison are very basic—essentially just self-consistency. Numerous approaches to hypothesis generation and prompting LLMs have been proposed previously, and it is reasonable to assume that many outperform simple self-consistency. Therefore, it was difficult to assess how well this method compares to more sophisticated alternatives.\n\nOf course, the comparisons with the chosen baselines are appropriate for validating the hypothesis, but to more convincingly demonstrate the utility and value of this approach to the AI community, I believe it would be beneficial to include comparisons with stronger baselines.\n\nRegarding the experimental design, it is important to evaluate alignment with ground truth, but in this case, such evaluation was conducted using LLMs as judges. It was unclear what specific instructions were given to the LLM judges or how effective they were—for example, how well their evaluations aligned with human judgments.\n\nHuman evaluations were conducted for the baseline comparisons, which I find desirable. However, the results only reported whether the proposed method outperformed existing methods or not. It was not clear how desirable the method’s properties were on their own, independent of its relative performance. While itemized evaluations are provided, only the overall human evaluation scores were reported. As a result, we cannot tell how each specific evaluation criterion was judged by human researchers. Therefore, I was not fully convinced of the overall persuasiveness of the experimental claims.\n\nFinally, while the use of metaphors and concept diagrams throughout the paper made the authors’ ideas more understandable, I felt that these descriptions were somewhat excessive in light of the need for more detailed explanations of the methodology, results, and additional analyses. For example, while the authors carefully explain that the process can be interpreted as combinatorial search, the actual proposal is just a particular way of prompting and combining outputs from an LLM—not a method involving any explicit combinatorial search technique. If that’s the case, I believe the paper would benefit more from devoting space to explaining the methodological details more concretely, as this would better convey the specifics of the research to readers.\n\nQUESTIONS:\nAm I correct in understanding that the core contribution of this paper lies in the design of prompts for the LLM and the method of combining its inputs and outputs? Furthermore, is the notion of “combinatorial search” intended as a conceptual framing rather than a literal explicit implementation of search space and search algorithm but rather leveraging prompt-based control of the LLM?\n\nLIMITATIONS:\nThe experiments could be made better by incorporating stronger benchmarks, more diverse datasets, and a larger number of annotators, and more.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces and formalizes the novel task of fine-grained scientific hypothesis discovery, which aims to generate detailed, experimentally actionable hypotheses from coarse-grained research directions. The authors frame this task as a combinatorial optimization problem over a vast, implicit space of possible experimental details. To tackle this, they propose a Hierarchical Heuristic Search (HHS) framework. This method decomposes the search problem into a series of smaller, more manageable subproblems arranged in a hierarchy of conceptual abstraction (e.g., from high-level reaction mechanisms down to specific parametric configurations in chemistry). The core intuition is that this hierarchical structure smooths the reward landscape, enabling an LLM-driven agent to more effectively navigate the search space and avoid poor local optima.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper's primary contribution is the identification and formalization of \"fine-grained scientific hypothesis discovery.\" This moves the field beyond generating high-level, often vague ideas towards producing concrete, testable, and actionable scientific plans.\n\nThe proposed Hierarchical Heuristic Search is well-motivated and intuitive. The analogy to coarse-to-fine refinement strategies is apt, and Figure 1 provides a concrete instantiation of this idea for the chemistry domain.\n\nThe paper is structured around three clear research questions (Q1-Q3), and the experiments are designed to answer them directly. The investigation into ensemble composition (Q3) is particularly insightful and offers practical guidance for future work.\n\n\nWeaknesses:\n\nThe HHS method, with its iterative search within each level, multiple independent runs for recombination, and progression through several hierarchical levels, appears to be computationally expensive in terms of LLM API calls and tokens. \n\nThe paper title aims at scientific discovery, but the paper does not discuss how this approach would generalize to other scientific domains (e.g., biology, materials science, physics) beyond chemistry. \n\nThe \"smoothing of the reward landscape\" is the central theoretical claim motivating HHS. However, this claim is supported only by illustrative schematics (Figures 3 & 4) and the downstream performance of the system. While the results are strong, they are indirect evidence. The paper would be significantly strengthened by a more direct, quantitative analysis attempting to measure this smoothing effect.\n\nQUESTIONS:\nCould you elaborate on the process of designing the 5-level chemistry hierarchy in Figure 1? How much expert effort was involved? More importantly, how sensitive is the performance of HHS to the specific design of this hierarchy? Can you easily extend the design to other scientific domain? For instance, what would happen if you collapsed levels 2 and 3, or used a different number of levels? This would help clarify the robustness and practical requirements of the method.\n\nCould you provide an analysis of the computational cost (e.g., approximate number of LLM calls or total tokens processed per problem instance) for HHS compared to the baselines?\n\nThe results show HHS consistently outperforming baselines. Were there any specific types of problems or hypotheses where HHS struggled or failed to produce a superior result? An error analysis could provide valuable insights into the limitations of the current framework.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on using large language models (LLMs) to solve the problem of fine-grained scientific hypothesis discovery. Addressing the shortcomings of existing methods, which generate rough hypotheses and lack experimental details, it formalizes this task as a combinatorial optimization problem for the first time. The study proposes a hierarchical heuristic search (HHS) framework, which refines hypotheses step by step from reaction mechanisms to complete experimental configurations, and utilizes internal heuristics of LLMs for candidate editing and quality assessment. The results show that HHS outperforms significantly in LLM self-assessment, expert scoring, and recall rate of real hypotheses, and that a single strong model ensemble performs better than a diverse model ensemble.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths :\nFormalizing \"fine-grained scientific hypothesis discovery\" as a combinatorial optimization problem, a hierarchical heuristic search (HHS) framework is proposed. It gradually refines hypotheses through a hierarchical structure (e.g., a five-level decomposition in the chemical field from reaction mechanisms to experimental configurations), effectively reducing the complexity of the search space. This hierarchical design reduces the traps of local optima by \"smoothing the reward landscape\", theoretically enhancing the stability of optimization.\n\nWeaknesses:\nThe definition of the fine-grained assumption relies on the annotation by domain experts, which may have subjective bias, and the benchmark scale is relatively small (based on 51 chemistry papers), so the robustness needs to be verified with larger-scale data.\n\nThe HHS framework cannot guarantee finding the global optimal solution (as admitted in the manuscript), and the hierarchical division relies on domain knowledge (such as the five-level structure in chemistry), which may carry the risk of over-reliance on prior assumptions. If the domain structure is complex or difficult to stratify, the method's effectiveness may decline.\n\nQUESTIONS:\nThe paper mentions using the internal heuristics of LLM as a reward signal to guide hypothesis search. Then, what is the specific form of the reward function? Is it defined solely through pairwise comparisons by LLM? During the optimization process, how can the stability and consistency of the reward function be ensured? Especially in different levels of search, will there be conflicts or deviations in the reward signals? For instance, the rewards at a higher level may favor the innovativeness of concepts, while those at a lower level focus more on the feasibility of experiments. How can the differences in rewards between these levels be balanced?\n\nThe paper defines the discovery of fine-grained hypotheses as a combinatorial optimization problem, with the complexity of the search space being C{^m}_n. When m and n are large, the computational complexity will increase exponentially. Although the hierarchical search framework alleviates this problem by restricting the search space at each level, in practical applications, how to determine the optimal search range at each level? When facing very complex hypotheses (such as those involving a large number of components and parameters), can HHS effectively avoid getting stuck in local optima? Are there theoretical analyses or experimental data to support the efficiency and effectiveness of HHS in high-dimensional search spaces?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates how LLMs can be used to generate fine-grained scientific hypotheses that could be directly implemented in laboratory settings, contrasting with prior studies that typically only produce coarse-grained ideas with low levels of detail. The authors propose a hierarchical heuristic search (HSS) approach to tackle this task, comparing it against two baseline methods that perform a flat search, namely greedy search with and without self-consistency. Using a benchmark set of 51 chemistry papers each annotated with a set of research background, a coarse-grained hypothesis, and a fine-grained hypothesis, the authors demonstrated that 1) HSS discovers superior local optima compared to the baseline methods in terms of effectiveness, novelty, detailedness and feasibility, 2) HSS-judged optimum is better aligned with the ground-truth hypotheses, and 3) using multiple instances of a strong single model is more effective at hypothesis optimization than combining different models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The work highlights and studies a timely and important challenge on how to leverage LLMs to generate experimentally actionable scientific hypotheses, which could significantly enhance the practical utility of LLMs in scientific discovery.  \n\n\n2. The proposed hierarchical heuristics search approach is conceptually sound, providing a logical framework for navigating the hypothesis space effectively. \n\n\nWeaknesses: \nThe specific questions/comments I have will be elaborated in the ‘Questions’ section but in general, the main weaknesses are: \n\n\n1. Evaluation metrics lack clarity - the definitions and evaluation methodology for the evaluation metrics are insufficiently explained, which limit interpretability of the results. \n\n\n2. Methodological details should provide clearer detail on how HSS is implemented. \n\n\n3. The conclusions drawn from the experiments for each research question would benefit from deeper analysis and clearer justification.\n\nQUESTIONS:\n1. Section 2.1 - The phrase “redundant or tangential elements” is somewhat vague. Could the authors provide specific examples of these elements in the context of hypothesis generation?\n\n\n2. Did the authors perform any quantitative analysis of the edits (e.g., number of insertions, deletions, or rewrites) across the hierarchy? This could strengthen their claims about the structured nature of the hierarchical heuristic search and its role in refining hypotheses.\n\n\n3. Section 2.5 - How are the hierarchical levels (p) and the number of hierarchical levels defined across different research topics or domains? Moreover, how are these levels conveyed to the LLM-driven agentic process? \n\n\n4. Section 3.2 - The paper mentions LLM-based dimension-specific evaluations. a) What scoring system is used for these assessments? b) How are these individual scores aggregated into an “overall” (LLM) evaluation? \n\n\n5. Section 3.2 - The authors claim to observe two trade-offs. How were these conclusions drawn? Could this be better described as a correlation, rather than a causal or generalizable trade-off? Similar trade-offs could arguably exist between novelty and feasibility, especially when feasibility depends on methodological availability.\n\n\n6. Section 3.3 - It is unclear whether the LLM-based evaluation of hypothesis alignment is simply matching keywords or phrases from the ground truth. If so, how do the authors ensure these matches are not out of context or semantically misleading (e.g., terms that appear similar but differ in meaning)? Concrete examples of both high and low scoring matches would help assess whether the 0–3 scale captures meaningful semantic overlap.\n\n\n7. Table 3 - There appear to be a high % of ties.  It would be helpful to explicitly discuss the implications of these ties on the strength of the conclusions drawn. I would imagine that the high tie rate might suggest that differences between committees are less pronounced than the win/loss counts alone indicate.\n\n\n8. Table 3 -  It has a confusing structure, as the comparison order is reversed between the left and right columns. For example, “Gemini 1.5 Flash committee vs. GPT-4o-mini committee” is in the left column, while the reverse appears in the right. Similarly, the row order for matchups is inconsistent (e.g., “Mixed committee vs. GPT-4o-mini” appears at the top on the left, but bottom on the right). Reformatting the table for consistent comparison direction and row order would improve clarity.\n\nLIMITATIONS:\nThe authors acknowledge that the hierarchical heuristic search (HHS) may not converge to the global optimum. However, it would strengthen the work to quantify how close HHS comes to the ground-truth hypotheses, assuming those can be treated as a proxy for the global optimum. Could the authors elaborate on whether Table 2 serves this purpose, i.e., does the recall of ground-truth components by HHS, greedy search, and self-consistent search reflect their respective distances from the global optimum?  If so, it would be helpful to explicitly frame Table 2 as a measure of this “distance to optimum” and discuss how often HHS actually recovers (or nearly recovers) the full ground-truth hypothesis. If not, are there alternative ways the authors could estimate or upper-bound the distance between generated hypotheses and the ground truth?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the task of fine-grained scientific hypothesis discovery and proposes a Hierarchical Heuristic Search (HHS) framework that leverages LLMs both to edit and evaluate hypotheses across multiple abstraction levels. Experiments on a small chemistry benchmark show that HHS outperforms greedy baselines in LLM self-assessments and expert rankings.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. Clear task definition & dataset – The authors articulate why coarse hypotheses are insufficient and release a contamination-controlled benchmark with expert annotations, filling a recognised gap in the literature.\n\n2. Coherent agentic framework – HHS is a self-contained, reproducible pipeline that factors an intractable search into tractable sub-problems and provides intuitive smoothing analysis.\n\n### Weaknesses\n1. The literature review is exceptionally thin for a NeurIPS-level submission—only ~15 references are listed, many of them generic model or dataset announcements rather than directly relevant prior art. Key recent work on hierarchical planning, tree-of-thought search, and agent-based scientific discovery published in NeurIPS/ICLR/ACL 2024-25 is absent. This shallow bibliography suggests the authors have not situated their contribution within the full landscape of related methods and undermines claims of novelty and significance. \n\n2. The contribution’s novelty is overstated. Prior systems, such as Chemcrow [1] and other retrieval-plus-refine agents already tackle hypothesis generation (some explicitly optimise for novelty and detail); the paper does not cite or benchmark against them nor clarify conceptual differences, leaving incremental value uncertain.\n\n3. HHS is essentially a coarse-to-fine or evolutionary search variant; the “hierarchy smooths reward” claim is supported only by schematic plots, lacking formal metrics or convergence theory, so theoretical depth is thin.\n\n4. Baselines omit standard combinatorial optimisers (reinforcement learning, MCTS, genetic algorithms) and recent retrieval-augment-edit pipelines, so HHS’s superiority is unproven outside the chosen comparisons.\n\n[1] Bran, Andres M., et al. \"Chemcrow: Augmenting large-language models with chemistry tools.\" arXiv preprint arXiv:2304.05376 (2023).\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nNo. Please do a thorough literature review and compare with SOTA methods.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-03"}