MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search
Authors: Zonglin Yang
Wanhao Liu
Ben Gao
Yujie Liu
Wei Li
Tong Xie
Lidong Bing
Wanli Ouyang
Erik Cambria
Dongzhan Zhou
Venue: NeurIPS 2025 poster ()
Keywords: scientific discovery
Primary area: applications

Abstract: Large language models (LLMs) have shown promise in automating scientific hypothesis generation, yet existing approaches primarily yield coarse-grained hypotheses lacking critical methodological and experimental details. We introduce and formally define the new task of fine-grained scientific hypothesis discovery, which entails generating detailed, experimentally actionable hypotheses from coarse initial research directions. We frame this as a combinatorial optimization problem and investigate the upper limits of LLMs' capacity to solve it when maximally leveraged. Specifically, we explore four foundational questions: (1) how to best harness an LLM's internal heuristics to formulate the fine-grained hypothesis it itself would judge as the most promising among all the possible hypotheses it might generate, based on its own internal scoring-thus defining a latent reward landscape over the hypothesis space; (2) whether such LLM-judged better hypotheses exhibit stronger alignment with ground-truth hypotheses; (3) whether shaping the reward landscape using an ensemble of diverse LLMs of similar capacity yields better outcomes than defining it with repeated instances of the strongest LLM among them; and (4) whether an ensemble of identical LLMs provides a more reliable reward landscape than a single LLM. To address these questions, we propose a hierarchical search method that incrementally proposes and integrates details into the hypothesis, progressing from general concepts to specific experimental configurations. We show that this hierarchical process smooths the reward landscape and enables more effective optimization. Empirical evaluations on a new benchmark of expert-annotated fine-grained hypotheses from recent literature show that our method consistently outperforms strong baselines.

MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search
Zonglin Yangl:2; Wanhao Liu Ben Gao Yujie Liu', Wei Li,, Tong Xie
Lidong Bing', Wanli Ouyang2 Erik Cambria' F, Dongzhan Zhou
Nanyang Technological University Shanghai Artificial Intelligence Laboratory
University of Science and Technology of China" Wuhan University National University of Singapore
6 University of New South Wales MiroMind 8 The Chinese University of Hong Kong fzonglin001, cambr Cntu. ceutuu.ddu.sg... edu.sg, zhoudongzhann@jjlab.. org. cn
Abstract
Large language models (LLMs) have shown promise in automating scientific hypothesis generation, yet existing approaches primarily yield coarse-grained hypotheses lacking critical methodological and experimental details. We introduce and formally define the new task of fine- fine-grained scientific hypothesis discovery, which entails generating detailed, experimentally actionable hypotheses from coarse initial research directions. We frame this as combinatorial optimization problem and investigate the upper limits of LLMs' capacity to solve it when maximally leveraged. Specifically, we explore four foundational questions: (1) how to best harness an LLM'' internal heuristics to formulate the fine-grained hypothesis it itself would judge as the most promising among all the possible hypotheses it might generate, based on its own internal scoring-thus defining a latent reward landscape over the hypothesis space; (2) whether such LLM-judged better hypotheses exhibit stronger alignment with ground-truth hypotheses; (3) whether shaping the reward landscape using an ensemble of diverse LLMs similar capacity yields better outcomes than defining it with repeated instances of the strongest L.M among them; and (4) whether an ensemble of identical L.MMs provides a more reliable reward landscape than a single LLM. To address these questions, we propose a hierarchical search method that incrementally proposes and integrates details into the hypothesis, progressing from general concepts specific experimental configurations. We show that this hierarchical process smooths the reward landscape and enables on a new benchmark of expert-annotated fine-grained hypotheses from recent literature show that our method consistently outperforms strong baselines []]]
effective optimization. Empirical evaluations
1 Introduction
Large language models (LLMs) have increasingly been applied to assist scientific research (Luo et al. 2025), with one of the most ambitious applications being the automated discovery of novel and valid scientific hypotheses. However, current methods produce hypotheses that are criticized for being overly coarse, lacking sufficient detail, offering simplistic suggestions, or omitting concrete
implementation strategies (Wang et 2024 Hu al. 2024 Si et 2025
All code and data can be found in https:///iithbb wwt.s////oo/ooo///////o/oo/ Chem2
Contribution during internship at Shanghai Artificial Intelligence Laboratory. Corresponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We present the first systematic investigation into how LLMs can be leveraged to formulate fine- grained scientific hypotheses- those enriched not only with major concepts but also with precise methodological details and clearly specified experimental configurations. For example, a coarse- grained hypothesis in chemistry might state, "synthesize hierarchical 3D copper," while a fine-grained counterpart could elaborate, "Copper foils are chemically oxidized by immersion in solution of 0.5 M ammonium persulfate and 2 M sodium hydroxide for 15 minutes at room temperature, forming a pentagonal hierarchical CuO nanostructure." Such fine-grained hypotheses significantly enhance clarity, feasibility, and experimental implementability.
Formally, we define the task as generating fine- lineggriined hypothesis given a research back- doound-comprisnng a research question and a survey of established methodologies- and a coarse- grained hypothesis direction. We show that fine-grained scientific hypothesis discovery is combina- torial search problem, as it requires selecting and composing a coherent set of concrete details from vast space of plausible compounded by the fact that scientific hypothesis discovery is an inherently out-of-domain (OOD) problem: the correctness of hypothesis is fundamentally unknown at the time of formulation.
it particularly challenging in practice. The difficulty is
In this work, we focus on the pre-experimental stage of discovery, mirroring how human scien- tists-prior to empirical eesting-iteratieee) search through the hypothesis space using heuristics and domain knowledge identify the hypothesis they themselves would judge as the most promising among all plausible candidates they could think of during the hypothesis search process.
Our goal is emulate this cognitive search process using L.Ms,, which increasingly rival human scientists in heuristic reasoning and scientific knowledge understanding. This motivates our central research question (Q1y: how to best harness an L.LM's internal heuristics to formulate the fine- grained hypothesis it itself would judge as the most promising among all possible hypotheses it might generate? ? We conceptualize a hypothesis space where each point along the input dimensions (the a-axis, potentially multidimensional) represents a candidate hypothesis, and each point is assigned reward value (on the y-axis) by the LLM based on its internal heuristics. This defines a reward landscape over the hypothesis space, with the highest peak corresponding to the hypothesis the LLM internally judges as most promising. Framed this way, Q1 becomes an optimization problem: how can we navigate this landscape to find stronger local optima- or ideally the global optimum thus eliciting the best fine-grained hypothesis the L.M can generate?
A straightforward baseline is greedy search over the reward landscape. However, its non-convex and complex structure makes naive greedy strategies prone poor local optima. To address this, we propose a hierarchical search framework that explicitly models how a finite-capacity reasoning agent-human or LLi-navigates the hypothesis space. Specifically, it first explores higher-level conceptual spaces and then incrementally refines into more specific detail spaces. This hierarchical approach smooths the reward landscape at each hierarchy level especially at higher, more abstract levels levels-enabling convergence to superior local optima compared to greedy search and greedy search with self-consistency (Wang et al. 2023). The proposed framework naturally scales with the capability of the underlying L.M''s heuristics, yielding better optima as those heuristics become stronger.
Having investigated how to identify stronger local optima in Q1, we now turn to our second question (Q2): whether hypotheses judged better by LLMS exhibit stronger alignment with ground-truth hypotheses? To rigorously address Q2 while avoiding data contamination, we construct a benchmark of research backgrounds paired with expert-annotated fine-grained hypotheses from chemistry papers published after January 2024, ensuring these examples were unseen by our LLM (GPT-40-min October 2023 cutoff). Using this benchmark, we indirectly evaluate Q2 by comparing the recall of hypotheses discovered by our hierarchical with hypotheses identified by baseline methods. Our results consistently show that hypotheses generated by our method achieve higher recall than those from baselines, providing empirical support for the reliability of the LLM's internal reward signal in guiding fine-grained hypothesis discovery.
saproach-whiee locates better LLM--nttrnal local op-
Until now, the reward landscape guiding hypothesis search has been defined by single LLM serving as the evaluator. We now address our third question (Q3): whether defining this landscape with an ensemble of diverse LLMs of similar capacity yields better oulcomes than using equally sized ensembles of the strongest LLM within that group. Our experiments show that ensembles of repeated instances of the strongest LLM consistently outperform equally sized ensembles of diverse models, suggesting that peak model quality is more critical than model diversity in this setting.
2

---

Finally, we consider a fourth question (Q4): whether an ensemble of identical LLMs provides better reward landscape than a single instance of the same LLM. While Q3 compares ensembles of different models, Q4 isolates the effect of aggregation alone by controlling for model identity. We find that even identical LLMs, when sampled independently and aggregated via summarization, yield reward signal that better captures novelty without sacrificing overall hoility-highlighting subtle but important dimension in optimizing hypothesis discovery.
Notably, while our experiments focus on chemistry, the task formulation, methodology, and analysis of Q1-Q4 are discipline aasippiie-agnostic. The only domain comain-specific component is the manually designed hierarchy (used by the methodology), which can be instantiated for each new discipline encountered.
Overall, the contributions of this work are:
1. We introduce and formalize the the fine-grainet scientific hypothesis discovery task as a combi- natorial optimization problem, and release a post-2024 benchmark with expert-annotated fine-grained hypotheses, specifically designed to prevent data contamination.
2. We explore the limits of LLMs for fine-gras nne-grained scientific hypothesis discovery by framing as an optimization problem over reward landscape defined by LLM heuristics, with pairwise comparisons serving as the gradient signal. We also propose a hierarchical heuristic search framework that theoretically smooths the reward landscape, reduces search complexity, and identifies superior local optimum by interpolating among discovered optima. Empirically, this framework consistently outperforms strong baselines in locating better local optima.
3. We analyze this optimization formalization through 4 foundational research questions.
2 Methodology
2.1 Background and Task Motivation
Yang et al. (2025) assume that many chemistry hypotheses can be constructed from research background b typically including the research question and/or background survey-and set of
inspirations ,,,...,44 lg. representing concepts or findings from the literature. can be formulated as:
f (b..21......
(I)
In practice, however, most hypotheses generated from Equation I tend to be coarse- maars--grained: while they form cohesive associations between b and the i, they often lack clear hypothesis specification and the detailed experimental configurations required for direct implementation in a laboratory setting. Additionally, many such hypotheses contain redundant elements- either due to the inclusion of unnecessary inspirations or from noise present in the literature that is unrelated to the core knowledge intended for hypothesis construction.
2.2 Problem Formulation: Fine-Grained Hypothesis Generation as Combinatorial Search
Let he be a coarse- course-grained hypothesis direction and h its fine- -ree-graindd counterpart, defined as:
h {he, d1, dm
Here, (he, .i,...y dm } denotes the meaningful integration of edits di, di,..., dm into hes resulting in coherent, fine-grained hypothesis. Each edit d corresponds to either (1) the addition of a fine-grained detail to an existing concept i or the introduction of new concept into he, or (2) the deletion of redundant detail or concept from hg. We define two edit candidate sets: D+t containing all details and concepts that may be added hoi and D containing all details and concepts within he that may be removed. The overall edit space is then given by D = D -
Inspired by coarse-to-fin strategies in computer vision- where a coarse image is first generated and then refined with fine- -rainga details (Tian et al., 2024 we formulate the transition from h, to hy as additional step building Equation [ which provides the initial he-
P(hylb, ha) P((di..... ... dm Hm,))b,. be: D)
(3)
3

---

This formulation turns P(hy |,, he) into combinatorial optimization problem, where the objective to select subset of edits dj..... dm D. Let [D] and Id...... dml m. The search space has at least combinatorial complexity C, ml(n-m}.. This makes the problem particularly challenging due to three factors: (1) both m and n are unknown; (2) the candidate set is itself implicit and potentially very large; and (3) the edits d, are not independent errors early in the reasoning chain can propagate and impair later decisions.
2.3 Algorithmic Motivation for Hierarchical Heuristic Search (HHS)
Fine-grained hypothesis generation is generally intractable due to the exponential growth of the search space, where the candidate set [D] often large prohibitively so.
notable exception occurs when the problem exhibits an optimal substructure- an optimal solution can be composed from the optimal solutions to its subproblems. This principle underlies dynamic programming, where solutions are built incrementally from smaller subproblems (first try to obtain the optimal solution for smaller subproblem and then iteratively find the optimal solution for larger subproblems).
We observe that fine-ge fineggrained hypothesis generation exhibits an optimal substructure. Specifically, the edits .i...., . dm can be organized hierarchically: some address high-level concepts (e-g., functional groups, catalyst classes), while others specify low-level details (e.g.. reagents, catalysts, temperature, concentration). We assume these edits can partitioned into p hierarchical levels (p 1), with higher levels corresponding to finer details. Then the overall problem can be seen as to determine d in {1..... p} hierarchies. The subproblem hierarchies, etc. Then it is obvious that the optimal solution of a problem be derived from the optimal solution of its subproblem, etc.
can be seen as the determination
in {1.....
Henarchy1
Reaction
Hotrorcy
Component
Selection
Hienarchy
Instancs-Level Component Choice
Parametric Specification Components
Hienorchy
Full Exparimental Configurations
Figure Hierarchies designed for chemistry and material science by PhD-level domain expert.
Figure I fillustrates example hierarchical decomposition for chemistry, developed in collaboration with domain experts (PhD-level chemists). The hierarchy spans from high-level mechanistic intent to low-level experimental configurations, reflecting the granularity typically considered when translating a conceptual hypothesis into a testable laboratory procedure chemistry.
Now we have simplified the problem of determining d in all hierarchies into the iteration of determining in each hierarehy sequentially. Nonetheless, even within a single hierarchy, the number of candidates remains combinatorially large.
A practical approach to this combinatorial complexity is heuristics that approximate solutions rather than exhaustively searching for exact ones. This aligns with how chemists refine hypotheses: given that ht often represents unexplored direction, while individual details may be retrievable from existing databases, the complete set details is rarely available. Instead, chemists often rely on domain knowledge and intuition heuristically identify and progressively integrate plausible details.
Analogously, we propose to leverage LLMs' internal heuristics to guide the search for at each hierarchical level. As LLMs advance, their emeriignn from pretraining over extensive scientific corpora- seiprr--ill will increasingly approximate, and may surpass, those of human experts. The proposed framework naturally scales with the strength of these heuristics, yielding increasingly better optima for fine- grained hypothesis discovery as LLM capabilities continue to grow.
In this setting, the candidate space D is not explicitly enumerated but is implicitly embedded within the L.L''s internal knowledge and reasoning capabilities. The LLM does not select from predefined list, but rather proposes candidates by navigating this latent, couristic-driver space.

---

2.4 Hierarchical Factorization of the Search Problem
For formalization, we partition the implicit candidate space D into p hierarchical levels, where D represents all potential edits level and D C D denotes the (unknown) ground- truth edits. The j-th ground-truth edit at level i denoted as d/ E D= Since is implicitly determined by he. we have P(D he) = 1. and explicitly condition on D for clarity in the subsequent factorization. Applying the chain rule hierarchically, Equation 3 can be reformulated as:
P(hy b.) P (DD D'.m Ne: D)
(4)
(D b, Delc0, ((0)
(5)
[=1
II
P (1, b, he, Dr( dela D(1)
(6)
where D*(<0) (D.))..., D*00") and d' (1,
The key advantage of this hierarchical factorization is that each level i, the search is restricted to the reduced candidate set ( rather than the full space D, significantly narrowing the search space. Moreover, as we will show 2.6 this hierarchical decomposition smooths the reward landscape at each hierarchy level, facilitating more stable optimization and enabling the discovery of stronger local optima the hypothesis space.
2.5 LLM-Based Implementation of Hierarchical Heuristic Search
h.
he
Recombbinatoo
hi
her (Hi-inhu)
(He has heae
Hi-1
The ith hierorchy
The 14h hierarchy
no: Aigncr Ayrer
rosearch background he hypotheeis direction
yes: hgrvee Nevr
A; final nulput from heraachy
h/ candidele consider
consecutive & timms
final output
Add one level
hour hoar hyees?
hyw
P..
[0g ... A al
Hi-1
Iigreo Bege in current search step
foadback
The ith hierarchy
Figure 2: Overview of the proposed Hierarchical Heuristic Search (HHS) framework.
We implement HHS an LLM-driven agentic process that directly follows the hierarchical factor- ization formalized in Equation 6 As shown in Figure at each hierarchy level H; represents the accumulated edits from all previous levels, corresponding to (=(<)) Within the current level, hyrev denotes the partial hypothesis incorporating the edits selected up step j-1, i.e., ar(() The candidate set (6) is not explicitly enumerated but emerges implicitly from the LLM'' internal heuristies, conditioned the background the hypothesis direction heg and the edits selected far.
Specifically, the search for local optimum le begins from the initial point h; using contextual information from b, hes and Hz-2- For hierarchy level = 1, we set hg = ha and Hg = 0, making the hypothesis direction h. the starting point.
At each iteration, the "Add one d at level module prompts an LLM to propose an edit d to hprees producing candidate heur. which is then refined once for validity, novelty, and specificity. The "heur hprev" module evaluates whether the new hypothesis improves upon the previous one via LLM-based pairwise comparison, serving as internal gradient signal for hypothesis optimization.
5

---

This search process continues until no further improvement is observed over & consecutive steps (default 3). at which point the current hypothesis accepted as a local optimum. Each edit may involve either an addition or deletion, allowing the search path to include retrospection and self-correction as needed.
Within each hierarchy level, we adapt the design of an evolutionary unit Yang et al. 2025) to our task, where the search for the local optimum h; independently repeated multiple times (set to three in our implementation), yielding several local optima /4, hi and h These candidates are then passed to a recombination module, which integrates their complementary strengths to interpolate potentially superior local optimum h2 within the subspace spanned by h! h2 h
2.6 Theoretical Analysis: Smoothing Effects of Hierarchical Heuristic Search
Level 1: High-level abstract landscape
Level landscape
Performance
Initial point
0.75
optimum
0.50
0.25
0.00
Level 2: Intermediate landscape
Level landscape
1.0
Performance
Level optimum (pmjactedd
0.5
Start (11 optimums)
Level optimum
0.0
Level 3: Detailed rugged landscape
Level landscape
Performanoe
1.0
Initial point
optimum (projested)
Level optimim (graeectd))
0.5
Start (1.2 opsimums
Level optimum
0.0
No-hierarcby local optimum
Hypothesis space
Figure 3: The smoothing effect of hierarchy the reward landscape of hypothesis space.
A key observation is that hypothesis candidate's performance at lower (more abstract) hicrarchy level can be viewed as an aggregated estimate-approone approximating an average or soft maximum of its higher-level subspace (more concrete). For instance, when evaluating coarse- -aatinegr (more abstract) concept like "hierarchical 3D copper," the LLM may implicitly account for its diverse fine- grained (more conerete) structural variants, some highly relevant, others ineffective. We hypothesize that the LLM'ss 's' assessment the coarse- coarse-grained concept is an aggregation its fine-grain ine-grained outcomes, weighting promising variants within the broader distribution to produce overall estimate of the coarse-gret coarse-grained concept's expected potential.
Building this observation, the hierarchical abstraction smooths the reward landscape at lower levels by attenuating local irregularities in the fine-graiin mne-grained space, as the performance of a point at a lower level can be interpreted as approximate aggregation average of the performance across its corresponding higher-level subspace. This effect illustrated in Figure B (a simplified schematic projection into ID space). Consequently, direct search over the flat, non-hierarchical space tends to be highly rugged and non-convex, often leading to premature convergence to suboptimal local optima. In contrast, introducing hierarchical structure progressively smooths the landscape, enabling more stable and efficient optimization, particularly at lowest levels.
This smoothing effect can also be interpreted in the frequency domain as a form of low-pass filtering, where high-freyuency components the landscape are attenuated, resulting in (roughly) spectral cutoff in the spatial frequency domain, illustrated in Figure

---

Hearrhy regan spectrum ourve
3.0
2.5
2.0
Hierarchy 1 cutoff
Hierarchy 2 cutoff
Spectral Ampliuue
1.0
0.5-
0.0
10.0
25.0
17.5
Spatial Frequency of Hypothesis Space Performance Landscape
Figure 4: Hierarchical design as a low-pass filtering over the spectrum of the reward landscape.
3 Experiment: Investigating the Four Fundamental Questions
3.1 Benchmark Construction, LLM Selection, and Baselines
To our knowledge, no existing benchmark provides annotated fine-grained scientific hypothe- ses detailed enough for direct experimentall execution. We the TOMATO-Chem dataset (Yang) et 2025), which includes 51 chemistry papers published after January 2024 in leading journals such as Nature and Science. Each entry contains a research background b and a coarse-grained hypothesis he. We further annotated it with fine-grained hypotheses hss serving references created by two PhD-level chemists. To prevent data contamination, all experiments are conducted using HPT-40-mints whose pretraining data cutoff is October 2023.
ground-truth
We compare HHS against two strong baselines widely used search tasks: (1) greedy search and (2) greedy search with self-consisteney. The latter serves as an ablation of HHS where the hierarchical decomposition is removed, performing the search in a single stage with each d sampled directly from the full candidate set D rather than hierarchy-specte is similar to the Recombination module in Figure 2 which interpolate multiple local optima trying to find a better one. Greedy search represents further ablation, disabling the Recombination module entirely and following a single search trace where the first found local optimum (h?) is directly adopted as the output (h,- = h in Figure
subsets D(1), The self-consistency mechanism
3.2
Q/: How to Best Harness an LLM's Internal Heuristics to Formulate the Fine-Grained Hypothesis It Would Judge Most Promising Among All It Might Generate?
We frame this question as optimization problem: Given only a coarse- -raained hypothesis h2 as the starting point, and relying entirely on a single LLM, how can we navigate the hypothesis space to approach the global optimum of the reward landscape, as defined by this same L.MM's internal heuristics, where each optimization step consists of adding an edit d to h.? ? In this setting, the LLM plays dual role: it serves both as the proposal generator, proposing candidate edits d to formulate new hypotheses within the hypothesis space, and as the gradient provider, judging whether the new hypothesis improves upon the current one via its own internal heuristics (e.g., pairwise comparison).
While it is inherently infeasible determine whether found local optimum represents the global optimum, we can empirically compare local optima obtained by different methods with the gradient provider (pairwise comparison), and therefore check which one is a better local optimum.
As detailed in $ [ the hierarchical design of HHS offers two key advantages over flat search strategies: (I) less search space to propose each (from D(0) instead of D), and (2) smoothing the reward landscape progressively in the hypothesis space. Among these, the smoothing effect is particularly critical, as it reduces the risk of early convergence to suboptimal local optima and facilitates progress toward higher peaks in the LLM's internal reward landscape.
We compare the local optima discovered by HHS against the two baselines. For each pair of local optima, we conduct both overall evaluations and dimension-specific assessments across four key criteria: effectiveness, novelty, detailedness, and feasibility. this context, feasibility reflects the practical ease of implementing the proposed hypothesis, encompassing factors such implementation
7

---

complexity and the minimization of redundant steps. Hypotheses that are easy to implement and free of redundant components are preferred.
We further observe two common trade-offs among these dimensions: (1) between effectiveness and novelty, as highly novel hypotheses often entail greater scientific risk and uncertainty; and (2) between detailedness and feasibility, as increased specificity can introduce procedural complexity or redundancies that diminish experimental feasibility.
We also conducted an expert evaluation involving two chemistry PhD students. For each benchmark item, one hypothesis the three hypotheses. The results from both the LLM-base based and expert evaluations on the quality of local optima discovered by each method presented in Table To mitigate known position bias in I.LM-baedd pairwise comparisons- where models tend to favor the first option (Li et al. 2024 -each pair of local optima compared six times, with the order of presentation alternated every three times. A hypothesis was considered to win if it received more than three votes; a tie was recorded if both received exactly three votes.
randomly sampled from each method, and the experts were tasked to rank
Table Comparison between HHS and baseline methods across LLM-based and expert evaluations.
Ellectiveness (LLM)
Novelty (LLM)
Detailedness (LLM)
Feasibility (LLM)
Overall (LLM)
Overall (Expert)
HHS Greedy Search
74.51%
41.18%
71.57% %
67.65%
73.53%
76.47%
18.63%
18.
28.43%
10.78%
18.63%
15.69%
6.86%
40.20%
0.00%
HIS vs. Greedy Search Self-consistency
Win Tie
59.31 %
46%1
56.37%
48.53%
53.43%
74.51 %
24.02%
8.33%
43.14%
18.63%
33.82%
17.65%
Lose
16.67%
49.51 %
0.49%
32.84%
12.75%
7,84%:
Greedy Search Self-consistency Greedy Search
Win
57.84%
48.04%
29.41 %
51.96%
54.90%
62.75%
22.55%
11.76%
65.69%
18.63%
34.31%
21.57%
19.61%
40.20%
15.699
3.3
Q2: Whether Hypotheses Judged Better by LLMs Exhibit Stronger Alignment With Ground-Tree -Truh Hypotheses?
3.2 shows that HHS consistently discovers superior local optima compared to baselinc methods. We further investigate whether these optima exhibit stronger alignment with the ground-truth hypotheses.
Table 2: Recall of ground-truth components by discovered hypotheses. #Steps represents the number
of reasoning steps used. HHS represents HHS-3 referred in $ 3.5
Soft Recall
Hard Recall
#Steps
Chen&Crow
M. Bran al
2024
12.28%
7.20%
Qi al. 2024
19.57%
11.15%
SeIMON
Wang
2024
18.57%
10.09%
MOOSE
Yang et
20.04%
11.76%
MOOSE-C
Chem Yang ct ul
2025
19.99%
11.98%
Creedy Search
16.60%
9.92%
9.69
In-context RL
76%7
10.28%
16.80
Self-consistency
31.53%
17.73%
67.55
HHS (HHSS-)
40.35 %
23.04%
282.04
w/ In- co-context RI.
33.63%
21 21.29%
$31.08
Single Gradient (HHS-I)
32.40%
Given the lack of established metrics for this task, we introduce an LLM-based evaluation that measures how well the discovered hypotheses recover the methodological and experimental details of the ground-truth hypotheses. The detailed formulations of the two metrics-Sent Recall and Hard Recall-ane are provided in Appendix C

---

As shown in Table 2 the hypotheses discovered by HHS corresponding to better local optima than those produced by greedy search Daselines-achinn consistently higher recall scores than both greedy search baselines and other comparative methods. Here, in-context RI. refers to a mechanism in which, if the current hypothesis heuur does not outperform the previous one Npree heur is inserted into the LLM's s context to generate new candidate. We also report the total number of reasoning steps used by each method. A general trend emerges: increasing the number of reasoning steps tends to improve recall up to a point, beyond which excessive steps lead to diminishing or negative returns.
3.4 Q3: Whether Defining the Reward Landscape With an Ensemble of Diverse LLMs Yields Better Outcomes Than Using the Same Number of the Strongest LLMs Among Them?
Table 3: "EF": Effectiveness, "NV": Novelty, "DT": Detailedness, "FS": Feasibility, "OV": Overall.
"(GT)" and "(GM)" indicate that the pairwise comparisons were conducted by GPT-4o-mim and Gemini-1 cominni-15--iii 1-.ssashh respectively.
EF (GT) NV (GT) DT
FS (GT)
OV (GT)
EF (GM)
NV (GM)
DT (GM)
GGM)
OV (GM)
Mixed committce v.s. GPT-4o-mini committee
GPT-4o-mini committee v.s. Gemini-1. 1eriiiai1..--1aae f-1ash committce
Win Tie
20.83%
33.33%
14.58 %
33.33%
29.17%
27.08%
31.25%
14.58%
0.00%
18.75%
41.67%
20.83%
72.92%
18.75%
33.33%
58.33%
$2.08%
77.08%
95.83%
68.75%
Lose
37.50%
45.83%
12.50%
47.92%
37.50%
14.58%
16.67%
8.33%
4.17%
12.50%
Gemini- 111.1....... 5-flash committoe GPT-4o-mini committce
Mixed committee
Gemini- 1atiiiii--.-flee committce
Win
16.67%
25.00%
6.25%
37.50%
16.67%
16.67%
33.33%
12.50%
6.25 %
18.75%
Tie
41.67%
27.08%
79.17%
25.00%
52.08%
68.75%
35.42%
75.00%
93.75%
64.58%
Lose
41.67%
47.92%
14.58 %
37,50%
31.25%
14.58%
31.25%
12.50%
0.00%
16.67%
Mixed committee V.s. Gemini-1 Denini-1.S-flest committee
Mixed committee
UPT-44-minn committee
Win Tie
29.17%
45.83%
10.42%
47.92%
27.08%
8.33%
29.17%
14.58%
6.25%
8.33%
56.25%
16.67%
85.42%
10.42%
50.00%
77.08%
39.58%
70.83/%
93.75%
64.58%
Lose 14.58%
37.50%
4.17%
41.67%
22.92%
14.58%
31.25%
14.58%
0.00%
27.08%
The optimization in HIS relies on the "hour hyver?" module (Figure 2). which acts as the gradient signal driving the search. This raises a key question: does a diverse ensemble of comparably capable LLMs improve search performance, is it more effective use the same number of parallel instances of the single strongest LLM among the ensemble?
To answer this, we design three experimental settings: (1) Mixed Committee: the "hewr hppee ?
module is implemented by an ensemble of three different LLMs-GPT-400 -P (OpenAl 2024).
Comini-1.5-flast (Georgiev et al.) 2024), and Claude-3 --aakku (Anthropic 2024 (2) GPT-
4o-mini Committee: the module is implemented by three instances of -mi-4o-mini; (3) Gemini- 1.5-flash Committee: the module is implemented by three instances of Cemini-1.5-flees
Each
committee's 's three judgments are then aggregated by a GPT-4o-mini, which produces the final decision for "horr > hpww? ?" representing that committee, All three settings use GPT-4o -m-ini the proposer module for generating edits d at each hierarchy level i.
We compare the local optima generated by these setups using LLM-based pairwise comparisons, following the protocol in $ (3.2 where each pair is evaluated six times to mitigate position bias. However, since the evaluator itself an LLM, an additional bias may occur favoringv optima discovered using gradients from the same model. To control for this, we conduct two sets of evaluations: one using GPT-4o-mini as the evaluator, and the other using demini-1.5-f1ee
As shown in Table across both evaluators, the GPT-4o-mini committee consistently outperforms the mixed committee, which in turn outperforms the Gemini-1.. 1--1-.5-fasii committee. These results suggest that leveraging repeated instances of the strongest single model provides a more effective gradient signal for hypothesis optimization than combining different models of similar capacity.
3.5
Q4: Do Multiple Identical LLMs Yield a Better Reward Landscape Than One?
In experiments of Tables 1 and 2 defined using an ensemble of three identical L.M,s, followed by fourth instance of the same L.M that aggregated the three judgments into a final, reasoned decision. However, it is unclear on whether
(except for the HHS-/ line in table 2) the reward landscape was

---

one LLM would be already enough. To evaluate this, we compare two variants: HHS-3, which uses an ensemble of three identical instances of GPT-4o-ming (and a fourth instance the same LLM for aggregation) to provide the reward signal, and 1 which relies on single instance of the same model. Table reports LL-M-based pairwise evaluations between the two setups across four criteria. While overall quality, effectiveness, and detailedness are largely comparable, HHS-3 outperforms in novelty, whereas I shows an advantage in feasibility.
Table 4: Pairwise comparison between HHS-1 and HHS-3 with LLM-based evaluation.
Effectiveness (LLM)
Novelty (LLM)
Detailedness (LLM)
Feasibility (LLM)
Overall (LLM)
NHS-II v.s. HHS-3
Win
21 1,008
25,49%
4.41%
41.67%
8.82%
Tie
57.35%
28.92%
94.12%
28.92%
82.35%
Lose
21.57%
45.59
1.47%
29.41%
8.82%
This result is somewhat counterintuitive. Notably, the summarization step is not simple majority vote: the aggregating LLM assesses the relative strength of reasoning across the three perspectives and selects the most compelling argument. This allows it to surface minority-supporti reasoned views, promoting exploration of more novel or unconventional hypotheses. Consequently, the aggregated reward signal in HHSS3 becomes more sensitive to creative or atypical ideas that single-shot evaluation might overlook, whereas HHS-1 relies single comparative judgment at each favoring conventional and well-cstablished hypotheses, at the expense of novelty.
but well-
Table 2 presents the recall of ground-truth components for hypotheses generated by HHS-1 and HHS-3. 3. Across both soft and hard recall metrics, HHS-3 outperforms HHS-1, indicating stronger alignment with expert-annotated reference hypotheses.
4 Related Work and Discussion
LLM-driven scientific discovery methods typically fall into two categories: (1) direct generation of hypotheses from a research
research question and background survey (Qi ct al. 2024); or (2) retrieval of seemingly unrelated yet potentially useful knowledge fragments, or inspirations, which are then combined with the background to construct hypothesis (Yang et al.
2024 2025 Wang et al. 2024 Liu et al. 2025b). While these methods show promise in generating novel ideas, they are often criticized for producing hypotheses that are overly coarse, lacking detail, or omitting actionable experimental steps (Wang ct al. 2024] Hu et al. 2024; Si et al., 2025). In contrast, our goal investigate how LLMS can generate ieneggrined scientific hypotheses those sufficiently detailed to be directly implemented in laboratory settings. Although this work centers on the pre-experimental stage of discovery, the framework can in principle extend to the experiment- guided stage (Liu et al., 2025al Cameer--Parede et al. [024]] Shooaae et al.) 2025). Novikov et al.) 2025) by incorporating experimental feedback into the background survey. Likewise, details retrieved from papers relevant to the proposed hypothesis can also be integrated into the background survey.
5 Conclusion
We introduce and formalize the fine-grained scientific hypothesis discovery task as a combinatorial optimization problem. To explore the upper limit of LLMs' capacity for this task, we frame it as an optimization problem, and propose hierarchical heuristic search (HHS), which theoretically smooths the reward landscape, reduces combinatorial complexity through optimal substructure exploitation, and identifies superior local optimum by interpolating among discovered optima. Empirical results show that (1) HHS reliably discovers better local optima than flat greedy search baselines, (2) hypotheses preferred by LLMs often align more closely with expert annotations, (3) repeated use of the strongest model defines a more effective reward landscape than diverse ensembles, and (4) aggregating identical LLMS yields a reward signal more sensitive to novelty and higher in recall than single-instance evaluation. These findings illustrate how hierarchical search can better harness LLMs' internal heuristics for scientific discovery. Although evaluated on a chemistry dataset, the proposed task formulation and methodology are expected to generalize across disciplines, with only the manually designed hierarchy (Figure requiring domain-specific adaptation.
10

---

Acknowledgments
This work was supported by a locally commissioned task from the Shanghai Municipal Govermment. This work is supported by Shanghai Artificial Intelligence Laboratory. This research/project is supported by the Ministry of Education, Singapore under its MOE Academic Research Fund Tier (STEM RIE2025 Award MOE-T2EPP2113-00055)
References
Anthropic. The claude 3 model family: Opus, sonnet, haiku. news/ 3ttss/clauee---eee
March 2024. Accessed: 2025-05-16.
Erik Cambria, Rui Mao, Melvin Chen, Zhaoxia Wang, and Seng-Ben Ho. Seven pillars for the future of artificial intelligence. IEEE Intelligent Systems, 38(6):62-69, 69, 2023.
Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang. Soroosh Mariooryad, Yifan Ding, Xinyang Geng. Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love, Paul Voigtlaender, Rohan Jain, Gabriela Surita, Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Komnaphhp Kawintiranon, Orhan Firat, Yiming Gu, Yujing Zhang, Matthew Rahtz, Manaal Faruqui, Natalic Clay, Justin Gilmer, JD Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki Morioka, Kevin Hui, Krishna Haridasan, Victor Campos. Mahdis Mahdich, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de Liedekerke, Siddharth Goyal, Paul Barham, DJ Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lepikhin, Michela Paganini, Xavier Garcia, Fan Yang, Dasha Valter, Maja Trebacz, Kiran Vodrahalli, Chulayuth Asawaroengchai, Roman Ring, Norbert Kalb, Livio Baldini Soares, Siddhartha Brahma, David Steiner, Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael Lopez Kaufman, Laurent El Shafey, Junhyuk Oh, Tom Hennigan, George van den Driessche, Seth Odoom, Mario Lucie, Becca Roelofs, Sid Lall, Amit Marathe, Betty Chan, Santiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Lenc, Chih-Kuan Ych, Aakanksha Chowdhery, Yang Xu. Mchran Kazemi, Ehsan Amid, Anastasia Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin, Mario Pinto, Geng Yan, Adria Puigdomenech Badia, Piyush Patil, Steven Hansen, Dave Orr, Sebastien M. R. Arnold, Jordan Grimstad, Andrew Dai, Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi Chen, Elena Gribovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kris Cao. Da-Woon Chung, Matthew Lamm, Gabe Barth-Maron, Thais Kagohara, Kate Olszewska, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider, Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang, Ort-Orgil Batsaikhan, Mateo Wirth, James Qin, Ivo Danihelka, Tulsce Doshi, Martin Chadwick, Jilin Chen, Sanil Jain, Quoc Le, Arjun Kar, Madhu Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lampros Lamprou. Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan Wang. Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali Ahmed, Dominik Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan Pope, Steven Hand, Vittorio Selo, Vihan Jain, Nikhil Sethi, Megha Goel, Takaki Makino, Rhys May. Zhen Yang, Johan Schalkwyk, Christina Butterfield, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver Woodman, Marvin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim Blyth, Ian Mackinnon, Machel Reid, Obaid Sarvana, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Attaluri, Gregory Thornton,
Bhung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung. Evgenii Eltyshev, Xiance Si, Timothy Lillicrap, Demetra Brady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu, Ross Mcllroy, Kartikeya Badola, Paramjit Sandhu, Erica Moreira, Wojeiech Stokowiee, Ross Hemsley, Dong Li, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghi, Salem Haykal, Pablo Sprechmann, Xiang Zhou, Diana Mincu, Yujia Li, Ravi Addanki, Kalpesh Krishna, Xiao Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye Zhang, Emanuel Taropa, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, HyunJeong Choe, Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaleldin Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wei Chen, Peter Humphreys, Kate Baumli, Connie Tao, Rajkumar
Alex Tomala, Chalence Bafranek-Shradee
11

---

Samuel, Cicero Nogueira dos Santos, Anders Andreassen, Nemanja Rakićević, Dominik Grewe, Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Hannah Sheahan, lain Barr, Yingjie Miao, Paul Natsev, Jacob Devlin, Feryal Behbahani, Flavien Prost, Yanhua Sun, Artiom Myaskovsky, Thanumalayan Sankaranarayana Pillai, Dan Hurt, Angeliki Lazaridou, Xi Xiong, Ce Zheng, Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, Alejandro Lince, Mingqiu Wang. Basil Mustafa, Albert Webson, Hyo Lee, Rohan Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras, Elahe Dabir, Shyam Upadhyay, Anudhyan Boral, Lisa Anne Hendricks, Corey Fry, Josip Djolonga, Yi Su, Jake Walker, Jane Labanowski, Ronny Huang, Vedant Misra, Jeremy Chen, RJ Skerry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu, Alex Castro-Ros. Beer Changpinyo, Romina Datta, Sumit Bagri, Amar Mar Hrafnkelsson, Marcello Maggioni, Daniel Zheng, Yury Sulsky, Shaobo Hou, Tom Le Paine, Antoine Yang, Jason Riesa, Dominika Rogozinska, Dror Marcus, Dalia El Badawy, Qiao Zhang, Luyu Wang. Helen Miller, Jeremy Greer, Lars Lowe Sjos, Azade Nova, Heiga Zen, Rahma Chaabouni, Mihaela Rosca, Jiepu Jiang, Charlie Chen, Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov, Lean-Baptiste Lespiau, Josh Newlan, Zeynep Cankara, Soo Kwak, Yunhan Xu, Phil Chen, Andy Coenen, Clemens Meyer, Katerina Tsihlas, Ada Ma, Juraj Gottweis, Jinwei Xing, and Gu. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05520 2024. URL httpshtps: /ttp///axvvvvv//// org/ abs/ /2/2240333 2400/05533000000
Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang. Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan, and Zhenzhong Lan. Nova: An iterative planning and search approach to enhance novelty and diversity of LLM generated ideas. CoRR, abs/2410. 14255, 2024, doi: 10.48550/ARXI.24410. 14255. htthttts::///////t https:///oott org/ 10. 48550/arXiv 2410. 14255
Zongjie Li, Chaozheng Wang. Pingchuan Ma. Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in lmm-based evaluators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11084-11108, 2024.
Wanhao Liu, Zonglin Yang, Jue Wang, Lidong Bing, Di Zhang, Dongzhan Zhou, Yugiang Li, Houqiang Li, Erik Cambria, and Wanli Ouyang. Moose-chem3: Toward experiment- guided hypothesis ranking via simulated experimental feedback. arXiv preprint arXiv:2505. 17873, 2025a.
Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, and Dongzhan Zhou. Researchbench: Benchmarking 1lms in scientific discovery via inspiration- task decomposition. arXiv preprint arXiv:2503.21245: 2025b.
Ziming Luo, Zonglin Yang. Zexin Xu, Wei Yang, and Xinya Du. Llm4sr: A survey large language models for scientific research. arXiv preprint 2it::2501003300 2025.
Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence. 6(5):525-535, 2024.
Alexander Novikov, Ngån Vū, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wag- ner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphae- volve: A coding agent for scientific and algorithmic discovery. arXiv preprint 2rXiv:2506.13131, 2025.
OpenAl. Gpt-4o mini: Advancing cost-efficient intelligence.
gpt- Sooommni-aae advancing cost- teestefffciient www.eeeeieeiineeeeleeeeeeeeeeeee intelligence// 2024. Accessed: 2025-05-16.
Biqing Qi, Kaiyan Zhang, Kai Tian, Haoxiang Li, Zhang-Ren Chen, Sihang Zeng, Ermo Hua, Hu Jinfang, and Bowen Zhou. Large language models as biomedical hypothesis generators: A comprehensive evaluation. In First Conference on Language Modeling, 2024.
Bernardino Porere-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan $ Ellenberg. Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 475, 2024.
Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy, Lim-sr: Scientific equation discovery via programming with large language models. In The Thirteenth International Conference Learning Representations, 2025.
12

---

Chenglei Si, Diyi Yang. and Tatsunori Hashimoto. Can llms generate novel research ideas? A large- scale human study with 100+ NLP researchers, In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via sext-scale prediction. Advances in neural information processing systems, 37:84839-8445 2024.
Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. Scimon: Scientific inspiration machines optimized for novelty. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisfics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 279-299. Association for Computational Linguistics, 2024, doi: 10. 11065///1/22224ALLLLLOGGG v1/2024. acl long. 18
Utt org 10 18653/
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning language models. In The Eleventh International Conference on Learning Representations, 2023.
Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng. Soujanya Poria, and Erik Cambria. Large language models for automated open-domain scientific hypotheses discovery. Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16. 2024. pp. 13545-13565, Association for Computational Linguistics, 2024. doi: 10. http:///112222..01I.OOCO ACL.804. URL https //doi.org/10 hatpa/orrr/..18533/////00000 11113//1/222220 aac.://sa..innnnnaaalll.......
Zonglin Yang. Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
13

---

A MOOSE-Chem2 Overview I/O Figure
Research Question
Background Survey
Fine-grained Research Hypothesis
Details MOOSE- MooSE-Chem Chapter 2
Coarse-grained
Research Direction
Figure 5: Overview of the input and output of the Hierarchical Hcuristic Search (HHS) framework, also known as MOOSE-Chem's
Figure 5 presents overview of the input and output of the Hierarchical Heuristic Search (HHS) framework, also referred as MOOSE-Cher Chapter 2 (MOOSE (MOOSE-Chem2). The input to the framework consists of three components: research question, background survey, and coarse- grained research direction.
The research question defines the scientific goal to be addressed. The background survey summarizes existing methodologies relevant the research question. It may optionally include methodologi- cal or experimental details from prior studies that investigate similar questions or employ similar methodologies the given research direction, enabling the framework to leverage richer contextual knowledge. It may also include experimental results from previously tested fine-grained hypotheses, which provide additional guidance for search and evaluation. The coarse-grans saare-grained research direc- tion potentially derived from MOOSE-Chem (Yang et al.) (0225) serves as a high-level starting point that can vary granularity, ranging from brief sentence describing general research direction to preliminary hypothesis with partial methodological details.
B Expert Evaluation Instructions
For each research question, you will be presented with three candidate hypotheses alongside ground-truth fine- gre-graaied hypothesis. The order of the hypotheses is randomized. Your task is to rank the three candidate hypotheses based on their quality, using the ground-truth hypothesis reference.
Please evaluate the hypotheses based on the following four criteria:
Effectiveness: How well the hypothesis addresses the research question.
Novelty: The degree of originality relative to existing knowledge.
Detailedness: The specificity and clarity of the hypothesis.
Feasibility: The practical plausibility of experimentally testing or implementing the hypoth- esis.
Note that two tradeoffs may arise:
Between effectiveness and novelty
Between detailedness and feasibility
Use your expert judgment to rank the hypotheses based on a holistic assessment of these criteria. In rare cases where two hypotheses appear be of similar quality, assigning them the same rank acceptable.
14

---

C Recall Metric Calculation
Each evaluation compares pair of hypotheses- ground-truth hypothesis (from literature) and candidate hypothesis (generated by the model). The goal is to quantify how much methodological or experimental content from the ground-truth hypothesis is covered by the candidate.
Step 1. Decomposition. Both hypotheses are decomposed into methodological and experimental components (details) using structured prompts co-designed with chemistry PhD students. The prompts guide an LLM to extract key components such as procedures, specific materials, or reaction parameters. Each component is annotated with its role, function, and context.
Step Scoring via LLM-Judge. For each ground-truti component, the LLM judge searches for functionally corresponding components in the candidate hypothesis and assigns a coverage score from 0-3:
0: no match;
1: vague or partial match;
2: close but not exact match;
3: exact specific match.
The scoring prompts and examples were jointly developed and audited by PhD students ensure calibration and consistency.
Step Aggregation.
Two recall metries are computed:
# components with score > 0}
Soft Recall
(7)
# (ground-truth components)
Σ; score;
Hard Recall
(8)
# # (ground-truth components]
Soft Recall captures whether relevant components recovered all, while Hard Recall further reflects how precisely they are reproduced.
D
Case Study
Example 1:
Research Question: How can cost-effective N-type quasi-solid-state thermocell be devel- oped to boost electricity production from low-grade heat by improving both ion transport efficiency and electrode performance?
Hypothesis Candidate (from HHS):
The development of cost-effective N-type quasi-solid- state thermocell will be achieved through the strategic integration three core components to enhance electricity production from low-grade heat ($100"C):
1. Hierarchical Metal Electrodes:
Constructed from a copper-nickel alloy (70% Cu, 30% Ni) optimized for conductivity and corrosion resistance, these electrodes will feature dual-layer design incorporating a titanium dioxide (TiO2; CAS number: 13463-67-7) coating, precisely controlled at thickness of 100 transport by providing a stable interface that reduces charge recombination losses. An aluminum oxide (Al9O3; CAS number: 1344-28-1) layer will be included improve corrosion resistance and reinforce mechanical stability, operating synergistically to enhance the overall electrochemical performance. The fabrication process will utilize an eco- cro-friendly dual-step electrochemical deposition in a 0.5 M potassium sulfate electrolyte at controlled temperature of 25%C, ensuring micro- and nanoscale porosity targeting 100 nm to maximize surface area as supported by literature demonstrating that this range optimally enhances charge transfer and ion migration efficiency. This degree of porosity is expected to lower charge transfer resistance significantly, fostering improved electrochemical kinetics, which will be verified using scanning electron microscopy (SEM) for monitoring thickness and porosity.
enhance charge
15

---

2. Metal-Based Redox Couples: The thermocell will utilize copper/copper() and nickel/nickel(1I) redox couples, selected for their favorable redox potentials minimize side reactions. An integrated cobalt co-catalyst (0.1 M) will serve as an effective stabilizing agent, enhancing electron transfer kinetics and maintaining the oxidation states of Cu and Ni during thermal cycling, as demonstrated by prior studies indicating its role in fostering electron transfer efficiency. Real-time monitoring will maintain pH levels between 4-7, with adjustable concentrations of redox couples systematically optimized between 0.5 to 1.5 M based on insights from the literature regarding their stability and reactivity under varying operational conditions, with specific methodologies for pH adjustments clearly defined to ensure minimal disruption during testing.
3. Anisotropic Polymer Materials: The polymer matrix will feature aligned functional groups (-COOH and -SOyH), which will be developed through controlled mechanical stretching (5 mm/min at 709C), a technique known to enhance ionic transport path- ways by promoting favorable interactions between the polymer backbone and ionic species. High-conductivity ionic species, specifically Nat and Lit, will be incorpo- rated at optimized concentrations (0.5 to 1.0 M), with clear justifications based on their advantageous solvation properties and diffusion characteristics. The impact of variations in thermal conditions on ionic conductivity will be quantitatively assessed through impedance spectroscopy, with diffusion coefficients measured to elucidate the mechanisms of ionic mobility, providing detailed connection between thermal gradients and polymer performance.
This approach clarifies the synergistic interactions and functional enhancements among hierarchical metal electrodes, optimized redox couples, and engineered polymer materials, establishing a cohesive mechanism that uses low-grade heat drive efficient ion migration and electron transfer within the quasi-solid-state thermocell.
Expert's 's Analysis: The study addresses the question of how develop a cost-effective N-type quasi-solid-state thermocell to enhance electricity production from low-grade heat by improving both ion transport efficiency and electrode performance. The response provides a detailed and logically structured explanation, covering key aspects related to electrodes, redox pairs, and polymer gel media. Specifically, the preparation of Hierarchical Metal Electrodes is highlighted, noting the primary use of copper-containis a clear principle of providing stable interface, which aligns well with the ground truth (gdth) description of 3D hierarchical copper electrodes. Additionally, the study includes specific characterization of the electrode surface using SEM. However, while the polymer gel mentions an anisotropic aligmment structure, it omits the specific operational methods for forming this alignment, which is inconsistent with the gdth. Nevertheless, the overall response encompasses detailed mechanistic explanations and core points related to the specific operations.
electrodes, with
Example 2:
Research Question: How can a Supramolecular amorphous glass material engineered to simultaneously exhibit strong mechanical properties, optical transparency, self-healing capabilities, and adhesive strength at room temperature?
Hypothesis Candidate (from HHS): The engineering of multifunctional peptide-based supramolecular amorphous glass materials will concentrate on creating a robust dynamic network facilitated by cooperative reversible hydrogen bonds derived from interactions among four key amino acids: lysine, glutamic acid, aspartic acid, and tyrosine, along with proline to enhance rigidity. The hypothesized molar ratios will be set 1:1 for lysine and glutamic acid, complemented by approximately 0.5:0.5 ratios for aspartic acid and tyrosine. These ratios are supported by empirical studies that have shown that such compositions can optimize hydrophilic and hydrophobic interactions, which are essential for improving mechanical strength, adhesion, andself- -healing capabilities.
Lysine's positively charged ammonium group is hypothesized to establish strong ionic interactions with the negatively charged carboxylate groups of glutamic acid and aspartic acid, enhancing the stability of the hydrogen-bonding framework critical for effective energy
16

---

dissipation during mechanical stress. Tyrosine will contribute to the network through m-- stacking interactions, which are expected to maintain the structural integrity and optical transparency of the material under load. Proline's 's unique eyclic structure is anticipated to provide localized rigidity, supporting favorable peptide conformations and facilitating effective stress distribution throughout the dynamic network.
A pivotal component of this dynamic system will be the incorporation of structured water, maintained at optimal concentration of 10-15% by weight. Structured water is theorized to engage in specific hydrogen bonding interactions with the peptide backbone, promoting molecular mobility and enabling rapid bond reformation necessary for self-healing capa- bilities at room temperature. The investigation will differentiate between structured and unstructured water forms, examining the specific interactions that influence bond lifetimes and recovery dynamics under mechanical stress. Previous research indicates that the interac- tions between structured water and the peptide backbone critical for maintaining bond lifetimes during stress, and these dynamics will be quantitatively assessed through methods such as differential scanning calorimetry (DSC) and rheological measurements.
The synthesis of these peptide assemblies will be executed under rigorously controlled environmental conditions. The reaction temperature will be maintained at 25%C to optimize amino acid interactions and prevent degradation, while systematic variations in pH (5.5 to 8.5) will be conducted to explore their effects on the ionization states of the amino acids and corresponding hydrogen bonding dynamics. Additionally, ionic strength will be regulated at approximately 0. 1515 M using sodium chloride, which is expected enhance electrostatic interactions and stabilize the hydrogen bonding network.
To thoroughly investigate these interactions and material properties, a combination of exper- imental methodologies will be utilized. Dynamic Mechanical Analysis (DMA) will assess mechanical properties such as tensile strength and clasticity, while rheological assessments will evaluate the material's 's response under stress. Spectroscopic techniques, including NMR spectroscopy and Fourier Transform Infrared (FTIR) spectroscopy, will be employed to elucidate hydrogen bonding dynamics and monitor molecular interactions. This comprehen- sive approach aims to clarify the intricate relationships among amino acids and structured water dynamics, well as the influences of environmental conditions on the multifunctional properties of the engineered peptide-based supramolecular materials.
Expert's Analysis: In addressing the research question-""Hw can supramolecular amor- phous glass material be engineered to simultaneously exhibit strong mechanical properties, optical transparency. self-healing capabilities, and adhesive strength at room tempera- ture?" the generated scientific hypothesis proposes notably complex system. This system is envisioned to comprise five distinct amino acids: lysine, glutamic acid, aspartic acid, tyrosine, and proline.
Despite the increased complexity of this multi colli-commpnnnt approach compared to the simpler system underlying the real hypothesis (the scientific finding concerning YYY peptide glass), several key conceptual parallels are evident: Shared Foundation in Peptide-Based Materials: At their core, both the generated hypothesis and the real scientific finding are centered on peptide-based materials as the fundamental building blocks for the desired amorphous Clase.Conerrent Aim for Dynamic Networks and Functional Properties: Both frameworks leverage their respective peptide systems with the goal of establishing a dynamic network. This network is considered crucial for imbuing the material with critical functionalities, particularly self-healing capabilities and effective adhesive strength. Emphhsii on the Role of Structural Water: In their mechanistic considerations, both hypotheses significantly highlight the indispensable role of structural water. The real hypothesis (the scientific finding on YYY glass) successfully demonstrated that dense, random hydrogen-bonding network, mediated by water molecules, is fundamental to the YYY glass's 's unique structure and its observed properties. The generated hypothesis also underscores the centrality of cooperative and reversible hydrogen bonds in the construction and operational dynamics of its proposed network.
Example 3:
17

---

Research Question:
How can computational methods be used to accurately predict and im- prove the reactivity and selectivity of modular diazo transfer (MoDAT) reactions, especially reactions with primary amines? And to design new reagents based on computational models.
Hypothesis Candidate 1 (from HHS); We propose to systematically investigate the reactivity and selectivity of modular diazo transfer (MoDAT) reactions utilizing azide-based reagents, with a specific focus on para-substituted benzyl azide derivatives modified with strong electron-withdrawen
groups (EWGs) such as nitro (-NO,) and eyano (-CN), as well as weaker electron- wiihrrowwwwiddawwiinn groups (e.g. fluoro (-F) and chloro (-CI), and electron- donating groups (EDGs) like methoxy (-OCH3). Our central hypothesis posits that the electronic nature and precise positioning of these substituents will significantly modulate the electrophilicity of the azide moiety, which will in turn influence the stability and geometrical configurations of intermediates and transition states during nucleophilic attacks by primary amines.
The experimental work will be executed under controlled laboratory conditions using a Schlenk line to maintain an inert nitrogen atmosphere for at least 30 minutes prior to reaction initiation, minimizing moisture exposure. Reactions will be conducted at a temperature of 90- chosen based on literature findings indicating optimal kinetic performance while preserving the stability of diazo intermediates. We will employ polar aprotic solvents such as limemthyloormmiid (DMF) and dimethyl sulfoxide (DMSO), which are anticipated to enhance solvation of the azide and improve nucleophilicity of the primary amines. A stoichiometric ratio of 1:1.5 (benzyl halide to sodium azide) will be applied, and reactant concentrations will be maintained at approximately 10-20 mM, a range supported by preliminary studies demonstrating optimal reactivity and solubility.
To deepen our mechanistic understanding, we will utilize advanced computational tech- niques, primarily Density Functional Theory (DFT) with specific emphasis on the B3LYP functional and a (d) basis set. This will allow us to thoroughly assess the impacts of substituent variations on charge distributions and transition state energies. In particular, we will quantify how the introduction of strong EWGs influences the electrophilicity of the azide, predicted to lower activation energy for nucleophilic attacks, while the EDGs may provide stabilization during transition state formation. A rigorous electronic structure analy- sis will be employed to visualize the overlap between the nucleophilic orbital of primary amines and the electrophilic centers of the diazo derivatives.
To ensure a comprehensive evaluation of solvent effects, we will systematically conduct reactions in both polar protic solvents, such as ethanol, and polar aprotic solvents, like DMSO, maintaining a concentration of 50% v/v relative to the reaction volume. This will elucidate how solvent polarity and cadrogen-bondingg capacity impact the stability of zwitte- rionic intermediates and transition states. Furthermore, we will establish robust iterative feedback mechanism that directly correlates our computational predictions empirical data from high- righuresolution techniques, including NMR spectroscopy and mass spectrometry. Dis- crepancies exceeding a predefined threshold such as a 10% difference between predicted and observed yields-will prompt specific adjustments to our computational models, thus refining our understanding of the factors influencing MoDAT reactions. This structured approach aims to enhance the design of azide hased reagents tailored for improved reactivity and selectivity in synthetic applications involving primary amines.
Hypothesis Candidate 2 (Greedy + self-consistency):: This study aims to investigate the re- activity and selectivity of modular diazo transfer (MoDAT) reactions by designing tailored azide-based reagents that incorporate strategically positioned glectron-withdrawiig groups (EWGs), specifically nitro (-NO2) and cyano (-CN) moieties. These groups will be sys- tematically evaluated at distances of 2 A, 4 A, and 6 Á from the azide functionality. This selection is grounded in existing literature, which suggests that such variations in EWG positioning can significantly influence the electrophilicity of the azide, thereby stabilizing transition states during nucleophilic interactions with primary amines. The specific rationale for these distances is based on prior studies indicating that alterations beyond these ranges may lead to diminished effects on reactivity or unfavorable sterics.
The primary amines selected for this varying degrees of steric hindrance and electronic
b-bulilannne,llllnnneneeeeee phenethylamine, ben-
zylamine, and ethylamine- characteristics, allowing for comprehensive analysis of their reactivity profiles when paired with engineered azide reagents. This choice effectively captures spectrum of nucleophilic
18

---

behavior and enhances our understanding of how both electronic and steric factors operate within MoDAT reactions.
An integrated computational strategy will be employed utilizing density functional theory (DFT) calculations focused on the B3LYP functional, supplemented by other functionals (e.g,, M06-2X and PBEO) to ensure accuracy and validation of outcomes. In addition to DFT, we will incorporate transition state theory and potentially machine learning approaches to analyze the reactivity patterns further. We will conduct extensive mapping of activation energy profiles and transition state geometries to uncover the relationships between EWG positioning, electron density distributions, and transition state stability. Molecular dynamics (MD) simulations will also be utilized explore solvation effects across differing solvent environments, including acetonitrile, methanol, and DMSO, examining how these solvents influence reaction pathways and stabilize charged intermediates.
Empirical validation of computational models will incorporate a structured approach to varying critical parameters such as azide reagent concentrations and molar ratios of primary amines to azides, along with solvent compositions, to derive quantitative metries, including reaction yields, rate constants, and activation energies. Statistical analyses will employ techniques such as ANOVA and regression models to extract significant trends from the experimental data. This iterative feedback mechanism will facilitate a dynamic refinement process, whereby experimental outcomes directly inform adjustments to computational predictions. Through this comprehensive methodological framework, we aim to elucidate the interplay between EWG distances and steric factors, ultimately leading to the design of innovative based reagents optimized for selective transformations of primary amines.
Hypothesis Candidate 3 (Greedy): This research aims to systematically investigate the re- activity and selectivity of modular diazo transfer (MoDAT) reactions utilizing azide-based reagents, focusing on a set of primary amines: benzylamine, 2-aminopropane, and cy- clohexylamine. This selection combines varying steric and electronic profiles, enabling comprehensive evaluation of how solvent and reaction conditions influence reactivity and selectivity across different nucleophilicity ranges. Initial studies will determine baseline reactivities through systematic kinetic measurements, assessing critical parameters such as rate constants and product ratios under controlled conditions.
Reactant concentrations will be evaluated at specific increments of 0. M (0.1 M, 0.5 M, and 1.0 M), and the temperature will be optimized through systematic approach involving stepwise assessments from 25 to 60CC, analyzing how these variations affect reaction progress. A comprehensive assessment of solvent effects will be performed. including the examination of solvent mixtures (e.g., varying concentrations of water, DMSO, and possible co-solvents) to quantify their influence on nucleophilicity and overall reactivity.
Advanced computational methods, including density functional theory (DFT) calculations with the M06-2X functional and a 6-31G basis set, will be employed to simulate the MoDAT reaction environment accurately. We will analyze key molecular descriptors such as nucleophilicity, clectrophilicity, and steric hindrance to construct predictive models of reactivity. These analyses will guide experimental design, with a feedback mechanism where discrepancies between computational predictions and experimental observations will result in specific adjustments to molecular descriptors or computational parameters, refining the predictive capabilities of the models.
Following these investigations, the design of innovative azide-based reagents will be under- taken to optimize MoDAT reactions. This design process will emphasize the incorporation of electron- withdrawing groups like trifluoromethyl and cyano, aimed at enhancing both sta- bility and selectivity by stabilizing the transition state. Rigorous standardized experimental protocols will ensure reproducibility. including specific techniques for measuring yields and selectivity ratios over controlled reaction durations. By integrating mechanistic insights from computational and empirical findings, this research will elucidate the key factors influencing reactivity and selectivity in diazo transfer reactions, enhancing our understanding of these critical processes.
Expert's 's Analysis: conducted relatively comprehensive analysis, for instance, suggesting that modifying the azide reagent with functional groups could improve it, which aligns with the original text. However, 2 and 3 did not. This time, has obvious emoor the speculated temperature is incorrect, and the proposed temperature experimentally unfeasible, as azide reagents are prone explosion at high temperatures. Of course, temperature is a minor
19

---

point, and overall, 1 is still acceptable. 2 deviates significantly from the original text in terms of the research design approach. Compared to 2, 3 lacks consideration of the group effect in the research design, making 3 the weakest. Finally, all three mentioned using DFT calculations, and although there are deviations details from the original text, the approach is correct.
Example 4:
Research Question: How can photoredox catalysis be used to exploit the latent reactivity of phosphorus ylides, allowing them to participate in formal three-component cycloaddition that converts inert C-H and C=P bonds into C-C and C=C bonds, creating versatile synthetic building blocks in an efficient, controlled manner?
Hypothesis Candidate 1 (from HHS):
The mechanism for activating phosphorus ylides in formal three-component cycloaddition via photoredox catalysis can be articulated in four key steps, each supported by optimized experimental conditions:
1. Initiation of Lingle-Electron Transfer (SET): Irradiation of phosphorus ylides with specific wavelengths of visible light (400-450 nm) from high-intensity LED source (approximately 20 mW/cm²), validated by studies demonstrating effective radical generation this intensity (Smith et al., 2020), promotes SET using suitable photoredox catalysts (e.g.. [Ru(bpy)s]= or The resulting radical cation exhibits enhanced electrophilicity due to significant charge localization, which is further assisted by strong centtronwiithrawwin substituents such as carbonyl or nitro groups. Empirical evidence indicates an increase in reactivity by up to 2.5-fold as supported by Hammett parameters.
2. Stabilization via Zwitterionic Intermediate: The radical cation transitions to zwitterionic intermediate, characterized by resonance stabilization through delocalized r-electrons and non-covalent interactions, such as hydrogen bonding in polar aprotic solvents like acetonitrile (dielectric constant x 37) and DMSO (dielectric constant R 47). To optimize stabilization, 1:1 (v/v) mixture of these solvents will be used, taking advantage of their combined dielectric properties (= 38) to enhance charge separation and stabilize reactive intermediates. Literature supports this approach, showing improved reaction kinetics (Miller et al., 2021).
3. Selective Nucleophilic Attack: The zwitterionic intermediate selectively engages in nucleophilic attacks on activated C-H and C=P bonds, particularly those adjacent to strong groups. Maintaining phosphorus ylide concentrations at 0.1 M and controlling reaction temperatures precisely within an optimized range of 10-25 "C as indicated by previous studies on radical stability, will minimize side reactions. An inert atmosphere (nitrogen or argon) will be established by purging the reaction vessel for 30 minutes before use, effectively mitigating oxidation. Real-time NMR (utilizing ID and 2D techniques) and GC-MS metrics will be employed to monitor yield and product distribution effectively, specifying analytical conditions (e.g., temperature settings and flow rates) to ensure accurate assessment of outcomes.
4. Concerted Formation of Products:
The reaction culminates in the concerted formation of new C-C and C-C bonds, facilitating the synthesis of valuable carbocycles and synthetic building blocks. The influence of substituent identity and positioning (ortho, meta, para) on reactivity will be quantitatively analyzed using NMR and HPLC techniques. This systematic approach will provide insight into the efficiency and selectivity of the cycloaddition process, explaining how each factor contributes to overall reactivity.
By integrating these components clearly and methodically, this hypothesis presents a com- prehensive exploration of how photoredox catalysis can unveil new reactivity pathways for phosphorus ylides, fully addressing the research question with explicitly defined roles of each mechanistic step and comprehensive definitions for specialized terms provided for clarity.
Hypothesis Candidate 2 (Greedy + self-coonistency):: This study aims to investigate how photoredox catalysis can elucidate specific reactivity mechanisms in diphenylp hpphenylphosphhny/ ylides, focusing on their participation as intermediates in formal three-component cycload- dition reactions that convert inert alkyl C-H bonds and coordinated C=P bonds into C-C and C=C bonds. We will concentrate on diphenylphosphing ylides due to their capacity to facilitate charge inversion processes, which are crucial for generating stabilized radical
20

---

intermediates. Specifically, we will examine how the electronic structure of diphenylphos- phinyl ylides transforms upon excitation under visible light (e.g., at wavelengths of 400-450 nm) into polar intermediates, leading to the generation of stable benzylic radicals from C-H bond cleavage and allylic radicals from C=P bond activation botth essential for effective cycloaddition.
1. Charge Inversion Mechanisms: We will clarify the mechanisms of charge inversion by detailing the electronic transitions involved, emphasizing the role of excited state lifetimes and how solvent polarity affects the stabilization of generated radicals. Our discussion will include specific references known photoredox systems and their operational parameters in literature, enhancing our findings with context.
2. Role of r-Stacking Interactions: We will investigate the influence of r--taccing inter- actions, specifying the optimal inter-ring distances and energy considerations between the stabilized radicals and electrophilic carbon centers from substrates like 2,6-dibromoaryl and 2,4- difllouoor.rr.. This section will include quantitative assessments of radical coupling reactivity, aiming to determine coupling efficiencies and product selectivity through methods such as H NMR and UV-Vis spectroscopy.
3. Solvent and Temperature Effects: We will analyze solvent effects by conducting reactions in range of solvents with varied dielectric constants, including thorough comparison of polar solvents like acetonitrile and non-polar solvents like dichloromethane. We will detail how these solvents are expected influence radical stabilization and the kinetics of cycloaddition, providing a theoretical framework based on established solvent interaction models. Additionally, to 50 by linking it to the expected stability of radical intermediates and the kinetics of the reactions, ensuring optimal conditions for product yield and selectivity.
will justify the controlled temperature range of room temperature
4. Experimental Conditions: Reactions will be conducted under carefully controlled condi- tions, with temperature justification focusing on maintaining the balance between radical stability and reaction kinetics. We will outline how these conditions directly relate to the completed cycloaddition mechanism and the anticipated outcomes of the study.
By systematically elucidating these mechanisms- speciifcally charge inversion, r-stacking interactions, and solvent effects- we aim to develop robust methodologies for the efficient generation of versatile synthetic building blocks from simple molecular precursors.
Hypothesis Candidate 3 (Greedy); Investigate how photoredox catalysis enhances the re- activity of phosphorus ylides through selective nucleophilic attack on a, --unsaturatedd carbonyl compounds, such crotonaldehyde, which are activated form stable radical cation intermediates via photoredox-dmee -drive single-electron transfer (SET) processes. These radical cations, characterized by their electrophilicity, promote effective nucleophilic attacks by phosphorus ylides, generating stabilized carbon radical intermediates that significantly enhance their reactivity in subsequent hond-forming transformations. Conduct a formal three-component cycloaddition by introducing a nucleophilic amine, such as ammonia or an aniline derivative, selected based on its electronic properties which influence the stabilization of the radical intermediates and affect product selectivity. Detail specific optimized reaction conditions, including the use of polar aprotic solvents like acetonitrile, which facilitate radical stability, and employ a specific light wavelength of 400 nm to ensure efficient exci- tation of the photocatalyst. These conditions will be designed to minimize potential side reactions and maximize the conversion of inert C-H H and C=P bonds into desired C-C and C=C bonds through well-defined mechanistic pathways, addressing the nuanced interplay between reaction parameters and final product outcomes.
Expert's 's Analysis: accurately predicted the light source wavelength range, metal catalyst system, and solvent system, such the use of Ir catalyst and acetonitrile as the solvent, all of which align with the original text. In contrast, 2 only correctly predicted the wavelength range and solvent system but failed to specify the metal catalyst system, which crucial in organic chemical reactions. Therefore, 2 is inferior to 1. Finally, 3 did not predict the light source wavelength range or the metal catalyst system, missing several key pieces of information, making it the weakest.
21

---

E
Expert Analysis of Hypothesis Quality
E.1
Convergence to Ground-Truie Local Optima
To complement our quantitative evaluations, asked domain experts qualitatively assess how well the hypotheses generated by HHS aligned with the expert-annotated fine-grained hypotheses in benchmark.
The distribution of expert assessments across all evaluated examples is as follows:
Reached completely different -iiionn-lkklyy distinct local optimum with scientific plausibility: 60%
Reached the vicinity of the ground-truth local optimum, but with differing details: 24%
Reached the vicinity of the ground-truth local optimum, but failed to fully elaborate or specify the key details: 16%
Here, the "ground-tue truth local optimum" refers to the expert-cxtracted fine-grained hypothesis from a publication, which serves the reference target. "Reaching the vicinity of local optimum" indicates that the generated hypothesis converges to coherent and internally consistent formulation that is conceptually close the ground-trun -ruth hypothesis, though not necessarily identical detail.
The relatively high divergence rate (60%) reflects an inherent tradeoff in our experimental setup. For many research questions, multiple hypotheses be plausible yet structurally distinct. Guiding the model toward the exact ground-truth hypothesis requires:
initializing the search process from a starting point sufficiently close to the ground-truth optimum:
2. but avoiding initialization that is too close or too specific, as this would risk leaking the ground-truth answer.
To strike this balance, we derive the initial search point from the annotated coarse-grained hypothesis he by applying an ambiguation procedure. This involves removing or abstracting key details to produce a generalized version of he-for example, replacing "a specific protein" " with "a protein" or "a catalyst" preserving the overall research direction while preventing answer leakage.
Consequently, even when the search begins in the correct conceptual region, the model may naturally diverge toward a nearby but distinct local optimum, especially given the openness of the hypothesis space and the heuristic-drive nature of the optimization process.
E.2 Coverage of Experimentally Critical Details
In addition to alignment with the reference hypotheses, we evaluated the extent to which the generated hypotheses captured the critical experimental details required for practical implementation.
Among all the details mentioned in the generated hypotheses, approximately 40% are experimentally important-regardless of their accuracy (which is not the focus of this analysis). The remaining 60% % are peripheral or have minimal impact on the actual experiment.
Among all the important details that should be included, about % are mentioned in the generated hypotheses.
Peripheral details refer to contextual or environmental factors with limited relevance to the core experiment- for instance, ambient humidity weather conditions, which may only affect specific reactions.
This highlights key challenge: while LLMs can generate rich and context-aware hypotheses, they often fail prioritize the most essential components for experimental planning. Future work may explore techniques to guide LLM attention toward experimentally salient information.
22

---

Table 5: Error analysis of hypotheses generated by HHS. Two PhD-level chemistry experts conducted the evaluation: one analyzed the top 30 samples and the other the remaining 21.
Missing key chemical substances
14/30
Excessive details in charactcrization methods Feasibility issues
28/30
18/30
Limitations of characterization methods
08/30
Insufficient basis for material selection
22/30
Lack of design comparison experiments Ignoring data validation and reproducibility
12/30
10/30
Severe deviation from feasibility
8/21
Missing or incorrect key chemicals or reaction systems
9/21
Incorrect explanation of chemical principles
12/21
Incorrect prediction of experimental system
10/21
Table 6: Error analysis why HHS's hypotheses better than the greedy search baselines'. Two PhD-level chemistry experts conducted the evaluation: one analyzed the top 30 sample pairs and the other the remaining 21.
Insufficient performance metrics
25/30
Complexity of experimental conditions
16/30
Insufficient explanation details
29/30
Inadequate description of preparation plan
22/30
Vague research objectives
28/30
Cost and scalability issues
13/30
Poor fcasibility
12/21
Errors research plan details
21/21
Insufficient explanation details
19/21
Clear experimental system
21/21
F Error Analysis
Two PhD-level chemistry experts conducted the analyses. SabbeeJssmmmriiess the main types observed hypotheses generated by HHS, while Table 6 analyzes the reasons why HHS outperforms the greedy search baseline.
G Hypothesis Search Prompt
The following prompt is used guide both the baseline methods and proposed method, HIS, during the hypothesis refinement process. To ensure fair comparison, the prompt is designed in controlled way: we use shared core prompt across all methods, with minimal differences. Specifically, the portion highlighted in orange is unique to HHS and introduces the hierarchical structure used in its search process.
This design isolates the effect of hierarchical search. As illustrated below, the only difference between HHS and the baseline (Greedy Search + Self-Consist)ncy) lies in the hierarchical prompting. The core content-ing incluuinn the role of the assistant, editing instructions, and structural expectations is kept identical.
This enables a controlled ablation-style comparison, attributing observed improvements specifically to the hierarchical design.
23

---

The complete prompt is as follows:
You are assisting with scientist's research. Given their research question, a survey on the past methods for the research question, and a preliminary coarse-grained research hypothesis for the research question, please help to make modifications into the coarse- grrise--ggndddde hypothesis, to make it one step closer to a more effective and more complete fine- fine-grained hypothesis.
The modification can be two-folds: (I); delete or change existing improper detail or information in the existing kypothesis: (2) add and integrate one detail to the existing hypothesis. you choose to add detail, do not simply append new information
the existing hypothesis. Instead, think thoroughly how the new detail relates to the existing components and integrate it seamlessly into the hypothesis to create a new coherent and unified hypothesis. In addition if you choose to add detail to a general information, if the corresponding general information is correct, you should try to keep the corresponding general information in the updated hypothesis and also mention the details, instead of replacing the general information with the details. In this way, it would be much easier for scientists to understand both the general infomration/structuree and the details from your generated hypothesis, It would be also easier for scientists to propose better details, inspired by your suggested details, following the general information.
Please remind that this is about research: research is about discover a new solution to the problem that ideally is more effective and can bring new insights. Usually we don't 't need the hypothesis to contain lots of known tricks to make it work better: we want to explore the unknown, which ideally is more effective than the known methods and can also bring in new insights. Therefore, research hypothesis is usually about a small set (usually less than eight) of major components (and lots of details on how to implement these major components), which overall composes a novel and complete solution to the research question, which potentially can bring in new insights. Hypotheses that include an excessive number of irrelevant or unnecessary major components, which do not contribute to addressing the research question, are less favorable, as we only want to know exactly what are the key components that fundamentally make the hypothesis work. you think any ancillary components that can truly assist with the research question, you may mention what are the key components and what are the ancillary components to avoid the ambiguity of which components are the key component. The reaction mechanism, however, is not classified as major component or detail (and therefore not limited by the number of major components). Instead, a novel and valid reaction mechanism can be good source of insights. If previous hypothesis already contains too many major components, you should consider to replace some of the major components with more effective ones (but not to add more major components), to give more details to the existing major components for clarity and ease of implementation (instead of adding or replacing major components).
Here we are searching for the fine-grained hypothesis in a hierarchical way. The rationale is that, we can classify any complete set of modifications into several hierarchy, with different levels of details. If we do not search in a hierarchical way, we need to consider all the available details in all hierarchy levels for each search step, which (1) has a very high complexity, and (2) first search low-level detail might largely influence the following search of a high-level detail: it might stuck in one high-level detail corresponding to the already searched low-level detail without considering the other low-level details corresponding to other high-level details, making the search process stuck in a local minumum at the beginning.
Here we roughly classify all possible modifications into five hierarchies: (1) Mechanism of the Reaction: Describes how the reaction proceeds a conceptual level, focusing on electron flow, bond formation and breaking, and any intermediates or transition states involved. This is the theoretical "blueprint" " that explains why the reaction works; (2) General Concept or General Component Needed: Identifies the type of reagent or functional group required (e.g., "a strong acid," "a Lewis base," "an activated aromatic ring") without committing to a specific chemical. outlines the broader roles that are necessary for the mechanism to proceed; (3) Specific Components for the General Concept: Narrows down from the general category to a particular substance (e.g., "concentrated HCI" for strong acid, "benzene" for an aromatic ring). This makes the reaction hypothesis testable by specifying which chemicals fulfill the roles; (4) Full Details of the Specific Components: Provides exact structural or molecular information-suce as SMILES strings, IUPAC names, purity, or CAS numbers. These details ensure clarity and reproducibility so researchers know precisely which substances to use: (5) Experimental Conditions: Specifies the practical setup- tempprraurrr,,r pressure, solvent system, reaction time, atmosphere, and any work-up procedures. This final layer describes
24

---

how to carry out the reaction in laboratory setting. And we are searching for modifications hierarchy by hierarchy: hierarchy (1) first, and then hierarchy (2), and so on. Hypothesis from higher hierarchy is an expansion of the hypothesis from its previous hierarchy, with additional information described above.
The research question is:
The survey is:
Now please kelp to make modifications into the coarse carse-grained hypothesis. to make one step closer to more effective and more complete fine- fine-grained hypothesis. Please do not include the expected performance or the significance of the hypothesis in your generation. Please answer the question in the following response format. (response format: "Reasoning Process: Revised Hypothesis:
H Experiment Compute Resources
We implement our proposed framework as an agentic LLM system using GPT-4o-mini using OpenAl's 's official API. Generating the final hypothesis via the HHS optimization process converging to the final local optimum at hicrarchy level 5 5-tpically involves several hundred or even to thousand iterative search steps.
I Limitation
While HHS consistently discovers higher-quality local optima compared to baseline methods, it does not guarantee convergence the global optimum. Addressing this limitation remains an open direction for future research.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we have summarized the contributions in the end of the introduction section. They accurately reflect the scope of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We mention it in the conclusion section: "The limitation is that HHS is not guaranteed to reach to a global optimum, and we leave it as a future research direction."
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification: In Section 2, we provide detailed information about the assumptions and corresponding proof.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed information about the experimental configurations and evaluation protocols. More details can be found in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code, data, and instructions
can
be
found
in
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
ollicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle Sote:////uudde//eeuuuuubiiii
(CodeSubiisss wwtaaabmmsssooni ssiiiissicccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details about hyperparameters provided in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All details about statistical significance are provided in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention that we use "GPT-4o-mini" More details are mentioned in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httptt/tt/////rrooroo.ooooooaoaa ittbbliic/thiiccccudeeiiiiee
Answer: [Yes]
Justification:
We check the code of Ethics, and we don't 't think this research has violated it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussions about social impacts are provided in the supplement.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this paper are properly credited and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code, data, and instructions
be
found in
http://ghhub..ommooZnniy////ooo Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
31

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
The core method development does not involve LLMs as any important components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to LLM policy for what should or should not be described.
32


Reviews:
[{"review_id": "9TK6PW04yc", "reviewer": "Reviewer_dwWU", "summary": "This paper introduces the task of fine-grained scientific hypothesis discovery and proposes a Hierarchical Heuristic Search (HHS) framework that leverages LLMs both to edit and evaluate hypotheses across multiple abstraction levels. Experiments on a small chemistry benchmark show that HHS outperforms greedy baselines in LLM self-assessments and expert rankings.", "questions": "No", "limitations": "No. Please do a thorough literature review and compare with SOTA methods.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "### Strengths\n\n1. Clear task definition & dataset – The authors articulate why coarse hypotheses are insufficient and release a contamination-controlled benchmark with expert annotations, filling a recognised gap in the literature.\n\n2. Coherent agentic framework – HHS is a self-contained, reproducible pipeline that factors an intractable search into tractable sub-problems and provides intuitive smoothing analysis.\n\n### Weaknesses\n1. The literature review is exceptionally thin for a NeurIPS-level submission—only ~15 references are listed, many of them generic model or dataset announcements rather than directly relevant prior art. Key recent work on hierarchical planning, tree-of-thought search, and agent-based scientific discovery published in NeurIPS/ICLR/ACL 2024-25 is absent. This shallow bibliography suggests the authors have not situated their contribution within the full landscape of related methods and undermines claims of novelty and significance. \n\n2. The contribution’s novelty is overstated. Prior systems, such as Chemcrow [1] and other retrieval-plus-refine agents already tackle hypothesis generation (some explicitly optimise for novelty and detail); the paper does not cite or benchmark against them nor clarify conceptual differences, leaving incremental value uncertain.\n\n3. HHS is essentially a coarse-to-fine or evolutionary search variant; the “hierarchy smooths reward” claim is supported only by schematic plots, lacking formal metrics or convergence theory, so theoretical depth is thin.\n\n4. Baselines omit standard combinatorial optimisers (reinforcement learning, MCTS, genetic algorithms) and recent retrieval-augment-edit pipelines, so HHS’s superiority is unproven outside the chosen comparisons.\n\n[1] Bran, Andres M., et al. \"Chemcrow: Augmenting large-language models with chemistry tools.\" arXiv preprint arXiv:2304.05376 (2023).", "was_revised": true, "final_justification": "Most of my questions have been addressed. However, I still find the paper difficult to follow due to the limited literature review. This lack of contextualization makes it challenging to compare the proposed approach with existing work and to clearly identify the paper’s contributions. Expanding the related work section and clarifying how this work differs from or builds upon prior studies would significantly improve the paper's clarity and accessibility.", "rebuttal": "We appreciate your insightful questions and helpful suggestions. For clarity, we have organized your queries along with our responses below:\n\n---\n\n**Q1**: Only ~15 references are listed, Key recent work on agent-based scientific discovery published in NeurIPS/ICLR/ACL 2024-25 is absent.\n\n**A1**: We thank the reviewer for raising concerns about the references. As this work introduces the novel task of fine-grained scientific hypothesis discovery—for which no prior research exists—there are inherently fewer directly relevant papers to cite.\n\nWe did cite key recent works on agent-based scientific discovery published in NeurIPS/ICLR/ACL 2024-25, including:\n\n[1] Scimon: Scientific inspiration machines optimized for novelty, ACL 2024.  \n[2] Large Language Models for Automated Open-domain Scientific Hypotheses Discovery, ACL 2024.  \n[3] Large language models are zero shot hypothesis proposers, COLM 2024.  \n[4] MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses, ICLR 2025.  \n[5] Can llms generate novel research ideas? A large-scale human study with 100+ NLP researchers, ICLR 2025.  \n[6] NOVA: An Iterative Planning Framework for Enhancing Scientific Innovation with Large Language Models, ACL 2025.  \n\n\n**Q2**: Previous work like Chemcrow already works on hypothesis generation. \n\n**A2**: We thank the reviewer for highlighting ChemCrow. While previous works like ChemCrow address hypothesis generation, it does not focus on fine-grained hypothesis discovery. We did not adopt it as a baseline initially because they won’t have a comparable performance on providing the hypothesis details. We have **added 5 additional baselines** including ChemCrow [7] as well as four other strong baselines (three published in top conferences in the recent two years, and one combinatorial optimization optimizer as required in Q3), with newly added baselines boldened. We will refer to them in the paper.\n\n\n|                            | Soft Recall | Hard Recall |\n|----------------------------|-------------|-------------|\n| **ChemCrow[7]**               |      12.28% |       7.20% |\n| **Reinforcement Learning** |      16.76% |      10.28% |\n| **[3]**                     |      19.57% |      11.15% |\n| **MOOSE[2]**                  |      20.04% |      11.76% |\n| **MOOSE-Chem[4]**             |      19.99% |      11.98% |\n| Greedy                     |      16.60% |       9.92% |\n| Greedy + Self-Consistency                         |      31.53% |      17.73% |\n| HHS                         |      **40.35%** |     **23.04%** |\n\n[7] ChemCrow: Augmenting large-language models with chemistry tools, Nature\n\n\n\n**Q3**: Baselines omit standard combinatorial optimisers such as reinforcement learning, MCTS, genetic algorithms.\n\n**A3**: We thank the reviewer for suggesting additional combinatorial optimizers as baselines. As fine-grained scientific hypothesis discovery is a new task introduced in this paper, and while we formalize it as a combinatorial optimization problem, the search space is inherently unknown (line 122: \"n is unknown\"), unlike traditional problems.\n\nMCTS and genetic algorithms require a clearly defined search space, precluding direct implementation here. We are aware that prior works like [8][9] use MCTS, but in tasks with explicit search spaces.\n\nReinforcement learning, however, does not require a defined search space, and we have implemented it as shown in the Table of A2.\n\n\n[8] Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design, EMNLP 2023.  \n[9] CHEMREASONER: Heuristic Search over a Large Language Model's Knowledge Space using Quantum-Chemical Feedback, ICML 2024.  \n\n\n\n\n**Q4**: HHS is essentially a coarse-to-fine or evolutionary search variant. \n\n**A4**: We appreciate the reviewer's observation that HHS incorporates coarse-to-fine and evolutionary elements. However, our core contributions lie in formalizing the novel fine-grained hypothesis discovery task as a combinatorial optimization problem and developing HHS to optimize it effectively for scientific discovery. No prior LLM-based scientific discovery work has framed the task through an optimization lens. Moreover, HHS advances coarse-to-fine search by adapting it to challenging, language-based open-ended tasks like hypothesis generation, where such implementations were previously unclear.\n\n**Q5**: The “hierarchy smooths reward” claim is supported only by schematic plots, lacking formal metrics or convergence theory.\n\n**A5**: \nWe appreciate the reviewer's feedback on the theoretical support for our \"hierarchy smooths reward\" claim. We connect fine-grained hypothesis discovery to established theories in image and signal processing, where smoothing convolution (averaging pixels in an image patch) acts as a low-pass filter.\n\nThe sufficient condition for this claim is our assumption that \"LLM’s evaluation for a hypothesis with a general concept can be viewed as an aggregated estimate--approximating an average or soft maximum--of LLM’s evaluation for hypotheses with specific concepts under that general concept\" (lines 197–199).\n\nWhile this assumption is commonsensically true but challenging to prove formally, we conducted additional experiments to verify it: We prompted LLMs to score hypotheses with general concepts and their corresponding specific variants on a 0–100 scale, finding an average difference of only 3.47 between the general score and the averaged specific scores (in total 204 pairs compared).\n\n---\n*We greatly appreciate your comprehensive feedback. We hope that our responses have satisfactorily addressed all your queries. Should you have further questions or suggestions for enhancing our manuscript, we warmly welcome your input.*"}, {"review_id": "r7oomOMC3C", "reviewer": "Reviewer_XFST", "summary": "The paper investigates how LLMs can be used to generate fine-grained scientific hypotheses that could be directly implemented in laboratory settings, contrasting with prior studies that typically only produce coarse-grained ideas with low levels of detail. The authors propose a hierarchical heuristic search (HSS) approach to tackle this task, comparing it against two baseline methods that perform a flat search, namely greedy search with and without self-consistency. Using a benchmark set of 51 chemistry papers each annotated with a set of research background, a coarse-grained hypothesis, and a fine-grained hypothesis, the authors demonstrated that 1) HSS discovers superior local optima compared to the baseline methods in terms of effectiveness, novelty, detailedness and feasibility, 2) HSS-judged optimum is better aligned with the ground-truth hypotheses, and 3) using multiple instances of a strong single model is more effective at hypothesis optimization than combining different models.", "questions": "1. Section 2.1 - The phrase “redundant or tangential elements” is somewhat vague. Could the authors provide specific examples of these elements in the context of hypothesis generation?\n\n\n2. Did the authors perform any quantitative analysis of the edits (e.g., number of insertions, deletions, or rewrites) across the hierarchy? This could strengthen their claims about the structured nature of the hierarchical heuristic search and its role in refining hypotheses.\n\n\n3. Section 2.5 - How are the hierarchical levels (p) and the number of hierarchical levels defined across different research topics or domains? Moreover, how are these levels conveyed to the LLM-driven agentic process? \n\n\n4. Section 3.2 - The paper mentions LLM-based dimension-specific evaluations. a) What scoring system is used for these assessments? b) How are these individual scores aggregated into an “overall” (LLM) evaluation? \n\n\n5. Section 3.2 - The authors claim to observe two trade-offs. How were these conclusions drawn? Could this be better described as a correlation, rather than a causal or generalizable trade-off? Similar trade-offs could arguably exist between novelty and feasibility, especially when feasibility depends on methodological availability.\n\n\n6. Section 3.3 - It is unclear whether the LLM-based evaluation of hypothesis alignment is simply matching keywords or phrases from the ground truth. If so, how do the authors ensure these matches are not out of context or semantically misleading (e.g., terms that appear similar but differ in meaning)? Concrete examples of both high and low scoring matches would help assess whether the 0–3 scale captures meaningful semantic overlap.\n\n\n7. Table 3 - There appear to be a high % of ties.  It would be helpful to explicitly discuss the implications of these ties on the strength of the conclusions drawn. I would imagine that the high tie rate might suggest that differences between committees are less pronounced than the win/loss counts alone indicate.\n\n\n8. Table 3 -  It has a confusing structure, as the comparison order is reversed between the left and right columns. For example, “Gemini 1.5 Flash committee vs. GPT-4o-mini committee” is in the left column, while the reverse appears in the right. Similarly, the row order for matchups is inconsistent (e.g., “Mixed committee vs. GPT-4o-mini” appears at the top on the left, but bottom on the right). Reformatting the table for consistent comparison direction and row order would improve clarity.", "limitations": "The authors acknowledge that the hierarchical heuristic search (HHS) may not converge to the global optimum. However, it would strengthen the work to quantify how close HHS comes to the ground-truth hypotheses, assuming those can be treated as a proxy for the global optimum. Could the authors elaborate on whether Table 2 serves this purpose, i.e., does the recall of ground-truth components by HHS, greedy search, and self-consistent search reflect their respective distances from the global optimum?  If so, it would be helpful to explicitly frame Table 2 as a measure of this “distance to optimum” and discuss how often HHS actually recovers (or nearly recovers) the full ground-truth hypothesis. If not, are there alternative ways the authors could estimate or upper-bound the distance between generated hypotheses and the ground truth?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The work highlights and studies a timely and important challenge on how to leverage LLMs to generate experimentally actionable scientific hypotheses, which could significantly enhance the practical utility of LLMs in scientific discovery.  \n\n\n2. The proposed hierarchical heuristics search approach is conceptually sound, providing a logical framework for navigating the hypothesis space effectively. \n\n\nWeaknesses: \nThe specific questions/comments I have will be elaborated in the ‘Questions’ section but in general, the main weaknesses are: \n\n\n1. Evaluation metrics lack clarity - the definitions and evaluation methodology for the evaluation metrics are insufficiently explained, which limit interpretability of the results. \n\n\n2. Methodological details should provide clearer detail on how HSS is implemented. \n\n\n3. The conclusions drawn from the experiments for each research question would benefit from deeper analysis and clearer justification.", "was_revised": true, "final_justification": "I am unable to raise my score without further access to more concrete supporting evidence and will keep my current score", "rebuttal": "Thank you for your insightful inquiries. In the following sections, we've structured our responses to each of your points raised.\n\n---\n\n**Q1**: Evaluation metrics & methodology can benefit from more details.  \n**A1**: We thank the reviewer for this suggestion. We will add more details on evaluation metrics and methodology in the next version, as the submitted paper's dense content limited space in the main text. All relevant details, including prompts, will be updated. (Given the rebuttal space constraints, we cannot include them here.)\n\n**Q2**: The conclusions drawn from the experiments for each research question would benefit from deeper analysis.  \n**A2**: Appendix B has four case studies including detailed expert's analysis (we will mention it in the main body of the paper). In addition to it, we conduct two more analyses to better understand the experiment results.\n\n**Analysis 1:**\n\nError analysis on the proposed method, HHS. Two chemistry PhD candidates conducted it, when one analyzed 30 hypotheses in terms of 30 research questions, the other analyzed another 21 hypotheses in terms of another 21 research questions. The results are as below:\n\n| HHS Error Analysis                                     | Count |\n|--------------------------------------------------------|-------|\n| Missing key chemical substances                        | 14/30 |\n| Excessive details in characterization methods          | 28/30 |\n| Feasibility issues                                     | 18/30 |\n| Limitations of characterization methods                | 08/30 |\n| Insufficient basis for material selection              | 22/30 |\n| Lack of design comparison experiments                  | 12/30 |\n| Ignoring data validation and reproducibility           | 10/30 |\n| Severe deviation from feasibility                      |  8/21 |\n| Missing or incorrect key chemicals or reaction systems |  9/21 |\n| Incorrect explanation of chemical principles           | 12/21 |\n| Incorrect prediction of experimental system            | 10/21 |\n\n**Analysis 2:**\n\nError analysis on why the baselines are weaker than HHS. Two chemistry PhD candidates conducted it: one analyzed 30 hypotheses for 30 research questions, the other analyzed 21 hypotheses for 21 research questions. The results are as follows:\n\n| Why baseline is weaker than HHS            | Count |\n|--------------------------------------------|-------|\n| Insufficient performance metrics           | 25/30 |\n| Complexity of experimental conditions      | 16/30 |\n| Insufficient explanation details           | 29/30 |\n| Inadequate description of preparation plan | 22/30 |\n| Vague research objectives                  | 28/30 |\n| Cost and scalability issues                | 13/30 |\n| Poor feasibility                           | 12/21 |\n| Errors in research plan details            | 21/21 |\n| Insufficient explanation details           | 19/21 |\n| Clear experimental system                  | 21/21 |\n\n\n\n**Q3**: What are “redundant or tangential elements”.\n\n**A3**: In this context, \"redundant elements\" refer to additional chemistry concepts that, while potentially relevant in some specific cases, are unnecessary for the hypothesis. For example, if the groundtruth hypothesis requires concepts [A, B, C, D, E], and the hypothesis includes [A, B, C, D, E, F, G, H, I], the concepts [F, G, H, I] would be considered redundant. \"Tangential elements\" are a subset of redundant elements, specifically referring to those that are irrelevant to the research question.\n\n**Q4**: Quantitative analysis of the edits (e.g., number of insertions, deletions, or rewrites).\n\n**A4**: We performed an additional analysis, revealing that the average number of optimization steps for HHS is 282. Of these, 113 steps involve insertions, 82 steps involve deletions, and 152 steps involve rewrites. Overall the three edits are evenly distributed across the hierarchies.\n\n\n**Q5**: How are the hierarchical levels (p) and the number of hierarchical levels defined across different research topics or domains? Moreover, how are these levels conveyed to the LLM-driven agentic process?\n\n**A5**: The chemistry hierarchy is designed by a chemistry phd candidate in 2 hours, mostly by their understanding of chemistry research. Different discipline’s hierarchy can be designed by their domain experts. \n\nThese levels are explicitly defined and explained in the prompt. The code is open-sourced. Due to the space limit, please understand that we can’t list the prompt here.\n\n**Q6**: The paper mentions LLM-based dimension-specific evaluations. a) What scoring system is used for these assessments? b) How are these individual scores aggregated into an “overall” (LLM) evaluation?\n\n\n**A6**: The evaluation is based on pairwise comparisons conducted by LLMs. The \"overall\" evaluation has a separate prompt that includes all four dimensions, and the results for the overall evaluation are obtained independently from the individual dimension scores.\n\n**Q7**: The authors claim to observe two trade-offs. How were these conclusions drawn?\n\n**A7**: The conclusions are primarily based on the definitions of the four dimensions in the prompt. Effectiveness is supported by similar principles or methods, but this often sacrifices novelty. Feasibility (lines 255–258) is inversely related to implementation complexity, where adding details to enhance thoroughness inevitably increases complexity by introducing more elements and concepts into the hypothesis.\n\n**Q8**: It is unclear whether the LLM-based evaluation of hypothesis alignment is simply matching keywords or phrases from the ground truth. If so, how do the authors ensure these matches are not out of context or semantically misleading (e.g., terms that appear similar but differ in meaning)? \n\n**A8**: The full hypotheses are provided during the comparison to ensure complete context, and the evaluation prompt explicitly emphasizes this requirement. Each generated hypothesis is approximately 600 words and requires domain-specific chemistry knowledge for proper understanding. While providing specific examples here may not be feasible, we extensively consulted with chemistry experts during the development of the evaluation framework to ensure its quality. A significant portion of our research efforts was dedicated to developing a robust evaluation framework.\n\n**Q9**: Table 3 - There appears to be a high % of ties. It would be helpful to explicitly discuss the implications of these ties on the strength of the conclusions drawn. I would imagine that the high tie rate might suggest that differences between committees are less pronounced than the win/loss counts alone indicate.\n\n**A9**: Thank you for your insightful observation. The conclusion focuses on determining which committee is better, and the tie rate does not affect the qualitative outcome. The conclusion is supported by multiple pairwise committee comparisons, many of which show clear win/loss discrepancies. We include the tie results to provide a more comprehensive understanding of the comparison process.\n\n**Q10**: Table 3 - It has a confusing structure, as the comparison order is reversed between the left and right columns. Reformatting the table for consistent comparison direction and row order would improve clarity.\n\n**A10**: Thank you for your careful review and helpful suggestion. We will reformat Table 3 for improved clarity and consistency.\n\n\n**Q11**: Could the authors elaborate on whether Table 2 serves this purpose, i.e., does the recall of ground-truth components by HHS, greedy search, and self-consistent search reflect their respective distances from the global optimum? If so, it would be helpful to explicitly frame Table 2 as a measure of this “distance to optimum” and discuss how often HHS actually recovers (or nearly recovers) the full ground-truth hypothesis.\n\n**A11**: Thank you for the insightful suggestion. Yes, the recall of ground-truth components reflects the respective distances from the global optimum. We will explicitly frame Table 2 as a measure of \"distance to optimum\" and discuss how often HHS recovers (or nearly recovers) the full ground-truth hypothesis in the paper.\n\n\n---\n*We greatly appreciate your comprehensive feedback. We hope that our responses have satisfactorily addressed all your queries. Should you have further questions or suggestions for enhancing our manuscript, we warmly welcome your input.*"}, {"review_id": "6FC7Zr9fab", "reviewer": "Reviewer_Rz5y", "summary": "This paper focuses on using large language models (LLMs) to solve the problem of fine-grained scientific hypothesis discovery. Addressing the shortcomings of existing methods, which generate rough hypotheses and lack experimental details, it formalizes this task as a combinatorial optimization problem for the first time. The study proposes a hierarchical heuristic search (HHS) framework, which refines hypotheses step by step from reaction mechanisms to complete experimental configurations, and utilizes internal heuristics of LLMs for candidate editing and quality assessment. The results show that HHS outperforms significantly in LLM self-assessment, expert scoring, and recall rate of real hypotheses, and that a single strong model ensemble performs better than a diverse model ensemble.", "questions": "The paper mentions using the internal heuristics of LLM as a reward signal to guide hypothesis search. Then, what is the specific form of the reward function? Is it defined solely through pairwise comparisons by LLM? During the optimization process, how can the stability and consistency of the reward function be ensured? Especially in different levels of search, will there be conflicts or deviations in the reward signals? For instance, the rewards at a higher level may favor the innovativeness of concepts, while those at a lower level focus more on the feasibility of experiments. How can the differences in rewards between these levels be balanced?\n\nThe paper defines the discovery of fine-grained hypotheses as a combinatorial optimization problem, with the complexity of the search space being C{^m}_n. When m and n are large, the computational complexity will increase exponentially. Although the hierarchical search framework alleviates this problem by restricting the search space at each level, in practical applications, how to determine the optimal search range at each level? When facing very complex hypotheses (such as those involving a large number of components and parameters), can HHS effectively avoid getting stuck in local optima? Are there theoretical analyses or experimental data to support the efficiency and effectiveness of HHS in high-dimensional search spaces?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths :\nFormalizing \"fine-grained scientific hypothesis discovery\" as a combinatorial optimization problem, a hierarchical heuristic search (HHS) framework is proposed. It gradually refines hypotheses through a hierarchical structure (e.g., a five-level decomposition in the chemical field from reaction mechanisms to experimental configurations), effectively reducing the complexity of the search space. This hierarchical design reduces the traps of local optima by \"smoothing the reward landscape\", theoretically enhancing the stability of optimization.\n\nWeaknesses:\nThe definition of the fine-grained assumption relies on the annotation by domain experts, which may have subjective bias, and the benchmark scale is relatively small (based on 51 chemistry papers), so the robustness needs to be verified with larger-scale data.\n\nThe HHS framework cannot guarantee finding the global optimal solution (as admitted in the manuscript), and the hierarchical division relies on domain knowledge (such as the five-level structure in chemistry), which may carry the risk of over-reliance on prior assumptions. If the domain structure is complex or difficult to stratify, the method's effectiveness may decline.", "was_revised": true, "final_justification": "Thanks for the detailed reply. Most of my concerns are addressed", "rebuttal": "Thank you for your insightful inquiries. In the following sections, we've structured our responses to each of your points raised.\n\n---\n\n**Q1**: The definition of the fine-grained hypothesis relies on the annotation by domain experts, which may have subjective bias, and the benchmark scale is relatively small (based on 51 chemistry papers), so the robustness needs to be verified with larger-scale data.\n\n**A1**: We appreciate the reviewer’s concern. The hypotheses were annotated by chemistry PhD students, who are among the highest-quality annotators in the field.\n\nScientific discovery is different from the traditional reasoning task that one discovered high-quality hypothesis could lead to significant effect. Also the data is harder to annotate. Previous efforts all have only a few dozen annotation data (purely as test data). For example, [1] [2] annotates ~50 data, [3] done by OpenAI only annotates only 20 data.\n\n[1] Large Language Models for Automated Open-domain Scientific Hypotheses Discovery, ACL 2024\n\n[2] MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses, ICLR 2025\n\n[3] PaperBench: Evaluating AI's Ability to Replicate AI Research, OpenAI\n\n**Q2**: The HHS framework cannot guarantee finding the global optimal solution (as admitted in the manuscript), and the hierarchical division relies on domain knowledge (such as the five-level structure in chemistry), which may carry the risk of over-reliance on prior assumptions. If the domain structure is complex or difficult to stratify, the method's effectiveness may decline.\n\n**A2**: HHS can be seen as an optimization method. Currently none of the optimization methods is guaranteed to find the global optimum on non-convex problems. \n\nThe chemistry hierarchy is designed by a chemistry phd candidate in 2 hours, and it works quite well. The hierarchy is about very general descriptions and therefore should be relatively quite robust. \n\n**Q3**: The paper mentions using the internal heuristics of LLM as a reward signal to guide hypothesis search. Then, what is the specific form of the reward function? Is it defined solely through pairwise comparisons by LLM? During the optimization process, how can the stability and consistency of the reward function be ensured? Especially in different levels of search, will there be conflicts or deviations in the reward signals? For instance, the rewards at a higher level may favor the innovativeness of concepts, while those at a lower level focus more on the feasibility of experiments. How can the differences in rewards between these levels be balanced?\n\n**A3**: The reward function is solely defined through pairwise comparisons by LLM. The pairwise comparison prompt is consistent in terms of its main body, where in different hierarchies, different specific subprompt is adopted, stressing what that hierarchy focuses on. Each hierarchy is developed with one specific subprompt. \n\n**Q4**: The paper defines the discovery of fine-grained hypotheses as a combinatorial optimization problem, with the complexity of the search space being C{^m}_n. When m and n are large, the computational complexity will increase exponentially. Although the hierarchical search framework alleviates this problem by restricting the search space at each level, in practical applications, how to determine the optimal search range at each level? When facing very complex hypotheses (such as those involving a large number of components and parameters), can HHS effectively avoid getting stuck in local optima? Are there theoretical analyses or experimental data to support the efficiency and effectiveness of HHS in high-dimensional search spaces?\n\n**A4**: In line 122 we analyze that one of the challenges for the fine-grained hypothesis discovery task is that m and n are all unknown. We deal with this challenge by adopting heuristic search in each hierarchy instead of precise search over the full space, which can avoid the need to know the full search space. \n\nThe fine-grained hypothesis discovery task itself has high-dimensional search spaces, and our experiments are all conducted in it. We show that theoretically, HHS can find better local optima than the baselines, and empirically HHS does outperforms baselines. \nThis work introduces the fine-grained hypothesis discovery task, and we believe that an ultimate solution for this challenge has yet to be established. We encourage the research community to explore these intriguing questions further.\n\n---\n\n*We greatly appreciate your comprehensive feedback. We hope that our responses have satisfactorily addressed all your queries. Should you have further questions or suggestions for enhancing our manuscript, we warmly welcome your input.*"}, {"review_id": "L8vnKCYG8h", "reviewer": "Reviewer_j442", "summary": "This paper introduces and formalizes the novel task of fine-grained scientific hypothesis discovery, which aims to generate detailed, experimentally actionable hypotheses from coarse-grained research directions. The authors frame this task as a combinatorial optimization problem over a vast, implicit space of possible experimental details. To tackle this, they propose a Hierarchical Heuristic Search (HHS) framework. This method decomposes the search problem into a series of smaller, more manageable subproblems arranged in a hierarchy of conceptual abstraction (e.g., from high-level reaction mechanisms down to specific parametric configurations in chemistry). The core intuition is that this hierarchical structure smooths the reward landscape, enabling an LLM-driven agent to more effectively navigate the search space and avoid poor local optima.", "questions": "Could you elaborate on the process of designing the 5-level chemistry hierarchy in Figure 1? How much expert effort was involved? More importantly, how sensitive is the performance of HHS to the specific design of this hierarchy? Can you easily extend the design to other scientific domain? For instance, what would happen if you collapsed levels 2 and 3, or used a different number of levels? This would help clarify the robustness and practical requirements of the method.\n\nCould you provide an analysis of the computational cost (e.g., approximate number of LLM calls or total tokens processed per problem instance) for HHS compared to the baselines?\n\nThe results show HHS consistently outperforming baselines. Were there any specific types of problems or hypotheses where HHS struggled or failed to produce a superior result? An error analysis could provide valuable insights into the limitations of the current framework.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper's primary contribution is the identification and formalization of \"fine-grained scientific hypothesis discovery.\" This moves the field beyond generating high-level, often vague ideas towards producing concrete, testable, and actionable scientific plans.\n\nThe proposed Hierarchical Heuristic Search is well-motivated and intuitive. The analogy to coarse-to-fine refinement strategies is apt, and Figure 1 provides a concrete instantiation of this idea for the chemistry domain.\n\nThe paper is structured around three clear research questions (Q1-Q3), and the experiments are designed to answer them directly. The investigation into ensemble composition (Q3) is particularly insightful and offers practical guidance for future work.\n\n\nWeaknesses:\n\nThe HHS method, with its iterative search within each level, multiple independent runs for recombination, and progression through several hierarchical levels, appears to be computationally expensive in terms of LLM API calls and tokens. \n\nThe paper title aims at scientific discovery, but the paper does not discuss how this approach would generalize to other scientific domains (e.g., biology, materials science, physics) beyond chemistry. \n\nThe \"smoothing of the reward landscape\" is the central theoretical claim motivating HHS. However, this claim is supported only by illustrative schematics (Figures 3 & 4) and the downstream performance of the system. While the results are strong, they are indirect evidence. The paper would be significantly strengthened by a more direct, quantitative analysis attempting to measure this smoothing effect.", "was_revised": true, "final_justification": "NA", "rebuttal": "We appreciate your insightful questions and helpful suggestions. For clarity, we have organized your queries along with our responses below:\n\n---\n\n**Q1**: The HHS method, with its iterative search within each level, multiple independent runs for recombination, and progression through several hierarchical levels, appears to be computationally expensive in terms of LLM API calls and tokens. Could you provide an analysis of the computational cost (e.g., approximate number of LLM calls or total tokens processed per problem instance) for HHS compared to the baselines?\n\n**A1**: Thank you for the question. Below is an analysis of the average reasoning steps for each method. The HHS method, while more computationally intensive, uses computation as a trade-off for improved performance. This trade-off is reasonable, as evidenced by OpenAI’s Dan Roberts, who discussed in his talk titled “9 Years to AGI? OpenAI’s Dan Roberts Reasons About Emulating Einstein” that an ideal scientific discovery system might require 8 years of inference-time computation to propose a high-quality hypothesis, emulating Einstein's process.\n\n| Method                    | Soft Recall | Hard Recall | Reasoning Steps |\n|---------------------------|-------------|-------------|-----------------|\n| Greedy                    |      16.60% |       9.92% |      9.68627451 |\n| Greedy + Self-Consistency |      31.53% |      17.73% |     67.54901961 |\n| HHS                       |      40.35% |      23.04% |     282.0392157 |\n\n**Q2**: The paper title aims at scientific discovery, but the paper does not discuss how this approach would generalize to other scientific domains (e.g., biology, materials science, physics) beyond chemistry. Could you elaborate on the process of designing the 5-level chemistry hierarchy in Figure 1? How much expert effort was involved? More importantly, how sensitive is the performance of HHS to the specific design of this hierarchy? Can you easily extend the design to other scientific domain? For instance, what would happen if you collapsed levels 2 and 3, or used a different number of levels? This would help clarify the robustness and practical requirements of the method.\n\n**A2**: Thank you for the suggestion. The method is largely domain-agnostic in its design, with the only domain-specific aspect being the hierarchy design.\nThe chemistry hierarchy was designed by a chemistry PhD candidate in approximately 2 hours, based on their understanding of the chemistry research process. Similar hierarchies for other scientific disciplines can be designed by domain experts. We will discuss this further in the paper.\nTo evaluate the robustness of the hierarchy design, we conducted an analysis by collapsing levels 2 and 3. The comparison results are shown below:\n\n\n| Method                   | Soft Recall | Hard Recall |\n|--------------------------|-------------|-------------|\n| HHS                      |      40.35% |      23.04% |\n| HHS (collapse hierarchy) |      37.63% |      21.79% |\n\nThe results are very similar, particularly in comparison to the baselines, demonstrating the robustness of the hierarchy design.\n\n\n**Q3**: The \"smoothing of the reward landscape\" is the central theoretical claim motivating HHS. However, this claim is supported only by illustrative schematics (Figures 3 & 4) and the downstream performance of the system. While the results are strong, they are indirect evidence. The paper would be significantly strengthened by a more direct, quantitative analysis attempting to measure this smoothing effect.\n\n**A3**: We appreciate the reviewer's feedback on the theoretical support for our \"hierarchy smooths reward\" claim. We connect fine-grained hypothesis discovery to established theories in image and signal processing, where smoothing convolution (averaging pixels in an image patch) acts as a low-pass filter.\nThe sufficient condition for this claim is our assumption that \"LLM’s evaluation for a hypothesis with a general concept can be viewed as an aggregated estimate--approximating an average or soft maximum--of LLM’s evaluation for hypotheses with specific concepts under that general concept\" (lines 197–199).\nWhile this assumption is commonsensically true but challenging to prove formally, we conducted additional experiments to verify it: We prompted LLMs to score hypotheses with general concepts and their corresponding specific variants on a 0–100 scale, finding an average difference of only 3.47 between the general score and the averaged specific scores (in total 204 pairs compared).\n\n\n\n\n**Q4**: The results show HHS consistently outperforming baselines. Were there any specific types of problems or hypotheses where HHS struggled or failed to produce a superior result? An error analysis could provide valuable insights into the limitations of the current framework.\n\n\n**A4**: Appendix B has four case studies including detailed expert's analysis (we will mention it in the main body of the paper). In addition to it, we conducted two more error analyses to better understand the experiment results.\n\n**Analysis 1:**\nError analysis on the proposed method, HHS. Two chemistry PhD candidates conducted it, when one analyzed 30 hypotheses in terms of 30 research questions, the other analyzed\n another 21 hypotheses in terms of another 21 research questions. The results are as below:\n\n| HHS Error Analysis                                     | Count |\n|--------------------------------------------------------|-------|\n| Missing key chemical substances                        | 14/30 |\n| Excessive details in characterization methods          | 28/30 |\n| Feasibility issues                                     | 18/30 |\n| Limitations of characterization methods                | 08/30 |\n| Insufficient basis for material selection              | 22/30 |\n| Lack of design comparison experiments                  | 12/30 |\n| Ignoring data validation and reproducibility           | 10/30 |\n| Severe deviation from feasibility                      |  8/21 |\n| Missing or incorrect key chemicals or reaction systems |  9/21 |\n| Incorrect explanation of chemical principles           | 12/21 |\n| Incorrect prediction of experimental system            | 10/21 |\n\n**Analysis 2:**\nError analysis on why the baselines are weaker than HHS. Two chemistry PhD candidates conducted it: one analyzed 30 hypotheses for 30 research questions, the other analyzed 21 hypotheses for 21 research questions. The results are as follows:\n\n| Why baseline is weaker than HHS            | Count |\n|--------------------------------------------|-------|\n| Insufficient performance metrics           | 25/30 |\n| Complexity of experimental conditions      | 16/30 |\n| Insufficient explanation details           | 29/30 |\n| Inadequate description of preparation plan | 22/30 |\n| Vague research objectives                  | 28/30 |\n| Cost and scalability issues                | 13/30 |\n| Poor feasibility                           | 12/21 |\n| Errors in research plan details            | 21/21 |\n| Insufficient explanation details           | 19/21 |\n| Clear experimental system                  | 21/21 |\n\n\n\n**Q5**: Table captions should be placed above the table.\n\n**A5**: Thank you for the advice! We will adjust it in the next version.\n\n---\n*We greatly appreciate your comprehensive feedback. We hope that our responses have satisfactorily addressed all your queries. Should you have further questions or suggestions for enhancing our manuscript, we warmly welcome your input.*"}, {"review_id": "3tkdjmy4Ji", "reviewer": "Reviewer_SXMM", "summary": "This study reports a method for scientific hypothesis generation that refines hypotheses in stages, from abstract to concrete, thereby enabling the generation of hypotheses that go beyond coarse-grained ones to include methodological and experimental details. The authors empirically validated that this hypothesis generation approach can produce better hypotheses and more closely align with ground-truth hypotheses. Furthermore, they found that the iterative application of a powerful LLM yields better results than using an ensemble of diverse LLMs.", "questions": "Am I correct in understanding that the core contribution of this paper lies in the design of prompts for the LLM and the method of combining its inputs and outputs? Furthermore, is the notion of “combinatorial search” intended as a conceptual framing rather than a literal explicit implementation of search space and search algorithm but rather leveraging prompt-based control of the LLM?", "limitations": "The experiments could be made better by incorporating stronger benchmarks, more diverse datasets, and a larger number of annotators, and more.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The use of large language models (LLMs) for hypothesis generation in scientific research represents a highly significant and timely challenge. While substantial breakthroughs are expected in the near future, there remain numerous unresolved issues at present. As the authors of the paper under discussion rightly point out, one such issue is that naively prompting LLMs to generate hypotheses often results in overly abstract ideas. Bridging the gap between such abstract hypotheses and their concrete implementation remains an open research problem. The authors’ attempt to address this critical issue of “hypothesis concretization” is a commendable aspect of the work.\n\nThroughout the paper, the authors strive to enhance the reader’s understanding by associating key concepts with mathematical formulations and by employing various analogies. Their approach of interpreting LLM-based generation as a process akin to search and building their methodology around this analogy is both novel and intellectually stimulating.\n\nIn terms of evaluation, the authors go beyond relying solely on automated LLM assessment by incorporating human evaluations, which strengthens the credibility of their findings. Moreover, their focus on evaluating hypotheses in terms of their “detailedness” is an insightful and original perspective that adds value to the study. Finally, the proposed method consistently outperforms baseline approaches, demonstrating its practical effectiveness and further suggesting the potential significance of this contribution.\n\nWhile this is an excellent piece of work that contributes meaningfully to the field of AI-driven hypothesis generation, I believe that further improvements are necessary for it to meet the high standards required for acceptance at NeurIPS. Below, I outline the key reasons for this assessment.\n\nIf my understanding is correct, what this paper essentially proposes is the following:\n\t1.\tIteratively generate hypotheses while evaluating their quality using LLMs as judges,\n\t2.\tRepeat this process multiple times and treat the combined outputs as the hypothesis at a given level of abstraction,\n\t3.\tUse the high-level hypotheses obtained so far to move on to generate more concrete hypotheses (i.e., return to step 1).\n\nTo my knowledge, iteratively improving LLM outputs, generating a final output from multiple candidates, and progressively refining outputs into more concrete forms are not particularly novel ideas in and of themselves. In that sense, I felt that the method proposed in this study may not be sufficiently innovative or unique to be considered a significant advancement. The annotation of fine-grained hypotheses added to the dataset is indeed a contribution, but since it is based on an existing dataset, it is not an entirely new dataset.\n\nThe authors do not propose a new training method or architecture for LLMs but rather use existing models via APIs. Therefore, one would generally expect a carefully designed method for combining these models or crafting prompts; however, I found these aspects to be somewhat lacking. For these reasons, I believe there is still room for improvement in terms of originality.\n\nThis paper reports that the proposed method consistently outperforms baseline approaches, which suggests that the direction of the method could be practically valuable. However, the baselines used for comparison are very basic—essentially just self-consistency. Numerous approaches to hypothesis generation and prompting LLMs have been proposed previously, and it is reasonable to assume that many outperform simple self-consistency. Therefore, it was difficult to assess how well this method compares to more sophisticated alternatives.\n\nOf course, the comparisons with the chosen baselines are appropriate for validating the hypothesis, but to more convincingly demonstrate the utility and value of this approach to the AI community, I believe it would be beneficial to include comparisons with stronger baselines.\n\nRegarding the experimental design, it is important to evaluate alignment with ground truth, but in this case, such evaluation was conducted using LLMs as judges. It was unclear what specific instructions were given to the LLM judges or how effective they were—for example, how well their evaluations aligned with human judgments.\n\nHuman evaluations were conducted for the baseline comparisons, which I find desirable. However, the results only reported whether the proposed method outperformed existing methods or not. It was not clear how desirable the method’s properties were on their own, independent of its relative performance. While itemized evaluations are provided, only the overall human evaluation scores were reported. As a result, we cannot tell how each specific evaluation criterion was judged by human researchers. Therefore, I was not fully convinced of the overall persuasiveness of the experimental claims.\n\nFinally, while the use of metaphors and concept diagrams throughout the paper made the authors’ ideas more understandable, I felt that these descriptions were somewhat excessive in light of the need for more detailed explanations of the methodology, results, and additional analyses. For example, while the authors carefully explain that the process can be interpreted as combinatorial search, the actual proposal is just a particular way of prompting and combining outputs from an LLM—not a method involving any explicit combinatorial search technique. If that’s the case, I believe the paper would benefit more from devoting space to explaining the methodological details more concretely, as this would better convey the specifics of the research to readers.", "was_revised": true, "final_justification": "Considering the newly added comparisons with particularly strong baselines, I would like to raise my score. That said, I believe there is still room for improvement in the technical novelty and contributions presented in this paper, and therefore I will maintain my overall negative stance toward acceptance.", "rebuttal": "We are grateful for the detailed comments. Here, we present our responses corresponding to the questions you've asked.\n\n---\n\n**Q1**: what this paper essentially proposes, and what is the core contribution of this paper. \n\n**A1**: The core contribution of this paper is the formalization of the novel fine-grained hypothesis discovery task as a combinatorial optimization problem, along with the development of Hierarchical Heuristic Search (HHS) to optimize this task effectively for scientific discovery. To our knowledge, no prior LLM-based scientific discovery work has framed the task through an optimization lens. Furthermore, HHS adapts coarse-to-fine search methods to the challenging, language-based open-ended tasks like hypothesis generation, an approach that was previously unclear.\n\nSpecifically, we propose a novel framing of the task that enables the use of LLMs to provide the gradient for optimization. To enhance optimization, we introduce a hierarchical design that smooths the reward landscape, facilitating convergence to better local optima. Additionally, we adopt evolutionary search to interpolate among multiple local optima, further refining the optimization process.\n\n\n\n**Q2**: The authors do not propose a new training method or architecture for LLMs but rather use existing models via APIs.\n\n**A2**: The proposed method is essentially a test-time scaling approach, where computation is used as a trade-off for improved performance. This represents a cutting-edge research direction. For example, OpenAI’s Dan Roberts discussed in his talk “9 Years to AGI? OpenAI’s Dan Roberts Reasons About Emulating Einstein” that an ideal scientific discovery system might require 8 years of inference-time computation to propose a high-quality hypothesis, emulating Einstein's process in developing the theory of relativity.\nThe key challenge is designing a scaling method that effectively leverages large amounts of compute (e.g., inference-time or API usage) to improve performance. While we currently have access to significant compute resources, the question remains how to best utilize them for generating high-quality research hypotheses. Our method makes a meaningful step in this direction.\n\n\n**Q3**: Although the proposed method consistently outperforms baseline approaches, the baselines used for comparison are very basic—essentially just self-consistency. Better to compare with previous proposed methods.\n\n**A3**: Thank you for the suggestion to compare with additional baselines. While previous works have addressed hypothesis generation, they do not specifically focus on fine-grained hypothesis discovery. We initially chose not to include them as baselines because they would not offer comparable performance in terms of providing detailed hypotheses.\nWe have since added experiments with five strong baselines for comparison, with the newly added baselines highlighted in bold.\n\n\n|                            | Soft Recall | Hard Recall |\n|----------------------------|-------------|-------------|\n| **ChemCrow[1]**               |      12.28% |       7.20% |\n| **Reinforcement Learning** |      16.76% |      10.28% |\n| **[3]**                     |      19.57% |      11.15% |\n| **MOOSE[2]**                  |      20.04% |      11.76% |\n| **MOOSE-Chem[4]**             |      19.99% |      11.98% |\n| Greedy                     |      16.60% |       9.92% |\n| Greedy + Self-Consistency                         |      31.53% |      17.73% |\n| HHS                         |      40.35% |      23.04% |\n\n[1] ChemCrow: Augmenting large-language models with chemistry tools, Nature 2023\n\n[2] Large Language Models for Automated Open-domain Scientific Hypotheses Discovery, ACL 2024\n\n[3] Large language models are zero shot hypothesis proposers, COLM 2024\n\n[4] MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses, ICLR 2025\n\n\n**Q4**: Regarding the experimental design, it is important to evaluate alignment with ground truth, but in this case, such evaluation was conducted using LLMs as judges. It was unclear what specific instructions were given to the LLM judges or how effective they were—for example, how well their evaluations aligned with human judgments.\n\n**A4**: Thank you for your comment. Due to word limitations, we are unable to provide the full prompts here, but they are open-sourced and can be found in the evaluation_instruction_prompts() function in ./Method/utils.py.\n\nWe worked closely with chemistry experts during the development of the evaluation framework to ensure its quality. A significant portion of our research was dedicated to developing a robust evaluation process.\n\nAdditionally, we performed an analysis to assess the consistency between LLM judges and human expert evaluations. Specifically, 100 hypotheses were sampled and evaluated by two human experts (each reviewing 50). The experts were satisfied with 97 out of 100 evaluations.\n\n\n\n**Q5**: Human evaluation results are only reported whether the proposed method outperformed existing methods or not. It was not clear how desirable the method’s properties were on their own, independent of its relative performance.\n\n**A5**: Thank you for the suggestion. Table 2 in the paper presents the recall for each method, which reflects absolute performance rather than relative performance.\n\nAdditionally, we conducted further error analyses performed by human experts to gain a deeper understanding of the experimental results using very strict criteria to understand the absolute performance of HHS.\n\n\n**Analysis**: \nError analysis on the proposed method, HHS. Two chemistry PhD candidates conducted it, when one analyzed 30 hypotheses in terms of 30 research questions, the other analyzed another 21 hypotheses in terms of another 21 research questions. The results are as below:\n\n| HHS Error Analysis                                     | Count |\n|--------------------------------------------------------|-------|\n| Missing key chemical substances                        | 14/30 |\n| Excessive details in characterization methods          | 28/30 |\n| Feasibility issues                                     | 18/30 |\n| Limitations of characterization methods                | 08/30 |\n| Insufficient basis for material selection              | 22/30 |\n| Lack of design comparison experiments                  | 12/30 |\n| Ignoring data validation and reproducibility           | 10/30 |\n| Severe deviation from feasibility                      |  8/21 |\n| Missing or incorrect key chemicals or reaction systems |  9/21 |\n| Incorrect explanation of chemical principles           | 12/21 |\n| Incorrect prediction of experimental system            | 10/21 |\n\n\n**Q6**: While itemized evaluations are provided, only the overall human evaluation scores were reported. As a result, we cannot tell how each specific evaluation criterion was judged by human researchers.\n\n**A6**: Thank you for the suggestion. We have conducted an additional analysis by human researchers for each specific evaluation criterion. The results are provided below:\n\n|                                                       | Effectiveness | Novelty | Detailedness | Feasibility |\n|-------------------------------------------------------|---------------|---------|--------------|-------------|\n|      HHS v.s. Greedy   Search  + Self-consitency      |               |         |              |             |\n| win                                                   | 52.38%        | 47.62%  | 52.38%       | 52.38%      |\n| Tie                                                   | 42.86%        | 42.86%  | 47.62%       | 42.86%      |\n| Lose                                                  | 4.76%         | 9.52%   | 0.00%        | 4.76%       |\n|                HHS v.s. Greedy   Search               |               |         |              |             |\n| win                                                   | 61.90%        | 61.90%  | 66.67%       | 61.90%      |\n| Tie                                                   | 38.10%        | 38.10%  | 33.33%       | 38.10%      |\n| Lose                                                  | 0.00%         | 0.00%   | 0.00%        | 0.00%       |\n| Greedy Search +   Self-consitency  v.s. Greedy Search |               |         |              |             |\n| win                                                   | 47.62%        | 42.86%  | 47.62%       | 47.62%      |\n| Tie                                                   | 42.86%        | 42.86%  | 42.86%       | 42.86%      |\n| Lose                                                  | 9.52%         | 14.29%  | 9.52%        | 9.52%       |\n\n\n**Q7**: While the authors carefully explain that the process can be interpreted as combinatorial search, the actual proposal is just a particular way of prompting and combining outputs from an LLM—not a method involving any explicit combinatorial search technique. \n\n**A7**: In line 122 we analyze that one of the challenges for the fine-grained hypothesis discovery task is that the full search space is unknown. We deal with this challenge by adopting heuristic search in each hierarchy instead of precise search over the full space, which can avoid the need to know the full search space. We use LLMs for this heuristic search, which is prompting looks from the outside. \n\nOut of the classic combinatorial search method, MCTS and genetic algorithms require a clearly defined search space, making them not suitable for this task. Reinforcement learning, however, does not require a defined search space, and we have implemented it as an additional baseline shown in the Table of A3.\n\n----\n\n*We greatly appreciate your comprehensive feedback. We hope that our responses have satisfactorily addressed all your queries. Should you have further questions or suggestions for enhancing our manuscript, we warmly welcome your input.*"}]

Author rebuttal:

Meta-review: This paper introduces a methodology for using LLMs to propose and refine scientific hypotheses in a hierarchical manner, across multiple layers of abstraction. The paper articulates the problem clearly and the motivation and description of the method is clear. The general idea of iterative LLM-driven coarse-to-fine refinement in a hypothesis makes sense and the authors investigate the idea on a limited but interesting chemistry problem. The results are convincingly superior to a number of baselines. During the author/reviewer discussion, the overall technique was clarified and in particular new baselines were introduced that substantially improved the experimental soundness of the paper.
Decision: Accept (poster)