MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses
Authors: Zonglin Yang
Wanhao Liu
Ben Gao
Tong Xie
Yuqiang Li
Wanli Ouyang
Soujanya Poria
Erik Cambria
Dongzhan Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: scientific discovery
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Scientific discovery contributes largely to the prosperity of human society, and recent progress shows that LLMs could potentially catalyst the process. However, it is still unclear whether LLMs can discover novel and valid hypotheses in chemistry. In this work, we investigate this main research question: whether LLMs can automatically discover novel and valid chemistry research hypotheses, given only a research question? With extensive discussions with chemistry experts, we adopt the assumption that a majority of chemistry hypotheses can be resulted from a research background question and several inspirations. With this key insight, we break the main question into three smaller fundamental questions. In brief, they are: (1) given a background question, whether LLMs can retrieve good inspirations; (2) with background and inspirations, whether LLMs can lead to hypothesis; and (3) whether LLMs can identify good hypotheses to rank them higher. To investigate these questions, we construct a benchmark consisting of 51 chemistry papers published in Nature or a similar level in 2024 (all papers are only available online since 2024). Every paper is divided by chemistry PhD students into three components: background, inspirations, and hypothesis. The goal is to rediscover the hypothesis given only the background and a large chemistry literature corpus consisting the ground truth inspiration papers, with LLMs trained with data up to 2023. We also develop an LLM-based multi-agent framework that leverages the assumption, consisting of three stages reflecting the more smaller questions. The proposed method can rediscover many hypotheses with very high similarity with the ground truth ones, covering the main innovations.

Published as a conference paper at ICLR 2025
MOOSE-CHEM: LARGE LANGUAGE MODELS FOR REDISCOVERING UNSEEN CHEMISTRY SCIENTIFIC HYPOTHESES
Zonglin Yang Wanhao Liu Ben Gao Tong Xie Yuqiang Li²,
Wanli Ouyang", Soujanya Poria' Erik Cambria', Dongzhan Zhou?t
Nanyang Technological University Shanghai Artificial Intelligence Laboratory
3 University of Science and Technology of China 4 Wuhan University 5 University of New South Wales
6 GreenDynamics Singapore University of Technology and Design
{zong] in. -nnyan cambria]entu. .g, 2houddnngzhandpjjlbbbbrr
ABSTRACT
Scientific discovery plays a pivotal role in advancing human society, and recent progress in large language models (LLMs) suggests their potential accelerate this process. However, it remains unclear whether LLMs can autonomously gener- ate novel and valid hypotheses chemistry. In this work, we investigate whether LLMs can discover high-quality chemistry hypotheses given only research back- cooundd-ompriinng a question and/or survey- without restriction on the do- main of the question. We begin with the observation that hypothesis discovery is a seemingly intractable task. address this, we propose a formal mathemat- ical decomposition grounded in a fundamental assumption: that most chemistry hypotheses can be composed from a research background and a set inspira- inspira- tions, composing hypotheses with inspirations, and ranking hypotheses- together constitute sufficient set of subtasks for the overall scientific discovery task. We further develop an agentic LLM framework, MOOSE-Chem, that is a di- rect implementation of this mathematical decomposition. To evaluate this frame- work, we construct a benchmark of 51 high-impact chemistry papers published and online after January 2024, each manually annotated by PhD chemists with background, inspirations, and hypothesis. The framework is able to rediscover many hypotheses with high similarity to the groundtruth, successfully capturing the core innovations while ensuring data contamination since it uses an L.LM with knowledge cutoff date prior to 2024. Finally, based on LLM's surprisingly high accuracy on inspiration retrieval, a task with inherently out-of-distributin nature, we propose a bold assumption: that LLMs may already encode latent sci- entific knowledge associations not yet recognized by humans.
tions. This decomposition leads to three practical
INTRODUCTION
Discovering new science has long been one of the deepest desires of humanity, which can not only satisfy our curiosity to understand the universe but also contribute largely to the prosperity of human society (Coccial 2019). Recently, there are some breakthroughs indicating that LLMs have the potential to assist scientists in accelerating the discovery process (Luo et al. 2025).
Yang et al. 2024b) first find that LLMs can generate novel and valid enough hypotheses evaluated by experts. They focus on the social science domain and make discoveries by developing a multi-agent system, leveraging an assumption that a majority of social science hypotheses can be divided into research background concept and an inspiration concept. This assumption is largely valid because a social science hypothesis about how an independent variable can influence another dependent variable (Hair et al. 2007).
"Contribution during internship Shanghai Artificial Intelligence Laboratory. 'Coresppondig author.

---

Published as a conference paper at ICLR 2025
Si et al. (2024) further validate this finding by employing large group of scientists to evaluate LLMs' generated hypotheses in the NLP domain and show that LLM can generate more novel but slightly less valid research hypotheses than human researchers. However, it is still unclear LLMs' scientific discovery ability in natural science such as the chemistry domain.
Sprueill al. (2023
2024 adopt LLMs to conduct search process for catalyst discovery. However, their method is limited in the catalyst discovery domain, and their evaluation relies on whether LLMs can rediscover existing commercially used catalysts, potentially influenced by data contamination problem. As a result, it is still unclear how good LLMs are for chemistry scientific discovery.
In this paper, we investigate this central research question: Can LLMs automatically discover novel and valid chemistry research hypotheses (even at the Nature level) given only chemistry research background (consisting of research question and/or background survey), without limitation on the domain of the research question? With extensive discussions with chemistry experts, we find that the assumption used in social science, that a hypothesis can be divided into background and inspiration, can also apply to a majority of chemistry hypotheses. It is not too surprising, since cognitive science research has shown that creative ideas often result from the cohesive association of two seemingly unrelated pieces of knowledge (Koestler) 1964; [enedek et al. 2012 Lee & Chung, 2024]. A main difference is that chemistry might need more than one inspiration (c.g.. adding several components to compose novel chemistry system). With this key insight, we break the scemingly impossible-to-solve central question into three smaller, more practical, and executable fundamental questions that, when summed up, should be very close to set of sufficient conditions for the central question. Specifically, the smaller questions are (1) whether LLM can identify inspiration papers that have the potential help with the given research question; (2) given only known knowledge (from background and inspirations), whether LLMs can infer unknown knowledge that is highly likely to be valid; and (3) whether LLM can identify good hypotheses and rank them higher.
To investigate these three questions, we build a benchmark consisting of 51 chemistry papers anno- tated by chemistry PhD students, breaking every paper into a background. several inspirations. and a hypothesis. The goal is to rediscover the hypothesis with only the background by using LLMs trained with data up to December 2023. The papers are all published in Nature, Science, or a similar level in 2024, and they are only made public on the internet in 2024. The benchmark is designed to be similar to the Mathematical Olympiad Competition (Trinh et al. 2024), to provide several dozens of very difficult and meaningful questions to solve. Along with the benchmark, we propose a rank- ing task for scientific discovery (along with evaluation criteria), which has been largely overlooked in previous works (Yang et al. 2024a| Wang et al.) 2024b). Ranking is important because although Al systems can generate a large number of hypotheses in a relatively short time, verifying them one by one requires lot of experimental costs.
Motivated by this breakup into three smaller questions, we design a multi-agent framework named MOOSE-CHEM for chemistry scientific discovery. It in general includes three stages: (1) searching through chemistry literature to find inspiration papers, (2) leveraging the inspirations to propose hypotheses for the background research question, and (3) identifying high-quality hypotheses to give them higher rank. Compared with Yang et al. (2024b)ss 's method in social science that assumes a similar separation between background and inspiration for hypothesis formulation, MOOSE-CHER adopts an evolutionary algorithm to foster a broader diversity of approaches in using inspiration for background, thereby capitalizing on the benefits derived from varied mutations. In addition, MOOSE- CCHEM also adopts a multi-step design to collect more than one inspirations for chemistry discovery. Finally, it uses an efficient ranking method for better reference for scientists.
We design experiments with the benchmark to test the three fundamental questions and find that LLMs are highly capable. We also test SOOSE-CCEED with the benchmark, mimicking the setting to run it in the wild by only giving background and corpus of up to 3000 chemistry papers to select inspiration. Even in this challenging setting, MOOSE-CHEM can still rediscover many hypotheses with very high similarity with the ground truth ones, covering the main innovations.
Overall, the contributions of this paper are:
We provide the first mathematical derivation on how to decompose the seemingly impossible-to-ssovve question P(hypothesis/ and practical smaller steps. This decomposition make P(hypothesis) researcch background) possible to be practical.
resseacc research background) into many executable
2

---

Published as a conference paper at ICLR 2025
We develop scientific discovery framework directly based on the mathematical derivation. Different from previous works, we propose an evolutionary algorithm- based method to bet- ter associate background and inspiration, multi-step inspiration retrieval and composition, and an efficient ranking method. In addition, the framework can be applied to chemistry and material science, which not covered by previous methods.
We construct a benchmark by three chemistry PhD students, consisting of 51 chemistry papers published on Nature, Science, or a similar level, decomposing each paper into the research background, inspirations, and hypothesis.
We propose an assumption, grounded in preliminary experiments, that LLMs may already possess numerous knowledge pairs capable of being associated to create novel knowl- edge-even when scientists have not previously recognized any relationship between them.
For the first time, we show that LLM-based framework can largely rediscover the main innovations of many chemistry hypotheses that have been published in Nature and Science The rediscovery not because of data contamination, because we have controlled the date of the training corpus of the LLM and the online date of the chemistry papers.
2 RELATED WORK
Zhong et al. (023) work finding the difference between two corpora propose hypotheses, but their evaluation is conducted by Turkers, which cannot lead to a novel discovery. Wang et al.] al.] (2024b) try to utilize LLMs to discover novel NLP and biochemical hypotheses, and find the hypotheses still fall far behind scientific papers in terms of novelty, depth, and utility. Yang et al. (2024b) first show that LLMs can generate novel and valid enough hypotheses evaluated by PhD students, but they only focus on social science. FunSearch (Romera-Paredee et al. (024) can discover specific solu- tions for mathematical conjecture but can't discover new math theorems. Qi et al. (2024) analyzes LLM's ability for scientific discovery in the biomedical domain by directly generating hypotheses with only the research background. Boiko et al. (2023); Back et al. (2024); [L et al. (2024); Lu et al.) (2024) focus on subsequent steps for scientific discovery, mainly developing and conducting experi- ments. Sprueill et al. (2023) 2024) focus on catalyst discovery, but their evaluation relies on whether rediscover existing commercially used catalysts, which might cause data contamination prob- lem. Kumar et al.] (2024) compare different LLMs on scientific discovery in different disciplines. Tshitoyan et al. (201)) show that word embedding obtained from large-scale chemistry literature can recommend materials years before their discovery. Xie et al. (2024) predict emerging thermoelectric materials by summarizing the sentiment in the existing literature.
BENCHMARK CONSTRUCTION
The goal of the benchmark, named TOMATO-Chem, is two-fold. Firstly, it is used to analyze LLM's ability in terms of the three smaller questions. Secondly, it serves as challenge rediscover nature- level chemistry hypotheses with only a research background. The setting of the challenge is very similar real copilot setting, where scientists tell the copilot about the specific research question they are interested in, and optionally a small survey consisting of several paragraphs summarizing the existing best- perforrnfo methods for the research question.
To achieve the goals, we split each collected paper into the following components: <background question, background question (strict), background survey, background survey (strict), one to three inspiration paper titles and their reason to serve as inspiration, research hypothesis, experiments, reasoning process, summarization of inspirations>. Every component is described by text.
The reason we add strict version for background question and background survey is that many hypotheses are making relatively minor modifications based on existing methods covered by the survey, and the question can be very insightful to provide hint on the general direction of the hypothesis. In practice, these situations are entirely possible, especially when the scientist users can provide more comprehensive survey on existing methods, contain deep insights in their question. Here, we also keep the strict version to make the task more challenging and encourage developing methods to better assist scientists even when they are also new to their research topic.
3

---

Published as a conference paper at ICLR 2025
The reasoning process indicates the relation between the components of background, inspirations, and hypothesis. For example, the reasoning process can be "background + inspiration + inspiration 2 = hypothesis", or "background + inspiration 1/inspiration 2 + inspiration 3 hypothesis".
The benchmark consists of 51 chemistry and material science papers and is constructed by multiple chemistry PhD students. We only select those papers published top chemistry venues and be public on the internet after January 2024. After constructing, the experts check again on (1) whether the identification of the inspirations correct and whether more inspirations are needed; (2) whether the background does not contain any information in inspirations or hypothesis; and (3) whether the background and the identified inspirations can roughly logically lead to the hypothesis. The complete instruction on the check process is shown in $ A.3
Category
Count
Publication Venue
Count
Polymer Chemistry
21
Nature / Science
27
Organic Chemistry
Nature Subjournals
20
Inorganic Chemistry
Analytical Chemistry
5
Other Top Journals
4
Total
51
Total
51
Table 1: Distribution of categories.
Table 2: Distribution of publication venues.
Table 1 and Table 2 show the statistics of the benchmark in terms of chemistry category and pub- lication venue. Material science is sub-category of chemistry and can belong to the categories in Table [ such as polymer material and organic material. Around 13 collected benchmark papers inside the material science domain. Beyond them, more papers have intersections with material science. In this paper, we target both chemistry and material science, but for simplicity, we only refer them as chemistry in this paper.
METHODOLOGY
4.1 FUNDAMENTAL ASSUMPTION AND FOLLOWING DECOMPOSITION
We propose an assumption that majority of chemistry hypotheses can originate from a research background and several inspirations. This assumption is not only supported by many chemistry researchers whom we have extensive discussions with but also by the cognitive science finding that "creative ideas often result from the cohesive association of two (or more) seemingly unrelated pieces of knowledge" (Koestler) 1964 Benedek et al., 2012 Lee & Chung 2024). We design our method based on this fundamental assumption.
Denoting background knowledge as b, inspiration knowledge as i, and hypothesis as h, we translate this assumption as:
(b..i.......4
(I)
Here, &n E Z represents the number of inspirations needed for particular h. Typically in chemistry, E [1,3]. In other words, given existing knowledge in the background, a majority of chemistry research about searching knowledge that previously not known be related to the background but in fact can assist the background, then associate the background knowledge and the searched knowledge in reasonable way to compose hypothesis.
Based on this assumption, we can transform the seemingly impossible-to-solve P(h b) into an equivalent form, where each step in the equivalent form is practical and executable.
P(h ()
II P(i b, P(hg) b, ig h)-1), where ho 0
Here, I denotes the full (chemistry) literature, representing the full inspiration space to search for every single i. The full proof along with detailed analyses is shown in A.I. which is the core of this paper (and therefore highly recommend to take a read).
4

---

obtain Published as a conference paper at ICLR 2025
one round
refine
I
1P
reiin
h1
12
h1
T3p
recombine
baccgground
6.
cosponation
f
21
22
121
24
Gypotthaa's
hypothosle mutetion
rate acore
Vevatuur corpun
Figure 1: The MOOSE-Ches framework. It receives b and / as input, and outputs list of ranked h. The bottom-right legend describes the symbols the figure.
Equation 2 is meaningful in that by decomposing P(h b) into more practical and executable smaller questions, the seemingly impossible-to-solve P(h b) itself becomes practical. We analyze how P(1 I)) and P(hy b, hz-1) are practical and executable by LLMs in $ 5.1 and
S 3.22 correspondingly.
Now we have clarified the steps obtain from b. However, still might not be enough helpful in practice, since can be on large scale, and the search process might find lots of i, and finally lead to lots of h. Moreover, it is very time-consuming for scientists to conduct experiments to verify every single h. Therefore, it would be very helpful if the generated h could be ranked based on quality, Here, we adopt a straightforward and efficient way for ranking. Specifically, we design a rating function R(h), such that R(h) - R. Denoting the full set of generated h as H,
P(Haaokcs)) P(H,R), R), where Hyanked {h1, fig hn [(hh)) 2 R(h(+1) for all i}
Supported by Equation 2 and Equation 3 as a result, model P(h b), the only three components we need to model are P(ù b, h T), P(h; h, ij h114)))). and R(h). The implementation details
of the three components are illustrated in the remaining subsections in $ 4 Analyses of L.LM's ability on the three components are provided in $
4.2
THE FRAMEWORK DEVELOPED BASED ON THE ASSUMPTION
4.2.1
THE GENERAL PICTURE
Our methodology is developed based on the fundamental assumption discussed in $ 4.1 Specif- ically, use LLMs to perform P(i, b, b, I), P(h, b, ig. hy-1), and R(h), and organize them into multi-agent -LMebased framework. The input to the framework is only background question and/or background survey, together with a (large) chemistry literature corpus search for inspiration. The output of the framework is a list of ranked research hypothesis.
direct implementation of Equation 2 and 3 We develop it as simply possible, retaining only the necessary parts.
The framework's design is shown in Figure (overview in Figure 2
In the general picture, given research background b (research question and/or research survey), the framework first performs P(ù b, Ao = 0. /) by screening through the literature corpus / to select many papers where each of them has the potential to serve as an inspiration. Then the framework performs P(h1 | b, i1, fno 0), associating b and each i together to compose h. Then, it ranks h by assigning an evaluation score r on each of h2 by R(h1). We call these three steps as one round. Another round means going through the three steps again, based the previous round's results.
Since normally in chemistry, no more than three inspirations are needed for one hypothesis (k E 3), the default setting for MOOSE- -hee- is to perform three rounds for each b. In every other round, the number of and h can expand exponentially. Here, we adopt beam search to select a fixed size of the top-ranked / to enter the next round. The default beam size is 15.
5

---

Published as a conference paper at ICLR 2025
4.2.2 DESIGN DETAILS OF P(i I) AND ITS MOTIVATION
We use LLMs to conduct a screening process for P(ij b, h5-1. Specifically, for each inference, (1) sequentially select a fixed number of papers from 1, where the fixed number is called the screening window size (default is 15); (2) set up prompt consisting of b, the title and abstract of the selected papers from I, and the previous h (if it is not 0): and (3) instruct the LLM to generate three titles from the input that can best serve as i for b (and optionally previous h), and give reasons.
In particular, we use LLMs choose potential inspiration but not choose i from citation nor semantic neighbors because is supposed to be previously not known to be related to b (we have discussed it in $ 4.1. If the chosen is already known to be related to b, then the composed probably would not be novel. If the chosen i contains similar semantic information with b, then probably not necessary to add i at all, since does not introduce much (any) extra information.
Our bold assumption here is that advanced LLMs, trained on vast scientific literature, may already recognize novel knowledge pairs unknown to any scientist that can be associated to create novel knowledge. However, this may not be too bold, as Tshitoyya et al. (2019) showed that unsupervised word embeddings from 3.3 million materials science abstracts could predict functiooal materials years before their discovery. Here, the functional applications mended materials can be seen as i, or even directly as h if is enough similar. It probably indicates that LLMs trained with significantly more literature tokens and parameters might already be able to identify the relation between many knowledge pairs that are unknown to be related by any scientist. We analyze this assumption in $5.
be seen b, and the recom-
4.2.3 DESIGN DETAILS OF P(hy b, 1, hy-1) AND ITS MOTIVATION
The retrieved i is expected to be not known to be related to b; therefore, it might be difficult to figure out an effective way to associate b and i together to compose h. Think of the time when backprop- agation is about to be invented. Even if we are very familiar with b (multi-layer logistic regression) and have successfully retrieved i (chain rule in mathematics), can invent backpropagation?
Our answer is, at least we might need to try multiple times and various ways to leverage the chain rule for multi-layer logistic regression. With this motivation, develop simple evolutionary aggorithm-based method, shown in the top-right of Figure We call it "evolutionary unit" (EU).
Specifically, given b and EU will first generate multiple hypothesis "mutations" m, where each m is a unique way to associate b and together. Then EU further develops each m independently by providing feedback to each m in terms of validness, novelty, clarity, and significance, and then refining them based on the feedback. Yang et al. (2024b) first propose to provide feedback in terms of validness, novelty, and clarity to refine hypotheses. Here, we add an additional aspect, signifi- cance, since significance is an important evaluation criterion in chemistry. We assume the refined hypothesis should be of better quality so that the refined hypothesis is "selected", while the previous hypothesis is "eliminated" by the "environment". Finally EU "recombines" the remaining selected m, leveraging the advantages from every m to propose h to better associate b and i.
4.2.4 DESIGN DETAILS OF R(h) AND ITS MOTIVATION
We adopt a simple and efficient way for R(h), which is to prompt an LLM to output evaluation scores for an input h in terms of validness, novelty, significance, and potential. Validness and novelty are two fundamental requirements for such an inductive reasoning process as scientific discovery (Yang) al. 2024a b). Significance is added because it important for chemistry. We additionally add potential, because the generated h are about to be further developed by scientists, so we might want to pick those h that not only are currently high quality but also have good potential to be further developed. We did not design R(h) in more complicated way, since there are lots of h to rank, and we might want to save more inference time.
Yang et al. (2024b) use the scores as automatic evaluation for generated social science hypotheses and have shown a high consistency score between automatic evaluation and expert evaluation. How- ever, in the chemistry domain, LLMs might not be reliable enough to directly evaluate the generated (Sprueill et al.) 2024). But it might still be able to provide a preliminary quality identifier to h: the ranking of the average score between the four aspects of an h. determines whether it will enter the

---

Published as a conference paper at ICLR 2025
Corpus Size
Hit Ratio (top 20%))
Hit Ratio (top 4%)
Hit Ratio (top 0.8%)
Hit Ratio (top 0.016%)
150
76.8%
61.4%
300
96.7%
83. 7%
60.8%
NA
1000
96.4%
46, 66.
3000
95.8%
86.9%
70.6%
52.0%
Table 3:
Main table for Q1. For each screen window of 15 papers, 3 papers are selected.
Screen window size
Hit Ratio (1 round)
Hit Ratio (2 round)
Hit Ratio mund)
Hit Ratio mund)
10
98 00.0%
88.9%
79.4%
56.5%
96.7%
20
91.2%
76.8%
58.8%
NA
60
71.6%
53.9%
NA
NA
Table 4: Ablation table on screen window size for Q1. The corpus size is 300. For each screen window no matter its size, 3 papers selected to remain for the next round of screening.
next round of MOOSE-Chem by beam search. To understand how well LLMs can perform R(h), we analyze "how well LLMs can rank chemistry hypotheses" in 5.3
5 INVESTIGATION ON FUNDAMENTAL QUESTIONS
P(h can be understood as the task discover high-quality chemistry research hypothesis, given only a background question and/or background survey. Our central question to investigate how well LLMs can perform P(h b). Supported by Equation 2 and we break up this main question into three smaller questions: how well can LLMs perform (T) P(1 h I). (2) P(hs
iy, 11)1). and (3) R(h)? All experiments are performed by Go (its training data up to October 2023).
5.1 How WELL CAN LLMS PERFORM P(i, 6, hj-1,
Here, investigate the question (denoted Q1): "whether LLM can identify inspiration papers which are unknown to be able to associate with the background (or at least unknown to associate in certain way) but in fact can associate with the background create novel knowledge?".
We first find 3000 most cited chemistry papers published in Nature, and construct a series of in size of 150, 300, 1000, and 3000. I is constructed by first adding the ground truth inspiration papers (around 120), then randomly selecting the remaining papers from the 3000 papers, and finally randomizing the order of all the collected papers. Only title and abstract are needed for each paper in I. The default setting is that each inference of LLMS will screen 15 papers from I, and generate three titles that LLMs think best assist b (and/or previous h). Screening through / for one round, only 20% of I will be selected. Screening another round will only leave 4%, and so on.
We use Hit Ratio the evaluation metric, which calculated by the number selected ground truth inspiration papers divided by the number of all ground truth inspiration papers. All the Hit Ratio numbers shown the tables are averaged across the 51 papers in the benchmark.
Table 3 shows the main experiment results. The Hit Ratio is surprisingly high: More than 75% of the ground truth inspirations are covered by even only the 4% chosen papers from the chemistry lit- erature corpus. It seems that L.MSs are quite capable of finding inspiration papers that are unknown to be able to associate with the background but fact, can associate with the background create novel knowledge. It means already know lots of knowledge pairs that are able to associate to create novel knowledge, where the knowledge pairs not known by any scientist be related" is possible to be true.
bold assumption in 4.2.2 that "the most advanced LLMs might
Table 4 shows the ablation study in terms screen window size. seems that a smaller window size lead to better performance: screen window size of 60 to keep 3 for one round will select 5% of the corpus, and the Hit Ratio is 71.6%; while a screen window size 15 to keep 3 for two rounds will select only 4% of the corpus, but the Hit Ratio is high 83.7%.
7

---

Published as a conference paper at ICLR 2025
Model
Hit Ratio (top 20%)
Hit Ratio (top 4%)
Hit Ratio (top 0.8%)
Llama-3. 31-88
71.6%
43.3%
26.8%
Llama-3.1-17
95.1%
83.0%
59.5%
1:1maa3.114058
95.7%
78,7%
52,7%
GPT-4o
96.7%
83.7%
60.8%
Table Comparison of alla series and GPT-4o 4o on inspiration retrieval. The corpus size is 300. For each screen window of 15 papers, 3 papers are selected.
Generated hypothesis covers three key points cor covers all the key points) and leverage them similarly as the groundtruth hypothesis; Extra key points do not have apparent flaws
5 points
Generated hypothesis covers three key points (or covers all the key points) and leverage similarly in the groundiruth hypothesis; Extra key points have apparent flaws.
4 points
Generated hypothesis covers two key points and leverage them similarly the groundtruth hypothesis, does not cover more all key points
3 points
Generated hypothesis covers one key point and leverage it similarly as in the groundtruth hypothesis, but does not cover more or all key points
2 points
Generated hypothesis covers at least one key point, but is used differently as in the groundtruth hypothesis
point
0 point
Generated hypothesis does not cover any key point
Table 6: Description of the Matched Score.
Table 5 compares I.LMs in different scales on inspiration retrieval ability. The results indicate that LLMs obtain the emergent ability for inspiration retrieval since rather small parameter size, but then quickly plateau. JA..diiscusses research background options' influence on inspiration retrieval.
5.2 How WELL CAN LLMS PERFORM P(h; b, ht-
Here, we investigate the question (denoted as Q2): "Given only known knowledge, whether LLM can reason to unknown knowledge that has high probability to be valid?".
The first challenge to answer Q2 is the evaluation method: The benchmark covers a large range of chemistry topics, and chemistry is very complex discipline that slight change of research topic would make chemist unable to provide reliable enough evaluation. In fact, chemistry researcher might not be able to provide reliable enough evaluation even if the hypothesis in his domain.
Therefore, we adopt a reference-based evaluation method called "Matched Score" (MS). The de- scriptions are shown in T It's on a 6-point Likert scale, roughly containing four stages. Denot- ing generated hypothesis as gh, and original hypothesis as oh. the four stages are (1) gh no = (0 point); (2) gh oh # 0 (1/2/3 points); (3) gh 2 oh (4 points); (4) gh oh (5 points).
We use MOOSE-Chem to investigate Q2. Specifically, we initialize T as only the ground truth inspiration papers and search for l round, where le is the number of ground truth needed for each
4
Total
w/ background survey
Average MS (GP T-4o) Top MS (GPT-4o)
9
17
28
1
51
Top MS (Experts)
9
12
22
6
2
51
w/o back hackground survey
Average MS (GPT-40)
7
17
19
7
51
Top MS (GFT-4c)
2
Table 7: Main table for Q2 Average/Top MS means the average/highest Matched Score of all generated h from one b. Table 12 is more complete version of this table including automatic evaluation results by Claude 3. 5 Sonnet and Gemini-11 1.5 Pro.
8

---

Published as a conference paper at ICLR 2025
#Matched
2
0
Average Rank Ratio Size
NA
0.411
0,474
0.521
302
2458
4899
Table 8: Relation between the number of matched ground truth and the average ranking ratio (+).
Matched
Average Rank Ratio
0,489
0.439
0.488
0,501
0.436
0,501
0.503
Size
Table Relation between the GPT-4o labeled Matched Score and average ranking ratio (4).
MOOSE-Chem will not retrieve the same i already retrieved in previous rounds, guaranteeing that before generating the final h, the framework has already seen all the ground truth inspirations.
Table 7 shows the results. For each b, the top two h with the highest MS by GPT-4o are selected for expert evaluation (by two chemistry PhD students). It indicates that LLMs are quite capable of associating known knowledge into unknown knowledge that has high probability to be valid (very close oh). addition, providing survey can assist the new mnwleegee-discoees process. We discuss the agreement between GPT-4o-based evaluation and expert evaluation in $ A.14
5.3 How WELL CAN LLMS PERFORM R(h)?
Here, we investigate Q3: "whether LLMs can select high-quality h to rank them higher?".
To investigate Q3, we run MOOSE-Chem with every 6 from the benchmark; |4] 300, containing all the ground truth i. Every h is given rating = R(h), and is ranked based on r. For every generated h, we get the number of ground truth i leveraged (#Matched i), and evaluate with GPT-4o evaluated MS (here MS is 11 means this h has not used any ground truth i).
Table g shows the relation between the #Matched and average ranking ratio (the lower, the better). It shows a clear trend that the more ground truth i is leveraged, the better ranking score h can have. indicates that h with a higher ranking ratio is more likely be matched with better i.
Shooss the relation between the GPT-4o evaluated MS and the average ranking ratio. There a trend that the higher MS, the better the average rank ratio (when MS [2,4]). However, the disadvantage of those h without a positive MS is not very significant. It seems that LLMs have certain ability to rank good h higher. But it is not sure how significant it is, because a part of the reason for these results is that those h generated without ground truth could be also in high quality.
EXPERIMENT AND ABLATION STUDY
We perform experiments in setting similar to the copilot in the wild setting. Only background question (strict), background survey (strict), and a chemistry corpus |/| 300 are provided to the framework. Only the top 4% of is selected and used to develop h. The evaluation metrics are Top MS and Average MS (the highest/average Matched Score of all generated h from one b), averaging across the benchmark. Experiments are conducted by GPT-4o (training data up to October 2023).
BASELINES
MOOSE is hypothesis discovery framework for the general social science domain. leverages LLMs to retrieve inspirations and uses self-refine (Madaan et al., (023) to improve the validness, novelty, and clarity aspects. The difference that (1) it does not adopt the mutation and recombina-
tion step to better associate background and inspiration; (2) it only retrieves one step of inspiration.
SciMON
is hypothesis discovery framework for the NLP and biochemical domain. relies on semantic and citation neighbors retrieve information to assist the background. As result, the retrieved information could be very related to the background that might not be able to serve as an inspiration. To make the generated hypothesis more novel, it adopts self-refine focus improving

---

Published as a conference paper at ICLR 2025
Method
Top MS
Average MS
SciMON
Wang
2024b
2.549
2.281
MOOSE
Yang al
2024a
2.882
2.464
Qi ct al.
2024
2.686
2.356
MOOSE-Chen
4.020
2.564
w/o multi-step
3.765
2.730
w/o multi-step EU
2.863
2.578
Table 10: Experiments and ablation study. The Matched Score (MS) is evaluated by GPT-4o (this
table), Claude-2 3. 1--55--omm Sonnet (Table 13 and Gemini 11.--1.6 5-Pro (Table [4]
4
Total
Top MS (Expert)
0
2
19
16
8
6
51
MOOSE- runs with =30=00,, mimicking the copilot setting. This table shows the statistics of the top Matched Score across the benchmark. The evaluation is done by experts.
Table 11:
the novelty aspect of the generated hypothesis. Here, implement SciMON with -LM-based inspiration retrieval, the same as MOOSE-Cher Table shows that the recall rate of LLM-based retrieval is 83.7%.
Qi al. (2024) work on hypothesis discovery in the biomedical domain. retrieves information pertinent to the keywords in the background to generate hypotheses. As a result, the retrieved infor- mation might compose of a background survey, but not as inspiration. Self-refine is also adopted.
6.2 RESULTS
Table T0 shows the baseline results and the ablation study of MOOSE-Chem. indicates that both mutation & recombination and the multi-step designs can significantly improve the pest-performing h. Mutation & recombination leads to a drop of Average MS compared to the MOOSE baseline; attribute the reason to that the mutation step forces LLMs generate h different from previous mutations from the same b and i, and therefore might generate many that do not make a lot of sense. The assigned MS to these mutation h is low, and therefore lower down the Average MS.
To better understand the performance of MOOSE-Chend in this real copilot setting, for each b the top 4 generated with the highest MS by GPT-4o evaluated again by experts in terms of MS. Table 11 shows the expert evaluation results. Here, the top MS is the highest MS for each b, out of the expert evaluated h for this Note that MS rated as three is already very high. Illustrated in Table 6 it means the generated h by MOOSE-CHE covers two main innovations of the chemistry hypothesis, which published in Nature, Science or similar level. Some case studies can be seen in A.16
(that has not seen h) in the real copilot setting
CONCLUSION
We investigated this central question: "Can LLMs automatically discover novel and valid chem- istry (including material science) research hypotheses (even those which deserve publication in Nature, Science, or a similar level) given only a chemistry research background (consisting of research question and/or background survey), without limitation on the domain of the research question?". We proposed fundamental assumption to break up this seemingly impossible-to-sole central question into three smaller, more practical, and executable fundamental questions. Then, we investigated LLM's ability on each of them.
To this end, we constructed a benchmark consisting of chemistry and material science papers pub- lished and only be public in 2024. We also developed an LLM-based multi-agent framework consist- ing of three stages reflecting the three smaller fundamental questions. Experiments showed that the framework (rums in copilot in in-the-wil setting, with LLMs with training data up October 2023) can rediscover many hypotheses with very high similarity with the ground truth ones, covering the main innovations.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the Shanghai Municipal Science and Technology Major Project. This work is supported by Shanghai Artificial Intelligence Laboratory. This rescarch/project is supported by the Ministry of Education, Singapore under its MOE Academic Research Fund Tier 2 (STEM RIE2025 Award MOE-T2EP20123-000
We thank Mengsong Wu for his insightful discussions with us, and we thank Yuwei Wan for her efforts support this research.
REFERENCES
Jinheon Baek, Sujay Kumar Jaubar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738, 2024.
Mathias Benedek, Tanja Könen, and Aljoscha C Neubauer. Associative abilities underlying creativ- ity. Psychology of Aesthetics, Creativity, and the Arts, 6(3):273, 2012.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpp:::/ppeeeeiieeetee/ www./fernffffree. Byglv 1v1HKDD
Danii A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes, Autonomous chemical research with large language models. Nat., 624((792):577--7777 2023, doi: 10.1008/541586-02--06222 038 4 586 6- 0000-023---0000 02300272-0--0-2222
URL http::://ooooiioo org.
Faxiang Bu, Yuqi Deng, Jie Xu, Dali Yang, Yan Li, Wu Li, and Aiwen Lei. Electrocatalytic reductive deuteration of arenes and heteroarenes. Nature, pp. 1-2, 2024.
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation. CoRR, abs/2311. 2023. doi: 10.48550AARXI..311...77 https://doi. http//oi.oggg...// org/l0. 48550/ arXiv. 2311.
URL
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In Christian Bessiere (ed.), Proceedings of the Twent-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3882-3890, ijcai.org, 2020, doi: 10.24963/ijcai 2020//337.
URL httpt:://///......... wtt.://ooooooooooooo////o./ooo// 10. 2496 963 / caai 2020/537
Mario Coccia. Why do nations produce science advances and new technology? society, 59:101124, 124, 2019.
Technology in
Joseph F Hair. Arthur H Money, Philip Samouel, and Mike Page. Research methods for business. Education+ Training. 49(4):336-337, 2007.
Arthur Koestler. The act of creation. London: Hutchinson, 1964.
Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal, and Asif Ekbal. Can large language models unlock novel scientific research ideas? arXiv preprint arXiv:2409.06188 2024.
Byung Cheol Lee and Jaeyeon Chung. An empirical investigation of the impact of chatgpt creativity. Nature Human Behaviour, pp. 1-9, 2024.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmmented generation for knowledge-intensive NLP tasks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. http////.//..tt////oooeaaaaaaaa. 80 bc 2tt.://00.00..02222552227a1aa-a0

---

Published as a conference paper at ICLR 2025
Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. Mlr-copilot:
Autonomous machine learn- ing research based on large language models agents. arXiv preprint arXiv:2408.14033 2024.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The Al scientist:
Towards fully automated open-ended scientific discovery. CoRR, abs/2408 0abss//4480002222. 2024. doi: 10.
48550/ARXIV.2400.00233 06292. httpttss::// //oii www.oo..oo.... 48550 r w.w.. 062 92
Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du LLM4SR: A survey on large language models for scientific research. CoRR, 2twp/2211.04406. 2025. doi: 10.48550/ARXIV. 2501.04300 URL htt.p::///o.iii.. org/10 48550 2501..43066.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference Neural Infor- mation Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: /pap:://pppppp.pppe .ww.ss..ppp.. nips. wwtpsseeepppppppp.eeeeeeeeeeee.e wetrrrffilsssapaaee wtt.://lleeaapee///////aaaaa Connttt-ooofffeee htt
Biqing Qi. Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng. Zhang- Chen, and Bowen Zhou. Large language models are zero shot hypothesis proposers. CoLM, abs/2311 2024. doi: 10.48550/ARXIV2231...
URL https: bttts:://oo/ 1tt.//......////......... 2311.
05965
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nat., 625(7995)4668 475, 2024. doi: 10.1038/
441586-023-06924--
Kaito Shibahara, Yoshihito Kayaki, Kairi Yamashiro. Yuki Nagashima, Kohei Fujii, and Ken Tanaka. Rh-catalysed enantioselective [2+ 2+ I] cycloaddition reactions using three different co-componetts. Nature Synthesis, pp. 1-13, 2024.
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? large- scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04/ 04109, 2024.
Henry Sprueill, Carl Edwards, Mariefel V. Olarte, Udishnu Sanyal, Heng Ji, and Sutanay Choudhury. Monte carlo thought search: Large language model querying for complex scientific reasoning in catalyst design. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 8348-8365. Association for Computational Linguistics, 2023. doi: 10. 10.88663//120222.IIIDI
EMNLP. 560
URL
Henry W. Sprueill, Carl Edwards, Khushbu Agarwal, Mariefel V. Olarte, Udishnu Sanyal, Conrad Johnston, Hongbin Liu, Heng Ji, and Sutanay Choudhury. CHEMREASONER, heuristic search over a large language model's knowledge space using quantum-chemical feedback. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: httptttttt/ppeeeeeeetss /ttp::///peeeeeeeeee
Ryuhei Suzuki, Taiga Ando, Fritz Deufel, Kohsuke Ohmatsu, and Takashi Ooi. Photocatalytic car- byne reactivity of phosphorus ylides for three-component formal cycloaddition reactions. Nature Synthesis, pp. 1-7, 2024.
Don R Swanson. Undiscovered public knowledge. The Library Quarterly, 56(2): 1986.
Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong, Solving olympiad geometry with- out human demonstrations, Nat, 625(7995) 482, 2024. doi: 10. 10110888444536-223007477575
12

---

Published as a conference paper at ICLR 2025
Vahe Tshitoyan, John Dagdelen, Leigh Weston, Alexander Dunn, Ziqin Rong, Olga Kononova, Kristin A. Persson, Gerbrand Ceder, and Anubhav Jain. Unsupervised word embeddings capture latent knowledge from materials science literature. Nat., 571((763))95 98, 2019. doi: 10. 1038/ 9t////415886-01 URL httth::/s////e http://too..rg.o.oo//// 0
S41586-019- 1335-88.
Jinpei Wang. Yuxin Song, Fanfei Yu, Yijun Zeng, Chenyang Wu, Xuezhi Qin, Liang Peng, Yitan Li, Yongsen Zhou, Ran Tao, et al. Ultrastrong, flexible thermogalvanic armor with a carnot-relative efficiency over 8%. Nature Communications, 15(1);6704, 2024a.
Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope, Scimon: Scientific inspiration machines optimized for novelty. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 279-299, Associ- ation for Computational Linguistics, 2024b. doi: 1ttpg/..1663/11/2222222222222222
URL
18
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learming Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:://ppeerreeeeeee hwtp///peeevieeeeeeeeeeeeeeeeeee net/ Forum?id=1PLINIIE
/
Tong Xie, Yuwei Wan, Haoran Wang, Ina Østrøm, Shaozhou Wang, Mingrui He, Rong Deng, Xinyuan Wu, Clara Grazian, Chunyu Kit, and Bram Hoex. Opinion mining by convolutional neural networks for maximizing discoverability of nanomaterials. J. Chem. Inf. Model., 64(7); 2759, 2024. doi: J0.002/ACSS.CIMM.CCC0 URL httts:://cc.../ org 100g/11..0 1021/ acs. jeim. 3c00746
Zonglin Yang, Xinya Du, Alexander M. Rush, and Claire Cardie. Improving event duration pre- diction via time-aware pre-training. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Find- ings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 3370-3378. Association for hwwp://ww...ininnnnmnnnn.......
Computational Linguistics, 2020. doi: 10. 22118863/1/2000..fmm
URL https:
//doi. 11w.11.116531111110
findings--nnig 302
Zonglin Yang, Xinya Du, Erik Cambria, and Claire Cardie. End-to-end case-based reasoning for commonsense knowledge base completion. In Proceedings of the 17th Conference of the Euro- pean Chupter of the Association for Computational Linguistics, pp. 3502-3522, Dubrovnik, Croa- tia, May 2023a. Association for Computational Linguisties. URL org 2023 eacl main. 255
ogy.
Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. Logical reasoning over natural lan- guage knowledge representation: A survey. In /st Workshop on Natural Language Reasoning and Structured Explanations ( (@ACL 2023), 2023b.
Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. Language models as inductive reasoners. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Associa- tion for Computational Linguistics. EACL 2024 Volume 1: Long Papers, St. Julian's 's, Malta, March 17-22, 2024, pp. 225. Association for Computational Linguistics, 2024a. URL https: acl antho ogy wwwp::///022.11011orrre....
Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. Large language models for automated open-domain scientific hypotheses discovery. In Lun-Wei Ku, Andre Mar- tins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties, ACL 2024, Bangkok, Thailand and virtual meeting. August 11-16, 2024, pp. 13545-13565. Associa- tion for Computational Linguistics, 2024b. doi: 10. 00655//V1/2024.INNDD ACL.8004 URL https://dob .org 000 11w131166112200 http://0//...//////0000000
inndddigg-acc 804
13

---

Published as a conference paper at ICLR 2025
Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal driven discovery of distributional differences via language descriptions. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023, URL h / www.:nii...p...ppoee.ee..mpp files/paperr2022//ssss 1e51Db2cccccSbobbbccaddeeeeeeeee Abstract - Itml
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROOF OF THE RIGOROUS DECOMPOSITION OF P(h (6) BASED ON THE FUNDAMENTAL ASSUMPTION
We propose assumption that majority of chemistry hypotheses can originate from research background and several inspirations. This assumption is not only supported by many chemistry researchers whom have extensive discussions with but also by the cognitive science finding that "creative ideas often result from the cohesive association of two (or more) seemingly unrelated pieces of knowledge" (Koestler) 1964 Benedek al. 2012 Lee & Chung. 2024). We design our method based on this fundamental assumption.
This assumption is reminiscent of Swanson Linking (Swanson 1986) in the domain of literature- based discovery (LBD), also known as the "ABC model", where two concepts A and C are hypoth- esized as linked if they both co-oceur with some intermediate concept differs in that: (1) for a chemistry hypothesis published in a good venue, usually more than one in- spirations are needed; (2) background and inspiration are not necessarily linked by a path of interme- diate papers; (3) our assumption is applied to majority of existing published chemistry hypotheses, while LBD has been considered to only focus on a very specific, narrow type of hypothesis (Wang et al. 2024b). It might indicate that similar proportion of future chemistry hypotheses can also be resulted from linkages of existing literature.
papers. Our assumption
Denoting background knowledge b, inspiration knowledge as i, and hypothesis as h, we translate this assumption as:
h = f(b, (b. ii))
(4)
Here, & E Z represents the number of inspirations needed for particular h. Typically in chemistry, E [1.3].
Equation 4 expresses the idea that, for the majority of chemistry hypotheses (if not all), each hy- pothesis can be formulated as a composition of background knowledge and additional knowledge elements, which refer to inspirations. This functional form reflects universal pattern in hy- pothesis formulation: regardless of where the inspiration originates- be prior literature, serendip- itous observation, discussions with peers the essential step is to identify correct inspirations and integrate them with existing background knowledge in a meaningful way. In this formulation, the process of first collecting and selecting the appropriate background knowledge and then identi- fying and integrating suitable inspirations constitutes both a necessary and sufficient condition for generating valid hypothesis h.
Here's 's an example in chemistry:
Research Question: How to obtain D2 gas more efficiently?
Background Knowledge: The best performing methods are electrocatalytic methods.
Inspiration Knowledge 1: Ruthenium catalyst
Inspiration Knowledge 2: Nitrogen-doped electrode
Inspiration Knowledge 3: D2O as chemical solution
Hypothesis: A nitrogen-doped ruthenium (Ru) electrode can effectively catalyze the ductive deuteration of (hetero) in the presence of D2O in an electrocatalytic method, leading to efficient D2 gas production.
Here's 's an example in AI:
Research Question:
How can we automatically update the parameters of a multi-layer lo- gistic regression model using data?
Background Knowledge: Multi-layer logistic regression
Inspiration Knowledge: The chain rule from calculus
15

---

Published as a conference paper at ICLR 2025
Hypothesis: Backpropagation
Here's s another example in Al:
Research Question: How can we improve reasoning performance in language models?
Background Knowledge: Chain-of- Thouuhh prompting
Inspiration Knowledge: Majority voting over multiple reasoning paths
Hypothesis: Self-consistency decoding
Here, "background knowledge" and "inspiration knowledge" as illustrated the example above, can be understood as specific, well-defined knowledge pieces. In practice, however, knowledge retrieval is rarely so clean. Instead of isolating single knowledge unit, we often retrieve a noisy cluster of information that contains the desired piece along with extraneous content. For instance, when retrieving paper that includes a relevant inspiration, the paper will inevitably also contain unrelated information that may not be useful for the current research question. Conversely, a single clean inspiration i may be embedded across multiple papers in the literature. This redundancy is beneficial- increases the likelihood of retrieving even when searching imperfectly.
In other words, given existing knowledge in the background, a majority of chemistry research is about searching knowledge that previously not known to related to the background but in fact can assis/ the background, then associate the background knowledge and the searched knowledge in a reasonable way to compose a hypothesis. Crucially, the inspiration should not be previously known to be related to the background-ad at least not in a way that has already been used to formulate hypotheses. Otherwise, the resulting hypothesis would lack novelty. This requirement positions the inspiration retrieval task surface connections that lie outside established knowledge associations.
inherently dittr--distribution (OOD) problem, where the goal is to
Our goal is to transform the seemingly impossible-to-solve P(h b) into an equivalent form, where each step in the equivalent form is practical and executable. Denoting the full inspiration knowledge space as such that P(T) 1. Then a straightforward way of decomposing P(h b) is by the
chain rule based on Equation 4
P(h b) i(h,i1,... .4 b)
(5)
P(h.b.))) P((.)))P((I)
if = 1 if
P(h.bii........ P(b,))....)e)
P(b.t) P(b) P(T)
P(ha)).P()) P(D)-P(I)
(6)
P(h b, P(i 1,, -IT P(1, 1......,,,,,.,, P(1 II
if
P(h b.ú.....,4))
if
Here, P(h, i....., b) can be interpreted the joint distribution over h and its associated inspi- rations conditioned on background knowledge Equation 5 holds because Equation 4 asserts that (i1,-..,44) constitutes necessary and sufficient set of additional knowledge required to compose h. denotes the full inspiration space-thar is, the set of all possible knowledge pieces that could serve inspiration for generating a new hypothesis. This space includes not only all existing chemistry knowledge but also potentially relevant knowledge from other disciplines that could be leveraged in formulating novel chemistry hypotheses. However, computing over the full space is computationally infeasible. To make the problem tractable, we approximate with a large but man- ageable subset i, consisting of approximately 3,000 top cited chemistry papers from the existing chemistry literature.
Equation 7 describes the process of P(h b) from a knowledge-searching perspective. However, may not fully capture how chemistry
the terms P(h | b, b,üj,...,úú) .x) and P(i. b. researchers actually discover new inspirations in practice. One key reason is that researchers typi- cally reason in an incremental fashion, composing hypotheses by integrating one or two knowledge components at time. It is cognitively and practically difficult to evaluate or integrate all candidate inspirations simultancously. Instead, researchers iteratively assess partial combinations -gadually building toward complete hypothesis.
16

---

Published as a conference paper at ICLR 2025
To mimic how chemistry researchers conduct research and make it more practicable, we break P(h fi....,.A, 4x) into a series of recursive smaller steps as
P(hk b,i,,..... ix) = P(hx
b, f [b, 11,.......... ......,.. 2k.).....))..
if > 1 if >
(8)
P(hk b, h [k)
(9)
Similarly, we can break P(iş+1 b, :1..... 1 as
P(ix+1 b. 1k, P(ix+1 b, f(b, ....,))))))..... = P(i+1 b, hk: I)
if >
(10)
if >
(11)
As a result, to achieve the final hgg we need to obtain {/i...., h--1) first (if k: > 1). In addition, seeing h "state", and i "action", obtaining h and é through P(h b. hg and P(i++1 b, hg, I) correspondingly indicates a Markov property: (1) a new only depends on b, its previous h, and the current i; and (2) an i only depends on b, and the previous h.
For brevity, we visualize this as:
h2
hg-1
h2
where each transition is still conditioned on the background knowledge b, though b omitted from the notation to emphasize the Markov structure of the progression.
Building on this visualization, we formalize the formation of a hypothesis (specifically. h = h4.) as constructive process that sequentially integrates of inspirations (i1,..., i into intermediate hypothesis states (h1,..., )..)
Formally, the conditional probability P(h b) be expressed marginal over all valid quences of inspirations that can generate ft:
P(h b)
Σ P(i-())...., (n(()) hi...., hx 16).
(12)
FEll.
where Th denotes the set of all permutations of (1..... k} applied to the inspirations {ir..... ig} such that the resulting composition yields the final hypothesis hg = h, under the assumption in Equation 4 which specifies that h is fully determined by b and {i .i).. In this context, h; represents the intermediate hypothesis state obtained after integrating (+() at step j.
The degree to which the order of inspirations (i....., 4 affects hypothesis formulation can vary across disciplines. In empirical sciences such as chemistry, the contributions of individual inspi- rations are largely interchangeable, and their order of integration has limited impact on the final hypothesis, resulting in large [11]]. Conversely, in disciplines like mathematics, where hypothe- ses (e.g., theorems) often require constructing a specific sequence of lemmas and prior results, the ordering of inspirations is more constrained and may follow detr-deetriiiiitic path.
Considering this variance depending on the disciplines, to make this problem tractable and casier to understand, adopt fixed constructive path assumption, i.c., [[g 1, selecting representative order inspirations {d1,-.., }
Under this inspiration-fied-orrde fixed-order assumption, where hypothesis h is constructed through specific sequence of inspirations {i1,..., i }. we have:
P(h b) = P(i,..... dn. h1. hi..h...hhh h b),
(13)
with hy denoting the intermediate hypothesis state after incorporating ij. and hy = h.
Therefore, if with this inspiration-fixedoodde assumption for simplicity, if l: 1,
17

---

Published as a conference paper at ICLR 2025
P(h 1)) P(i ik. h1.... ....,,,, hx (6)
(14)
P(11, 11 1 (8) P(iz, h2 b, 81, 11)..... .... P(ix. by 0, 1 .1......
ha-1
(15)
P(ii, h1 18) P( h2 6,
P(ix, he 1,, b,hx-1)
(16)
P( P(b,ù1.01)
P(b, in. hu 1x-1: P(b, ix. hw-1, hk)
(17)
I(b,)) P(b,ū))
P(b,ha-1,1)
P(b, iny hx-
P(h b,1) P(h b,,1) II P(ij+1 16, f) P(hg+1 by h44 -
(18)
I f=1
P(ù 16, 1,, P(h.) b, bj 11-1),, where ho = 0
(19)
Although starting from l > 1, Derivation 19 covers the situation when k: = 1 in Equation 7
Therefore, in sum, break up the seemingly impossible question P(h b) into many practical and executable smaller questions as:
R II
P(h b)
P(ù 16, P(hj) b, ig h1-1), where ho = 0 and k >=
(20)
Of course, without the Inspiration-fieed fixed-order assumption, more complete derivation of P(h | involves marginalizing over all valid permutations in IIg:
b)
P(h = Σ
P(ix(a)...... (x()) hy..................... hy b)
(21)
TElls
ΣI xlll ==1
P( 6, 16,6 P(h)5) (7) | 16, (
(22)
where hg (z) 0. and >= Here, h(()) denotes the intermediate hypothesis state at step
j in the permutation æ,, which results from sequentially incorporating inspirations in the order
A.2 MOOSE-CHEM OVERVIEW I/O FIGURE
Ch
Research Question
(Optional) Background Survey
(Ranked) Chemistry Research Hypotheses
Inspiration Corpus:
Ch
MOOSE-CHER CHEM
Ch
Lots Chemistry
Papers
Figure 2: Overview of the input and output of the MOOSE-Chem framework.
Figure shows the input and output overview of the MOOSE-Chem framework.
18

---

Published as a conference paper at ICLR 2025
A.3 THE FULL INSTRUCTION FOR BENCHMARK CHECKING
Please help us check again before finalizing the decomposition of each paper in the benchmark:
Whether the background question is correct.
2. Background survey shouldn't t contain any information/mmethod in inspiration or hypothesis (ex- cept if this information/mettodd has been used for this particular background question before). is encouraged to include the most similar existing method to the proposed method. For example, the proposal to change BaC12 to BaSO4. It is encouraged to include BaC12 in the survey, but SO4 must not be included in the survey (since SO4 belongs to the inspiration).
Background question cannot contain any information in inspiration or hypothesis as well: should be little bit general question, instead of a specific question asking about how the inspiration can be leveraged to help with the question. It also shouldn't be too general that we can't understand which specific research domain it works on.
3. Whether the identification of inspirations really the main inspirations for this paper, and whether need more main inspiration(s).
4. Whether the main hypothesis is correct and covers the main key points.
5. Whether the background survey + background question + identified inspirations can logically lead to the hypothesis (if not, we might need to identify more inspirations).
Thank you for the efforts! Your contribution is indispensable for the success of this research. Please let me know if you have any questions.
A.4 PROMPT TO OBTAIN R(h)
You known diligent and harsh reviewer in Chemistry and Material Science that will spend much time to find flaws when reviewing and therefore usually gives a relatively much lower score than other reviewers. But when you meet with hypothesis you truly appreciate, you don't mind to give it good scores. Given a not yet peer reviewed research hypothesis in Chemistry or Material Science domain, try to evaluate the research hypothesis from four research aspects and give score according to evaluation guidelines provided below. All four aspects should be evaluated in a 5 point scale.
Aspect Validness. 5 points: The hypothesis is a logical next step from current research, strongly supported by theory, perhaps with some indirect experimental evidence or highly predictive computational results. The experimental verification seems straightforward with a high probability of confirming the hypothesis; 4 points: Here, the hypothesis well-rooted existing theory with some preliminary data computational models supporting it. It extends known science into consistent areas, where experiments are feasible with current technology, and there's a reasonable expectation of positive results; 3 points: This hypothesis is within the realm of theoretical possibility but stretches the boundaries of what's ' known. It might combine existing knowledge in very novel ways or predict outcomes for which there's no direct evidence yet. There's 's a conceptual framework for testing, but success is uncertain; 2 points: While the hypothesis might be grounded in some theoretical aspects, significantly deviates from current understanding or requires conditions or materials that are currently impossible or highly improbable to achieve or synthesize; 1 point: The hypothesis proposes concepts or outcomes that are not only unsupported by current theory but also contradict well-established principles or data. There's no clear path to experimental testing due to fundamental theoretical or practical barriers.
but logically
Aspect 2: Novelty.
5 points: This level of novelty could fundamentally alter understanding of chemistry create entirely new fields. It often involves predictions or discoveries that, if proven, would require a significant overhaul of existing chemical theories; points: The hypothesis significantly departs from established norms, potentially redefining how certain chemical phenomena are understood or applied. It might involve entirely new materials or theoretical frameworks; 3 points: This level
19

---

Published as a conference paper at ICLR 2025
involves hypothesis that could potentially lead to new insights applications. It might challenge minor aspects of current theories or introduce new methodologies or materials; 2 points: The hypothesis introduces new angle method within established framework. It might involve known compounds or reactions but in contexts or combinations not previously explored; 1 point: The hypothesis involves minor tweaks applications of well-known principles techniques. It might slightly extend existing knowledge but doesn't 't introduce fundamentally new concepts.
Aspect 3: Significance.
points: This hypothesis could fundamentally change one or more branches of chemistry. might introduce entirely new principles, theories, or methodologies that redefine the boundaries of chemical science; points: This hypothesis challenges current understanding or introduces concept that could lead to substantial changes in how particular area of chemistry is viewed or applied. It might lead to new technologies or significant theoretical advancements; points: this hypothesis proposes something new advancements specific area of chemistry. might open new avenues for research or application but doesn't 't revolutionize the field; 2 points: This hypothesis might offer small variation or incremental improvement on existing knowledge. It could potentially refine a known concept but doesn't significantly alter the field; 1 point: The hypothesis addresses a very narrow already well-established aspect of chemistry. It might confirm what is already known without adding much new insight.
innovative approach that could lead to noticeable
Aspect 4: Potential.
5 points: The hypothesis, while potentially intriguing now, holds the promise of being revolutionary with the addition of key methodological component. This could introduce entirely new concepts or fields, fundamentally changing our understanding or capabilities in chemistry; 4 points: The hypothesis, though promising, could be transformative with the right methodological enhancement. This enhancement might lead to groundbreaking discoveries or applications, significantly advancing the field; points: The hypothesis, while interesting in its current form, could be significantly elevated with the right methodological addition. This might lead to new insights applications that go beyond the initial scope; points: The hypothesis currently offers some value but has the potential for more substantial contributions if enhanced with could lead to incremental advancements in understanding or application; I point: The hypothesis, it stands, might be straightforward or well-trodden. Even with methodological enhancements, it's unlikely to significantly expand current knowledge or applications beyond minor improvements.
methodological approach. This
The hypothesis is:
Please give a response to the initial question on scoring the hypothesis from four aspects. Remember that you are a diligent and harsh reviewer.
A.5 AUTOMATIC EVALUATION BY CLAUDE AND GEMINI
To investigate whether the results and corresponding conclusions in the main text are caused by the usage of GPT-4o for automatic evaluation, here we use Claude-3 Sonnett and Gemini-1.5-Pra Pro evaluate all of the results that have been evaluated by GPT-4o.
Table 12 covers the contents in Table 7 but with more results on using C1 Claude-3. 5-Soneet and Gemini-1. 5-Pro for automatic evaluation. When using different LLMs for automatic eval- uation, the instruction is the same (can be found in $ A2) The robust results indicate again that LLMs are quite capable of associating known knowledge into unknown knowledge that has high probability to be valid (very close to oh).
evaluate the same hypotheses with Table 10 but using Claude- S-Sonnet and Gemini-1e 5-Pro for automatic evaluation correspondingyy (in- stead of GPT- 4o). The results indicate the robustness of MOOSE-Chem and its components.
Table 13 and Table 14
20

---

Published as a conference paper at ICLR 2025
2
Total
background survey
Average MS (GPT-40 40)
18
17
5
0
51
Average MS (Claude-3, S-Sonnet)
19
15
10
3
0
51
Average MS (Gemini-1.5-Pra
13
17
8
11
0
51
Top MS (GPT-4o)
28
19
3
0
0
$1
Top MS (Claude-3. 5--onneett
33
7
10
1
0
0
51
Top MS Gemini-1.2 ProoP
20
18
0
12
1
0
51
Top MS (Experts)
12
22
6
2
51
background survey
Average MS (GPT-40)
19
7
0
Average MS (Claude-3.) 3.eee3..5.oseee 5-Sonnet)
2
0
Average MS (Gemini 1..iii--..
1.5 5-rr))
15
5
Top MS (GPT-4o) Top MS 3 5onnnett)
25
2
19
5
0
0
51
31
19
1
0
0
0
51
Top MS (Gemini-1.5-Pro)
19
19
11
0
51
Table 12: Main table for Q2. Average/ Top MS means the average/highest Matched Score of all gen- crated h from one The numbers represent the statistics of Average/Top MS over the benchmark.
Method
Top MS
Average MS
SciMON Wang
2024b
3.824
3,529
MOOSE Yang 2024a
3.902
3,559
Qi
3.431
3.092
MOOSE-Chin CheSE-Chem
4.471
3.697
multi-step
4.216
3,592
multi-step EU
3.941
3.614
Table 13: Experiments and
ablation study.
The
Matched
Score is evaluated by
Claude-23 5.55-nnnt...
Method
Top MS
Average MS
SciMON WWng
2024b
2.980
2.618
MOOSE Yang 2024a
3.039
2.690
Qi
2.216
1.846
MOOSE- -hhom
3,686
2.443
multi-step
3,588
2.529
multi-stop EU
2.902
2.631
Table Experiments and ablation study, The Matched Score is evaluated by Gemini- 1omini--1.55 -1.5-Pro.
MS threshold
only non-EU branch
only EU branches
only EU-recombination branch
16
46
19
54
24
Table 15: Number of hypotheses receiving high Matched Score (MS) from only non-EU branch, only EU branches, and only EU-recombination branch. Only the hypotheses with a MS that is higher than the MS threshold are counted.
21

---

Published as a conference paper at ICLR 2025
5
w/ significance Teedback
Average MS
4
19
15
Top MS
33
10
w/o significance feedback
Average MS
8
28
Top MS
34
13
0
0
Table 16: Effect of significance feedback (evaluated
Overall
Validness
Novelty
Significance
Potential
Average Rank Ratio
0.65
0.75
0.76
0.73
0.70
Table 17: Average rank ratio (L) of the ground truth hypotheses (mixed with generated hypotheses)
A.6 MORE ANALYSIS ON EU
Table 15 shows the number of hypotheses receiving high Matched Score from only non-EU branch, only EU branches, and only EU-recombination branch. Here, only non-EU branch can be seen as the hypotheses obtained directly without mutations. The hypotheses from the same experiment in Table 10
The result indicates that about one-third of high-quality hypotheses be obtained directly without mutations. In addition, the recombination branch contains more high-quality hypotheses than the only non-EU branch.
A.7 EFFECT OF SIGNIFICANCE FEEDBACK
Table 16 presents ablation study on the significance feedback. The results with significance feedback are from Table I2
The results indicate that not using significance feedback can even lead to better performance in terms of the Matched Score metric. We attribute this phenomenon to LLM'' ability on creativity: when asked to generate significant hypotheses, LLMs tend to be more deviate from the existing information for more possible significance, resulting in lower matched score. However, we should note that the matched score only measures the matching degree of one given ground truth hypothesis, and it possible that the more deviated one is more significant.
A.8 RANKING OF GROUND TRUTH HYPOTHESES
Intuitively if rank the original hypothesis with the generated hypothesis, the original hypothesis may be ranked at the top for most of the time. But is it?
Table shows the result, where we assign each ground truth hypothesis with reward value R(h) (in terms of validness, novelty, significance, and potential), and calculate its average rank ratio regarding the namework-genneater hypotheses.
Surprisingly, the ground truth hypotheses are not ranked to the top. There are three possible reasons:
1. LLM does poorly on ranking hypotheses;
2. The generated hypotheses tend describe their novelty and significance (although they are prompted to not to), which might influence the judgment;
3. The generated hypotheses may surpass the original in quality.
4. The generated hypotheses may sometimes have more details than the ground truth (since the iterative usage of clarity feedback and refinement).
22

---

Published as a conference paper at ICLR 2025
Striet Background
Background Survey (top 20%)
Ratio (top 4%%)
(top 0.8%)
96.7%
83.7%
60.8%
95.1%%
54.2% 77.8%
57.8%
Ablation table background options for Q1. The corpus size is 300. For each screen window of 15 papers, 3 papers are selected.
Table 18:
A.9 INFLUENCE OF RESEARCH BACKGROUND OPTIONS TO INSPIRATION RETRIEVAL
Table 18 shows the ablation study in terms whether to use striet background (discussed in $ or survey not. It indicates that survey can largely help with the inspiration retrieval process. Sur- prisingly, without strict background, the Hit Ratio goes down bit. We attribute it to the reason that mentioning information related the inspiration will discourage retrieving that inspiration, since in the prompt, we ask LLMs to search for inspirations, and the demonstration example indicates that inspirations should not be similar the background (to bring in additional information).
A.10 DISCUSSION ON HALLUCINATION AND SCIENTIFIC DISCOVERY
In contrast to the traditional understanding that ballucination is purely a bad thing, LLM's scientific discovery ability in fact counts on its hallucination ability find novel hypotheses: a novel hypoth- esis would not have been observed by itself, therefore all novel hypotheses come from the class of hallucination.
In essence, the research development of LLMs for automated scientific hypothesis discovery is to develop how to better leverage LLMs to hallucinate an unseen hypothesis that has more possibility to be valid.
A.11 OTHER RELATED WORKS
A.11.1 REASONING
Scientific discovery is highly related reasoning, since it requires of very complex reasoning processes to lead to new discovery.
Inductive reasoning (Yang al. 2024a) is the most relevant reasoning type. It is about finding rules or hypotheses from observations. Scientific discovery is naturally an ultimate goal of inductive reasoning.
Inductive reasoning a sub-reasoning type of logical reasoning. The other two sub-reasoning types are deductive reasoning (Clark et al. 2020) and abductive reasoning (Bhagavatula 2020 Yang et al. (2023b) discuss their definitions and differences in detail.
Another relevant reasoning type commonsense reasoning (Yang al. 2020 2023a Scientific discovery can be seen as an opposite task, which is to reason far outside of commonsense, even to discover unknown knowledge.
A. 11.2 RETRIEVAL
The retrieval of inspiration is retrieval task, and RAG (Lewis et al.) 2020) also works retrieval. The main difference is that the current RAG method would most likely retrieve the information that semantically the most similar to the input information (research background), while here our goal is to retrieve those information that was not known to be related to the input information before, but in fact related. We assume that LLMs might have the ability do it.
A.11.3 SELF CONSISTENCY
Self-consistency Wang et al. 2023 Chen et al. 2023 might have similar looking to the "evolu- tionary unit" (EU), as they all have expansion to several branches, and finally collect these branches into one.
23

---

Published as a conference paper at ICLR 2025
A key difference is that EU is to explore more diverse options to choose the optimal one, while self-consistency is to find consistent voting between options.
A.12 PROMPT TO GPT-40 FOR MATCHED SCORE
You are helping to evaluate the quality of a proposed research hypothesis in Chemistry by a phd student. The ground truth hypothesis will also be provided to compare. Here, mainly focus on whether the proposed hypothesis has covered the key points in terms of the methodology in the ground truth hypothesis. You will also be given summary of the key points in the methodology of the ground truth hypothesis for reference. Please note that for the proposed hypothesis to cover one key point, it is not necessary to explicitly mention the name of the key point, but might also can integrate the key point implicitly in the proposed method. The evaluation criteria is called 'Matched score', which in 6-point Likert scale (from 5 0). Particularly, 5 points mean that the proposed hypothesis (1) covers all the key points and leverage them similarly as in the methodology of the ground truth hypothesis, and (2) does not contain any extra key point that has apparent flaws; points mean that the proposed hypothesis (1) covers all the key points (or at least three key points) and leverage them similarly as in the methodology of the ground truth hypothesis, (2) but also with extra key points that have apparent flaws; 3 points mean that the proposed hypothesis (1) covers at least two key points and leverage them similarly as in the methodology of the ground truth hypothesis, (2) but does not cover all key points in the ground truth hypothesis, (3) might or might not contain extra key points: 2 points mean that the proposed hypothesis (I) covers at least one key point in the methodology of the ground truth hypothesis, and leverage similarly as in the methodology of ground truth hypothesis, (2) but does not cover all key points in the ground truth hypothesis, and (3) might or might not contain extra key points; 1 point means that the proposed hypothesis (1) covers at least one key point in the methodology of the ground truth hypothesis, (2) but is used differently as in the methodology of ground truth hypothesis, and (3) might or might not contain extra key points; O point means that the proposed hypothesis does not cover any key point in the methodology of the ground truth hypothesis at all. Please note that the total number of key points in the ground truth hypothesis might be less than three, so that multiple points can be given. E.g., there's only one key point in the ground truth hypothesis, and the proposed hypothesis covers the one key point, it's possible to give 2 points, 4 points, and points. In this case, should choose score from 4 points and 5 points, depending on the existence and quality of extra key points. "Leveraging key point similarly in the methodology of the ground truth hypothesis' means that in the proposed hypothesis, the same (or very related) concept (key point) is used in similar way with similar goal compared to the ground truth hypothesis (not necessarily for the proposed hypothesis to be exactly the same with the groudtruth hypothesis to be classified as 'similar'). When judging whether an extra key point has apparent flaws, you should use your own knowledge to judge, but rather than to rely on the count number of pieces of extra key point to judge.
Please evaluate the proposed hypothesis based on the ground truth hypothesis. The proposed hypothesis is:
The ground truth hypothesis is:
The key points in the ground truth hypothesis are:
Please evaluate the proposed hypothesis based the ground truth hypothesis, and give score.
A.13 GENERATED HYPOTHESES WITH Low MATCHED SCORE ARE NOT NECESSARILY BAD
MS only measures the similarity between the generated h and the ground truth h. Receiving an MS as 0 or does not mean the generated h is bad. Only real lab experiments can check each h.
A.14 EVALUATION AGREEMENT BETWEEN EXPERT EVALUATION AND GPT-40 EVALUATION
Table 19 shows the agreement between expert evaluation and automatic evaluation (by GPT-4o) on MS. Hard consistency is assigned to 1 only if the two scores are exactly the same, else is assigned to O. Soft consistency is assigned to only if the absolute difference between the two scores is less than 2, else is assigned to O.
24

---

Published as a conference paper at ICLR 2025
#Comparison Pairs
Hard Consistency Score
Soft Consistency Score
392
0.345
0.542
Table 19: Consistency score between expert evaluation and GPT -o evaluation.
#Comparison Pairs
Hard Consistency Score
Soft Consistency Score
48
0,438
0.854
Table 20: Consistency score between experts in expert evaluation.
The results show medium to high consistency between expert evaluation and automatic evaluation. points higher than
The main reason that, in practice, the automatic evaluation usually expert evaluation, since GPT-4o can usually find a way to explain how the generated hypothesis related to the ground truth hypothesis in terms of the main innovations. While this explanation usually is not wrong, the experts might find that compared to the MS given by GPT-4o, the generated hypotheses might not be clear enough to deliver the new innovation information to scientists when using it as copilot advice.
A.15 AGREEMENT BETWEEN EXPERT EVALUATION
Table 20 shows the consistency score between experts. It is conducted by having the third chemistry expert check the randomly sampled expert evaluation from the previous two chemistry experts. Hard consistency is assigned to only if the two scores are exactly the same, else is assigned to 0. Soft consistency is assigned to only if the absolute difference between the two scores less than 2, else is assigned to 0.
The results show high consistency between experts, indicating high reliability of reference-based expert evaluation.
A.16
CASE STUDIES
A.16.1 CASE (MS BY EXPERT: 3)
Ground truth h (Bu et al. 2024): The main hypothesis is that a nitrogen-doped ruthenium (Ru) electrode can effectively catalyze the reductive deuteration of (hetero)arenes the presence of D2O, leading to high deuterium incorporation into the resulting saturated cyclic compounds. The findings validate this hypothesis by demonstrating that this electrocatalytic method is highly effi- cient, scalable, and versatile, suitable for wide range of substrates.
Generated h: pioneering integrated electrocatalytic system leveraging ruthenium nanoparticles embedded in nitrogen-doped graphene, combined with a dual palladium-cone coated ion- ton-exchange mem- brane reactor, will catalyze efficient, scalable, and site-selective reductive deuteration of aromatic hydrocarbons and heteroarenes. Utilizing deuterium sources from both D2 gas and D2O. this sys- tem will optimize parameters through real-time machine learning-drivee dynamic adjustments. Spe- cific configurations include ruthenium nanoparticle sizes (2-4 nm), nitrogen doping levels (12-14%), precisely engineered palladium membranes (5 micrometers, ensuring 98% deuterium-selectin per- meability), and advanced cyclic voltammetry protocols (1-5 Hz, -0.5V to -1.5V).
Expert's ' analysis: The proposed hypothesis effectively covers two key points from the ground truth hypothesis: the incorporation of ruthenium (Ru) and the use of D₂O as a deuterium source within the electrocatalytic system. However, the current content does not detail the mechanism by which Ru-D is produced, which is essential for explaining the process of reductive deuteration. Nevertheless, the results are still insightful. The specific level of nitrogen doping, for example, highly suggestive and warrants further investigation. Overall, the match remains strong in its alignment with the original hypothesis while also presenting opportunities for deeper exploration.
25

---

that are consistent with those of the Published as a conference paper at ICLR 2025
A.16.2 CASE 2 (MS BY EXPERT: 4)
Ground truth h (Wang et al. 2024a): By integrating guanidine sulfate (Gdm)2SO; into poly vinyl alcohol (PVA) hydrogel and employing directional freezing create aligned channels, it is possible to achieve a flexible thermogalvanic armor (FTGA) with a Carnot-relative efficiency exceeding 8% while maintaining high mechanical strength. This integration allows for enhanced thermopower and mechanical robustness, exceeding the performance of traditional quasi-solid thermocells.
Generated h: Leveraging an integrated multi-layer hydrogel-ion assembly, constructed through ad- vanced freete-casting, salting-out processes, and bioinfornatics-driven macromolecular design, will enhance the Carnot-relative efficiency and mechanical robustness of flexible thermogalvanic devices. This approach reimagines the use of poly(viny alcohol) hydrogels and incorporates novel ion- speci-ecific interactions informed by the Hofmeister series with uniquely potent ions like magnesium and lithium. Precision ion concentrations (0.15 to 0.45 M) are optimized to amplisy thermoelectric responses and ionic conductance. Molecular dynamics simulations, employing detailed quantum chemistry models, will validate the enhanced ionic interactions and entropy modulation. Compre- hensive characterization through nanoindentation, electron microscopy. and advanced computa- tional platforms will verify crystalline polymer structures and stratified ionic gel configurations. By setting new benchmarks in efficiency and flexibility, this innovation is poised to revolutionize energy solutions in wearable technologies, outperforming current models in both scalability and applica- tion feasibility.
Expert's Analysis:
The proposed hypothesis effectively covers key points, mirroring the ground truth hypothesis. It in- corporates the Hofmeister series by mentioning "ion-specific interactions informed by the Hofmeis- ter series," which aligns with the ground truth's 's reference hydrogel integration, acknowledging that sulfate ions are a significant component of the Hofmeister series. Additionally, the proposed hypothesis employs "freeze-casting truth's "directional freezing" to create structured channels within the hydrogel. This approach en- hances both mechanical strength and thermopower- ground truth hypothesis.
"guanidine sulfate (Gdm)2SO)
in the
which reflects the ground
Moreover, the inclusion of salting-out processes in the proposed hypothesis does indeed contribute to improving the mechanical properties of thermoelectric materials. However, the selection of mag- nesium and lithium as specific examples problematic. Overall, the proposed hypothesis provides valuable insights and serves as a source of inspiration for further exploration within this domain.
A.16.3 CASE 3 (MS BY EXPERT: 3)
Ground truth h (Shibahara et al. 2024): Rh(l)-catalyzed vinylidene formation and enantioselective control via chiral diphosphine ligands can work together to enable selective [2+2+1] cycloud- dition reaction using three different 27-components. The formation of the vinylidene intermediate ensures reaction selectivity, while the chiral ligands, through their dihedral angles, control the enan- tiomeric outcome. Together, faeses will result in the highly enantioselective synthesis of chiral methylenecccloopen-- I-ene derivatives.
Generated h: An enantioselective [2+2+1] cycloaddition involving cycloalkenes, acetylenecarboxy- lates, and terminal alkynes can achieved using a dynamically tunable Rh(1) pincer complex system, specifically incorporating PNP and PONOP ligands, to control the -ovyn--t-vinnylddene transformation and enhance selectivity towards chiral -melhyleeeeccclopnn---ee l-ene derivatives. By adjusting the P-Rh-P bite angle with novel ligand modifications and utilizing a blended solvent system containing toluene and dichloromethane at controlled temperatures (25cC to 80CC), reaction conditions can be fine-tuned to suppress side reactions. Computational methods (DFT. QTAIM, and NBO) will guide the optimization process, supported by real- monitoring using FTR and NMR. Exploring dual-ligand strategies, including phosphine/phosphinii combinations, will offer insights into previously unreported mechanistic pathways, ensuring a reproducible and scalable synthesis method applicable to pharmacenticals and advanced material synthesis.
Expert's Analysis: The hypothesis generated in Case 2 receives a score of 3, reflecting its accurate identification of the metal and ligand necessary for the catalytic process, as well as its articulation of the mechanism underlying chiral control via ligand geometry. However, the inclusion of mixed
26

---

Published as a conference paper at ICLR 2025
solvents, which were not utilized in the actual study, prevents higher rating. This prediction cessfully asserts the necessity of rhodium metal and a phosphine ligand, while also proposing the significance the the chelating angle between the phosphine ligand and rhodium, in alignment with the hypothesis presented in the referenced article. Furthermore, although the DFT calculations men- tioned in the hypothesis not addressed in the article, they represent promising avenue for future research in organic chemistry.
A.16.4 CASE (MS EXPERT:
Ground truth h (Suzuki et al. 2024): The hypothesis is that phosphorus ylides, when exposed to photoredox catalysis, undergo single-electron oxidation generate radical cations, which gage in radical-polar crossover reactions. These intermediatess can then sequentially form C-C and C=C bonds through formal cycloaddition process, offering powerful method for constructing six-membered carbocycles from simple substrates.
Generated h: A sophisticated dual-modal photoredox strategy synergistically integrates phospho- rus ylides with synchronized dual-wavelength excitation (450 nm and 520 nm) system to incite regioselective ttree-component cycloaddition. This method exploits both net-neutral radical/polar crossover pathways and metal-free carbyne generation to convert inert C-H and C=P bonds into C-C and C=C bonds efficiently. Implemented within an optimized mixed solvent system (ace- tone/ucetomitrile) under ambient nilrogen almosphere, the approach utilizes precise kinetic mod- ulation and catalytic control, with computational validation (employing DFT models) underpinning mechanistic insights and regioselectivity. Experimental verification through in situ spectroscopic techniques (e.g., UV-Vis and NMR) enhances synchronization precision, curtailing side reactions. This methodology pledges substantial advancements in sustainable pharmaceutical synthesis and innovative polymer architecture formation, specifically by improving production efficiency for com- plex molecules and novel materials.
The generated hypothesis also merits a score of 3, as correctly anticipates the use of photocatalysis and highlights the significant influence of solvent on the reaction. However, since dual wavelength catalysis and solvent mixing were not employed in the actual experiment, higher score is not warranted. Notably, despite the proposed mixed solvents not being used in the study, their composition comprises the best rest-performing single solvents from the actual research, thus providing valuable insights that remain relevant to the ongoing investigation.
Expert's Analysis:
27


Reviews:
[{"review_id": "6wqTKd3LtF", "reviewer": "Reviewer_zDBz", "summary": "This paper investigates the potential of LLMs to automatically discover novel and valid hypotheses in the field of chemistry. The study introduces a framework called MOOSE-Chem, which decompose the main research question into three smaller questions, respectively focusing on (1) the identification of inspiration papers; (2) the inference of unknown knowledge that is highly likely to be valid; and (3) the identification and ranking of hypotheses generated by the LLMs.", "questions": "1. There is a lack of experimental exploration for mutation settings.   I am curious about the impact of mutations on the final results. \n     For example, according to the ablation studies, what proportion of high-quality hypotheses can be obtained directly without mutations?\n\n2. How can we explain that a smaller window size leads to better performance in the inspiration retrieval phase?\n\n3. Did the authors replace GPT-4o for all comparison methods for generation? Please specify.\n\n4. How helpful is it to add new dimensions of significance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality:\nFirstly, while LLMs have been utilized for scientific discovery in social science and NLP, this paper is the first to investigate their potential in chemistry. \nBesides, The MOOSE-CHEM framework employs a three-step approach to retrieve inspiration papers, inference valid knowledge, identify hypotheses and rank them, which hasn’t been used in previous research. \nMoreover, the use of the evolutionary algorithm to foster a broader diversity in hypothesis generation is also an innovation point.\n\nQuality:\nThe paper includes extensive comparative experiments and ablation studies, providing a thorough evaluation of LLMs' performance. It also demonstrates the consistency between expert evaluation and automated evaluation of MS, enhancing the reliability of the results.\n\nClarity: \nThe methodology is well-structured, and the three steps based on smaller questions are well-defined. Besides, detailed evaluation methods and corresponding formulas are provided for the three research questions, Q1, Q2 and Q3.  The practical implementation process is also relatively clear.\n\nSignificance:\nThe ability of LLMs to generate hypotheses can significantly accelerate the pace of scientific discovery by reducing the time and effort required for hypothesis generation. It can also open researchers’ minds for the application of LLMs in scientific discovery.", "weaknesses": "Firstly, using the same large language model to evaluate its own generated results may introduce bias. It is recommended to try using different LLMs to evaluate the results so as to guarantee the reliability of the results. For example, consider using models like LLaMa[1], Claude[2], Gemini[3], or other recent LLMs to compare outputs. If using the same LLM is necessary, you could collect hypotheses generated by humans and also have both experts and GPT-4 evaluate them. Then, compare their Hard/Soft Consistency Scores as well as distribution of MS with those of the hypotheses generated by the LLM.\n\n[1] Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint arXiv:2307.09288, 2023.\n[2] Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://www.anthropic.com, 2024.\n[3] Team G, Anil R, Borgeaud S, et al. Gemini: a family of highly capable multimodal models[J]. arXiv preprint arXiv:2312.11805, 2023.\n\nBesides, during the evaluation process (as demonstrated in section 5.1), in the constructed dataset, other papers besides the ground truth inspiration papers may also be helpful for the current background. This step may also require further evaluation to obtain more reliable results. You might select several high-ranking papers that are not included in set I, particularly those that repeatedly appear in experiments with varying corpus sizes or screen window sizes, and conduct case studies on them.\n\nAdditionally, this paper only employs the titles and abstracts for retrieval, which might overlook inspirations that could arise from the detailed content of the articles. Sections like Conclusion or Discussion often contain insights into the paper's limitations and suggestions for future work. These elements are usually crucial for generating inspiration.\n\nMoreover, the benchmarks mentioned in the paper contain only 51 references, which may not be sufficient to fully assess the effectiveness of the proposed method. You are advised to increase the number of samples, especially selecting literature from different chemical fields. In addition to traditional categories such as Analytical Chemistry, Organic Chemistry, and Inorganic Chemistry, broader research topics like Environmental Chemistry, Medicinal Chemistry, and Nuclear Chemistry can also be considered. Furthermore, Nature Chemistry includes interdisciplinary areas such as Bioinorganic Chemistry, Bioorganic Chemistry, and Organometallic Chemistry, which could be investigated as well. Of course, this may involve considerations related to the resource consumption of LLMs.\n\nFinally, although the paper used chemistry PhD students for the evaluation, there is still subjectivity that may affect the reliability of the evaluation results. You might consider inviting more experts to participate in the evaluation and analyzing the consistency of the assessments from different human individuals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GJxAuqP0OV", "reviewer": "Reviewer_rMit", "summary": "This work investigates whether large language models (LLMs) can autonomously generate novel and valid hypotheses in chemistry based solely on a research background. Specifically, the study explores if LLMs, when provided with a research question or background survey, can independently identify inspirations, synthesize hypotheses, and evaluate their quality. Building on discussions with chemistry experts, the authors hypothesize that most chemistry hypotheses can be derived from combining research background with relevant inspirations. They break down the central question into three key tasks: retrieving valuable inspirations, generating hypotheses, and ranking the quality of these hypotheses. To test this approach, the authors create a benchmark of 51 high-impact chemistry papers published in 2024. LLMs, trained only on data up to 2023, attempt to rediscover these hypotheses from the background and an extensive chemistry literature corpus. The results, achieved through a multi-agent LLM framework, show promising success, with many rediscovered hypotheses closely matching the originals and capturing core innovations.", "questions": "See the weaknesses listed above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tGenerating research hypotheses is a complicated task, and the authors heuristically decomposed hypothesis generation into two steps: \n\n(1). inspiration retrieval, and \n\n(2). hypothesis refinement. In the hypothesis refinement step, the authors propose a novel “mutate and recombine” trick to help generate good hypotheses.\n\n2.\tThe experiments to verify each of the research questions are well-designed with good quality.", "weaknesses": "1.\tThe introduction section could be written better and more clear. (a) It would be great if the authors could provide a summary of the major contributions of this work at the end of the introduction section. What are really the contribution to the field? (b) It would be great if the authors could briefly discuss why the decomposition of the major question is necessary, what’s the difference or connection between the proposed inspiration identification (the first step of the three) and Retrieval Augmented Generation (RAG).\n\n2.\tAs the end goal is to rediscover the chemistry scientific hypotheses, the upper bound of “rediscovery” is the exact match of the original hypothesis, intuitively if we rank the original hypothesis with the generated hypothesis, the original hypothesis may be ranked at the top for most of the time. Experiment on this and analysis about the cases where the original hypothesis is not ranked at the top would be interesting to better reflect LLM’s ability to perform R(h).\n\n3. Although the proposed approach constructed a benchmark as the basis to evaluate the performance, it is unclear to what degree the generated hypotheses can be trusted or reliable, not made up from baseless hallucination.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pojsHiILGK", "reviewer": "Reviewer_TMW5", "summary": "This paper introduces a framework for scientific hypothesis discovery\nand generation using large language models (LLMs). The authors gathered\na carefully curated dataset of scientific papers where questions and hypotheses were extracted for evaluation. The authors propose \"MOOSE-CHEM\", a\nmethod based on previous work \"MOOSE\" but with modifications\nto adapt it for chemistry literature and that improve Overall\nperformance (refinement, mutation). The framework is also formally motivated.\nThe proposed approach is evaluated in each of the main three tasks\nthat compose it: P (i|b), P (h|b, i), and R(h). Overall the results show great\nperformance in the tasks with GPT-4o, with evaluations that include both \nexperts and GPT-4o, and where various hyperparameters are tested (e.g. corpus size).", "questions": "This includes a list of minor issues, questions and suggestions:\n\n- Through the paper, it is mentioned that the inspiration shouldn't be related\nto the given background. Is this enforced in the method, or is this verified\nsomehow?\n\n- The proposed method of Evolutionary Algorithm (EA) sounds to be related\nto other methods in LLMs that try to improve the consistency/quality of\nthe answers, e.g. \n  - Self-Consistency (SC) (Wang et al., \"Self-Consistency Improves Chain of Thought Reasoning in Language Models\")\n  - Universal SC (Chen et al. \"Universal Self-Consistency for Large Language Model Generation\")\n  - Multiple Chain of Thought (MCT), Self reflection and/or self-correction approaches, etc. I thus believe that some related work could be included regarding this algorithm in the paper.\n\n\n- On line 351, claim that \"is possible to be true\" that LLMs are able to create\nnovel knowledge. While the authors are correct here since it is not clear\nwhether this is true or not, could the authors provide other explanations\non why the LLMs are able to correctly find inspiration papers?\n\n\n- Please if possible provide some description of how P (i|b), P (h|b, i), and R(h)\ninputs and outputs look like. From the example provided, I can guess that\nhypothesis is one paragraph long, but it would be great for the reader \nto have this stated clearly.\n\n- I honestly don't fully understand the experimental setting reported in section 5.1.\nWhat are the \"ground truth inspiration papers\" and how are these related to \nthe 51 papers from the benchmark? In Table 3, does this mean that when a corpus size\nof 150 is used, 120 papers are considered ground truth? \n\n- From Table 3, it almost seems like corpus size doesn't have much effect on the Hit ratio,\nis this correct? If so, could you elaborate more in this regard?\n\n- If I got the idea right, in the results of Table 1, if a screen window size is used, the corpus is 300 and 3 papers are selected for each window, then round 1 has (300/10)*3 = 90, round 2 has\n(90/10)*3 = 27, and so on, is this correct? It would be more clarifying to include the amount of papers in each round in the table or in a separate table. It would be also clarifying \nto include a baseline comparison of the performance based on pure random screening (this is, select papers randomly).\n\n- From the results reported in Table 7, it seems that the background survey (almost) doesn't have any \neffect on MS, however, this is somehow counterintuitive. Shouldn't the background survey improve performance as it happens in Q1?\n\n- I would suggest the authors format the rounds in Table 4 to go along\nwith the direction of the text, this is, from left to right.\n\n- Could you please clarify why there are NAs in Tables 3 and 4?\n\n- On line 134: \"contrusted by multiple chemistry PhD students\".\nFor clarity, it would be great to know how many students.\n\n- While it is not mandatory, given the stochasticity and prone-to-hallucinations nature of LLMs,\nit is a good practice to use countermeasures to reduce hallucinations, e.g. sampling multiple times with majority voting,\nor using different models for comparison. This is not an issue, but it would be\ninteresting to see how much variability the results reported in the experiments\nare or if the method is robust to hallucinations.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is generally well-written, and in good English. The text is clear and\nthe authors did a good job guiding the reader through the motivation,\nthe derivation of the method and motivating each of the proposed steps\nand experiments. The topic of the paper is very relevant and the results\nare positive. Related work is well covered, and experiments are included\nthat compare the proposed method with previous work. Every claim made \non the performance of the method is generally backed up with experiments.\nI think the paper is generally a great and novel contribution to the field.", "weaknesses": "My main concerns with the paper are regarding the reproducibility,\nclarity and discussion of the approach:\n\n- Reproducibility: we note that the authors introduce a scientific benchmark,\nalong with a novel framework for hypothesis generation. However,\nthe authors do not provide access to the novel-introduced benchmark,\nwhich hampers the ability to really discriminate the difficulty of the tasks\nat hand. Additionally, this impedes the ability to reproduce the results or\nfor future work to compare the performance with the proposed method.\nI would suggest the authors to make the benchmark data available if possible.\nSimilarly, no access to the source code is provided. While available source code is not\na requirement, for a method that relies heavily on LLMs prompting, it would\nbe very beneficial to at least have access to the prompts used in the\nwork, just for the sake of reproducibility.\n\n- (Related to the previous issue): There are parts of the paper that lack\na formal definition, which again, hampers the \nability to reproduce the work. For instance, the authors mention that they\nintroduce a novel \"Evolutionary Algorithm (EA)\" that mutates, refines and\nrecombines hypotheses, however, it is unclear how the mechanisms of\nmutation, refinement and recombination are implemented. I would suggest \nthe authors to provide, when possible more technical details on the\nproposed method.\n\n- While the work includes a fair comparison with previous work, the work\ndoesn't include any discussion on the limitations of the proposed method,\nnor any future work. It would be very much appreciated if the authors\ncould provide a discussion on the limitations of the proposed method\nto guide potential future work in the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hwdijx9eT7", "reviewer": "Reviewer_km2P", "summary": "This paper explores the potential of Large Language Models (LLMs) to generate novel, valid hypotheses in the chemistry domain by employing a multi-agent framework. It targets the rediscovery of scientific hypotheses using LLMs based on recent high-impact chemistry publications (since 2024), using a framework that separates tasks into three stages: (1) finding \"inspiration\" papers from a predefined literature corpus, (2) generating hypotheses by associating these inspirations with a background question, and (3) ranking hypotheses based on quality. To solve this, the authors proposed a multi-agent framework existing framework with an evolutionary algorithm to iteratively mutate hypotheses based on reward feedback (validness, novelty, clarity, and significance).", "questions": "1. Can the authors clarify how/whether MOOSE-Chem substantially differs from previous work in multi-agent scientific discovery systems? \n2. How would the authors envision MOOSE-Chem supporting a chemist in hypothesis generation beyond rediscovering existing knowledge? Have any of the generated hypotheses been practically tested, or do they align with ongoing chemistry research needs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The use of LLMs for scientific hypothesis generation is a growing area, and extending this work to complex fields like chemistry holds practical importance if achieved meaningfully.\n2. The use of beam search and evolutionary algorithms for hypothesis refinement seems simple and powerful, as it systematically explores multiple possibilities.\n3. The framework does not require expensive training or fine-tuning.", "weaknesses": "1. There is a key fundamental assumption that the authors did not discuss here: reliance on general-purpose LLMs containing implicit domain knowledge. While this may be reasonable for powerful, large-scale models like GPT-4 or domains like social sciences, it is overly strong and may not apply to smaller or open-source models in this case. A deeper analysis (maybe more experiments) of how this assumption impacts performance across other LLMs (apart from GPT-4 or closed-source ones) is needed.\n2. The assumption that LLMs have sufficient knowledge to connect novel and unrecognized scientific relations without any domain-specific tuning seems too bold. The model's effectiveness may degrade in more niche areas or specific research questions.\n3. Much of this framework — multi-agent, hypothesis scoring, and the background-inspiration format closely resembles existing work in this area. Given the limited scale of benchmarking and experiments in this paper, the technical contribution seems limited for this venue.\n3. The selection of inspiration papers seems unclear and might not align with typical challenges or open research questions in chemistry. A more detailed explanation of the selection process of those 51 papers and corresponding inspiration papers, including how relevance, significance, and novelty are ensured, is needed.\n3. The paper presentation could be improved, particularly in the introduction and methodology. The technical details regarding probabilistic inference seem unnecessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)