Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses

## Abstract

Abstract Scientific discovery contributes largely to human society’s prosperity, and recent progress shows that LLMs could potentially catalyze this process. However, it is still unclear whether LLMs can discover novel and valid hypotheses in chemistry.
In this work, we investigate this central research question: Can LLMs automatically discover novel and valid chemistry research hypotheses given only a chemistry research background (consisting of a research question and/or a background survey), without limitation on the domain of the research question?
After extensive discussions with chemistry experts, we propose an assumption that a majority of chemistry hypotheses can be resulted from a research background and several inspirations.
With this key insight, we break the central question into three smaller fundamental questions.
In brief, they are:
(1) given a background question, whether LLMs can retrieve good inspirations;
(2) with background and inspirations, whether LLMs can lead to hypothesis; and
(3) whether LLMs can identify good hypotheses to rank them higher.
To investigate these questions, we construct a benchmark consisting of 51 chemistry papers published in Nature, Science, or a similar level in 2024 (all papers are only available online since 2024).
Every paper is divided by chemistry PhD students into three components: background, inspirations, and hypothesis.
The goal is to rediscover the hypothesis, given only the background and a large randomly selected chemistry literature corpus consisting the ground truth inspiration papers, with LLMs trained with data up to 2023.
We also develop an LLM-based multi-agent framework 1 1 1 Code and Benchmark are available at https://github.com/ZonglinY/MOOSE-Chem.git that leverages the assumption, consisting of three stages reflecting the three smaller questions.
The proposed method can rediscover many hypotheses with very high similarity with the ground truth ones, covering the main innovations. † † † : Corresponding author

## 1 Introduction

Discovering new science has long been one of the deepest desires of humanity, which can not only satisfy our curiosity to understand the universe, but also contribute largely to the prosperity of human society (Coccia, ).
Recently, there are some breakthroughs indicating that LLMs have the potential to assist scientists in accelerating the discovery process.

Yang et al. () first find that LLMs can generate novel and valid enough hypotheses evaluated by experts. They focus on the social science domain and make discoveries by developing a multi-agent system, leveraging an assumption that a majority of social science hypotheses can be divided into a research background concept and an inspiration concept.
This assumption is largely valid, because social science hypothesis is about how an independent variable can influence another dependent variable (Hair et al., ).
Si et al. () further validate this finding by employing a large group of scientists to evaluate LLMs’ generated hypotheses in the NLP domain, and show that LLM can generate more novel but slightly less valid research hypotheses than human researchers.
However, it is still unclear LLMs’ scientific discovery ability in natural science such as the chemistry domain.

Sprueill et al. (; ) adopt LLMs to conduct a search process for catalyst discovery.
However, their method is limited in the catalyst discovery domain, and their evaluation relies on whether LLMs can rediscover existing commercially used catalysts, potentially influenced by a data contamination problem.
As a result, it is still unclear how good LLMs are for chemistry scientific discovery.

In this paper, we investigate this central research question:
Can LLMs automatically discover novel and valid chemistry research hypotheses (even in the Nature level) given only a chemistry research background (consisting of a research question and/or a background survey), without limitation on the domain of the research question?
With extensive discussions with chemistry experts, we find that the assumption used in social science, that a hypothesis can be divided into background and inspiration, can also apply to a majority of chemistry hypotheses.
It is not too surprising, since cognitive science research has shown that creative ideas often result from the cohesive association of two seemingly unrelated pieces of knowledge (Koestler, ; Benedek et al., ; Lee & Chung, ).
A main difference is that chemistry might need more than one inspiration (e.g., adding several components to compose a novel chemistry system).
With this key insight, we break the seemingly-impossible-to-solve central question into three smaller, more practical and executable fundamental questions that, when summed up, should be very close to a set of sufficient conditions for the central question.
Specifically, the smaller questions are
(1) whether LLM can identify inspiration papers that have the potential to help with the given research question;
(2) given only known knowledge (from background and inspirations), whether LLMs can infer unknown knowledge that is highly likely to be valid; and
(3) whether LLM can identify good hypotheses and rank them higher.

Figure: Figure 1: Overview of the input and output of the MOOSE-Chem framework.
Refer to caption: x1.png

To investigate these three questions, we build a benchmark consisting of 51 chemistry papers annotated by chemistry PhD students, breaking every paper into a background, several inspirations, and a hypothesis.
The goal is to rediscover the hypothesis with only the background by using LLMs trained with data up to December 2023.
The papers are all published in Nature, Science, or a similar level in 2024, and they are only made public on internet in 2024.
The benchmark is designed to be similar to the Mathematical Olympiad Competition (Trinh et al., ), to provide several dozens of very difficult and meaningful questions to solve.
Along with the benchmark, we propose a ranking task for scientific discovery (along with evaluation criteria), which has been largely overlooked in previous works (Yang et al., ; Wang et al., ).
Ranking is important because although AI systems can generate a large number of hypotheses in a relatively short time, verifying them one by one requires a lot of experimental costs.

Motivated by this breakup into three smaller questions, we design a multi-agent framework named MOOSE-Chem for chemistry scientific discovery.
It in general includes three stages:
(1) searching through chemistry literature to find inspiration papers,
(2) leveraging the inspirations to propose hypotheses for the background research question, and
(3) identifying high-quality hypotheses to give them a higher rank.
Compared with Yang et al. ()’s method in social science that assumes a similar separation between background and inspiration for hypothesis formulation, MOOSE-Chem adopts an evolutionary algorithm to foster a broader diversity of approaches in using inspiration for background, thereby capitalizing on the benefits derived from varied mutations.
In addition, MOOSE-Chem also adopts a multi-step design to collect more than one inspirations for chemistry discovery.
Finally, it uses an efficient ranking method for better reference for scientists.

We design experiments with the benchmark to test the three fundamental questions, and find that LLMs are highly capable.
We also test MOOSE-Chem with the benchmark, mimicking the setting to run it in the wild by only giving a background and a corpus of up to 3000 chemistry papers to select inspiration.
Even in this challenging setting, MOOSE-Chem can still rediscover many hypotheses with very high similarity with the ground truth ones, covering the main innovations.

## 2 Related Work

Zhong et al. () work on finding the difference between two corpora to propose hypotheses, but their evaluation is conducted by Turkers, which cannot lead to a novel discovery.
Wang et al. () try to utilize LLMs to discover novel NLP and biochemical hypotheses, and find the hypotheses still fall far
behind scientific papers in terms of novelty, depth, and utility.
Yang et al. () first show that LLMs can generate novel and valid enough hypotheses evaluated by PhD students, but they only focus on social science.
FunSearch (Romera-Paredes et al., ) can discover specific solutions for mathematical conjecture but can’t discover new math theorems.
Qi et al. () analyzes LLM’s ability for scientific discovery in the biomedical domain by directly generating hypotheses with only the research background.
Boiko et al. (); Baek et al. (); Li et al. (); Lu et al. () focus on subsequent steps for scientific discovery, mainly developing and conducting experiments.
Sprueill et al. (; ) focus on catalyst discovery, but their evaluation relies on whether can rediscover existing commercially used catalysts, which might cause data contamination problem.
Kumar et al. () compare different LLMs on scientific discovery on different disciplines.
Tshitoyan et al. () show that word embedding obtained from large-scale chemistry literature can recommend materials for functional applications years before their discovery by controlling the date of the training corpus.
(Xie et al., ) predict emerging thermoelectric materials by summarizing the sentiment in the existing literature.

## 3 Benchmark Construction

The goal of the benchmark, named TOMATO-Chem, is two-fold.
Firstly, it is used to analyze LLM’s ability in terms of the three smaller questions.
Secondly, it serves as a challenge to rediscover nature-level chemistry hypotheses with only a research background.
The setting of the challenge is very similar to a real copilot setting, where scientists tell the copilot about the specific research question they are interested in, and optionally a small survey consisting of several paragraphs summarizing the existing best-performing methods for the research question.

To achieve the goals, we split each collected paper into the following components:
$<$background question, background question (strict), background survey, background survey (strict), one to three inspiration paper titles and their reason to serve as an inspiration, research hypothesis, experiments, reasoning process, summarization of inspirations$>$.
Every component is described by text.

The reason we add a strict version for background question and background survey is that many hypotheses are making relatively minor modifications based on existing methods covered by the survey, and the question can be very insightful to provide a hint on the general direction of the hypothesis.
In practice, these situations are entirely possible, especially when the scientist users can provide a more comprehensive survey on existing methods, or contain deep insights in their question.
Here we also keep the strict version to make the task more challenging, and encourage developing methods to better assist scientists even when they are also new to their research topic.

The reasoning process indicates the relation between the components of background, inspirations, and hypothesis.
For example, the reasoning process can be “background + inspiration 1 + inspiration 2 = hypothesis”, or “background + inspiration 1/inspiration 2 + inspiration 3 = hypothesis”.

The benchmark consists of 51 chemistry and material science papers, and is constructed by multiple chemistry PhD students.
We only select those papers published on top chemistry venues and be public on the internet after January 2024.
After constructing, the experts check again on
(1) whether the identification of the inspirations is correct and whether more inspirations are needed;
(2) whether the background does not contain any information in inspirations or hypothesis; and
(3) whether the background and the identified inspirations can roughly logically lead to the hypothesis.
The complete instruction on the check process is shown in § A.2.

**Table 1: Distribution of categories.**
| Category | Count |
| --- | --- |
| Polymer Chemistry | 21 |
| Organic Chemistry | 22 |
| Inorganic Chemistry | 3 |
| Analytical Chemistry | 5 |
| Total | 51 |

Table 2 and Table 2 show the statistics of the benchmark in terms of chemistry category and publication venue.
Material science is a sub-category of chemistry and can belong to the categories in Table 2, such as polymer material and organic material.
Around 13 collected benchmark papers are inside the material science domain.
Beyond them, more papers have intersections with material science.
In this paper, we target both chemistry and material science, but for simplicity, we only refer to them as chemistry in this paper.

## 4 Methodology

### 4.1 Fundamental Assumption and Following Decomposition

We propose an assumption that a majority of chemistry hypotheses can originate from a research background and several inspirations.
This assumption is not only supported by many chemistry researchers whom we have extensive discussions with but also by the cognitive science finding that “creative ideas often result from the cohesive association of two (or more) seemingly unrelated pieces of knowledge” (Koestler, ; Benedek et al., ; Lee & Chung, ).
We design our method based on this fundamental assumption.

This assumption is reminiscent of Swanson Linking (Swanson, ) in the domain of literature-based discovery (LBD), also known as the “ABC model”, where two concepts A and C are hypothesized as linked if they both co-occur with some intermediate concept B in papers.
Our assumption differs in that:
(1) for a chemistry hypothesis published in a good venue, usually more than one inspirations are needed;
(2) background and inspiration are not necessarily linked by a path of intermediate papers;
(3) our assumption is applied to a majority of existing published chemistry hypotheses, while LBD has been considered to only focus on a very specific, narrow type of
hypothesis (Wang et al., ).
It might indicate that a similar proportion of future chemistry hypotheses can also be resulted from linkages of existing literature.

Denoting background knowledge as $b$, inspiration knowledge as $i$, and hypothesis as $h$, we translate this assumption as:

$$ $h=f(b,i_{1},\ldots,i_{k})$ (1) $$

Here $k\in\mathbb{Z}$ represents the number of inspirations needed for a particular $h$. Typically in chemistry, $k\in[1,3]$.

In other words, given existing knowledge in the background, a majority of chemistry research is about searching knowledge that previously *not known* to be related to the background but in fact *can assist* the background, then associate the background knowledge and the searched knowledge in a reasonable way to compose a hypothesis.
For example, the proposal of backpropagation can be seen as a hypothesis.
In this case, the background knowledge is multi-layer logistic regression, and the searched knowledge is the chain rule in calculus.

Here, we call the searched knowledge as “inspiration”.
It is vital that the inspiration should not be known to be related to the background before, or at least should not be used to associate with the background in a known way.
Otherwise the hypothesis would not be novel.

Our goal is to transform the seemingly impossible-to-solve $P(h|b)$ into an equivalent form, where each step in the equivalent form is practical and executable.
A straightforward way of decomposing $P(h|b)$ is by the chain rule based on Equation 1:

$$ $P(h|b)=\begin{cases}P(h|b,i_{1})\cdot P(i_{1}|b,I)&\text{if }k=1\\ P(h|b,i_{1},\ldots,i_{k})\cdot\prod_{j=2}^{k}P(i_{j}|b,i_{1},\ldots,i_{j-1},I) \cdot P(i_{1}|b,I)&\text{if }k>1\end{cases}$ (2) $$

Here $I$ is the full inspiration space to search for every single $i$ (here we use the existing chemistry literature, containing up to 3000 papers as $I$).
The order of $i_{j}$ is exchangeable.

Equation 2 describes the process of $P(h|b)$ in terms of the knowledge-searching perspective.
However,
$P(h|b,i_{1},\ldots,i_{k})$ and $P(i_{j}|b,i_{1},\ldots,i_{j-1},I)$
might not be enough practicable, and do not precisely reflect how chemistry researchers find a new $i$.
One of the main reasons is that researchers tend to think small step by small step.
It would be very challenging to think in terms of a big step without breaking it into several small steps.
As a result, they do it similarly but propose a new updated $h$ every time they find a new $i$.
A newer $i$ is searched by the updated $h$, which could contain the key information of all previous $i$, together with $b$.
So we can rewrite Equation 2 as ($k>=1$):

$$ $\displaystyle P(h|b)\approx\prod_{j=1}^{k}P(i_{j}|b,h_{j-1},I)\cdot P(h_{j}|b, i_{j},h_{j-1}),\text{ where }h_{0}=\emptyset$ (3) $$

The full derivation is shown in § A.1.
Equation 3 is meaningful in that it successfully breaks up the seemingly impossible question $P(h|b)$ into many more practical and executable smaller questions.

Now it is clear on the steps to obtain $h$ from $b$.
But it still might not be enough helpful in practice, since $I$ can be on a large scale, and the search process might find lots of $i$, and finally lead to lots of $h$.
Moreover, it is very time-consuming for scientists to conduct experiments to verify every single $h$.
Therefore, it would be very helpful if the generated $h$ could be ranked based on quality.

Here we adopt a straightforward and efficient way for ranking. Specifically, we design a rating function $R(h)$, such that $R(h)\rightarrow\mathbb{R}$.
Denoting the full set of generated $h$ as $H$, we can obtain

$$ $P(H_{\text{ranked}})=P(H,R),\text{ where }H_{\text{ranked}}=\{h_{1},h_{2}, \ldots,h_{n}\mid R(h_{i})\geq R(h_{i+1})\text{ for all }i\}$ (4) $$

Supported by Equation 3 and Equation 4, as a result, to model $P(h|b)$, the only three components we need to model are $P(i_{j}|b,h_{j-1},I)$, $P(h_{j}|b,i_{j},h_{j-1})$, and $R(h)$.
The implementation details of the three components are illustrated in the remaining subsections in § 4.
Analyses on LLM’s ability on the three components are provided in § 5.

### 4.2 The Framework Developed Based on the Assumption

#### 4.2.1 The General Picture

Our methodology is developed based on the fundamental assumption discussed in § 4.1.
Specifically, we use LLMs to perform $P(i_{j}|b,h_{j-1},I)$, $P(h_{j}|b,i_{j},h_{j-1})$, and $R(h)$, and organize them into a multi-agent LLM-based framework.
The input to the framework is only a background question and/or background survey, together with a (large) chemistry literature corpus to search for inspiration.
The output of the framework is a list of ranked research hypothesis.

Figure: Figure 2: The MOOSE-Chem framework. It receives $b$ and $I$ as input, and outputs a list of ranked $h$. The bottom-right legend describes the symbols in the figure.
Refer to caption: x2.png

The MOOSE-Chem framework is shown in Figure 2.
It is a direct implementation of Equation 3 and 4.
We try to develop it as simply as possible, only keeping the necessary parts.

In the general picture, given a research background $b$ (research question and/or research survey), the framework first performs $P(i_{1}|b,h_{0}=\emptyset,I)$ by screening through the literature corpus $I$ to select many papers $i$, where each of them has the potential to serve as an inspiration.
Then the framework performs $P(h_{1}|b,i_{1},h_{0}=\emptyset)$, associating $b$ and each $i$ together to compose $h$.
Then, it ranks $h$ by assigning an evaluation score $r$ on each of $h_{1}$ by $R(h_{1})$.
We call these three steps as one round.
Another round means going through the three steps again, based on the previous round’s results.

Since normally in chemistry, no more than three inspirations are needed for one hypothesis ($k\in[1,3]$), the default setting for MOOSE-Chem is to perform three rounds for each $b$.
In every other round, the number of $i$ and $h$ can expand exponentially. Here, we adopt beam search to select a fixed size of the top-ranked $h$ to enter the next round.
The default beam size is 15.

#### 4.2.2 Design Details of P ⁢ ( i j | b , h j − 1 , I ) 𝑃 conditional subscript 𝑖 𝑗 𝑏 subscript ℎ 𝑗 1 𝐼 P(i_{j}|b,h_{j-1},I) italic_P ( italic_i start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_b , italic_h start_POSTSUBSCRIPT italic_j - 1 end_POSTSUBSCRIPT , italic_I ) And Its Motivation

We use LLMs to conduct a screening process for $P(i_{j}|b,h_{j-1},I)$.
Specifically, for each inference, we
(1) sequentially select a fixed number of papers from $I$, where the fixed number is called the screening window size (default is 15);
(2) set up a prompt consisting of $b$, the title and abstract of the selected papers from $I$, and the previous $h$ (if it is not $\emptyset$); and
(3) instruct the LLM to generate three titles from the input that can best serve as $i$ for $b$ (and optionally previous $h$), and give reasons.

Here we use LLMs to choose potential inspiration $i$, but not choose $i$ from citation nor semantic neighbors because $i$ is supposed to be previously not known to be related to $b$ (we have discussed it in § 4.1).
If the chosen $i$ is already known to be related to $b$, then the composed $h$ probably would not be novel.
If the chosen $i$ contains similar semantic information with $b$, then probably it is not necessary to add $i$ at all, since it does not introduce much (any) extra information.

Here we have a bold assumption that the most advanced LLMs, after training on hundreds of millions of scientific literature, might already know many knowledge pairs that can be associated to create novel knowledge, where the knowledge pairs are not known by any scientist to be related.
It might not be too bold, since Tshitoyan et al. () have shown that word embedding obtained from unsupervised learning on 3.3 million material science publication abstracts can recommend materials for functional applications several years before their discovery.
Here, the functional applications can be seen as $b$, and the recommended materials can be seen as $i$, or even directly as $h$ if it is enough similar.
It probably indicates that LLMs trained with significantly more literature tokens and significantly more parameters might already be able to identify the relation between many knowledge pairs that are unknown to be related by any scientist.
We analyze this assumption in § 5.1.

#### 4.2.3 Design Details of P ⁢ ( h j | b , i j , h j − 1 ) 𝑃 conditional subscript ℎ 𝑗 𝑏 subscript 𝑖 𝑗 subscript ℎ 𝑗 1 P(h_{j}|b,i_{j},h_{j-1}) italic_P ( italic_h start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_b , italic_i start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_j - 1 end_POSTSUBSCRIPT ) And Its Motivation

The retrieved $i$ is expected to be not known to be related to $b$; therefore, it might be difficult to figure out an effective way to associate $b$ and $i$ together to compose $h$.
Think of the time when backpropagation is about to be invented. Even if we are very familiar with $b$ (multi-layer logistic regression) and have successfully retrieved $i$ (chain rule in mathematics), can we invent backpropagation?

Our answer is, at least we might need to try multiple times and various ways to leverage the chain rule for multi-layer logistic regression.
With this motivation, we develop a simple evolutionary algorithm based method, shown in the top-right of Figure 2.
We call it “evolutionary unit” (EU).

Specifically, given $b$ and $i$, EU will first generate multiple hypothesis “mutations” $m$, where each $m$ is a unique way to associate $b$ and $i$ together.
Then EU further develops each $m$ independently by providing feedback to each $m$ in terms of validness, novelty, clarity, and significance, and then refine them based on the feedback.
Yang et al. () first propose to provide feedback in terms of validness, novelty, and clarity to refine hypotheses.
Here we add an additional aspect, significance, since significance is an important evaluation criterion in chemistry.
We assume the refined hypothesis should be in better quality, so that the refined hypothesis is “selected”, while the previous hypothesis is “eliminated” by the “environment”.
Finally EU “recombines” the remaining selected $m$, leveraging the advantages from every $m$ to propose $h$ to better associate $b$ and $i$.

#### 4.2.4 Design Details of R ⁢ ( h ) 𝑅 ℎ R(h) italic_R ( italic_h ) And Its Motivation

We adopt a simple and efficient way for $R(h)$, which is to prompt an LLM to output evaluation scores for an input $h$ in terms of validness, novelty, significance, and potential.
Validness and novelty are two fundamental requirements for such an inductive reasoning process as scientific discovery (Yang et al., ; ).
Significance is added because it is important for chemistry.
We additionally add potential, because the generated $h$ are about to be further developed by scientists, so we might want to pick those $h$ not only is currently in high quality, but also have good potential to be further developed.
We didn’t design $R(h)$ in a more complicated way, since there are lots of $h$ to rank, and we might want to save more inference time.

Yang et al. () use the scores as automatic evaluation for generated social science hypotheses and have shown a high consistency score between automatic evaluation and expert evaluation.
However, in the chemistry domain, LLMs might not be reliable enough to directly evaluate the generated $h$ (Sprueill et al., ).
But it might still be able to provide a preliminary quality identifier to $h$: the ranking of the average score between the four aspects of an $h$ determines whether it will enter the next round of MOOSE-Chem by beam search.
To understand how well LLMs can perform $R(h)$, we analyze “how well LLMs can rank chemistry hypotheses” in § 5.3.

## 5 Investigation on Fundamental Questions

$P(h|b)$ can be understood as the task to discover high-quality chemistry research hypothesis, given only a background question and/or background survey.
Our central question to investigate is how well LLMs can perform $P(h|b)$.
Supported by Equation 3 and 4, we break up this main question into three smaller questions:
how well can LLMs perform (1) $P(i_{j}|b,h_{j-1},I)$, (2) $P(h_{j}|b,i_{j},h_{j-1})$, and (3) $R(h)$?
All experiments are performed by GPT-4o (its training data is up to October 2023).

### 5.1 How Well can LLMs perform P ⁢ ( i j | b , h j − 1 , I ) 𝑃 conditional subscript 𝑖 𝑗 𝑏 subscript ℎ 𝑗 1 𝐼 P(i_{j}|b,h_{j-1},I) italic_P ( italic_i start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_b , italic_h start_POSTSUBSCRIPT italic_j - 1 end_POSTSUBSCRIPT , italic_I ) ?

Here we investigate the question (denoted as $Q1$): “whether LLM can identify inspiration papers which *are unknown* to be able to associate with the background (or at least unknown to associate in a certain way) but in fact can associate with the background to create novel knowledge?”.

We first find 3000 most cited chemistry papers published in Nature, and construct a series of $I$ in size of 150, 300, 1000, and 3000.
$I$ is constructed by first adding the ground truth inspiration papers (around 120), then randomly selecting the remaining needed papers from the 3000 papers, and finally randomizing the order of all the collected papers.
Only title and abstract are needed for each paper in $I$.
The default setting is that each inference of LLMs will screen 15 papers from $I$, and generate three titles that LLMs think can best assist $b$ (and/or previous $h$).
Screening through $I$ for one round, only 20% of $I$ will be selected. Screening another round will only left 4%, and so on.

We use Hit Ratio as evaluation metric, which is calculated by the number of selected ground truth inspiration papers divided by the number of all ground truth inspirations papers.
All the Hit Ratio numbers shown in the tables are averaged across the 51 papers in the benchmark.

Table 3 shows the main experiment results. The Hit Ratio is surprisingly high: More than 75% of the ground truth inspirations are covered by even only the 4% chosen papers from the chemistry literature corpus.
It seems that LLMs are quite capable of finding inspiration papers that are unknown to be able to associate with the background but in fact can associate with the background to create novel knowledge.
It means our bold assumption in § 4.2.2 that “the most advanced LLMs might already know lots of knowledge pairs that are able to associate to create novel knowledge, where the knowledge pairs are not known by any scientist to be related” is possible to be true.

**Table 3: Main table for $Q1$. For each screen window of 15 papers, 3 papers are selected.**
| Corpus Size | Hit Ratio (top 0.016%) | Hit Ratio (top 0.8%) | Hit Ratio (top 4%) | Hit Ratio (top 20%) |
| --- | --- | --- | --- | --- |
| 150 | NA | 61.4% | 76.8% | 92.8% |
| 300 | NA | 60.8% | 83.7% | 96.7% |
| 1000 | 46.7% | 69.0% | 88.9% | 96.4% |
| 3000 | 52.0% | 70.6% | 86.9% | 95.8% |

**Table 4: Ablation table on screen window size for $Q1$. The corpus size is 300. For each screen window no matter its size, 3 papers are selected to remain for the next round of screening.**
| Screen window size | Hit Ratio (4 round) | Hit Ratio (3 round) | Hit Ratio (2 round) | Hit Ratio (1 round) |
| --- | --- | --- | --- | --- |
| 10 | 56.5% | 79.4% | 88.9% | 98.0% |
| 15 | NA | 60.8% | 83.7% | 96.7% |
| 20 | NA | 58.8% | 76.8% | 91.2% |
| 40 | NA | NA | 54.9% | 88.9% |
| 60 | NA | NA | 53.9% | 71.6% |

**Table 5: Ablation table on background options for $Q1$. The corpus size is 300. For each screen window of 15 papers, 3 papers are selected.**
| Strict Background | Background Survey | Hit Ratio (top 0.8%) | Hit Ratio (top 4%) | Hit Ratio (top 20%) |
| --- | --- | --- | --- | --- |
| ✓ | ✓ | 60.8% | 83.7% | 96.7% |
| ✓ | ✗ | 54.2% | 77.8% | 95.1% |
| ✗ | ✓ | 57.8% | 80.1% | 96.7% |

Table 4 shows the ablation study in terms of screen window size.
It seems that smaller window size can lead to better performance: screen window size of 60 to keep 3 for one round will select 5% of the corpus, and the Hit Ratio is 71.6%; while screen window size of 15 to keep 3 for two rounds will select only 4% of the corpus, but the Hit Ratio is as high as 83.7%.

Table 5 shows the ablation study in terms of whether to use strict background (discussed in § 3) or survey or not.
It indicates that a survey can largely help with the inspiration retrieval process.
Surprisingly, without a strict background, the Hit Ratio goes down a bit.
We attribute it to the reason that mentioning information related to the inspiration will discourage retrieving that inspiration, since in the prompt, we ask LLMs to search for inspirations, and the demonstration example indicates that inspirations should not be too similar to the background (to bring in additional information).

### 5.2 How Well can LLMs perform P ⁢ ( h j | b , i j , h j − 1 ) 𝑃 conditional subscript ℎ 𝑗 𝑏 subscript 𝑖 𝑗 subscript ℎ 𝑗 1 P(h_{j}|b,i_{j},h_{j-1}) italic_P ( italic_h start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_b , italic_i start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_j - 1 end_POSTSUBSCRIPT ) ?

**Table 6: Description of the Matched Score.**
| 5 points | Generated hypothesis covers all the key points and leverage them similarly<br>as in the groundtruth hypothesis; Extra key points do not have apparent flaws. |
| --- | --- |
| 4 points | Generated hypothesis covers all the key points (or at least three key points) and leverage<br>them similarly as in the groundtruth hypothesis; Extra key points have apparent flaws. |
| 3 points | Generated hypothesis covers at least two key point and leverage it similarly<br>as in the groundtruth hypothesis, but does not cover all key points |
| 2 points | Generated hypothesis covers at least one key point and leverage it similarly<br>as in the groundtruth hypothesis, but does not cover all key points |
| 1 point | Generated hypothesis covers at least one key point, but is used differently<br>as in the groundtruth hypothesis |
| 0 point | Generated hypothesis does not cover any key point |

Here we investigate the question (denoted as $Q2$): “Given only known knowledge, whether LLM can reason to unknown knowledge that has high probability to be valid?”.

The first challenge to answer $Q2$ is the evaluation method:
The benchmark covers a large range of chemistry topics, and chemistry is a very complex discipline that a slight change of research topic would make a chemist unable to provide a reliable enough evaluation.
In fact, a chemistry researcher might not be able to provide a reliable enough evaluation even if the hypothesis is in his domain.

Therefore, we adopt a reference-based evaluation method called “Matched Score” (MS).
The descriptions are shown in Table 6.
It’s on a 6-point Likert scale, roughly containing four stages.
Denoting generated hypothesis as $gh$, and original hypothesis as $oh$, the four stages are
(1) $gh\cap oh=\emptyset$ (0 point);
(2) $gh\cap oh\neq\emptyset$ (1/2/3 points);
(3) $gh\supseteq oh$ (4 points);
(4) $gh\approx oh$ (5 points).

**Table 7: Main table for $Q2$. Average/Top MS means the average/highest Matched Score of all generated $h$ from one $b$. The numbers represent the statistics of Average/Top MS over the benchmark.**
|  | 5 | 4 | 3 | 2 | 1 | 0 | Total |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | w/ background survey |  |  |  |  |  |  |
| Average MS (GPT-4o) | 2 | 9 | 18 | 17 | 5 | 0 | 51 |
| Top MS (GPT-4o) | 28 | 1 | 19 | 3 | 0 | 0 | 51 |
| Top MS (Experts) | 9 | 12 | 22 | 6 | 2 | 0 | 51 |
|  | w/o background survey |  |  |  |  |  |  |
| Average MS (GPT-4o) | 1 | 7 | 17 | 19 | 7 | 0 | 51 |
| Top MS (GPT-4o) | 25 | 2 | 19 | 5 | 0 | 0 | 51 |

We use MOOSE-Chem to investigate $Q2$.
Specifically, we initialize $I$ as only the ground truth inspiration papers and search $i$ for $k$ round, where $k$ is the number of ground truth $i$ needed for each $b$.
MOOSE-Chem will not retrieve the same $i$ already retrieved in previous rounds, guaranteeing that before generating the final $h$, the framework has already seen all the ground truth inspirations.

Table 7 shows the results.
For each $b$, the top two $h$ with the highest MS by GPT-4o are selected for expert evaluation (by two chemistry PhD students).
It indicates that LLMs are quite capable to associate known knowledge into an unknown knowledge that has high probability to be valid (very close to $oh$).
In addition, providing a survey can assist the new knowledge discovering process.
We discuss the agreement between GPT-4o based evaluation and expert evaluation in § A.7.

### 5.3 How Well can LLMs perform R ⁢ ( h ) 𝑅 ℎ R(h) italic_R ( italic_h ) ?

**Table 8: Relation between the number of matched ground truth $i$ and the average ranking ratio ($\downarrow$).**
| #Matched $i$ | 3 | 2 | 1 | 0 |
| --- | --- | --- | --- | --- |
| Average Rank Ratio | NA | 0.411 | 0.474 | 0.521 |
| Size | 0 | 302 | 2458 | 4899 |

Here we investigate $Q3$: “whether LLMs can select high-quality $h$ to rank them higher?”.

To investigate $Q3$, we run MOOSE-Chem with every $b$ from the benchmark; $|I|=300$, containing all the ground truth $i$.
Every $h$ is given a rating $r=R(h)$, and is ranked based on $r$.
For every generated $h$, we get the number of ground truth $i$ it leveraged (#Matched $i$), and evaluate it with a GPT-4o evaluated MS (here MS is -1 means this $h$ has not used any ground truth $i$).

Table 8 shows the relation between the #Matched $i$ and average ranking ratio (the lower, the better).
It shows a clear trend that the more ground truth $i$ is leveraged, the better ranking score $h$ can have.
It indicates that that $h$ with a higher ranking ratio are more likely to be matched with better $i$.

Table 9 shows the relation between the GPT-4o evaluated MS and the average ranking ratio.
For $h$ with a positive MS, there is a trend that the higher the MS, the better the average rank ratio (if MS $\in$ [2,4]).
However, the disadvantage of those $h$ without a positive MS is not very significant.
It seems that LLMs have a certain ability to rank good $h$ higher. But it is not sure how significant it is, because a part of the reason of this results is that those $h$ generated without groundtruth $i$ could be also in high quality.

**Table 9: Relation between the GPT-4o labeled Matched Score and average ranking ratio ($\downarrow$).**
| Matched Score | 5 | 4 | 3 | 2 | 1 | 0 | -1 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Average Rank Ratio | 0.489 | 0.439 | 0.488 | 0.501 | 0.436 | 0.501 | 0.503 |
| Size | 210 | 36 | 404 | 427 | 29 | 102 | 6451 |

## 6 Experiment and Ablation Study

Here, we perform experiments in a setting similar to the copilot in the wild setting.
Specifically, only background question (strict), background survey (strict), and a chemistry corpus $|I|=300$ are provided to the framework.
Only the top 4% of $I$ is selected and used to develop $h$.
The evaluation metrics are Top MS and Average MS (the highest/average Matched Score of all generated $h$ from one $b$), averaging across the benchmark.
All experiments are conducted by GPT-4o (its training data is up to October 2023).

### 6.1 Baselines

##### MOOSE

is a hypothesis discovery framework for the general social science domain.
It leverages LLMs to retrieve inspirations and uses self-refine (Madaan et al., ) to improve the validness, novelty, and clarity aspects.
The difference is that
(1) it does not adopt the mutation and recombination step to better associate background and inspiration;
(2) it only retrieves one step of inspiration.

##### SciMON

is a hypothesis discovery framework for the NLP and biochemical domain.
It relies on semantic and citation neighbors to retrieve information to assist the background.
As a result, the retrieved information could be very related to the background that might not be able to serve as an inspiration.
To make the generated hypothesis more novel, it adopts self-refine to focus on improving the novelty aspect of the generated hypothesis.
Here we implement SciMON with LLM-based inspiration retrieval, the same as MOOSE-Chem.
Table 3 shows that the recall rate of LLM-based retrieval is 83.7%.

##### Qi et al. ( 2024 )

work on hypothesis discovery in the biomedical domain.
It retrieves information pertinent to the keywords in the background to generate hypotheses.
As a result, the retrieved information might compose of a background survey, but not as inspiration.
Self-refine is also adopted.

### 6.2 Results

**Table 10: Experiments and ablation study. The Matched Score is evaluated by GPT-4o.**
| Method | Top MS | Average MS |
| --- | --- | --- |
| SciMON (Wang et al., ) | 2.549 | 2.281 |
| MOOSE (Yang et al., ) | 2.882 | 2.464 |
| Qi et al. () | 2.686 | 2.356 |
| MOOSE-Chem | 4.020 | 2.564 |
| w/o EU | 3.765 | 2.730 |
| w/o multi-step | 3.588 | 2.452 |

Table 10 shows the baseline results and the ablation study of MOOSE-Chem.
It indicates that both mutation & recombination and the multi-step designs can significantly improve the best-performing $h$.
Mutation & recombination leads to a drop of Average MS compared to the MOOSE baseline; we attribute the reason to that the mutation step forces LLMs to generate $h$ different from previous $h$ mutations from the same $b$ and $i$, and therefore might generate many $h$ that do not make a lot of sense.
The assigned MS to these mutation $h$ is low, and therefore lower down the Average MS.

To better understand the performance of MOOSE-Chem in this real copilot setting, for each $b$ the top 4 generated $h$ with the highest MS by GPT-4o are evaluated again by two experts in terms of MS.
Table 11 shows the expert evaluation results.
Here the top MS is the highest MS for each $b$, out of the 4 expert evaluated $h$ for this $b$.
Note that MS rated as three is already very high.
Illustrated in Table 6, it means the generated $h$ by MOOSE-Chem (that has not seen $h$) in the real copilot setting covers two main innovations of the chemistry hypothesis, which is published in Nature, Science or a similar level.

**Table 11: MOOSE-Chem runs with $|I|$=300, mimicking the copilot setting. This table shows the statistics of the top Matched Score across the benchmark. The evaluation is done by experts.**
|  | 5 | 4 | 3 | 2 | 1 | 0 | Total |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Top MS (Expert) | 0 | 2 | 19 | 16 | 8 | 6 | 51 |

## 7 Case Study

Here, we show a case study of a generated $h$ (from MOOSE-Chem with $|I|$=300), corresponding ground truth $h$,
and an expert’s analysis (gives a MS as 3).
More case studies can be seen in § A.8.

Generated $h$:
A pioneering integrated electrocatalytic system leveraging ruthenium nanoparticles embedded in nitrogen-doped graphene, combined with a dual palladium-coated ion-exchange membrane reactor, will catalyze efficient, scalable, and site-selective reductive deuteration of aromatic hydrocarbons and heteroarenes. Utilizing deuterium sources from both $D_{2}$ gas and $\mathbf{D_{2}O}$, this system will optimize parameters through real-time machine learning-driven dynamic adjustments. Specific configurations include ruthenium nanoparticle sizes (2-4 nm), nitrogen doping levels (12-14%), precisely engineered palladium membranes (5 micrometers, ensuring 98% deuterium-selective permeability), and advanced cyclic voltammetry protocols (1-5 Hz, -0.5V to -1.5V).

Ground truth $h$:
The main hypothesis is that a nitrogen-doped ruthenium (Ru) electrode can effectively catalyze the reductive deuteration of (hetero)arenes in the presence of $\mathbf{D_{2}O}$, leading to high deuterium incorporation into the resulting saturated cyclic compounds. The findings validate this hypothesis by demonstrating that this electrocatalytic method is highly efficient, scalable, and versatile, suitable for a wide range of substrates.

Expert’s analysis:
The proposed hypothesis effectively covers two key points from the ground truth hypothesis: the incorporation of ruthenium (Ru) and the use of $\mathbf{D_{2}O}$ as a deuterium source within the electrocatalytic system. However, the current content does not detail the mechanism by which Ru-D is produced, which is essential for explaining the process of reductive deuteration. Nevertheless, the results are still insightful. The specific level of nitrogen doping, for example, is highly suggestive and warrants further investigation. Overall, the match remains strong in its alignment with the original hypothesis while also presenting opportunities for deeper exploration.

## 8 Conclusion

We investigate this central question: “Can LLMs automatically discover novel and valid chemistry (including material science) research hypotheses (even those which deserve a publication in Nature, Science, or a similar level) given only a chemistry research background (consisting of a research question and/or a background survey), without limitation on the domain of the research question?”.
We propose a fundamental assumption to break up this seemingly-impossible-to-solve central question into three smaller, more practical and executable fundamental questions.
Then, we investigate LLM’s ability on each of them.
To do it, we construct a benchmark consisting of chemistry and material science papers published and only be public in 2024.
We also develop an LLM-based multi-agent framework consisting of three stages reflecting the three smaller fundamental questions.
Experiments show that the framework (runs in a copilot in-the-wild setting, with LLMs with training data up to October 2023) can rediscover many hypotheses with very high similarity with
the ground truth ones, covering the main innovations.

#### Acknowledgments

This research/project is supported by the Ministry of Education, Singapore under its MOE Academic Research Fund Tier 2 (STEM RIE2025 Award MOE-T2EP20123-0005).

We thank Mengsong Wu for his insightful discussions with us, and we thank Yuwei Wan for her efforts to support this research.

## Appendix A Appendix

### A.1 Full Proof / Derivation for the Fundamental Assumption

We propose an assumption that a majority of chemistry hypotheses can originate from a research background and several inspirations.
This assumption is not only supported by many chemistry researchers whom we have extensive discussions with but also by the cognitive science finding that “creative ideas often result from the cohesive association of two (or more) seemingly unrelated pieces of knowledge” (Koestler, ; Benedek et al., ; Lee & Chung, ).
We design our method based on this fundamental assumption.

This assumption is reminiscent of Swanson Linking (Swanson, ) in the domain of literature-based discovery (LBD), also known as the “ABC model”, where two concepts A and C are hypothesized as linked if they both co-occur with some intermediate concept B in papers.
Our assumption differs in that:
(1) for a chemistry hypothesis published in a good venue, usually more than one inspirations are needed;
(2) background and inspiration are not necessarily linked by a path of intermediate papers;
(3) our assumption is applied to a majority of existing published chemistry hypotheses, while LBD has been considered to only focus on a very specific, narrow type of
hypothesis (Wang et al., ).
It might indicate that a similar proportion of future chemistry hypotheses can also be resulted from linkages of existing literature.

Denoting background knowledge as $b$, inspiration knowledge as $i$, and hypothesis as $h$, we translate this assumption as:

$$ $h=f(b,i_{1},\ldots,i_{k})$ (5) $$

Here $k\in\mathbb{Z}$ represents the number of inspirations needed for a particular $h$. Typically in chemistry, $k\in[1,3]$.

We denote the full (chemistry) literature as $I$, such that $P(I)=1$. Then we can have:

$$ $\displaystyle P(h|b)$ $\displaystyle=P(h,i_{1},\ldots,i_{k}|b)$ (6) $\displaystyle=\begin{cases}\frac{P(h,b,i_{1})}{P(b,i_{1})}\cdot\frac{P(b,i_{1} )\cdot P(I)}{P(b)\cdot P(I)}&\text{if }k=1\\ \frac{P(h,b,i_{1},\ldots,i_{k})}{P(b,i_{1},\ldots,i_{k})}\cdot\frac{P(b,i_{1}, \ldots,i_{k})\cdot P(I)}{P(b,i_{1},\ldots,i_{k-1})\cdot P(I)}\cdot\ldots\cdot \frac{P(b,i_{1})\cdot P(I)}{P(b)\cdot P(I)}&\text{if }k>1\\ \end{cases}$ (7) $\displaystyle=\begin{cases}P(h|b,i_{1})\cdot P(i_{1}|b,I)&\text{if }k=1\\ P(h|b,i_{1},\ldots,i_{k})\cdot\prod_{j=2}^{k}P(i_{j}|b,i_{1},\ldots,i_{j-1},I) \cdot P(i_{1}|b,I)&\text{if }k>1\end{cases}$ (8) $$

Here, $P(h|b,i_{1},\ldots,i_{k})$ is used to obtain $h$, and $P(i_{j+1}|b,i_{1},\ldots,i_{j},I)$ is used to obtain each $i$.
However, they do not precisely reflect how chemistry researchers do, and might not be practicable.
One of the main reasons is that researchers tend to think small step by small step.
It would be very challenging to think in terms of a big step without breaking it into several small steps.

To mimic how chemistry researchers conduct research and make it more practicable, we break $P(h|b,i_{1},\ldots,i_{k})$ into a series of recursive smaller steps as

$$ $\displaystyle P(h_{k}|b,i_{1},\ldots,i_{k})$ $\displaystyle\approx P(h_{k}|b,f(b,i_{1},\ldots,i_{k-1}),i_{k})$ $\displaystyle\text{if }k>1$ (9) $\displaystyle=P(h_{k}|b,h_{k-1},i_{k})$ $\displaystyle\text{if }k>1$ (10) $$

Similarly, we can break $P(i_{j+1}|b,i_{1},\ldots,i_{j},I)$ as

$$ $\displaystyle P(i_{k+1}|b,i_{1},\ldots,i_{k},I)$ $\displaystyle\approx P(i_{k+1}|b,f(b,i_{1},\ldots,i_{k}),I)$ $\displaystyle\text{if }k>1$ (11) $\displaystyle=P(i_{k+1}|b,h_{k},I)$ $\displaystyle\text{if }k>1$ (12) $$

As a result, to achieve the final $h_{k}$, we need to obtain $h_{1},\ldots,h_{k-1}$ first (if $k>1$).
In addition, seeing $h$ as a “state”, and $i$ as an “action”, obtaining $h$ and $i$ through $P(h_{k}|b,h_{k-1},i_{k})$ and $P(i_{k+1}|b,h_{k},I)$ correspondingly indicates a property very similar to the Markov property:
(1) a new $h$ only depends on $b$, its previous $h$, and the current $i$; and
(2) an $i$ only depends on $b$, $I$, and the current $h$.

Therefore, if $k>1$,

$$ $\displaystyle P(h|b)$ $\displaystyle=P(i_{1},\ldots,i_{k},h_{1},\ldots,h_{k}|b)$ (13) $\displaystyle=P(i_{1},h_{1}|b)\cdot P(i_{2},h_{2}|b,i_{1},h_{1})\cdot\ldots \cdot P(i_{k},h_{k}|b,i_{1},\ldots,i_{k-1},h_{1},\ldots,h_{k-1})$ (14) $\displaystyle\approx P(i_{1},h_{1}|b)\cdot P(i_{2},h_{2}|b,h_{1})\cdot\ldots \cdot P(i_{k},h_{k}|b,h_{k-1})$ (15) $\displaystyle=\frac{P(b,i_{1},I)}{P(b,I)}\cdot\frac{P(b,i_{1},h_{1})}{P(b,i_{1 })}\cdot\ldots\cdot\frac{P(b,i_{k},h_{k-1},I)}{P(b,h_{k-1},I)}\cdot\frac{P(b,i _{k},h_{k-1},h_{k})}{P(b,i_{k},h_{k-1})}$ (16) $\displaystyle=P(i_{1}|b,I)\cdot P(h_{1}|b,i_{1})\cdot\prod_{j=1}^{k-1}P(i_{j+1 }|b,h_{j},I)\cdot P(h_{j+1}|b,i_{j+1},h_{j})$ (17) $\displaystyle=\prod_{j=1}^{k}P(i_{j}|b,h_{j-1},I)\cdot P(h_{j}|b,i_{j},h_{j-1} ),\text{ where }h_{0}=\emptyset$ (18) $$

Although starting from $k>1$, Derivation 18 covers the situation when $k=1$ in Equation 8.

Therefore, in sum, we break up the seemingly impossible question $P(h|b)$ into many practical and executable smaller questions as:

$$ $\displaystyle P(h|b)\approx\prod_{j=1}^{k}P(i_{j}|b,h_{j-1},I)\cdot P(h_{j}|b, i_{j},h_{j-1}),\text{ where }h_{0}=\emptyset\text{ and }k>=1$ (19) $$

### A.2 The Full Instruction for Benchmark Checking

Please help us check again before finalizing the decomposition of each paper in the benchmark:

1. Whether the background question is correct.

2. Background survey shouldn’t contain any information/method in inspiration or hypothesis (except if this information/method has been used for this particular background question before). It is encouraged to include the most similar existing method to the proposed method. For example, the proposal is to change BaCl2 to BaSO4. It is encouraged to include BaCl2 in the survey, but SO4 must not be included in the survey (since SO4 belongs to the inspiration).

3. Background question cannot contain any information in inspiration or hypothesis as well: It should be a little bit general question, instead of a specific question asking about how the inspiration can be leveraged to help with the question. It also shouldn’t be too general that we can’t understand which specific research domain it works on.

3. Whether the identification of inspirations really the main inspirations for this paper, and whether we need more main inspiration(s).

4. Whether the main hypothesis is correct and covers the main key points.

5. Whether the background survey + background question + identified inspirations can logically lead to the hypothesis (if not, we might need to identify more inspirations).

Thank you for the efforts! Your contribution is indispensable for the success of this research. Please let me know if you have any questions.

### A.3 Prompt to obtain R ⁢ ( h ) 𝑅 ℎ R(h) italic_R ( italic_h )

You are known as a diligent and harsh reviewer in Chemistry and Material Science that will spend much time to find flaws when reviewing and therefore usually gives a relatively much lower score than other reviewers. But when you meet with a hypothesis you truly appreciate, you don’t mind to give it good scores. Given a not yet peer reviewed research hypothesis in Chemistry or Material Science domain, try to evaluate the research hypothesis from four research aspects and give score according to evaluation guidelines provided below. All four aspects should be evaluated in a 5 point scale.

Aspect 1: Validness.
5 points: The hypothesis is a logical next step from current research, strongly supported by theory, perhaps with some indirect experimental evidence or highly predictive computational results. The experimental verification seems straightforward with a high probability of confirming the hypothesis; 4 points: Here, the hypothesis is well-rooted in existing theory with some preliminary data or computational models supporting it. It extends known science into new but logically consistent areas, where experiments are feasible with current technology, and there’s a reasonable expectation of positive results; 3 points: This hypothesis is within the realm of theoretical possibility but stretches the boundaries of what’s known. It might combine existing knowledge in very novel ways or predict outcomes for which there’s no direct evidence yet. There’s a conceptual framework for testing, but success is uncertain; 2 points: While the hypothesis might be grounded in some theoretical aspects, it significantly deviates from current understanding or requires conditions or materials that are currently impossible or highly improbable to achieve or synthesize; 1 point: The hypothesis proposes concepts or outcomes that are not only unsupported by current theory but also contradict well-established principles or data. There’s no clear path to experimental testing due to fundamental theoretical or practical barriers.

Aspect 2: Novelty.
5 points: This level of novelty could fundamentally alter our understanding of chemistry or create entirely new fields. It often involves predictions or discoveries that, if proven, would require a significant overhaul of existing chemical theories; 4 points: The hypothesis significantly departs from established norms, potentially redefining how certain chemical phenomena are understood or applied. It might involve entirely new materials or theoretical frameworks; 3 points: This level involves a hypothesis that could potentially lead to new insights or applications. It might challenge minor aspects of current theories or introduce new methodologies or materials; 2 points: The hypothesis introduces a new angle or method within an established framework. It might involve known compounds or reactions but in contexts or combinations not previously explored; 1 point: The hypothesis involves minor tweaks or applications of well-known principles or techniques. It might slightly extend existing knowledge but doesn’t introduce fundamentally new concepts.

Aspect 3: Significance.
5 points: This hypothesis could fundamentally change one or more branches of chemistry. It might introduce entirely new principles, theories, or methodologies that redefine the boundaries of chemical science; 4 points: This hypothesis challenges current understanding or introduces a concept that could lead to substantial changes in how a particular area of chemistry is viewed or applied. It might lead to new technologies or significant theoretical advancements; 3 points: this hypothesis proposes something new or an innovative approach that could lead to noticeable advancements in a specific area of chemistry. It might open new avenues for research or application but doesn’t revolutionize the field; 2 points: This hypothesis might offer a small variation or incremental improvement on existing knowledge. It could potentially refine a known concept but doesn’t significantly alter the field; 1 point: The hypothesis addresses a very narrow or already well-established aspect of chemistry. It might confirm what is already known without adding much new insight.

Aspect 4: Potential.
5 points: The hypothesis, while potentially intriguing now, holds the promise of being revolutionary with the addition of a key methodological component. This could introduce entirely new concepts or fields, fundamentally changing our understanding or capabilities in chemistry; 4 points: The hypothesis, though promising, could be transformative with the right methodological enhancement. This enhancement might lead to groundbreaking discoveries or applications, significantly advancing the field; 3 points: The hypothesis, while interesting in its current form, could be significantly elevated with the right methodological addition. This might lead to new insights or applications that go beyond the initial scope; 2 points: The hypothesis currently offers some value but has the potential for more substantial contributions if enhanced with a new methodological approach. This could lead to incremental advancements in understanding or application; 1 point: The hypothesis, as it stands, might be straightforward or well-trodden. Even with methodological enhancements, it’s unlikely to significantly expand current knowledge or applications beyond minor improvements.

The hypothesis is:
Please give a response to the initial question on scoring the hypothesis from four aspects. Remember that you are a diligent and harsh reviewer.

### A.4 Related Works on Reasoning

Scientific discovery is highly related to reasoning, since it requires a set of very complex reasoning processes to lead to new discovery.

Inductive reasoning (Yang et al., ) is the most relevant reasoning type.
It is about finding rules or hypotheses from observations.
Scientific discovery is naturally an ultimate goal of inductive reasoning.

Inductive reasoning is a sub-reasoning type of logical reasoning. The other two sub-reasoning types are deductive reasoning (Clark et al., ) and abductive reasoning (Bhagavatula et al., ).
Yang et al. () discuss their definitions and differences in detail.

Another relevant reasoning type is commonsense reasoning (Yang et al., ; ).
Scientific discovery can be seen as an opposite task, which is to reason far outside of commonsense, even to discover the unknown knowledge.

### A.5 Prompt to GPT-4o for Matched Score

You are helping to evaluate the quality of a proposed research hypothesis in Chemistry by a phd student. The groundtruth hypothesis will also be provided to compare. Here we mainly focus on whether the proposed hypothesis has covered the key points in terms of the methodology in the groundtruth hypothesis. You will also be given a summary of the key points in the methodology of the groundtruth hypothesis for reference. Please note that for the proposed hypothesis to cover one key point, it is not necessary to explicitly mention the name of the key point, but might also can integrate the key point implicitly in the proposed method. The evaluation criteria is called ’Matched score’, which is in a 6-point Likert scale (from 5 to 0). Particularly, 5 points mean that the proposed hypothesis (1) covers all the key points and leverage them similarly as in the methodology of the groundtruth hypothesis, and (2) does not contain any extra key point that has apparent flaws; 4 points mean that the proposed hypothesis (1) covers all the key points (or at least three key points) and leverage them similarly as in the methodology of the groundtruth hypothesis, (2) but also with extra key points that have apparent flaws; 3 points mean that the proposed hypothesis (1) covers at least two key points and leverage them similarly as in the methodology of the groundtruth hypothesis, (2) but does not cover all key points in the groundtruth hypothesis, (3) might or might not contain extra key points; 2 points mean that the proposed hypothesis (1) covers at least one key point in the methodology of the groundtruth hypothesis, and leverage it similarly as in the methodology of groundtruth hypothesis, (2) but does not cover all key points in the groundtruth hypothesis, and (3) might or might not contain extra key points; 1 point means that the proposed hypothesis (1) covers at least one key point in the methodology of the groundtruth hypothesis, (2) but is used differently as in the methodology of groundtruth hypothesis, and (3) might or might not contain extra key points; 0 point means that the proposed hypothesis does not cover any key point in the methodology of the groundtruth hypothesis at all. Please note that the total number of key points in the groundtruth hypothesis might be less than three, so that multiple points can be given. E.g., there’s only one key point in the groundtruth hypothesis, and the proposed hypothesis covers the one key point, it’s possible to give 2 points, 4 points, and 5 points. In this case, we should choose score from 4 points and 5 points, depending on the existence and quality of extra key points. ’Leveraging a key point similarly as in the methodology of the groundtruth hypothesis’ means that in the proposed hypothesis, the same (or very related) concept (key point) is used in a similar way with a similar goal compared to the groundtruth hypothesis (not necessarily for the proposed hypothesis to be exactly the same with the groudtruth hypothesis to be classified as ’similar’). When judging whether an extra key point has apparent flaws, you should use your own knowledge to judge, but rather than to rely on the count number of pieces of extra key point to judge.

Please evaluate the proposed hypothesis based on the groundtruth hypothesis.
The proposed hypothesis is:
The groundtruth hypothesis is:
The key points in the groundtruth hypothesis are:
Please evaluate the proposed hypothesis based on the groundtruth hypothesis, and give a score.

### A.6 Generated Hypotheses With Low Matched Score Are Not Necessarily Bad

MS only measures the similarity between the generated $h$ and the ground truth $h$.
Receiving a MS as 0 or 1 does not mean the generated $h$ is in bad.
Only real lab experiments can check each $h$.

### A.7 Agreement Between Expert Evaluation and GPT-4o Evaluation

**Table 12: Consistency score between expert evaluation and GPT-4o evaluation.**
| #Comparison Examples | Hard Consistency Score | Soft Consistency Score |
| --- | --- | --- |
| 392 | 0.345 | 0.542 |

Table 12 shows the agreement between expert evaluation and automatic evaluation (by GPT-4o) on MS.
Hard consistency is assigned to 1 only if the two scores are exact the same, else is assigned to 0.
Soft consistency is assigned to 1 only if the absolute difference between the two scores is less than 2, else is assigned to 0.

The results show a medium to high consistency between expert evaluation and automatic evaluation.
The main reason is that, in practice, the automatic evaluation is usually 1 to 2 points higher than expert evaluation, since GPT-4o can usually find a way to explain how the generated hypothesis is related to the ground truth hypothesis in terms of the main innovations.
While this explanation usually is not wrong, the experts might find that compared to the MS given by GPT-4o, the generated hypotheses might not be clear enough to deliver the new innovation information to scientists when using it as a copilot advice.

### A.8 More Case Study

#### A.8.1 Case 1 (MS by Expert: 4)

Generated $h$:
Leveraging an integrated multi-layer hydrogel-ion assembly, constructed through advanced freeze-casting, salting-out processes, and bioinformatics-driven macromolecular design, will enhance the Carnot-relative efficiency and mechanical robustness of flexible thermogalvanic devices. This approach reimagines the use of poly(vinyl alcohol) hydrogels and incorporates novel ion-specific interactions informed by the Hofmeister series with uniquely potent ions like magnesium and lithium. Precision ion concentrations (0.15 to 0.45 M) are optimized to amplify thermoelectric responses and ionic conductance. Molecular dynamics simulations, employing detailed quantum chemistry models, will validate the enhanced ionic interactions and entropy modulation. Comprehensive characterization through nanoindentation, electron microscopy, and advanced computational platforms will verify crystalline polymer structures and stratified ionic gel configurations. By setting new benchmarks in efficiency and flexibility, this innovation is poised to revolutionize energy solutions in wearable technologies, outperforming current models in both scalability and application feasibility.

Groundtruth $h$ (Wang et al., ):
By integrating guanidine sulfate $(Gdm)_{2}SO_{4}$ into a poly vinyl alcohol (PVA) hydrogel and employing directional freezing to create aligned channels, it is possible to achieve a flexible thermogalvanic armor (FTGA) with a Carnot-relative efficiency exceeding 8% while maintaining high mechanical strength. This integration allows for enhanced thermopower and mechanical robustness, exceeding the performance of traditional quasi-solid thermocells.

Expert’s Analysis:

The proposed hypothesis effectively covers key points, mirroring the ground truth hypothesis. It incorporates the Hofmeister series by mentioning ”ion-specific interactions informed by the Hofmeister series,” which aligns with the ground truth’s reference to ”guanidine sulfate $(Gdm)_{2}SO_{4}$” in the hydrogel integration, acknowledging that sulfate ions are a significant component of the Hofmeister series. Additionally, the proposed hypothesis employs ”freeze-casting,” which reflects the ground truth’s ”directional freezing” to create structured channels within the hydrogel. This approach enhances both mechanical strength and thermopower—objectives that are consistent with those of the ground truth hypothesis.

Moreover, the inclusion of salting-out processes in the proposed hypothesis does indeed contribute to improving the mechanical properties of thermoelectric materials. However, the selection of magnesium and lithium as specific examples is problematic. Overall, the proposed hypothesis provides valuable insights and serves as a source of inspiration for further exploration within this domain.

#### A.8.2 Case 2 (MS by Expert: 3)

Generated $h$:
An enantioselective [2+2+1] cycloaddition involving cycloalkenes, acetylenecarboxylates, and terminal alkynes can be achieved using a dynamically tunable Rh(I) pincer complex system, specifically incorporating PNP and PONOP ligands, to control the alkyne-to-vinylidene transformation and enhance selectivity towards chiral 3-methylenecyclopent-1-ene derivatives. By adjusting the P–Rh–P bite angle with novel ligand modifications and utilizing a blended solvent system containing toluene and dichloromethane at controlled temperatures (25°C to 80°C), reaction conditions can be fine-tuned to suppress side reactions. Computational methods (DFT, QTAIM, and NBO) will guide the optimization process, supported by real-time monitoring using FTIR and NMR. Exploring dual-ligand strategies, including phosphine/phosphinite combinations, will offer insights into previously unreported mechanistic pathways, ensuring a reproducible and scalable synthesis method applicable to pharmaceuticals and advanced material synthesis.

Groundtruth $h$ (Shibahara et al., ):
Rh(I)-catalyzed vinylidene formation and enantioselective control via chiral diphosphine ligands can work together to enable a selective [2+2+1] cycloaddition reaction using three different 2$\pi$-components. The formation of the vinylidene intermediate ensures reaction selectivity, while the chiral ligands, through their dihedral angles, control the enantiomeric outcome. Together, these factors will result in the highly enantioselective synthesis of chiral 3-methylenecyclopent-1-ene derivatives.

Expert’s Analysis:
The hypothesis generated in Case 2 receives a score of 3, reflecting its accurate identification of the metal and ligand necessary for the catalytic process, as well as its articulation of the mechanism underlying chiral control via ligand geometry. However, the inclusion of mixed solvents, which were not utilized in the actual study, prevents a higher rating. This prediction successfully asserts the necessity of rhodium metal and a phosphine ligand, while also proposing the significance of the chelating angle between the phosphine ligand and rhodium, in alignment with the hypothesis presented in the referenced article. Furthermore, although the DFT calculations mentioned in the hypothesis are not addressed in the article, they represent a promising avenue for future research in organic chemistry.

#### A.8.3 Case 3 (MS by Expert: 3)

Generated $h$:
A sophisticated dual-modal photoredox strategy synergistically integrates phosphorus ylides with a synchronized dual-wavelength excitation (450 nm and 520 nm) system to incite a regioselective three-component cycloaddition. This method exploits both net-neutral radical/polar crossover pathways and metal-free carbyne generation to convert inert C–H and C=P bonds into C–C and C=C bonds efficiently. Implemented within an optimized mixed solvent system (acetone/acetonitrile) under an ambient nitrogen atmosphere, the approach utilizes precise kinetic modulation and catalytic control, with computational validation (employing DFT models) underpinning mechanistic insights and regioselectivity. Experimental verification through in situ spectroscopic techniques (e.g., UV-Vis and NMR) enhances synchronization precision, curtailing side reactions. This methodology pledges substantial advancements in sustainable pharmaceutical synthesis and innovative polymer architecture formation, specifically by improving production efficiency for complex molecules and novel materials.

Groundtruth $h$ (Suzuki et al., ):
The hypothesis is that phosphorus ylides, when exposed to photoredox catalysis, can undergo single-electron oxidation to generate radical cations, which engage in radical-polar crossover reactions. These intermediates can then sequentially form C–C and C=C bonds through a formal cycloaddition process, offering a powerful method for constructing six-membered carbocycles from simple substrates.

Expert’s Analysis:
The generated hypothesis also merits a score of 3, as it correctly anticipates the use of photocatalysis and highlights the significant influence of solvent on the reaction. However, since dual wavelength catalysis and solvent mixing were not employed in the actual experiment, a higher score is not warranted. Notably, despite the proposed mixed solvents not being used in the study, their composition comprises the two best-performing single solvents from the actual research, thus providing valuable insights that remain relevant to the ongoing investigation.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. There is a lack of experimental exploration for mutation settings.   I am curious about the impact of mutations on the final results. 
     For example, according to the ablation studies, what proportion of high-quality hypotheses can be obtained directly without mutations?

2. How can we explain that a smaller window size leads to better performance in the inspiration retrieval phase?

3. Did the authors replace GPT-4o for all comparison methods for generation? Please specify.

4. How helpful is it to add new dimensions of significance?
soundness: 3
strengths: Originality:
Firstly, while LLMs have been utilized for scientific discovery in social science and NLP, this paper is the first to investigate their potential in chemistry. 
Besides, The MOOSE-CHEM framework employs a three-step approach to retrieve inspiration papers, inference valid knowledge, identify hypotheses and rank them, which hasn’t been used in previous research. 
Moreover, the use of the evolutionary algorithm to foster a broader diversity in hypothesis generation is also an innovation point.

Quality:
The paper includes extensive comparative experiments and ablation studies, providing a thorough evaluation of LLMs' performance. It also demonstrates the consistency between expert evaluation and automated evaluation of MS, enhancing the reliability of the results.

Clarity: 
The methodology is well-structured, and the three steps based on smaller questions are well-defined. Besides, detailed evaluation methods and corresponding formulas are provided for the three research questions, Q1, Q2 and Q3.  The practical implementation process is also relatively clear.

Significance:
The ability of LLMs to generate hypotheses can significantly accelerate the pace of scientific discovery by reducing the time and effort required for hypothesis generation. It can also open researchers’ minds for the application of LLMs in scientific discovery.
summary: This paper investigates the potential of LLMs to automatically discover novel and valid hypotheses in the field of chemistry. The study introduces a framework called MOOSE-Chem, which decompose the main research question into three smaller questions, respectively focusing on (1) the identification of inspiration papers; (2) the inference of unknown knowledge that is highly likely to be valid; and (3) the identification and ranking of hypotheses generated by the LLMs.
weaknesses: Firstly, using the same large language model to evaluate its own generated results may introduce bias. It is recommended to try using different LLMs to evaluate the results so as to guarantee the reliability of the results. For example, consider using models like LLaMa[1], Claude[2], Gemini[3], or other recent LLMs to compare outputs. If using the same LLM is necessary, you could collect hypotheses generated by humans and also have both experts and GPT-4 evaluate them. Then, compare their Hard/Soft Consistency Scores as well as distribution of MS with those of the hypotheses generated by the LLM.

[1] Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint arXiv:2307.09288, 2023.
[2] Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://www.anthropic.com, 2024.
[3] Team G, Anil R, Borgeaud S, et al. Gemini: a family of highly capable multimodal models[J]. arXiv preprint arXiv:2312.11805, 2023.

Besides, during the evaluation process (as demonstrated in section 5.1), in the constructed dataset, other papers besides the ground truth inspiration papers may also be helpful for the current background. This step may also require further evaluation to obtain more reliable results. You might select several high-ranking papers that are not included in set I, particularly those that repeatedly appear in experiments with varying corpus sizes or screen window sizes, and conduct case studies on them.

Additionally, this paper only employs the titles and abstracts for retrieval, which might overlook inspirations that could arise from the detailed content of the articles. Sections like Conclusion or Discussion often contain insights into the paper's limitations and suggestions for future work. These elements are usually crucial for generating inspiration.

Moreover, the benchmarks mentioned in the paper contain only 51 references, which may not be sufficient to fully assess the effectiveness of the proposed method. You are advised to increase the number of samples, especially selecting literature from different chemical fields. In addition to traditional categories such as Analytical Chemistry, Organic Chemistry, and Inorganic Chemistry, broader research topics like Environmental Chemistry, Medicinal Chemistry, and Nuclear Chemistry can also be considered. Furthermore, Nature Chemistry includes interdisciplinary areas such as Bioinorganic Chemistry, Bioorganic Chemistry, and Organometallic Chemistry, which could be investigated as well. Of course, this may involve considerations related to the resource consumption of LLMs.

Finally, although the paper used chemistry PhD students for the evaluation, there is still subjectivity that may affect the reliability of the evaluation results. You might consider inviting more experts to participate in the evaluation and analyzing the consistency of the assessments from different human individuals.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: See the weaknesses listed above.
soundness: 3
strengths: 1.	Generating research hypotheses is a complicated task, and the authors heuristically decomposed hypothesis generation into two steps: 

(1). inspiration retrieval, and 

(2). hypothesis refinement. In the hypothesis refinement step, the authors propose a novel “mutate and recombine” trick to help generate good hypotheses.

2.	The experiments to verify each of the research questions are well-designed with good quality.
summary: This work investigates whether large language models (LLMs) can autonomously generate novel and valid hypotheses in chemistry based solely on a research background. Specifically, the study explores if LLMs, when provided with a research question or background survey, can independently identify inspirations, synthesize hypotheses, and evaluate their quality. Building on discussions with chemistry experts, the authors hypothesize that most chemistry hypotheses can be derived from combining research background with relevant inspirations. They break down the central question into three key tasks: retrieving valuable inspirations, generating hypotheses, and ranking the quality of these hypotheses. To test this approach, the authors create a benchmark of 51 high-impact chemistry papers published in 2024. LLMs, trained only on data up to 2023, attempt to rediscover these hypotheses from the background and an extensive chemistry literature corpus. The results, achieved through a multi-agent LLM framework, show promising success, with many rediscovered hypotheses closely matching the originals and capturing core innovations.
weaknesses: 1.	The introduction section could be written better and more clear. (a) It would be great if the authors could provide a summary of the major contributions of this work at the end of the introduction section. What are really the contribution to the field? (b) It would be great if the authors could briefly discuss why the decomposition of the major question is necessary, what’s the difference or connection between the proposed inspiration identification (the first step of the three) and Retrieval Augmented Generation (RAG).

2.	As the end goal is to rediscover the chemistry scientific hypotheses, the upper bound of “rediscovery” is the exact match of the original hypothesis, intuitively if we rank the original hypothesis with the generated hypothesis, the original hypothesis may be ranked at the top for most of the time. Experiment on this and analysis about the cases where the original hypothesis is not ranked at the top would be interesting to better reflect LLM’s ability to perform R(h).

3. Although the proposed approach constructed a benchmark as the basis to evaluate the performance, it is unclear to what degree the generated hypotheses can be trusted or reliable, not made up from baseless hallucination.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: This includes a list of minor issues, questions and suggestions:

- Through the paper, it is mentioned that the inspiration shouldn't be related
to the given background. Is this enforced in the method, or is this verified
somehow?

- The proposed method of Evolutionary Algorithm (EA) sounds to be related
to other methods in LLMs that try to improve the consistency/quality of
the answers, e.g. 
  - Self-Consistency (SC) (Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models")
  - Universal SC (Chen et al. "Universal Self-Consistency for Large Language Model Generation")
  - Multiple Chain of Thought (MCT), Self reflection and/or self-correction approaches, etc. I thus believe that some related work could be included regarding this algorithm in the paper.


- On line 351, claim that "is possible to be true" that LLMs are able to create
novel knowledge. While the authors are correct here since it is not clear
whether this is true or not, could the authors provide other explanations
on why the LLMs are able to correctly find inspiration papers?


- Please if possible provide some description of how P (i|b), P (h|b, i), and R(h)
inputs and outputs look like. From the example provided, I can guess that
hypothesis is one paragraph long, but it would be great for the reader 
to have this stated clearly.

- I honestly don't fully understand the experimental setting reported in section 5.1.
What are the "ground truth inspiration papers" and how are these related to 
the 51 papers from the benchmark? In Table 3, does this mean that when a corpus size
of 150 is used, 120 papers are considered ground truth? 

- From Table 3, it almost seems like corpus size doesn't have much effect on the Hit ratio,
is this correct? If so, could you elaborate more in this regard?

- If I got the idea right, in the results of Table 1, if a screen window size is used, the corpus is 300 and 3 papers are selected for each window, then round 1 has (300/10)*3 = 90, round 2 has
(90/10)*3 = 27, and so on, is this correct? It would be more clarifying to include the amount of papers in each round in the table or in a separate table. It would be also clarifying 
to include a baseline comparison of the performance based on pure random screening (this is, select papers randomly).

- From the results reported in Table 7, it seems that the background survey (almost) doesn't have any 
effect on MS, however, this is somehow counterintuitive. Shouldn't the background survey improve performance as it happens in Q1?

- I would suggest the authors format the rounds in Table 4 to go along
with the direction of the text, this is, from left to right.

- Could you please clarify why there are NAs in Tables 3 and 4?

- On line 134: "contrusted by multiple chemistry PhD students".
For clarity, it would be great to know how many students.

- While it is not mandatory, given the stochasticity and prone-to-hallucinations nature of LLMs,
it is a good practice to use countermeasures to reduce hallucinations, e.g. sampling multiple times with majority voting,
or using different models for comparison. This is not an issue, but it would be
interesting to see how much variability the results reported in the experiments
are or if the method is robust to hallucinations.
soundness: 4
strengths: The paper is generally well-written, and in good English. The text is clear and
the authors did a good job guiding the reader through the motivation,
the derivation of the method and motivating each of the proposed steps
and experiments. The topic of the paper is very relevant and the results
are positive. Related work is well covered, and experiments are included
that compare the proposed method with previous work. Every claim made 
on the performance of the method is generally backed up with experiments.
I think the paper is generally a great and novel contribution to the field.
summary: This paper introduces a framework for scientific hypothesis discovery
and generation using large language models (LLMs). The authors gathered
a carefully curated dataset of scientific papers where questions and hypotheses were extracted for evaluation. The authors propose "MOOSE-CHEM", a
method based on previous work "MOOSE" but with modifications
to adapt it for chemistry literature and that improve Overall
performance (refinement, mutation). The framework is also formally motivated.
The proposed approach is evaluated in each of the main three tasks
that compose it: P (i|b), P (h|b, i), and R(h). Overall the results show great
performance in the tasks with GPT-4o, with evaluations that include both 
experts and GPT-4o, and where various hyperparameters are tested (e.g. corpus size).
weaknesses: My main concerns with the paper are regarding the reproducibility,
clarity and discussion of the approach:

- Reproducibility: we note that the authors introduce a scientific benchmark,
along with a novel framework for hypothesis generation. However,
the authors do not provide access to the novel-introduced benchmark,
which hampers the ability to really discriminate the difficulty of the tasks
at hand. Additionally, this impedes the ability to reproduce the results or
for future work to compare the performance with the proposed method.
I would suggest the authors to make the benchmark data available if possible.
Similarly, no access to the source code is provided. While available source code is not
a requirement, for a method that relies heavily on LLMs prompting, it would
be very beneficial to at least have access to the prompts used in the
work, just for the sake of reproducibility.

- (Related to the previous issue): There are parts of the paper that lack
a formal definition, which again, hampers the 
ability to reproduce the work. For instance, the authors mention that they
introduce a novel "Evolutionary Algorithm (EA)" that mutates, refines and
recombines hypotheses, however, it is unclear how the mechanisms of
mutation, refinement and recombination are implemented. I would suggest 
the authors to provide, when possible more technical details on the
proposed method.

- While the work includes a fair comparison with previous work, the work
doesn't include any discussion on the limitations of the proposed method,
nor any future work. It would be very much appreciated if the authors
could provide a discussion on the limitations of the proposed method
to guide potential future work in the field.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Can the authors clarify how/whether MOOSE-Chem substantially differs from previous work in multi-agent scientific discovery systems? 
2. How would the authors envision MOOSE-Chem supporting a chemist in hypothesis generation beyond rediscovering existing knowledge? Have any of the generated hypotheses been practically tested, or do they align with ongoing chemistry research needs?
soundness: 3
strengths: 1. The use of LLMs for scientific hypothesis generation is a growing area, and extending this work to complex fields like chemistry holds practical importance if achieved meaningfully.
2. The use of beam search and evolutionary algorithms for hypothesis refinement seems simple and powerful, as it systematically explores multiple possibilities.
3. The framework does not require expensive training or fine-tuning.
summary: This paper explores the potential of Large Language Models (LLMs) to generate novel, valid hypotheses in the chemistry domain by employing a multi-agent framework. It targets the rediscovery of scientific hypotheses using LLMs based on recent high-impact chemistry publications (since 2024), using a framework that separates tasks into three stages: (1) finding "inspiration" papers from a predefined literature corpus, (2) generating hypotheses by associating these inspirations with a background question, and (3) ranking hypotheses based on quality. To solve this, the authors proposed a multi-agent framework existing framework with an evolutionary algorithm to iteratively mutate hypotheses based on reward feedback (validness, novelty, clarity, and significance).
weaknesses: 1. There is a key fundamental assumption that the authors did not discuss here: reliance on general-purpose LLMs containing implicit domain knowledge. While this may be reasonable for powerful, large-scale models like GPT-4 or domains like social sciences, it is overly strong and may not apply to smaller or open-source models in this case. A deeper analysis (maybe more experiments) of how this assumption impacts performance across other LLMs (apart from GPT-4 or closed-source ones) is needed.
2. The assumption that LLMs have sufficient knowledge to connect novel and unrecognized scientific relations without any domain-specific tuning seems too bold. The model's effectiveness may degrade in more niche areas or specific research questions.
3. Much of this framework — multi-agent, hypothesis scoring, and the background-inspiration format closely resembles existing work in this area. Given the limited scale of benchmarking and experiments in this paper, the technical contribution seems limited for this venue.
3. The selection of inspiration papers seems unclear and might not align with typical challenges or open research questions in chemistry. A more detailed explanation of the selection process of those 51 papers and corresponding inspiration papers, including how relevance, significance, and novelty are ensured, is needed.
3. The paper presentation could be improved, particularly in the introduction and methodology. The technical details regarding probabilistic inference seem unnecessary.

Meta-review: The authors propose to combine LLMs with traditional search strategies for generating/refining hypotheses in scientific domains (e.g., chemistry). Hence, the problem setting is very challenging and rather unexplored until now. The technical solution may not be the end of the story but appropriate. The paper has also been well received by the reviewers. Particularly the empirical evaluation is strong and provides convincing results.

Decision: Accept (Poster)