MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses

Authors: Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, Dongzhan Zhou
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: scientific discovery

Abstract:
Scientific discovery contributes largely to the prosperity of human society, and recent progress shows that LLMs could potentially catalyst the process. However, it is still unclear whether LLMs can discover novel and valid hypotheses in chemistry. In this work, we investigate this main research question: whether LLMs can automatically discover novel and valid chemistry research hypotheses, given only a research question? With extensive discussions with chemistry experts, we adopt the assumption that a majority of chemistry hypotheses can be resulted from a research background question and several inspirations. With this key insight, we break the main question into three smaller fundamental questions. In brief, they are: (1) given a background question, whether LLMs can retrieve good inspirations; (2) with background and inspirations, whether LLMs can lead to hypothesis; and (3) whether LLMs can identify good hypotheses to rank them higher. To investigate these questions, we construct a benchmark consisting of 51 chemistry papers published in Nature or a similar level in 2024 (all papers are only available online since 2024). Every paper is divided by chemistry PhD students into three components: background, inspirations, and hypothesis. The goal is to rediscover the hypothesis given only the background and a large chemistry literature corpus consisting the ground truth inspiration papers, with LLMs trained with data up to 2023. We also develop an LLM-based multi-agent framework that leverages the assumption, consisting of three stages reflecting the more smaller questions. The proposed method can rediscover many hypotheses with very high similarity with the ground truth ones, covering the main innovations.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1730419028467

thread_type:
official_review

<thread_object>
creator: reviewer_km2p

<content>
summary:
This paper explores the potential of Large Language Models (LLMs) to generate novel, valid hypotheses in the chemistry domain by employing a multi-agent framework. It targets the rediscovery of scientific hypotheses using LLMs based on recent high-impact chemistry publications (since 2024), using a framework that separates tasks into three stages: (1) finding "inspiration" papers from a predefined literature corpus, (2) generating hypotheses by associating these inspirations with a background question, and (3) ranking hypotheses based on quality. To solve this, the authors proposed a multi-agent framework existing framework with an evolutionary algorithm to iteratively mutate hypotheses based on reward feedback (validness, novelty, clarity, and significance).

strengths:
1. The use of LLMs for scientific hypothesis generation is a growing area, and extending this work to complex fields like chemistry holds practical importance if achieved meaningfully.
2. The use of beam search and evolutionary algorithms for hypothesis refinement seems simple and powerful, as it systematically explores multiple possibilities.
3. The framework does not require expensive training or fine-tuning.

weaknesses:
1. There is a key fundamental assumption that the authors did not discuss here: reliance on general-purpose LLMs containing implicit domain knowledge. While this may be reasonable for powerful, large-scale models like GPT-4 or domains like social sciences, it is overly strong and may not apply to smaller or open-source models in this case. A deeper analysis (maybe more experiments) of how this assumption impacts performance across other LLMs (apart from GPT-4 or closed-source ones) is needed.
2. The assumption that LLMs have sufficient knowledge to connect novel and unrecognized scientific relations without any domain-specific tuning seems too bold. The model's effectiveness may degrade in more niche areas or specific research questions.
3. Much of this framework — multi-agent, hypothesis scoring, and the background-inspiration format closely resembles existing work in this area. Given the limited scale of benchmarking and experiments in this paper, the technical contribution seems limited for this venue.
3. The selection of inspiration papers seems unclear and might not align with typical challenges or open research questions in chemistry. A more detailed explanation of the selection process of those 51 papers and corresponding inspiration papers, including how relevance, significance, and novelty are ensured, is needed.
3. The paper presentation could be improved, particularly in the introduction and methodology. The technical details regarding probabilistic inference seem unnecessary.

questions:
1. Can the authors clarify how/whether MOOSE-Chem substantially differs from previous work in multi-agent scientific discovery systems? 
2. How would the authors envision MOOSE-Chem supporting a chemist in hypothesis generation beyond rediscovering existing knowledge? Have any of the generated hypotheses been practically tested, or do they align with ongoing chemistry research needs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730593755240

thread_type:
official_review

<thread_object>
creator: reviewer_tmw5

<content>
summary:
This paper introduces a framework for scientific hypothesis discovery
and generation using large language models (LLMs). The authors gathered
a carefully curated dataset of scientific papers where questions and hypotheses were extracted for evaluation. The authors propose "MOOSE-CHEM", a
method based on previous work "MOOSE" but with modifications
to adapt it for chemistry literature and that improve Overall
performance (refinement, mutation). The framework is also formally motivated.
The proposed approach is evaluated in each of the main three tasks
that compose it: P (i|b), P (h|b, i), and R(h). Overall the results show great
performance in the tasks with GPT-4o, with evaluations that include both 
experts and GPT-4o, and where various hyperparameters are tested (e.g. corpus size).

strengths:
The paper is generally well-written, and in good English. The text is clear and
the authors did a good job guiding the reader through the motivation,
the derivation of the method and motivating each of the proposed steps
and experiments. The topic of the paper is very relevant and the results
are positive. Related work is well covered, and experiments are included
that compare the proposed method with previous work. Every claim made 
on the performance of the method is generally backed up with experiments.
I think the paper is generally a great and novel contribution to the field.

weaknesses:
My main concerns with the paper are regarding the reproducibility,
clarity and discussion of the approach:

- Reproducibility: we note that the authors introduce a scientific benchmark,
along with a novel framework for hypothesis generation. However,
the authors do not provide access to the novel-introduced benchmark,
which hampers the ability to really discriminate the difficulty of the tasks
at hand. Additionally, this impedes the ability to reproduce the results or
for future work to compare the performance with the proposed method.
I would suggest the authors to make the benchmark data available if possible.
Similarly, no access to the source code is provided. While available source code is not
a requirement, for a method that relies heavily on LLMs prompting, it would
be very beneficial to at least have access to the prompts used in the
work, just for the sake of reproducibility.

- (Related to the previous issue): There are parts of the paper that lack
a formal definition, which again, hampers the 
ability to reproduce the work. For instance, the authors mention that they
introduce a novel "Evolutionary Algorithm (EA)" that mutates, refines and
recombines hypotheses, however, it is unclear how the mechanisms of
mutation, refinement and recombination are implemented. I would suggest 
the authors to provide, when possible more technical details on the
proposed method.

- While the work includes a fair comparison with previous work, the work
doesn't include any discussion on the limitations of the proposed method,
nor any future work. It would be very much appreciated if the authors
could provide a discussion on the limitations of the proposed method
to guide potential future work in the field.

questions:
This includes a list of minor issues, questions and suggestions:

- Through the paper, it is mentioned that the inspiration shouldn't be related
to the given background. Is this enforced in the method, or is this verified
somehow?

- The proposed method of Evolutionary Algorithm (EA) sounds to be related
to other methods in LLMs that try to improve the consistency/quality of
the answers, e.g. 
  - Self-Consistency (SC) (Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models")
  - Universal SC (Chen et al. "Universal Self-Consistency for Large Language Model Generation")
  - Multiple Chain of Thought (MCT), Self reflection and/or self-correction approaches, etc. I thus believe that some related work could be included regarding this algorithm in the paper.


- On line 351, claim that "is possible to be true" that LLMs are able to create
novel knowledge. While the authors are correct here since it is not clear
whether this is true or not, could the authors provide other explanations
on why the LLMs are able to correctly find inspiration papers?


- Please if possible provide some description of how P (i|b), P (h|b, i), and R(h)
inputs and outputs look like. From the example provided, I can guess that
hypothesis is one paragraph long, but it would be great for the reader 
to have this stated clearly.

- I honestly don't fully understand the experimental setting reported in section 5.1.
What are the "ground truth inspiration papers" and how are these related to 
the 51 papers from the benchmark? In Table 3, does this mean that when a corpus size
of 150 is used, 120 papers are considered ground truth? 

- From Table 3, it almost seems like corpus size doesn't have much effect on the Hit ratio,
is this correct? If so, could you elaborate more in this regard?

- If I got the idea right, in the results of Table 1, if a screen window size is used, the corpus is 300 and 3 papers are selected for each window, then round 1 has (300/10)*3 = 90, round 2 has
(90/10)*3 = 27, and so on, is this correct? It would be more clarifying to include the amount of papers in each round in the table or in a separate table. It would be also clarifying 
to include a baseline comparison of the performance based on pure random screening (this is, select papers randomly).

- From the results reported in Table 7, it seems that the background survey (almost) doesn't have any 
effect on MS, however, this is somehow counterintuitive. Shouldn't the background survey improve performance as it happens in Q1?

- I would suggest the authors format the rounds in Table 4 to go along
with the direction of the text, this is, from left to right.

- Could you please clarify why there are NAs in Tables 3 and 4?

- On line 134: "contrusted by multiple chemistry PhD students".
For clarity, it would be great to know how many students.

- While it is not mandatory, given the stochasticity and prone-to-hallucinations nature of LLMs,
it is a good practice to use countermeasures to reduce hallucinations, e.g. sampling multiple times with majority voting,
or using different models for comparison. This is not an issue, but it would be
interesting to see how much variability the results reported in the experiments
are or if the method is robust to hallucinations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730641737090

thread_type:
official_review

<thread_object>
creator: reviewer_rmit

<content>
summary:
This work investigates whether large language models (LLMs) can autonomously generate novel and valid hypotheses in chemistry based solely on a research background. Specifically, the study explores if LLMs, when provided with a research question or background survey, can independently identify inspirations, synthesize hypotheses, and evaluate their quality. Building on discussions with chemistry experts, the authors hypothesize that most chemistry hypotheses can be derived from combining research background with relevant inspirations. They break down the central question into three key tasks: retrieving valuable inspirations, generating hypotheses, and ranking the quality of these hypotheses. To test this approach, the authors create a benchmark of 51 high-impact chemistry papers published in 2024. LLMs, trained only on data up to 2023, attempt to rediscover these hypotheses from the background and an extensive chemistry literature corpus. The results, achieved through a multi-agent LLM framework, show promising success, with many rediscovered hypotheses closely matching the originals and capturing core innovations.

strengths:
1.	Generating research hypotheses is a complicated task, and the authors heuristically decomposed hypothesis generation into two steps: 

(1). inspiration retrieval, and 

(2). hypothesis refinement. In the hypothesis refinement step, the authors propose a novel “mutate and recombine” trick to help generate good hypotheses.

2.	The experiments to verify each of the research questions are well-designed with good quality.

weaknesses:
1.	The introduction section could be written better and more clear. (a) It would be great if the authors could provide a summary of the major contributions of this work at the end of the introduction section. What are really the contribution to the field? (b) It would be great if the authors could briefly discuss why the decomposition of the major question is necessary, what’s the difference or connection between the proposed inspiration identification (the first step of the three) and Retrieval Augmented Generation (RAG).

2.	As the end goal is to rediscover the chemistry scientific hypotheses, the upper bound of “rediscovery” is the exact match of the original hypothesis, intuitively if we rank the original hypothesis with the generated hypothesis, the original hypothesis may be ranked at the top for most of the time. Experiment on this and analysis about the cases where the original hypothesis is not ranked at the top would be interesting to better reflect LLM’s ability to perform R(h).

3. Although the proposed approach constructed a benchmark as the basis to evaluate the performance, it is unclear to what degree the generated hypotheses can be trusted or reliable, not made up from baseless hallucination.

questions:
See the weaknesses listed above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648156098

thread_type:
official_review

<thread_object>
creator: reviewer_zdbz

<content>
summary:
This paper investigates the potential of LLMs to automatically discover novel and valid hypotheses in the field of chemistry. The study introduces a framework called MOOSE-Chem, which decompose the main research question into three smaller questions, respectively focusing on (1) the identification of inspiration papers; (2) the inference of unknown knowledge that is highly likely to be valid; and (3) the identification and ranking of hypotheses generated by the LLMs.

strengths:
Originality:
Firstly, while LLMs have been utilized for scientific discovery in social science and NLP, this paper is the first to investigate their potential in chemistry. 
Besides, The MOOSE-CHEM framework employs a three-step approach to retrieve inspiration papers, inference valid knowledge, identify hypotheses and rank them, which hasn’t been used in previous research. 
Moreover, the use of the evolutionary algorithm to foster a broader diversity in hypothesis generation is also an innovation point.

Quality:
The paper includes extensive comparative experiments and ablation studies, providing a thorough evaluation of LLMs' performance. It also demonstrates the consistency between expert evaluation and automated evaluation of MS, enhancing the reliability of the results.

Clarity: 
The methodology is well-structured, and the three steps based on smaller questions are well-defined. Besides, detailed evaluation methods and corresponding formulas are provided for the three research questions, Q1, Q2 and Q3.  The practical implementation process is also relatively clear.

Significance:
The ability of LLMs to generate hypotheses can significantly accelerate the pace of scientific discovery by reducing the time and effort required for hypothesis generation. It can also open researchers’ minds for the application of LLMs in scientific discovery.

weaknesses:
Firstly, using the same large language model to evaluate its own generated results may introduce bias. It is recommended to try using different LLMs to evaluate the results so as to guarantee the reliability of the results. For example, consider using models like LLaMa[1], Claude[2], Gemini[3], or other recent LLMs to compare outputs. If using the same LLM is necessary, you could collect hypotheses generated by humans and also have both experts and GPT-4 evaluate them. Then, compare their Hard/Soft Consistency Scores as well as distribution of MS with those of the hypotheses generated by the LLM.

[1] Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint arXiv:2307.09288, 2023.
[2] Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://www.anthropic.com, 2024.
[3] Team G, Anil R, Borgeaud S, et al. Gemini: a family of highly capable multimodal models[J]. arXiv preprint arXiv:2312.11805, 2023.

Besides, during the evaluation process (as demonstrated in section 5.1), in the constructed dataset, other papers besides the ground truth inspiration papers may also be helpful for the current background. This step may also require further evaluation to obtain more reliable results. You might select several high-ranking papers that are not included in set I, particularly those that repeatedly appear in experiments with varying corpus sizes or screen window sizes, and conduct case studies on them.

Additionally, this paper only employs the titles and abstracts for retrieval, which might overlook inspirations that could arise from the detailed content of the articles. Sections like Conclusion or Discussion often contain insights into the paper's limitations and suggestions for future work. These elements are usually crucial for generating inspiration.

Moreover, the benchmarks mentioned in the paper contain only 51 references, which may not be sufficient to fully assess the effectiveness of the proposed method. You are advised to increase the number of samples, especially selecting literature from different chemical fields. In addition to traditional categories such as Analytical Chemistry, Organic Chemistry, and Inorganic Chemistry, broader research topics like Environmental Chemistry, Medicinal Chemistry, and Nuclear Chemistry can also be considered. Furthermore, Nature Chemistry includes interdisciplinary areas such as Bioinorganic Chemistry, Bioorganic Chemistry, and Organometallic Chemistry, which could be investigated as well. Of course, this may involve considerations related to the resource consumption of LLMs.

Finally, although the paper used chemistry PhD students for the evaluation, there is still subjectivity that may affect the reliability of the evaluation results. You might consider inviting more experts to participate in the evaluation and analyzing the consistency of the assessments from different human individuals.

questions:
1. There is a lack of experimental exploration for mutation settings.   I am curious about the impact of mutations on the final results. 
     For example, according to the ablation studies, what proportion of high-quality hypotheses can be obtained directly without mutations?

2. How can we explain that a smaller window size leads to better performance in the inspiration retrieval phase?

3. Did the authors replace GPT-4o for all comparison methods for generation? Please specify.

4. How helpful is it to add new dimensions of significance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732194658913

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank All Reviewers

comment:
To all reviewers:

We would like to thank all reviewers for their thoughtful insights and valuable comments. 

It seems that the vacancy of a summarization of the contributions of this paper have caused many misunderstandings. We summarize here and will add them to the paper:

1. We provide the first mathematical derivation on how to decompose the seemingly-impossible-to-solve question P(hypothesis | research background) into many executable and practical smaller steps. This decomposition make P(hypothesis | research background) possible to be practical;
2. We develop a scientific discovery framework directly based on the mathematical derivation. Different from previous work, we propose an evolutionary algorithm based method to better associate background and inspiration, multi-step inspiration retrieval and composition, and an efficient ranking method. In addition, the framework can be applied to chemistry and material science, where previous methods only covered social science, NLP, and biomedical domains;
3. We construct a benchmark by three chemistry PhD students, consisting of 51 chemistry papers published on Nature, Science, or a similar level, decomposing each paper into the research background, inspirations, and hypothesis;
4. **For the first time, we show that an LLM-based framework can largely rediscover many chemistry hypotheses that have published in Nature and Science**. It is guaranteed that the rediscovery is not because of data contamination, because we have controlled the date of the training corpus of the LLM and the date of the chemistry papers that are available online.

We are excited that you recognized our contributions. We quote correspondingly as below:

1. "the authors did a good job guiding the reader through the motivation, the derivation of the method and motivating each of the proposed steps and experiments" [reviewer TMW5]; "Generating research hypotheses is a complicated task, and the authors heuristically decomposed hypothesis generation into ..." [reviewer rMit]; "corresponding formulas are provided for the three research questions, Q1, Q2 and Q3." [reviewer zDBz]
2. "The MOOSE-CHEM framework employs a three-step approach to retrieve inspiration papers, inference valid knowledge, identify hypotheses and rank them, which hasn’t been used in previous research." [reviewer zDBz]; "The use of beam search and evolutionary algorithms for hypothesis refinement seems simple and powerful" [reviewer km2P]; "this paper is the first to investigate LLM's potential in chemistry" [reviewer zDBz]; 
3. "To test this approach, the authors create a benchmark of 51 high-impact chemistry papers published in 2024." [reviewer rMit]; "The authors gathered a carefully curated dataset of scientific papers where questions and hypotheses were extracted for evaluation."[reviewer TMW5]
4. "The results, ..., show promising success, with many rediscovered hypotheses closely matching the originals and capturing core innovations." [reviewer rMit]

We are grateful that you also found that:
1. The paper is generally well-written, and in good English. [reviewer TMW5]
2. The experiments to verify each of the research questions are well-designed with good quality. [reviewer rMit]
3. Every claim made on the performance of the method is generally backed up with experiments. [reviewer TMW5]
4. This paper provides a thorough evaluation of LLM's performance on the scientific discovery task. [reviewer zDBz]
5. Related work is well covered, and experiments are included that compare the proposed method with previous work. [reviewer TMW5]


We also appreciate many helpful suggestions, based on which we have improved our manuscript. The main changes are:
1. We added Claude-3.5-Sonnet and Gemini-1.5-pro for automatic evaluation for all experiments, replacing GPT-4o. 
2. We added the consistency score between experts for expert evaluation.
3. We analyzed the proportion of high-quality hypotheses from the mutation branch and not from the mutation branch.
4. We added the analyse on the effect of the significance feedback.
5. We added the ranking ratio of ground truth hypotheses.
6. We added the results of random screening baseline for inspiration retrieval.
7. We added different size of llama's experiments (Llama-3.1-8b, Llama-3.1-70b, and Llama-3.1-405b), in addition to GPT-4o.


We would again like to thank all reviewers for their time and effort, and we hope that our changes adequately address all concerns. We are glad to have further discussions and take additional suggestions to help improve our manuscript.

Sincerely,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose to combine LLMs with traditional search strategies for generating/refining hypotheses in scientific domains (e.g., chemistry). Hence, the problem setting is very challenging and rather unexplored until now. The technical solution may not be the end of the story but appropriate. The paper has also been well received by the reviewers. Particularly the empirical evaluation is strong and provides convincing results.