Two Heads Are Better Than One: A Multi-Agent System Has the Potential to Improve Scientific Idea Generation

Authors: Haoyang Su, Renqi Chen, SHIXIANG TANG, Xinzhe Zheng, Jinzhe Li, Zhenfei Yin, Wanli Ouyang, Nanqing Dong
Primary area: applications to robotics, autonomy, planning
Keywords: Large Language Model, Multi-agent System, Collaboration Strategy, Automatic Scientific Discovery, Science of Science

Abstract:
The rapid advancement of scientific progress requires innovative tools that can accelerate discovery. While recent AI methods, particularly large language models (LLMs), have shown promise in tasks such as hypothesis generation and experimental design, they fall short in replicating the collaborative nature of real-world scientific practices, where diverse teams of experts work together to tackle complex problems. To address the limitation, we propose an LLM-based multi-agent system, i.e., Virtual Scientists (VirSci), designed to mimic the teamwork inherent in scientific research. VirSci organizes a team of agents to collaboratively generate, evaluate, and refine research ideas. Through comprehensive experiments, we demonstrate that this multi-agent approach outperforms the state-of-the-art method in producing novel and impactful scientific ideas, showing potential in aligning with key insights in the Science of Science field. Our findings suggest that integrating collaborative agents can lead to more innovative scientific outputs, offering a robust system for autonomous scientific discovery.

Decision: reject
Reviewer scores: 5, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730433758423

thread_type:
official_review

<thread_object>
creator: reviewer_wxag

<content>
summary:
The paper introduces VIRSCI, a Large Language Model (LLM)-based multi-agent system designed to simulate the collaborative nature of scientific research teams. The system comprises virtual scientists that engage in collaborative discussions to generate, evaluate, and refine research ideas. The authors aim to replicate the teamwork inherent in scientific discovery and explore how multi-agent collaboration can enhance the novelty and impact of generated scientific ideas.

strengths:
**Novel Exploration of Team Size and Idea Novelty**: The paper provides an insightful exploration into how team size affects the novelty of generated ideas. It demonstrates that there is an optimal team size (e.g., eight members) that maximizes creativity without overwhelming the collaborative process. This finding is interesting and contributes to the understanding of team dynamics in scientific idea generation.

**Rich Experiments and Ablation Studies**: The authors conduct extensive experiments and ablation studies, examining various factors such as team size, team freshness, research diversity, and discussion patterns. This comprehensive approach strengthens the validity of their conclusions and provides valuable insights into key variables affecting multi-agent systems in research.

**Simulation of Collaborative Scientific Processes**: The paper presents a structured framework that simulates the collaborative process of scientific research, from team formation to idea generation and evaluation. This approach aligns closely with real-world scientific practices and showcases how LLM-based multi-agent systems can be applied in this context.

weaknesses:
**Possible Data Leakage Affecting Novelty Evaluation**: In Section 4.1, the authors utilize a dataset comprising papers published between 2000 and 2014, with a particular emphasis on data from 2011 to 2014. Given that contemporary LLMs may have been trained on data from this period, there is a significant risk of data leakage. This means the models might have already been exposed to the data used in the experiments, potentially generating ideas that are not genuinely novel but rather reproductions of existing concepts. This compromises the ability to accurately assess the novelty of the generated ideas and undermines the validity of the experimental results.

**Lack of Human Evaluation to Validate LLM Metrics**: The paper relies heavily on LLM-based evaluations (e.g., using GPT-4) to assess the quality and novelty of the generated ideas and abstracts. However, there is no involvement of human annotators or experts to validate these metrics. Without human evaluation, it is difficult to ascertain the effectiveness and reliability of the LLM review metrics. The absence of human validation raises concerns about the robustness of the conclusions drawn from these evaluations.

**Limited Innovation in the Framework**: The proposed framework seems to be an application of existing LLM-based multi-agent systems to the specific domain of scientific idea generation. While the application is interesting, the methodological innovation appears limited. The framework primarily extends basic LLM multi-agent methodologies without introducing significant novel approaches or mechanisms specific to the challenges of research idea generation.

questions:
Suggestions for Improvement:

**Address Data Leakage Concerns**: To mitigate the impact of data leakage, the authors should consider using more recent or entirely new datasets that the LLMs have not been exposed to during training. Alternatively, they could implement techniques to control for data overlap and assess how much the models' prior knowledge influences the results.

**Include Human Evaluation**: Incorporating human annotators or expert reviewers in the evaluation process would enhance the validity of the findings. Human judgments could corroborate the LLM-based assessments and provide nuanced insights into the quality and novelty of the ideas.

**Highlight Methodological Innovations**: The authors should emphasize any unique aspects of their framework that advance multi-agent systems for scientific research. Detailed explanations of innovative strategies or mechanisms would strengthen the contribution.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730443103405

thread_type:
official_review

<thread_object>
creator: reviewer_poux

<content>
summary:
This paper introduces VIRSCI, an LLM-based multi-agent system, a framework specifically designed to model the teamwork collaboration style in scientific research. This is a framework of multiple agents collectively working on research idea generation made up of five phases, similar to how a human would do.  To do so, this paper proposes a novel team discussion mechanism based on a paper database as the knowledge bank, leading to generating abstracts and ideas from the LLM agents. To evaluate this framework, the authors also introduce a benchmark focusing on the novelty of the idea to measure performances of their model from three aspects.

strengths:
1. Novel framework with a new dataset and metric: The authors propose a novel research generation framework with LLMs collaborating on this task through a 'team discussion' mechanism to simulate discussion scenarios in real life. With abstracts as the novel ideas output of the model, the authors also construct a benchmark dataset and evaluation metrics to measure the performances. 
2. Comprehensive experiment design: the authors consider multiple aspects of the framework, for example, the # of team members, components of the system, etc. would make a difference to the result as an ablation study.

weaknesses:
1. Limited domain and scope in the benchmarking dataset: the authors only include computer science as the scientific research domain in the dataset and with a limited number of years. Given the rapid development in the field, this limited temporal scope may fail to capture the latest developments and trends, especially since the number of papers and ideas is not growing linearly. Expanding the dataset to include a broader range of years or domains would provide a more comprehensive foundation for idea generation and make this paper more sound.
2. Limited evaluation via a self-defined metric focused on novelty: the paper evaluates the model's performance mainly through a novelty metric derived from embedding distances and based on citation counts. While novelty is important, it should not be the only focus of research idea generation. Additionally, semantic comparisons are made with papers from only 2000-2010 and 2010-2014, which may not fully represent this field’s progress. To better validate their metric, the authors could supply human evaluation or case studies to validate that semantic distance accurately reflects the quality of this metric.

questions:
1. Limitation of the dataset: I like the idea of the collaborative approach of the agents in generating research ideas, but I am curious about the model’s potential for generalization as the knowledge bank expands to include papers from additional domains and recent years. The current threshold, limited to authors from 2000 to 2014, may overlook recent growth in CS publications. Could you discuss how extending the dataset to include more contemporary research might impact the model’s ability to generalize and its applicability beyond historical data?
2. Validity of the ideas: the authors propose three metrics for measuring their VIRSCI’s performances compared to AI Scientist, but those mainly focus on the novelty and the potential impact of the idea by comparing embedding distances. However, since this is a research idea generation system and if we potentially want to put it into actual use, we would not only care about the novelty of the idea but also the feasibility and validity of the idea, if the idea makes sense, and how reasonable they are, which is currently missing in the currently proposed metrics. The whole paper puts a lot of emphasis on measuring the novelty of the ideas but does not mention if they manually review if the outputs make sense, there is a chance that the idea is novel but is due to the model's hallucinations. 
3. Practical use of the tool: Given that this framework is proposed as a system for generating research ideas, I am curious about its practical value beyond generating novel concepts. While the focus on novelty is valuable, real-world research applications require ideas to be both feasible and sound. Could you expand on how this system might be used practically and how you see it balancing novelty with the reliability or reasonableness of its outputs? This could be done with human evaluation, or comparing generated ideas with more recent papers and see if there are actual matches, etc.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730496466899

thread_type:
official_review

<thread_object>
creator: reviewer_6tuq

<content>
summary:
The paper introduces VIRSCI, a multi-agent, LLM-based system designed to simulate teamwork-driven scientific research. The system organizes agents to mimic collaborative processes, including selecting collaborators, generating research ideas, assessing novelty, and drafting abstracts.

strengths:
1. The multi-agent approach proposed in this paper has the potential to greatly enhance the quality and breadth of scientific research. The discussion-oriented idea generation closely mirrors real scientific processes.
2. The 5-step approach proposed in this paper—comprising Collaborator Selection, Topic Selection, Idea Generation, Idea Novelty Assessment, and Abstract Generation—presents a promising and robust framework for idea generation. The evaluation metrics (CD, CI, and HD) are logically sound.
3. VIRSCI, a multi-agent system for scientific collaboration, shows clear advantages over single-agent methods.

weaknesses:
1. Generating the abstract is a solid starting point, but conducting at least some preliminary experiments based on the abstract would add greater impact.
2. There is no metric to assess the practical feasibility of the abstract or idea. While VIRSCI may generate highly unique or novel ideas, these are less valuable if experimental designs cannot support them within practical constraints.
3. The system’s effectiveness may vary with the underlying LLM’s capabilities.

questions:
1. Could "usefulness" and "practically-feasible" metrics be added during the novelty assessment to provide a broader evaluation of the generated ideas?
2. In section 3.1 under adjacency matrix section, why choose a simple increment of 1 in the adjacency matrix? Would a distribution function, like a normal distribution, or an explore-exploit model provide better results?
3. Given the potential variation in capabilities across LLMs, have you assessed how team size or turn count might need adjusting for different models?
4. How do you ensure that high-scoring abstracts are practically feasible for real-world scientific research?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618744266

thread_type:
official_review

<thread_object>
creator: reviewer_2x9t

<content>
summary:
This paper proposes a new multi-agent system VIRSCI, designed to mimic the teamwork inherent in scientific research. Multi-agent methods can produce more novel and influential scientific ideas than single agent. This indicates that integrating collaborative agents can lead to more innovative scientific outputs, offering a robust system for autonomous scientific discovery.  The contributions of this paper are mainly in the following three aspects:

1. A new multi-agent system VIRSCI was proposed, which constructed the entire pipeline from team organization to final idea formation.

2. The experiment proves that VIRSCI has better performance than single-agent. At the same time, the paper explores the impact of different settings in the VIRSCI system on the final results.

3. The simulation results are consistent with key findings in Science of Science, such as that new teams tend to produce more innovative research, demonstrating VIRSCI's potential as a powerful tool for future research in this field.

strengths:
1. Originality

This paper is the first to apply the LLM-based multi-agent system to the problem of scientific discovery. It realizes the generation of research ideas in autonomous scientific discovery.

2. Quality

This paper not only implements a multi-agent system for automatic scientific discovery, but also conducts a large number of experiments to verify the effectiveness of the system, and explores the different effects of system settings on the results. It also conducts a lot of experiments to verify the effectiveness of many methods in the system, which is a high-quality work.

3. clarity

This paper clearly explains the construction and operation process of the system, and details the implementation of each step of the process. At the same time, the language description of the paper is very clear, and the technical explanation is detailed. The experiment part also clearly describes the experimental settings and experimental process. Many details in the experiment are explained and demonstrated in the appendix, which allows people to clearly understand all the technical details.

4. significance

This paper is the first attempt to apply the LLM-based multi-agent system to the field of scientific exploration, which allows us to see greater possibilities of AI for science. This shows that in the future, multi-agent technology may really be able to make new and valuable scientific discoveries.

weaknesses:
This paper not only pioneered the use of multi-agent technology in a completely new field, but also conducted sufficient experiments to verify the effectiveness of this system. The paper has a clear structure and logic, and the language is clear, which allows people to clearly understand the core ideas and innovations of the paper. However, this paper still has the following weaknesses:

1. The indicators of the experimental part of the paper focus more on the novelty and dissimilarity of the ideas generated by VIRSCI, but are there any experiments that can illustrate the usability of scientific discoveries generated by VIRSCI? Is there an analysis of the feasibility of all the abstracts generated by the system, and what proportion of them are likely to provide exploration for our scientific discoveries after preliminary screening?

2. The paper is very clear in its drawings and processes, but the organizational structure and interaction mode of each agent in each process are relatively poorly described. In addition, the differences and characteristics of the other scientist agents, except for the team leader, are not well described.

3. The experimental scenario of the paper is relatively simple, and experiments were only conducted in the field of computer science, which is also mentioned in the paper.

questions:
1. The experimental part of the paper cannot illustrate the usability of scientific findings, which is relatively fatal. Can you provide some analysis or examples to prove that the scientific findings generated by VIRSCI can be instructive to us?

2. From the paper, I saw that the multi-agent system team adopted a loop execution strategy during the discussion. Does this part of the paper propose a novel organizational structure and interaction mode? Can sequential execution guarantee the final effect of the experiment?

3. The experiment setting in the ablation study was conducted under a 5-turn discussion. Is it possible that the effect of other turn discussions cannot produce such a conclusion? Experiments conducted under only one setting lack certain persuasiveness to show that these technologies are necessary, and whether it can be strengthened through examples to prove that these technologies do improve the overall performance of the system.
</content>
</thread_object>

</thread>


Metareview: