Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

Authors: Chenglei Si, Diyi Yang, Tatsunori Hashimoto
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, Idea Generation, Human Study

Abstract:
Recent advancements in large language models (LLMs) have sparked optimism about their potential to accelerate scientific discovery, with a growing number of works proposing research agents that autonomously generate and validate new ideas. Despite this, no evaluations have shown that LLM systems can take the very first step of producing novel, expert-level ideas, let alone perform the entire research process. We address this by establishing an experimental design that evaluates research idea generation while controlling for confounders and performs the first comparison between expert NLP researchers and an LLM ideation agent. By recruiting over 100 NLP researchers to write novel ideas and blind reviews of both LLM and human ideas, we obtain the first statistically significant conclusion on current LLM capabilities for research ideation: we find LLM-generated ideas are judged as more novel (p < 0.05) than human expert ideas while being judged slightly weaker on feasibility. Studying our agent baselines closely, we identify open problems in building and evaluating research agents, including failures of LLM self-evaluation and their lack of diversity in generation.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730272200151

thread_type:
official_review

<thread_object>
creator: reviewer_r2ms

<content>
summary:
This paper investigates the potential of LLMs to autonomously generate innovative research ideas comparable to those created by human experts. 
In a controlled study, over 100 NLP researchers evaluated ideas generated by both humans and an LLM agent across metrics like novelty, excitement, feasibility, and effectiveness. The findings reveal that LLM-generated ideas were rated significantly more novel than human ideas, though slightly lower on feasibility.

The authors addressed multiple experimental challenges, including ensuring a balanced comparison between human and LLM ideas by standardizing topics and using blind reviews. 
They also tested the robustness of results through various statistical methods. The LLM utilized a retrieval-augmented generation approach and scaled inference techniques to produce numerous ideas, from which a ranking mechanism selected the best ones. Despite the promising novelty scores, the study identifies several limitations of LLMs in research ideation, particularly a lack of diversity in ideas and challenges with reliable self-evaluation.

The study highlights open questions in designing research agents and emphasizes the potential and current limitations of LLMs in academic innovation.

strengths:
1. The topic is interesting, as employing LLMs to generate research ideas holds practical value for researchers. Additionally, the findings offer valuable insights that can help researchers determine whether to adopt LLMs as part of their research toolkit.

2. The study’s experimental design is well-structured, effectively controlling for potential confounding factors. Through the use of blind reviews and standardized styles and topics, the authors achieve a fair and balanced comparison between ideas generated by humans and those by LLMs.

3. This paper incorporates a large-scale human evaluation, involving over 100 expert NLP researchers to create and assess research ideas. This approach provides a robust evaluation of the ideas across multiple metrics.

weaknesses:
1. As noted in L434-L435, I appreciate that the authors acknowledge the subjectivity involved in reviewing ideas rather than executed papers. However, it would still be valuable to compare LLM-generated ideas with those from published papers (as ground truth), as the latter have been deemed at least ``feasible''.

2. According to L175-L179, the LLM agent was created with a minimalist approach, incorporating only paper retrieval, idea generation, and ranking. However, in real-world research, high-quality ideas often benefit from iterative refinement and prototyping, which are essential for improving clarity and feasibility. As such, I view this paper as an initial investigation into whether LLMs, in their current, straightforward form, can generate high-quality research ideas. The limitations observed, such as the lower feasibility of LLM-generated ideas, could potentially be addressed by a more sophisticated agent framework that includes stages for refining and prototyping ideas.

3. I would not consider this as a major weakness, but given the insights provided, it would be beneficial if the authors offered best practices or recommendations for using LLMs effectively to generate research ideas.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692628904

thread_type:
official_review

<thread_object>
creator: reviewer_yqjh

<content>
summary:
They constructed a meticulously controlled experiment comparing human-generated and LLM-generated ideas, thus overcoming the sample size and baseline issues present in previous small-scale evaluation studies. The study recruited 138 outstanding natural language processing researchers to generate human baseline ideas and conduct blind reviews (49 were responsible for writing ideas, and 79 for blind review). To reduce the impact of confounding factors, the study strictly controlled the style of the ideas and standardized the topic distribution between human and LLM-generated ideas. After a year-long high-cost evaluation, the authors provided a human expert baseline and a standardized evaluation protocol, laying a foundation for future related research.

In nearly 300 reviews, the authors found that under multiple hypothesis tests and different statistical tests, AI-generated ideas were considered more novel than those of human experts, but less feasible. In addition to evaluating the ideas, the study also analyzed the LLM agent, revealing its limitations and unresolved issues. Despite high hopes for expanding LLMs' reasoning abilities, the authors found that they still show deficiencies in the diversity of idea generation, and the results of human consistency analysis indicate that LLMs cannot yet serve as reliable evaluators of scientific idea quality.

In summary, the contributions of the paper are as follows:

1. Standardized some evaluation criteria for scientific ideas generated using large language models while demonstrating the limitations of LLMs as judges in evaluating scientific ideas.
2. Conducted a blind review evaluation of the quality of scientific ideas generated by LLMs through the recruitment of a large number of scientists, verifying both the potential and shortcomings of LLMs in the field of scientific research.

strengths:
1. The results are quality assured due to significant investment. The author recruited a total of 138 excellent natural language processing researchers and verified their quality through Google Scholar. On average, each idea took the researchers 5.5 hours to generate, and there was a reward of $300. As for the evaluators, each evaluation opinion had a reward of $25. This incentive mechanism, to a certain extent, ensures the quality of idea generation and evaluation.
2. It is relatively fair. By standardizing the topics and providing human experts and large language models with the same writing templates, and finally passing both outputs through the same style transformation module for rewriting, the consistency of style between the two is ensured, minimizing human bias towards writing style as much as possible.
3. It is very meaningful. This large-scale manual evaluation verifies the ability of large language models to generate scientific ideas comparable to those of human experts, providing motivation for the development of subsequent work.

weaknesses:
1. The author concluded that ideas generated by large language models lack diversity. However, using the same prompts, especially longer prompts, it is easy for large models to produce similar responses. Therefore, the lack of diversity may not be solely due to the models themselves; it could also be due to a lack of diversity in the prompts. The author needs to test more prompts to sufficiently reach such a conclusion for this experiment.
2. The paper only verifies the generation quality of the author's simple implementation of an agent, lacking evaluation against other baselines.
3. The research is limited to the NLP field and lacks studies in other domains.

questions:
1. Could you provide the specific prompt used for the Agent to generate ideas?
2. Observing that some researchers in Table 3 and Table 4 have a familiarity level of 1 with the domain (which could be considered as low-quality annotation), what are the experimental results in Table 5 after removing this data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709282364

thread_type:
official_review

<thread_object>
creator: reviewer_hdan

<content>
summary:
- Study: This paper presents a novel study comparing research ideas generated by Large Language Models (LLMs) with those crafted by expert NLP researchers.
- Methodology: The study involves hired expert researchers who generate ideas and conduct blind reviews of both human- and AI-generated ideas.
- Conclusion: This work is the first to conclude that AI-generated ideas are judged as significantly more novel than those from human experts (p < 0.05).

strengths:
* Thorough Study Design: This study is detailed and carefully planned, taking a year to complete and requiring significant resources. It includes human expert baselines and a clear evaluation process, with strong statistical methods that make the results reliable.

* Key Conclusion: The study makes an important finding about the differences between AI-generated and human-generated research ideas, adding to our understanding of AI's potential in coming up with research ideas. — more discussions on the absolute quality, in addition to the comparison between AI and human, would be more apprieciated (e.g. are the quality of both ideas bad).

* Interesting Findings: The study also reveals some surprising insights, such as AI-generated ideas becoming repetitive as more ideas are created, and that current AI systems aren’t yet good at reliably evaluating research ideas

weaknesses:
* While this work is interesting and presents important conclusions, it does not center on machine learning methodology or technical analysis. Although it is relevant and intriguing for the ML community, the paper leans more towards evaluating scientific idea generation, aligning more with applications rather than with core ML research.
* The study is limited by its exclusive focus on prompting-based NLP research, which restricts the generalizability of its findings to other fields or other ML directions.
* Although the authors worked to standardize evaluation criteria, research ideation remains inherently subjective, as reflected in the study's inter-reviewer agreement of 56.1%

questions:
- The authors used reviews from Openreview, are there any licensing issues?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714872630

thread_type:
official_review

<thread_object>
creator: reviewer_3xgm

<content>
summary:
This paper evaluates if LLM based RAG agents are capable of generating novel research ideas for prompting techniques by comparing them with expert researchers. The key contributions of their work are 

1) Evaluating the capabilities of LLMs to generating research ideas and analyzing their strengths (novelty) and weaknesses (feasibility and diversity)
2) They focus on removing confounders by conducting a large scale human evaluations and controlling experimental setting

strengths:
1) Extensive human evaluation with over 100 NLP researchers
2) Great care taken to remove confounders through style normalization, review form, grant submission form for research ideas etc. 
3) Interesting insights like lack of ability of LLMs to generate diverse research ideas.

weaknesses:
1) Not strong enough motivation for restricting to prompting based research. This could have been a confounder since we did we are not sure about the ability of LLMs to generate research where the solutions are more complex. 

2) Usefulness: The paper does not talk about the usefulness of the generated ideas research ideas, which is more important than novelty. Research ideas are born out of deep insights into the limitations of previous research. This paper neither gives the discussion/limitations section of research papers to the LM as context nor does it do any analysis of the usefulness of the generated ideas. Using abstracts and paper titles sounds like an anti-causal way of generating ideas which I think could affect the usefulness.

questions:
1) RAG pipeline ablation tests: "We keep the top k = 20 papers from each executed function call and stop the action generation when a max of N = 120 papers have been retrieved." , "For retrieval augmentation, we randomly select k = 10 papers from the top-ranked retrieved papers" . Can you please provide some small scale (self-evaluated) tests ablating these parameters to ensure their optimality?

2) Researcher Background: I understand that the researcher familiarity numbers are provided for idea generation and reviewing processes, however it is slightly different from expertise. So, a better way to ensure the robustness of results is to look at the distribution of the expertise of the reviewers. 

3) Interestingness: Can you please clarify the meaning of the term? Is it that the insight behind the proposed idea a non-trivial one or is it that the human evaluating it finds it personally interesting?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732227881854

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all the reviewers for their insightful reviews and constructive feedback. We notice a common question raised by the reviewers (3XGM, HdAN, and YqJH): why do we restrict to prompting-based NLP research for the study? We address this concern below. 

As we explained in our paper (lines 118-122), our motivation for restricting to prompting-based NLP research is twofold. 

First, prompting-based research requires minimal computing hardware and is generally easy to implement and execute compared to many other types of AI research. We intend for our ideation study and the data / ideas generated from this to serve as a useful set of resources and a foundation from which to study the excitability of AI-generated ideas, as well as automated research execution agents. Therefore the feasibility of execution is important for facilitating such future work. 

Second, prompting is an active area of research, and many prompting-based research papers have been published at AI/NLP/ML conferences in recent years. For example, in this current ICLR conference itself, 1194 submissions mentioned prompting in the title or abstract. Successful prompting research (such as chain-of-thought prompting) has had a big impact on the research community.

We believe our study in its current scope and design has substantial value for the community. Future work can take our evaluation framework and extend it to other research domains.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**

This paper aims to study whether LLMs can be scientifically creative and then evaluates if LLM based RAG agents are capable of generating novel research ideas for prompting techniques by comparing them with expert researchers. 
This paper addresses the question that we all don't want to be answerd as discussed in [1]. 


**Strengths**

The extensive analysis made using NLP researchers

**Weaknesses**
Clearly, this kind of paper has many weaknesses as we don't want LLMs to succed in this task.
- The usefulness of the generated ideas does not seems to be proven (reviewer *3XGM* is not convinced)


**Final remarks**

The paper touches important issues.

** References **

[1] GASP! Generating Abstracts of Scientific Papers from Abstracts of Cited Papers