Title: Predicting Empirical AI Research Outcomes with Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Research Idea Evaluation

Automated Research
Abstract: Many promising-looking ideas in AI research fail to deliver, but their validation takes substantial human labor and compute. Predicting an idea's chance of success is thus crucial for accelerating empirical AI research, a skill that even expert researchers can only acquire through substantial experience. We build the first benchmark for this task and compare LMs with human experts. Concretely, given two research ideas (e.g., two jailbreaking methods), we aim to predict which will perform better on a set of benchmarks. 
We scrape ideas and experimental results from conference papers, yielding 1,585 human-verified idea pairs \textit{published after our base model's cut-off date} for testing, and 6,000 pairs for training.
We then develop a system that combines a fine-tuned GPT-4.1 with a paper retrieval agent, and we recruit 25 human experts to compare with.
In the NLP domain, our system beats human experts by a large margin (64.4\% v.s. 48.9\%).
On the full test set, our system achieves 77\% accuracy, while off-the-shelf frontier LMs like o3 perform no better than random guessing, even with the same retrieval augmentation.
We verify that our system does not exploit superficial features like idea complexity through extensive human-written and LM-designed robustness tests.
Finally, we evaluate our system on unpublished novel ideas, including ideas generated by an AI ideation agent.
Our system achieves 63.6\% accuracy, demonstrating its potential as a reward model for improving idea generation models.
Altogether, our results outline a promising new direction for LMs to accelerate empirical AI research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework to predict the experimental performance of machine learning methods from descriptions of research ideas. The authors create a dataset that includes pairwise comparisons of performance on benchmarks, constructed using results presented in conference papers. Their experiments show that existing models, including GPT-4.1 and o3, cannot effectively predict experimental results even when provided with related papers. The result shows that GPT-4.1 fine-tuned on their dataset achieves better performance compared to human experts in the pairwise performance prediction task.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* This paper targets a crucial and useful research topic. Predicting the experimental performance of research ideas would be a crucial part of research automation.\n\n* Novel dataset made with manual verification. The dataset that includes pairs of research ideas and experimental results is novel and useful. While the dataset is automatically created using LLMs, the quality is verified by manual annotation.\n\n### Weaknesses\n\nAlthough the idea is promising, the paper does not provide sufficient information to establish the reliability of its claims. Additionally, I have concerns regarding potential dataset contamination. I recommend a substantial revision before the work is considered for publication.\n\n* Concern about data contamination\n\nLines 72 and 93 suggest that the authors use conference publication dates as the publication dates of papers. However, many papers are publicly available as preprints several months before their conference publication, and LLMs are often exposed to these preprints. Moreover, the paper retrieval module (Line 135) retrieves documents from arXiv, and the retrieved papers may contain information about the target ideas. To more reliably avoid data contamination, the authors should consider using the preprint publication dates rather than the conference publication dates.\n\nIn addition, according to [the official information from OpenAI](https://openai.com/index/gpt-4-1/), the knowledge cutoff date of GPT-4.1 is June 2024, though the exact date is not specified. To ensure a more reliable exclusion of training data, I believe it would be safer to include only papers published after July 2024.\n\n* Concern about the shortcut in the proposed method\n\nAs stated in Line 130, some of the compared methods were proposed prior to the knowledge cutoff date and may be included in the retrieved documents. Since newer methods often outperform older ones, a trivial baseline would be to recommend methods not present in the retrieved documents. I am concerned that the proposed fine-tuned model may exploit this shortcut when making predictions.\n\nAlthough I consider the experiments in Section 6 essential for addressing this concern, the results are not compared with baseline methods or expert human performance.\n\n* Limited information about the dataset creation process, the dataset, and the results\n\nThe paper lacks detailed statistics, example prompts, samples from the datasets, and raw outputs from the evaluated methods. I recommend including raw examples from the datasets as well as unprocessed outputs from all evaluated methods.\n\n* Too many missing references\n\nThere exist many previous studies in automated research, but this paper cites very few papers. Particularly, the first two paragraphs of the introduction and the second paragraph of the related work section do not cite any papers, and the total number of cited papers is only 19.\n\n* Minor suggestions\n\n  * I consider the majority voting over all benchmarks (Line 88) to be too sensitive to noise. For example, when Idea A outperforms Idea B on 3 out of 5 benchmarks, I am unsure whether Idea A is better than Idea B in a statistically significant manner. I recommend pairing ideas with a larger gap.\n\nQUESTIONS:\nPlease let me know if there are any inaccuracies or misunderstandings in the points raised under Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-12"}

comments: SUMMARY:
This paper is about predicting which of two research ideas is more promising. To realize this, the authors collect a dataset and evaluate LLMs, including a novel AI based ideation agent. Results show that the best models compare favorably to humans.

STRENGTHS AND WEAKNESSES:
**Strengths** 
This could be a potentially very impactful paper, as the results are quite convincing, and the idea and task seem very novel. The paper is also thorough and presents several analyses.

**Weaknesses**
My main concern is that I do not understand the core Section 3. For example, what is happening in query generation? An example would really be extremely helpful and much appreciated.

QUESTIONS:
1) In Section 3, how does the system determine whether sufficient information has been collected?

2) In Section 3 Step 1, what is a query?

3) In l. 130, why is at least one idea completely novel?

4) What is the role of paper summarization in Section 3?

5) Why are the four steps necessary in Section 3 and what is their main purpose?

LIMITATIONS:
The limitation section is extremely brief. One could mention for example, that evaluating the outcome of new ideas could drastically increase human reviewing burdens, among many other.
TITLE: Official Review
CLARITY: 2
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors create a new benchmark to compare LLMs and human experts in predicting which of two research ideas will succeed. The benchmark is composed by scraping ideas and experimental results from conference papers, resulting in 1585 human-verified idea pairs published after the base model's cut-off date, and 6000 pairs for training. Four rounds of students checks every test pair, and a panel of 25 senior researchers supplies the human baseline. Off-the-shelf o3 and Claude 3.5 land at roughly 55% accuracy, just above the experts. Fine-tuning GPT-4.1 on the 6 000 training pairs and letting it read brief paper summaries through RAG results in 77%.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n- The authors introduce a new strongly curated benchmark which measures the ability of LLMs to assess the potential success of research ideas.\n- The authors show that fine-tuning GPT-4.1 outperforms other SOTA LLMs and verify the robustness of these results through many stress-testing studies (including evaluation on novel unpublished ideas).\n\nWeaknesses:\n- While it is a first attempt at such a dataset, the limitation to binary choice is quite limiting to real-life use cases.\n\nQUESTIONS:\n- \"We only consider the predictions valid when both predictions are consistent; otherwise, inconsistent predictions are considered incorrect by default.\" --> Could the authors indicate how many times this happens?\n- The authors state in footnote 1 that Claude 3.7 is not evaluated, but mention Claude 3.7 in the introduction. I believe this should be Claude 3.5?\n\nLIMITATIONS:\nThe authors indicate some limitations. However, I believe they could add that reproducibility is hard since a closed-source model has been fine-tuned, which is not publicly accessible. Also, the data is not shared, but the checklist indicates they will release it.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the topic of predicting outcomes of AI research ideas. A benchmark is proposed to faithfully evaluate the soundness of the AI-generated idea. The construction of the benchmark requires laborious efforts, including selecting papers, extracting ideas. The authors also propose a fine-tuned GPT-4.1 with a paper retrieval agent for this task. The proposed method outperforms existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. It is worthwhile to construct a benchmark to evaluate the soundness of the generated ideas in the domain of AI research.\n2. It is laborious to construct such a benchmark, and the authors provided detailed construction procedures in the paper and it looks reasonable.\n3. The paper proposes a framework to predict the outcomes, and the performance surpasses existing methods.\n4. The paper includes extensive and robust experiments to verify the performance of the proposed framework, including an ablation study on retrieved information, CoT, and stress-testing etc.\n\nWeaknesses:\n1. What are the retrieved papers used for in the final proposed system? Section 3.1 only mentions how to retrieve papers from the database, but it does not mention how retrieval results are used. Are these results simply used as parts of inputs, or are some specific modules proposed for processing this information? \n2. Will the proposed benchmark be released upon the acceptance of the paper? The benchmark is one of the core contributions of the paper, so it is important to have the benchmark publicly available to the community.\n\nQUESTIONS:\n1. What are the retrieved papers used for in the final proposed system? Section 3.1 only mentions how to retrieve papers from the database, but it does not mention how retrieval results are used. Are these results simply used as parts of inputs, or are some specific modules proposed for processing this information? \n2. Will the proposed benchmark be released upon the acceptance of the paper? The benchmark is one of the core contributions of the paper, so it is important to have the benchmark publicly available to the community.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}