Predicting Empirical AI Research Outcomes with Language Models
Authors: Jiaxin Wen
Chenglei Si
Chen Yueh-Han
He He
Shi Feng
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Research Idea Evaluation
Automated Research
Primary area: applications
TL;DR: We build a LM-based system that can outperfrom expert AI researchers in predicting the outcomes of empirical AI research ideas, without running actual experiments.

Abstract: Many promising-looking ideas in AI research fail to deliver, but their validation takes substantial human labor and compute. Predicting an idea's chance of success is thus crucial for accelerating empirical AI research, a skill that even expert researchers can only acquire through substantial experience. We build the first benchmark for this task and compare LMs with human experts. Concretely, given two research ideas (e.g., two jailbreaking methods), we aim to predict which will perform better on a set of benchmarks. 
We scrape ideas and experimental results from conference papers, yielding 1,585 human-verified idea pairs \textit{published after our base model's cut-off date} for testing, and 6,000 pairs for training.
We then develop a system that combines a fine-tuned GPT-4.1 with a paper retrieval agent, and we recruit 25 human experts to compare with.
In the NLP domain, our system beats human experts by a large margin (64.4\% v.s. 48.9\%).
On the full test set, our system achieves 77\% accuracy, while off-the-shelf frontier LMs like o3 perform no better than random guessing, even with the same retrieval augmentation.
We verify that our system does not exploit superficial features like idea complexity through extensive human-written and LM-designed robustness tests.
Finally, we evaluate our system on unpublished novel ideas, including ideas generated by an AI ideation agent.
Our system achieves 63.6\% accuracy, demonstrating its potential as a reward model for improving idea generation models.
Altogether, our results outline a promising new direction for LMs to accelerate empirical AI research.

Predicting Empirical AI Research Outcomes with Language Models
Jiaxin Wen', Chenglei Si?, Chen Yueh-Han', He He², Shi Feng' UC Berkeley *Stanford New York University *George Washington University
Abstract
Many promising-looking ideas research fail deliver, but their validation takes substantial human labor and compute. Predicting an idea's 's chance of success is thus crucial for accelerating empirical AI research, skill that even expert researchers can only acquire through substantial experience. We build the first benchmark for this task and compare L.Ms with human experts. Concretely, given two research ideas (e.g., two jailbreaking methods), we aim to predict which will perform better on a set of benchmarks. We scrape ideas and experimental results from conference papers, yielding 1,444 human-verifies idea pairs published after our base model's 's cut-off date for testing, and 6,000 pairs for training. We then develop system that combines fine-tuned GPT-4.1 with paper retrieval agent, and we recruit human experts to compare with. In the NLP domain, our system beats human experts by large margin (64.4% 48.9%). On the full test set, system achieves 77% accuracy, while off-the-shelf frontier LMs like o3 perform better than random guessing, even with the same retrieval augmentation. We verify that our system does not exploit superficial features like idea complexity through extensive evaluate our system on unpublished novel ideas, including ideas generated by an Al ideation agent. Our system achieves 63.6% accuracy, demonstrating potential as a reward model for improving idea generation models. Altogether, our results outline promising new direction for LMs to accelerate empirical AI research.
and L.M L.M-designed robustness tests. Finally, we
General Accuracy
Stress-test Accuracy
64.4
65.4
60.0
60
60
50
48.9
50.0
38.5
30
30
20
10
10
Ours
Expert (Majority)
Expert (Best)
Ours
Expert (Majority)
Expert (Best)
Figure 1: Our system is more accurate than human NLP experts. Majority aggregates predictions from all annotators, while Best keeps only the best-performing annotator per research topic. For the stress test, we select a subset of idea pairs where the mathematically complex one is actually ineffective. While humans often get misled by this feature, our model does not.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Many promising-looking Al research ideas turn out to be ineffective when executed. Unfortunately, the only way to find out is actually implement them. The failed ideas can add up significant cost in both human labor and computational resources. Better more promising ideas first-can thus significantly improve research efficiency. But doing so requires predicting the outcomes of experiments without actually running them, a seemingly impossible task.
implementing the
We hypothesize that language models (LMs) do this task better than human experts. Humans develop such research intuition through experience, but LMs can acquire it more efficiently by consuming countless research papers, analyzing experimental results, and potentially discovering subtle patterns that are difficult for humans to identify.
Concretely, given the description of a pair of research ideas, goal is predict which one works better on a set of benchmarks. For example, given two jailbreaking methods, we aim to predict which one achieves higher attack success rates. This is objective task: can obtain the groundtruth by actually implementing both ideas. But making a prediction without implementing the ideas is difficult, and any non- non-trivial accuracy could be valuable as it informs resource allocation, prioritization of experiments, and iterative refinement of ideas. Crucially, this task provides more actionable information than evaluating subjective aspects of ideas such as novelty excitement which existing work focuses on.
14 9 4.
We construct benchmark to facilitate the study of this task by scraping both ideas and results from existing conference papers. Each example consists of a research goal specified by a set of benchmarks, two competing ideas, and binary outcome label indicating which idea performs better across these benchmarks. After four rounds of human verification, we obtain 1,444 verified test examples. To avoid data contamination, we ensure that each test example contains at least one idea published after July 1st, 2024, the knowledge cut-off date of our base model GPT-4.1.
We then develop system that combines a specialized GPT-4.1 fine-tuned on 6,000 historical idea pairs from the training set and paper retrieval agent. This system achieves a promising 77% accuracy on the test set. Off-the-shelf frontier models (e.g., o3, Claude 3.5 Sonnet)-even when augmented by the same paper retrieval importance of proper capability elicitation. We then identify challenging test subset consisting of 45 NLP idea pairs, and recruit 25 NLP experts to establish a human baseline. Specifically, each human prediction is made by ensemble of 5 experts who collectively spent over 45 minutes. Our system beats this strong expert baseline by a large margin (64.4% v.s. 48.9%).
no better than random guessing, demonstrating the
To test the system's 's generalizability, we design stress tests to measure its sensitivity to superficial features like idea recency and complexity. On three -umanndesigned stress tests and hundreds tests proposed by LMs, our system demonstrates robust behavior.
Lastly, we test our system on brand new, unpublished research projects from recent study on using AIs to generate ideas [14. This is a set of 35 ideas with experiments implemented by NLP researchers (recruited by the study's authors). These projects have never been released publicly anywhere on the internet, thus avoiding any possible contamination. Our system achieves an accuracy of 63.6%, indicating its generalizability and its potential to improve existing automated research systems 14 as an idea ranker or a reward model.
2 Benchmark
We define our task predicting the outcomes of two competing research ideas for given research goal. We focus on pairwise evaluation, where the binary outcome label is aggregated over multiple benchmarks avoid the ambiguity and noise when evaluating individual ideas [14 9 example involves the following components (Figure 2):
Each
Idea pair, each defined by a detailed description following standard format.
Empirical research goal, defined by set of benchmarks, each with quantitative metric.
Binary outcome label, indicating which idea wins on more benchmarks.

---

Idea A: DiffusionAttacker
Idea B: AdvPrompter
DiffusionAttacker uses seq2seq pre- trained diffusion language models rewrite jallbreaking prompts.
AdvPrompter two-step process iteratively generate adversarial suffixes combined with supervised fine-tuning.
Empirical research goal: Jailbreak LMs
Benchmarkc AdvBench
B
A
Outcome label:
Metric: Attack Success
Benchmark: AdvBench
Metric: Fluency (Perplexity)
A
>
Idea
B
wins
Benchmark: AdvBench
Metric: Diversity (Self-BLEU)
B
>
A
Figure 2: Comparing two ideas for jailbreaking methods on three benchmarks. Each idea is defined by detailed summary. The goal is to predict the outcome, whose label is determined by actually evaluating the ideas on these benchmarks and seeing who wins more.
2.1 Benchmark Construction
We develop an automatic pipeline to extract ideas and their corresponding empirical results from existing papers. The pipeline operates in four steps using Claude 3.5 Sonnet the main LM.
Step Paper Collection. We start by collecting papers from top AI conferences various domains. Table outtiness our data sources. To collect test examples published after frontier LMs' knowledge ouf- we mainly collect papers published in the past two years. After scraping, we use LMs filter out non-empirical papers, such as those focus on "rethinking". and "understanding" topics.
Step 2: Data Extraction. For each paper PDF, prompt LMs to extract the research goal, idea names and their outcomes, mainly from the result tables. Next, we generate summaries for each idea based on the PDF. While the main idea is explained in detail, baseline ideas often described only briefly and potentially in biased way. To address this, we extract the reference for each baseline idea and download the corresponding paper PDF. we fail to find references (c.g.. the reference is missing or the extracted reference is invalid), we discard these ideas. To avoid information leakage, we explicitly instruct LMs
mention any actual empirical results in summaries.
Step 3: Idea Pairing. We then group ideas into comparison pairs. Pairs are created only within the same paper, since cross-paper comparisons might suffer from confounding factors (e.g., differences in hyperparameters or data preprocessing). For example, comparing the results of two parameter- efficient tuning methods reported different papers (e.g., LoRA and prompt tuning) could be invalid they use different numbers of trainable parameters.
Step 4: Label Aggregation. To determine the final comparison label for each idea pair, we do majority voting over all benchmarks. For example, Idea A outperforms Idea on 4 out of benchmarks, we label the pair "Idea A is better". This aggregation strategy mitigates label noise in individual benchmark comparisons. To further improve label quality, we remove idea pairs that result in tie evaluated on fewer than three benchmarks.
Train/Test Splitting. We then split our collected data into train and test sets using our main base model GPT-4.1's ouf- date, July 1st, 2024. In particualr, we use the date that the paper first appeared on arxiv (which we extract from its arxiv ID), not the date that is published at the conference. This ensures no leakage from the fine-tuning process. So the model must predict ideas proposed in the "future work". Data statistics are shown in Table
2.2 Human Verification of Extracted Examples
Since our benchmark is automatically constructed by parsing paper PDFs with LMs, parsing errors inevitable. To ensure the quality and reliability of our test set, recruit human annotators to

---

Table
Data source.
Table 2: Data statisties.
Domain
Conferences
Split
Train
Test
NLP
ACL, EMNLP, NAACL, COLM NeurIPS, ICLR
Size
6,000
1,444
ML
# Benchmark
3.2
3.4
CV
CVPR, ECCV, ACMMM CoRL
After Cut-off'
No
Yes
Robotics
Human-
No
Yes
carefully go through our test set, rewriting or removing problematic examples. Note that this set of human annotators are tasked only to verify the extracted examples; the annotators for our expert baseline have much higher expertise in their respetive domains.
Recruiting Human Annotators. We select college students majoring in Computer Science or Electronic Engineering out of 30 pre-screened candidates. All of them experienced in reading AI papers, and 25% of them have published papers at Al conferences. We train annotators to do the verification task with 50 warmup examples and verify their skills through their evaluation error rate.
Annotators verify and correct each example by consulting the original paper PDF. Table in the Appendix showcases typical errors found by our annotators, sorted by frequency. The most common error is "Incorrect Win Condition", e.g., mistakenly marking the perplexity metric as higher is better. Additionally, the win conditions for certain metrics are context-dependent. For example, Attack Success Rate (ASR) should maximized adversarial attack papers but minimized in defense papers. Annotators fix these errors by rewriting examples removing them when the results are fabricated or incomparable due to unfair experiment settings.
Annotation Process.
To incentivize our annotators to provide high-quality annotations, we design the following bonus schemes. Each annotation initially earns S5. However, if an annotation found incorrect during cross cross-validation by another annotator, the original annotator gets a penalty of $3, while the annotator who catches the eтrоr gets an additional $3 reward. We conduct four rounds of cross- volss-vali.tion.. In each round, we randomly sample 400 examples and send them to review by new annotators. From the first to the last round, the rate of identified mislabels decreased from 1% to 2.5%.
3 Building a System for Research Outcome Prediction
3.1 Retrieval
When predicting outcomes of new ideas, human researchers often draw inspiration from existing literature. While the exact same ideas do not exist, prior studies can offer indirect insights, transferable knowledge, or analysis of similar sub-components. Therefore, we develop an agentic retrieval module that iteratively performs the following four steps: query generation, paper retrieval, paper summarization, relevance checking and filtering. The final retrieval results will be used as parts of inputs for prompting fine-tuning.
Step 1: Query Generation. At iteration f, given research goal, two research ideas, and previous queries and retrieval results, our system first determines whether sufficient information has been collected, thus allowing for early exit. If further research is required, the system prompts an LM to generate new query distinct from previous queries. Importantly, since at least one idea in the given comparison pair is entirely novel, the system would not try to directly search for identical idea comparisons. Instead, the query generation employs two strategies: 1) retrieving ideas that offer indirect or transferable insights, and 2) decomposing the novel idea into sub-components and retrieving related literature for these sub-components.
Step Paper Retrieval. We use htttt://peeeee ai as the paper search engine. Unlike conventional key- word search, they support neural search that can di- rectly take natural language queries as inputs, e.g., "effectiveness of confidence calibration and iterative refinement in question answering". EXA also sup- ports automatic query optimization [12] for better retrieval performance. For each query, retrieve
Table 3: Impacts of paper summarization
methods: reuse abstracts summarize the full paper. Evaluated on 600 test examples.
Method
Accuracy
GPT-4.
42.0
w/ RAG (abstract-only)
38.8
w/ RAG (whole paper)
53.0

---

Prior work shows that LMs often yield inconsistent predictions when swapping orders of inputs in pair-wise comparison tasks (3]. To mitigate potential order bias, obtain two predictions for each test example by swapping the order of two ideas. We only consider the predictions valid when both the top-15 relevant papers from arxiv.org. To prevent information leakage from retrieval, we only retrieve papers published before July 1st, 2024.
Step 3: Paper Summarization. Prior work mainly resues paper abstracts as summaries 14 9
However, abstracts often too brief cover rich details (e.g., ablation studies comparison with specific baselines). Therefore, we download each paper's PDF, and prompi LMs to summarize the paper with respect the current query. As shown in Table 3 this substantially boosts the accuracy from 38.8% to 53.0%. We find that using abstracts alone biases the model towards favoring old ideas, while summarizing the whole paper alleviates such biases.
Step 4: Relevance Checking and Filtering. The initially retrieved 15 papers per query ensure coverage, but may also introduce irrelevant papers that may negatively mislead LMs' predictions. To alleviate this, we prompt LMs to judge the relevance of each paper summary (binary classification: relevant irrelevant), and subsequently filter out those irrelevant ones.
3.2 Fine-tuning
Next, we fine-tune LMs to reason over the research goal. two research ideas, and all retrieval results, to make the final prediction. A straightforward setting is fine-tuning LMs with golden outcome labels.
In addition, we also explore fine-tuning LMs to generate chain-of-thougle (CoT) reasoning [[5] before the final prediction. High-quality Co'Ts are not available for this task. Thus, following recent practices [17], we use the LM to augment the training data with its own generated CoTs. Specifically, sample multiple CoTs for each query, filter out CoTs that lead incorrect predictions, and try several strategies to select one CoT from all candidates (e.g., random or LM-based selection).
Evaluation on Human-written Ideas
4.1 Experiment Setup
We use GPT-4.1 as the main base model. Additionally, we evaluate multiple frontier LMs such as o3 and Claude 3.5 Sonnet. The knowledge cut-off of all these tested models is before July 1st, 2024
predictions consistent; otherwise, inconsistent predictions considered incorrect by default.
4.2
Automatic Evaluation
80
Vanilla
70
Fine-tuned
60
Table 4: Fine-tuning GPT-4.1 with raw outcome
labels or self- augmened CoTs.
50
Accurary.
40
Model
Accuracy
30
GPT-4.1
51.4
20
FT on Raw Label
77.0
10
FT Random CoT
48.0
FT on LM-selected CoT
47.8
GPT-3.5 turbo
GPT-4.1 Claude-3.5
03
Sonnet
Figure 3: Automatic evaluation results.
Finding 1: LMs are not naturally good at the task. We start with evaluating the zero-shot
performance as baseline. We select the best-performing zero-shot prompt among 10 candidates. As shown in Figure 3 even with the same retrieval augmentation, none of the frontier models are
We didn't evaluate Claude 3.7 Sonnet since its knowledge cut-off is November 1st, 2024.

---

naturally good at this task. For example, GPT-4.1 only achieves an accuracy of 51.9%; reasoning models like ol and o3 also just achieve similar or slightly better performance.
Finding 2: LMs can learn to predict idea outcome via fine-tuning on past ideas. Fine-tuning GPT-4.1 on past ideas substantially boosts its accuracy in predicting "future" ideas, from 51.9% to 77%. This demonstrates the importance of capability clicitation on this task.
However, fine-tuning GPT-3.5-turke on the same data only yields moderate improvement (45.0% to 53.7%). Given that GPT-3.5-turbe has a much earlier cut-off (September 1st, 2021), we suspect that GPT-4.11' Is performance arises from both its stronger general reasoning abilities and the updated knowledge of recent Al research - a crucial advantage given the rapid pace of Al in recent years.
In addition, the limited gains with GPT-3.5-turbo, a weaker but still capable model, reveal that our dataset does not have some trivial shortcuts that can easily fit.
Prior work [17] shows that fine- tuning LMs on their own generated CoTs can improve performance, e.g., in mathematical reasoning. On our benchmark, however, this approach yields no gain. As Table 4 shows, neither randomly sampled nor LM-selected CoTs improves upon the baseline. We suspect that because of GPT-4.1's near-chance performance, its generated CoT's are often not sound. Consequently, these low-quality CoTs provide little useful signals for fine-tuning.
Finding 3: Self-augmented CoTs do not improve performance.
4.3 Comparison with Expert Researchers
Data Preparation. Since our recruited annotators are primarily NLP researchers, we select five popular NLP topies: long-context, alignment, reasoning, agent, and efficient LM. For each topic, we sample 6 to 12 idea pairs from our test set, resulting in a total of 45 NLP idea pairs.
Human Annotation. For each topic, recruit 5 re- searchers with relevant backgrounds. Among them, 68% have published papers on their assigned topics, while the remaining 32% also have extensive paper reading experience. [5 summarizes their profiles.
Table 5: Statistics of research profile and the time cost (minutes) per annotation of our recruited Al researchers.
Metric
Mean
Min
Max
Given a test example, annotators first indicate whether they have read any of these two ideas. Then, they provide a binary prediction and a natural language ra- tionale. The average time cost per annotation is 9.1 minutes. They are paid S50 for completing examples under their assigned topics. In total, collect 225 la-
Paper Citation
18.6
2
90
1273.8
14
14,378
H-index
8.9
2
41
i10-index
9.3
56
Time Cost
9.1
4.5
15
bels; three are excluded since the annotators have read both ideas before. The final data thus has 222 valid annotations for the 45 idea pairs.
Results. As shown in Figure D (left), even expert NLP researchers struggle to predict research outcomes in their own research domains. Specifically, doing majority voting over five experts' predictions on each example achieves only 48.9% accuracy. We further measure inter-annotator agreement, i.c., how often does each annotator's label agree with the majority voting label. The agreement score is 75. %, which is similar to the inter-reviewer agreement on conference paper decisions, e.g.. 75.8% in NeurlPS 2021 [1I.
We also establish ceiling human baseline by aggregating the best-performing researchers per research topic, where the best- rest-performing researcher is selected based on the exact same test set. This ceiling human baseline yields a 60.0% accuracy. Overall, these results show that predicting an idea's is chance of success is difficult task that even domain experts struggle with. In comparison, our system achieves an accuracy of 64.4%, substantially surpassing human experts.
5 Stress-testing Our System's Robustness
We run extensive stress-testing to ensure that our system does not exploit ideas" superficial features, such as complexity and stylistic cues. Each test is based on one potential superficial feature that the model can potentially rely on. We divide our test set based on whether the label is correlated with the feature, e.g., examples where the more complex method performs better vs. examples

---

Accuracy
50
20
10-
Concurrent New Win
Old Win
Simiar Length
Long
Short Win
Original
Perturbed
(a) Idea recency.
(b) Idea length.
(c) Famous lab names.
Figure 4: Stress-testing our system's 's sensitivity to three biases that humans might be prone to. The low accuracy variance across subsets of the test data indicates that system is robust.
Supported
Unsupported
Task-/DDmainn-peccfic
Novel Representation Leariing
Complex Multi-Component
80
80
80
70
Accuracy
60
60
50
30
30
20
20
20
10
Novel Architectures
Complex Math
Not Incremental
80
80
70
70
Accuracy
60
50
50
40
40
30
30
30
20
Figure 5: Our system is robust based on LM-designed stress tests. The Supported subset includes examples where the feature is noticeably more prominent in the winning idea, e.g., includes more complex math; the Unsupported subset the opposite. Our system shows a slight (potentially justifiable) bias, but is generally robust.
where the simpler one is better. We then compare the model's accuracy on these subsets: small accuracy difference indidates robust model that's not overly sensitive to superficial features. We hand-designed three tests, then use an LM to propose many more tests.
5.1 Human-Designed Tests
We design three tests based on biases that human experts might be prone to when evaluating ideas.
Recency, i.e., favoring new ideas. We divide our test set into three sets based on whether the newer idea is better, or if the two ideas concurrent (published within three months). Although we strip date information from idea descriptions, LMs might be able to infer that information based on prior knowledge and indirect cues such as references. As shown in Figure 4al our system's accuracy not sensitive to idea recency.
Length, i.c., favoring long and complicated ideas. We divide the test set into three based whether the longer idea is better, and ties. Results shown in Figure 4b indicate that our system has slight bias but still identify complicated yet ineffective ideas.
Famous Lab Names, i.e., favoring ideas from "big names". study this, we append the names of 10 famous labs (e.g., Anthropic, OpenAl, Google Deepmind) to the summary of the losing ideas. If our system exploits the shorteut, accuracy should notably drop after this perturbation. Results shown in Figure 4c indicate our system's 's resilience.

---

5.2 LM-Designed Tests
To improve the coverage of our stress-testing, we follow 18 and use LMs to automatically propose robustness tests at scale.
Method. Our pipeline consists of three steps: 1) proposing hypotheses about differences between two idea groups (model-preferred v.s. model-dispreferred) 2) verifying hypotheses across the test set, and 3) splitting the test set into supported and unsupported subsets according each hypothesis.
In step I, due to the limited model context length, each time we randomly sample idea pairs and group them into model-preferred or model-dispreferree sets. We then query the LM: "how is the idea in Group A different from the idea in Group B". We repeat this 500 times to obtain 500 hypotheses. An example of LM- -.o-poooseed hypothesis is: "Each research idea in Group A is more focused on utilizing advanced probabilistic and mathematical techniques for optimization and leamnigg in machine learning models compared to Group B."
Since hypotheses are derived from only examples, we further verify them across the whole test set. Specifically, for each example, we query the LM: "Compared to (losing idea), is (winning idea) (Hypothesis)) ?". If so, this example is classified as supported under that hypothesis, because simply exploiting this shortcut can get perfect accuracy on such examples. Each hypothesis thus divides our test set into supported and unsupported subsets. For each hypothesis, we calculate a validity score, defined as the proportion of supported examples. We then discard hypotheses with scores outside the 25%-75% range, leaving us with 289 hypotheses.
For each test, the decrease in accuracy from the supported to the unsupported subset indicates sensitivity to the corresponding feature.
Results. Hypotheses are classified as Flagged if the accuracy the corresponding unsup- ported subset drops below 62% (given the av- erage accuracy of our system over the entire test set is 77%), otherwise they are marked as Cleared. Across all 289 verified hypotheses, our system achieves an average unsupported accuracy of 68.5%, and notably, over 88% (255 out of 289) are Cleared, indicating ro- bustness against most hypothesized superficial biases.
Table 6: Results on LM- LM-poopoeed hypothetical bi- ases. Hypotheses are Flagged if the unsupported sub- set's accuracy falls below 62%, otherwise Cleared.
Metric Size
All
Flagged
Cleared
289
34
255
Ace (Supported)
81.3
82.1
81.1
Ace (Unsupported)
68.5
61.3
69.4
Even among the 34 Flagged hypotheses, our model maintains a considerable average unsupported accuracy of 3..3% The first row in Figure[5]showe the evaluation results of three flagged cases. In particular, we find two representative hypotheses: model-preferred ideas are often more complex or more specialized (i.e., task- or domain-specifice. However, predicting more complicated specialized yet ineffective ideas is inherently more challenging. Therefore, under such tests, our system's 's > 61% accuracy (Figure 5 non-trivial.
Further, examining the 255 Cleared hypotheses reveals potential shortcuts that our model does not rely on. In the second row of Figure 5 present the valuation results of three cleared cases, which stree-tests whether our system always prefers favor ideas that are 1) focused on novel architectures, 2) mathematically complex, 3) entirely novel instead of doing incremental changes to existing methods. As we can see, our system remains robust under these features.
Evaluation on Novel Unpublished Ideas
So far, our evaluation is based on published ideas with control measures for contamination. The real test, however, is to make predictions about unpublished ideas. The bottleneck to such an evaluation is to obtain groundtruth by actually implementing the ideas.
We conduct a small-scale experiment with unpublished ideas with actual groundtruths. We start with a dataset of 35 ideas from an Al ideation study [14], half generated by their LM agent, and the other half generated by their recruited NLP experts on the spot. None of these ideas have been released anywhere on the internet. The research topics focus on novel prompting methods to improve various capabilities of L.s.. The outcomes are obtained by expert researchers implementing the ideas, costing 103.4 hours per idea on average. For each idea, the experiments and results are documented
8

---

paper, from which we can extract the idea into our standard format for our benchmarking. We pair the proposed novel idea with its baseline and derive the golden label based on empirical outcomes. After excluding ties, we finally collect 33 labeled idea pairs.
Our system achieves accuracy of 63.6%, demonstrating the potential use our system reward model in a complete automated research pipeline, either to guide sampling at inference time, or optimizing the idea generation model using reinforcement learning.
7 Related Work
Accelerating Research with AL With the overarching goal of using Al to build AI, recent work has explored integrating AI into the entire research workflow, including literature review (2), ideation
paper writing [9] and review [8] Notably, one fully Al- generated paper passed the peer-review process of an ICLR workshop [66. However, discovering such workshop-level idea is non-trivial: Starting from 256 candidate papers, the authors manually select the top-3 papers since LM judges are often unreliable, while only one paper gets accepted. Importantly, executing 256 AI research ideas into whole papers is expensive and slow because of the significant GPU or API costs. To accelerate AI research, our paper aims to answer the question: "can we predict the outcomes of ideas before actually implementing them?"
14 experimental validation 6 9
Research Evaluation. Prior work mainly focuses on peer-review style evaluation; the goal is to predict scores that align with human reviewers based on fully written papers. However, such evaluations still require implementing ideas. Moreover, peer-review scores are often subjective. Critiques of novelty, excitement, and paper presentation are often independent of the ideas' actual empirical effectiveness. Consequently, humans can get misled to prefer "fancy" (c.g.. mathematically complex) yet ineffective ideas. Instead, this papers focuses objective empirical effectiveness, for each we can reliably establish ground truth labels.
Research Outcome Prediction. Prior work also attempts to use AI for predicting empirical research outcomes. The most relevant work is [IO] which uses Als to predict neuroscience results. However, their actual experiment setup is entirely different from ours. Specifically, given a published paper abstract, they use LMs to alter descriptions about results while keeping the method and background unchanged. The goal is then to distinguish the original abstract from its altered counterpart. This task easy, even fine-tuned Llama-2-78 model achieves an 80%+ accuracy. In contrast, we study a more realistic and challenging scenario: given two independently written idea summaries that exclude any empirical results, predict which is more effective.
Forecasting, Our task is type of forecasting, i.c., forecasting the outcomes of ideas before running actual experiments to get the outcome. Prior attempts on neural forecasting methods demonstrate the gains from retrieval and scale (e.g., model and data scale) [19 13 [7) For example, [5] builds human-level forecasting system by retrieval augmentation and fine-tuning on historical data. Inspired by these works, our work demonstrates the potential of building specialized LMs to surpass human experts in forecasting outcomes of Al research ideas.
Discussion
Limitations. Our current system functions as black-box and is not conducive to human-Al cooperation beyond prioritizing research ideas based on model predictions. And despite our best attempts verify its robustness, we cannot rule out the possible reliance on spurious features.
Future Work. Our fine-tuning method is straightforward but works well. Future work can explore advanced modeling methods, e.g., inference-time simulation of experiments. Our system can also be used as a reward model to improve automated ideation systems.
Conclusion. A crucial skill in empirical research is being able to predict which ideas are more likely to work out. Human experts develop this skill by reading papers and observing experiments, we show that LMs can do this more efficiently; our LM system outperforms human experts at making direct predictions about the outcome of AI research without running experiments. In addition to higher accuracy, our system is robust to biases like recency that humans are prone to. The system also generalizes to completely novel, unpublished ideas, including Al-generated ones, demonstrating the potential of further accelerations of Al research.

---

Acknowledgements
This work was supported by Eric and Wendy Schmidt and Open Philanthropy.
References
[1] Mahdi Abolghasemi, Odkhishig Ganbold, and Kristian Rotaru, Humans vs large language models: Judgmental forecasting in era of advanced ai. arXiv preprint arXiv:2312.06941, 2024.
[2] Shubham Agarwal, Issam H. Laradji, Laurent Charlin, and Christopher Pal. Litllm: A toolkit for scientific literature review. arXiv preprint arXiv:2402.01788 2024.
[3] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039- 2023.
[4] Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Vikram Dhillon, Eeshit Dhaval Vaishnav, Byron Lee, Tiago Yunhan Xu, Annalisa Pawlosky, Alan Karthikesalingam, and Vivek Natarajan. Towards ai co-scientist. arXiv preprint arXiv:2502. 18864, 2025. CC BY-NC-ND 4.0.
Costa, José Penadés, Gary Peltz,
[5] Danny Halawi, Fred Zhang. Yueh-Han Chen, and Jacob Steinhardt, Approaching human-level forecasting with language models. arXiv preprint arXiv:2402.1856. 2024.
[6] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:2310.03302, 2023.
[7] Ezra Karger, Houtan Bastani, Yueh-Han Chen, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock. Forecastbench: A dynamic benchmark of ai forecasting capabilities. arXiv preprint arXiv:2409. 19839, 2025.
[8] Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. Can large language models provide useful feedback on research papers? large-scale empirical analysis. NEJM AI, (S)::lo22400116. 2024.
[9] Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scien- tist: Towards fully automated open-ended scientific discovery. arXiv preprini arXiv:2408. 062222.0..06... 2024.
[10] Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O Cohen, Valentina Borghesani, Anton Pashkov, et al. Large language models surpass human experts in predicting neuroscience results. Nature human behaviour. 9(2):305-315, 2025.
[11] None. Neurips 2021 summary, 2021.
[12] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng, Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.0344 03345, 2023.
[13] Philipp Schoenegger and Peter S. Park. Large language model prediction capabilities: Evidence from a real-world forecasting toumament. arXiv preprint 0rXiv:2310.130// 2023.
[14] Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04100 2024.
10

---

Table 7: Typical errors found the human verification process sorted by frequency. Human annotators will rewrite or directly remove the incorrect data.
Error Type
Example
Solution
Incorrect Win Condition
The metric (e.g., perplexity) is the lower the better, but is extracted as the higher the better.
Rewrite
Incorrect Empirical Results
The numerical results are incorrect or hallucinated (i.c., do not exist in the paper).
Rewrite Remove
Confusing Benchmark & Metric Description
The metric name is mistakenly extracted as the bench- mark name.
Rewrite
Incomparable Ideas
One of the idea adopts unfair experiment setup to serve as a ceiling or flooring baseline
Remove
[15] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of -hain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 248327 2022.
[16] Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-22: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504, 08066, 2025.
[17] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Self-taught reasoner bootstrapping reasoning with reasoning. In Proc. the 36th International Conference on Neural Information Processing Systems, volume 126, 2024.
[18] Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal driven discovery of distributional differences via language descriptions. Advances in Neural Information Processing Systems, 36:40204-40277, 2023.
[19] Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks. arXiv preprint arXiv:2206. 1474, 2022.
Appendix
A Human Verification of Test Examples
Table 7 presents the typical errors found by our human annotators during the verification process.
B
LM-Designed Tests

---

Table Examples of LM- Aro-poooeed hypotheses.
Hypothesis
Ace (Pos.)
Acc (Neg.)
Each research idea in Group more focused specific domains task improvements with detailed mechanisms and methodologies tailored for unique problem sets.
80.6
62.0
Group ideas focus more on novel conceptual frameworks and mecha- 81.6 nisms for learning representations and transferring knowledge.
61.2
Group A ideas focus more novel machine learning and Al model architectures while Group ideas concentrate on practical methodologies and algorithmic improvements in specific domains.
81.2
73.5
Each research idea in Group A is more focused on utilizing advanced probabilistic and mathematical techniques for optimization and learning machine learning models compared Group B
79.7
73.8
Each research idea in Group A is more focused on specific application ar- tasks compared to Group B's broader methodological innovations.
79.2
73.0
Group provides more detailed descriptions and technical depth about model architectures and implementations whereas Group B offers more concise summaries and emphasizes key features and differences from existing methods.
77.9
70.6
Each research idea in Group A is more focused innovative techniques for enhancing Al model efficiency and effectiveness without extensive model training fine-tuning.
80.2
71.0
Each research idea Group A is more focused on innovative architectural and training strategies tailored to specific types of data or tasks compared Group B's emphasis on optimizing existing models and frameworks for efficiency and performance.
83.2
64.6
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See experiments Sec4, Sec5 and Sec6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Sec8 Discussion.
Guidelines:
12

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
See Sec4, Sec5 and Sec6.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
13

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will open source code and publicly available data we used our experi- ments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
14

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Sec2, Sec3 and Sec4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: See Sec5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emorr bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: See Sec3 and Sec4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
15

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:p/p///aaap.....ooooo//o///t
Answer: [Yes]
Justification: See Sec3 and Sec4.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: See Sec7 Conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
16

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See references.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessii thcode. www.com//a/oo/oc datasat has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Sec2.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
See See 2, see and Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
17

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [No]
Justification: The paper does not collect demographic information from annotators.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy lattps:/nnerrips http://errrreeccc/oooooeeerrrres for what should or should not be described.
18


Reviews:
[{"review_id": "AtiTGeuIbI", "reviewer": "Reviewer_pjfg", "summary": "The paper studies the topic of predicting outcomes of AI research ideas. A benchmark is proposed to faithfully evaluate the soundness of the AI-generated idea. The construction of the benchmark requires laborious efforts, including selecting papers, extracting ideas. The authors also propose a fine-tuned GPT-4.1 with a paper retrieval agent for this task. The proposed method outperforms existing methods.", "questions": "1. What are the retrieved papers used for in the final proposed system? Section 3.1 only mentions how to retrieve papers from the database, but it does not mention how retrieval results are used. Are these results simply used as parts of inputs, or are some specific modules proposed for processing this information? \n2. Will the proposed benchmark be released upon the acceptance of the paper? The benchmark is one of the core contributions of the paper, so it is important to have the benchmark publicly available to the community.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. It is worthwhile to construct a benchmark to evaluate the soundness of the generated ideas in the domain of AI research.\n2. It is laborious to construct such a benchmark, and the authors provided detailed construction procedures in the paper and it looks reasonable.\n3. The paper proposes a framework to predict the outcomes, and the performance surpasses existing methods.\n4. The paper includes extensive and robust experiments to verify the performance of the proposed framework, including an ablation study on retrieved information, CoT, and stress-testing etc.\n\nWeaknesses:\n1. What are the retrieved papers used for in the final proposed system? Section 3.1 only mentions how to retrieve papers from the database, but it does not mention how retrieval results are used. Are these results simply used as parts of inputs, or are some specific modules proposed for processing this information? \n2. Will the proposed benchmark be released upon the acceptance of the paper? The benchmark is one of the core contributions of the paper, so it is important to have the benchmark publicly available to the community.", "was_revised": true, "final_justification": "I will keep my rating to accept the paper.", "rebuttal": "Thanks for appreciating the strength of our paper! We will address each of your questions below, and are willing to expand or provide further responses if necessary.\n\n> What are the retrieved papers used for in the final proposed system? Section 3.1 only mentions how to retrieve papers from the database, but it does not mention how retrieval results are used. Are these results simply used as parts of inputs, or are some specific modules proposed for processing this information?\n\nYes, the retrieved results are simply used as parts of inputs during prompting and fine-tuning.\n\n> Will the proposed benchmark be released upon the acceptance of the paper?\n\nYes, we will release the benchmark."}, {"review_id": "kxU24jmeEX", "reviewer": "Reviewer_2pTv", "summary": "The authors create a new benchmark to compare LLMs and human experts in predicting which of two research ideas will succeed. The benchmark is composed by scraping ideas and experimental results from conference papers, resulting in 1585 human-verified idea pairs published after the base model's cut-off date, and 6000 pairs for training. Four rounds of students checks every test pair, and a panel of 25 senior researchers supplies the human baseline. Off-the-shelf o3 and Claude 3.5 land at roughly 55% accuracy, just above the experts. Fine-tuning GPT-4.1 on the 6 000 training pairs and letting it read brief paper summaries through RAG results in 77%.", "questions": "- \"We only consider the predictions valid when both predictions are consistent; otherwise, inconsistent predictions are considered incorrect by default.\" --> Could the authors indicate how many times this happens?\n- The authors state in footnote 1 that Claude 3.7 is not evaluated, but mention Claude 3.7 in the introduction. I believe this should be Claude 3.5?", "limitations": "The authors indicate some limitations. However, I believe they could add that reproducibility is hard since a closed-source model has been fine-tuned, which is not publicly accessible. Also, the data is not shared, but the checklist indicates they will release it.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n- The authors introduce a new strongly curated benchmark which measures the ability of LLMs to assess the potential success of research ideas.\n- The authors show that fine-tuning GPT-4.1 outperforms other SOTA LLMs and verify the robustness of these results through many stress-testing studies (including evaluation on novel unpublished ideas).\n\nWeaknesses:\n- While it is a first attempt at such a dataset, the limitation to binary choice is quite limiting to real-life use cases.", "was_revised": true, "final_justification": "The rebuttal did not change my acceptance recommendation.", "rebuttal": "Thanks for appreciating the strength of our paper! We will address each of your questions below. Happy to expand or provide further responses if necessary.\n\n> \"We only consider the predictions valid when both predictions are consistent; otherwise, inconsistent predictions are considered incorrect by default.\" --> Could the authors indicate how many times this happens?\n\nThe table below reports the frequency of inconsistent predictions (i.e. inconsistency score).\nOur choice to consider inconsistent predictions as incorrect is in line with prior work ([1][2]) in pair-wise evaluation .\n\n| Model         | Inconsistency|\n| ------------- | ------------------ |\n| GPT-3.5-turbo | 14.1               |\n| + FT          | 20.8               |\n| GPT-4.1       | 16.3               |\n| + FT          | 11.4               |\n\n[1]  AlpacaEval : An Automatic Evaluator for Instruction-following Language Models. github\n\n[2] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv 2024\n\n\n> The authors state in footnote 1 that Claude 3.7 is not evaluated, but mention Claude 3.7 in the introduction. I believe this should be Claude 3.5?\n\nYes, it should be Claude 3.5. Thanks for pointing out this typo!\n\n> While it is a first attempt at such a dataset, the limitation to binary choice is quite limiting to real-life use cases.\n\nWe actually think pair-wise comparison is well aligned with real-life use cases: 1) human researchers often seek to know whether a new idea is better than specific baselines, 2) in an automated research pipeline, a pair-wise reward model can be used to rerank model-generated ideas or directly optimize the idea generation model."}, {"review_id": "G5I7CDPFq8", "reviewer": "Reviewer_pUcN", "summary": "This paper is about predicting which of two research ideas is more promising. To realize this, the authors collect a dataset and evaluate LLMs, including a novel AI based ideation agent. Results show that the best models compare favorably to humans.", "questions": "1) In Section 3, how does the system determine whether sufficient information has been collected?\n\n2) In Section 3 Step 1, what is a query?\n\n3) In l. 130, why is at least one idea completely novel?\n\n4) What is the role of paper summarization in Section 3?\n\n5) Why are the four steps necessary in Section 3 and what is their main purpose?", "limitations": "The limitation section is extremely brief. One could mention for example, that evaluating the outcome of new ideas could drastically increase human reviewing burdens, among many other.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths** \nThis could be a potentially very impactful paper, as the results are quite convincing, and the idea and task seem very novel. The paper is also thorough and presents several analyses.\n\n**Weaknesses**\nMy main concern is that I do not understand the core Section 3. For example, what is happening in query generation? An example would really be extremely helpful and much appreciated.", "was_revised": true, "final_justification": "The authors and other reviews motivated me to increase my score, even though I continue not to be an expert for this paper", "rebuttal": "Thanks for appreciating the strength of our paper! We will address each of your questions below, and are willing to expand or provide further responses if necessary.\n\n\n> In Section 3, how does the system determine whether sufficient information has been collected?\n\nWe prompt LMs to automatically determine this.\n\nSystem prompt:\n```\nYou are an expert researcher in comparing and predicting other proposed research ideas.\nYou will be given access to a paper retrieval API, which you may use to survey the literature and find relevant papers to help you make your decision.           \n\nYou will be given 5 rounds to decide on which idea would work better, but you do not need to use them all.\nAt any round, you may exit early and decide on the comparison result.\n```\n\nExcerpt of Prompt\n```\nFirst, determine if you need to search for relevant research papers to make an informed assessment \n```\n\n> In Section 3 Step 1, what is a query?\n\nIt’s the query sent to the paper retrieval engine.\nSpecifically, we prompt LMs as follows:\n```\nFirst, determine if you need to search for relevant research papers to make an informed assessment (e.g., check if there are similar ideas or supporting empirical evidence). If yes, provide a search query in this format: \n<search>\nyour query (a natural language sentence, e.g., effectiveness of weight decay in noisy label learning, effectiveness of diffusion models in image style transfer compared to vision transformer, effectiveness of model editing in sequential editing setting)\n</search>\nGuideline for search queries: \n- At least one of the ideas is entirely new. This means that you won't find exact comparisons in the literature. Instead, search for ideas that could provide indirect or transferable insights, or you can break the new idea into smaller sub-ideas to identify related work for each sub-component.\n- Make queries as precise and context-specific as possible to maximize the relevance of the retrieved information.\n- Avoid redundancy:\n   - Do not repeat previously made queries.\n   - Avoid gathering excessive evidence about components you already understand well. Instead, prioritize components that need further investigation.\n```\n\n\nHere are two examples of generated queries:\n- ```effectiveness of state space models v.s. GRU architectures in sequence modeling tasks```\n- ```effectiveness of code-based prompting v.s. Chain of thought prompting for logical reasoning tasks```\n\n> In l. 130, why is at least one idea completely novel?\n\nBecause each test example contains at least one idea published after June 1st, 2024, the knowledge cut-off date of our base model GPT 4.1 (Line 93-95).\n\n> What is the role of paper summarization in Section 3?\n\nAdding the whole content of all retrieved relevant papers would 1) exceed the model context limits, and 2) introduce too much irrelevant, non-informative information that might negatively mislead LMs’ predictions (e.g. experiment setup, hyperparameter details, irrelevant results). \n\n> Why are the four steps necessary in Section 3 and what is their main purpose?\n\nAs explained in Line 122-126, similar to how human researchers draw inspiration from existing literature, we empower LMs with this agentic retrieval engine to improve its predicting ability.\n\nThe four steps in Section 3.1 are standard practice for building retrieval-augmented systems [1] [2] [3]. In particular, our paper summarization step is quite different: while prior work directly reuses paper abstracts as summaries, we download each paper’s PDF and prompt LMs to summarize the whole paper with respect to the current query. As shown in Table 3, this substantially boosts the accuracy from 38.8% to 53.0%.\n\n[1] The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv 2024.\n\n[2] The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. arXiv 2025\n\n[3] Can Language Models Generate Novel Research Ideas. ICLR 2025\n\n> The limitation section is extremely brief. One could mention for example, that evaluating the outcome of new ideas could drastically increase human reviewing burdens, among many other.\n\nWe are happy to include more discussion on the social impact of our work and AI-automated research in general (e.g., the potential increase in human reviewing burdens, as the reviewer suggested) in camera ready with the extra space. Our current discussion focuses on the technical limitations of our approach."}, {"review_id": "PqSrEdzYST", "reviewer": "Reviewer_SEun", "summary": "This paper proposes a framework to predict the experimental performance of machine learning methods from descriptions of research ideas. The authors create a dataset that includes pairwise comparisons of performance on benchmarks, constructed using results presented in conference papers. Their experiments show that existing models, including GPT-4.1 and o3, cannot effectively predict experimental results even when provided with related papers. The result shows that GPT-4.1 fine-tuned on their dataset achieves better performance compared to human experts in the pairwise performance prediction task.", "questions": "Please let me know if there are any inaccuracies or misunderstandings in the points raised under Weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* This paper targets a crucial and useful research topic. Predicting the experimental performance of research ideas would be a crucial part of research automation.\n\n* Novel dataset made with manual verification. The dataset that includes pairs of research ideas and experimental results is novel and useful. While the dataset is automatically created using LLMs, the quality is verified by manual annotation.\n\n### Weaknesses\n\nAlthough the idea is promising, the paper does not provide sufficient information to establish the reliability of its claims. Additionally, I have concerns regarding potential dataset contamination. I recommend a substantial revision before the work is considered for publication.\n\n* Concern about data contamination\n\nLines 72 and 93 suggest that the authors use conference publication dates as the publication dates of papers. However, many papers are publicly available as preprints several months before their conference publication, and LLMs are often exposed to these preprints. Moreover, the paper retrieval module (Line 135) retrieves documents from arXiv, and the retrieved papers may contain information about the target ideas. To more reliably avoid data contamination, the authors should consider using the preprint publication dates rather than the conference publication dates.\n\nIn addition, according to [the official information from OpenAI](https://openai.com/index/gpt-4-1/), the knowledge cutoff date of GPT-4.1 is June 2024, though the exact date is not specified. To ensure a more reliable exclusion of training data, I believe it would be safer to include only papers published after July 2024.\n\n* Concern about the shortcut in the proposed method\n\nAs stated in Line 130, some of the compared methods were proposed prior to the knowledge cutoff date and may be included in the retrieved documents. Since newer methods often outperform older ones, a trivial baseline would be to recommend methods not present in the retrieved documents. I am concerned that the proposed fine-tuned model may exploit this shortcut when making predictions.\n\nAlthough I consider the experiments in Section 6 essential for addressing this concern, the results are not compared with baseline methods or expert human performance.\n\n* Limited information about the dataset creation process, the dataset, and the results\n\nThe paper lacks detailed statistics, example prompts, samples from the datasets, and raw outputs from the evaluated methods. I recommend including raw examples from the datasets as well as unprocessed outputs from all evaluated methods.\n\n* Too many missing references\n\nThere exist many previous studies in automated research, but this paper cites very few papers. Particularly, the first two paragraphs of the introduction and the second paragraph of the related work section do not cite any papers, and the total number of cited papers is only 19.\n\n* Minor suggestions\n\n  * I consider the majority voting over all benchmarks (Line 88) to be too sensitive to noise. For example, when Idea A outperforms Idea B on 3 out of 5 benchmarks, I am unsure whether Idea A is better than Idea B in a statistically significant manner. I recommend pairing ideas with a larger gap.", "was_revised": true, "final_justification": "My primary concerns regarding (1) data contamination and (2) a possible shortcut by the proposed method have been addressed in the rebuttal. Accordingly, I consider this paper to be above the threshold.\n\nHowever, I still believe that the paper should provide more detailed information about the dataset and experimental settings. Although the authors mentioned that they plan to include additional details, the current lack of clarity remains the primary reason I am not assigning a higher score. I am also concerned about certain aspects of the dataset configuration, as noted in my response to the rebuttal.", "rebuttal": "Thanks for appreciating the strength of our paper! We will address each of your questions below, and are willing to expand or provide further responses if necessary.\n\n> Concern about data contamination. Lines 72 and 93 suggest that the authors use conference publication dates as the publication dates of papers.\n\nWe will clarify this confusion in revision. “Publication date” here refers to the date that the paper first appeared on arxiv (which we extract from its arxiv ID), not the date that it’s published at the conference.\n\n> the knowledge cutoff date of GPT-4.1 is June 2024, though the exact date is not specified. To ensure a more reliable exclusion of training data, I believe it would be safer to include only papers published after July 2024.\n\nWe pick a subset of our original test set that includes 1,444 ideas published after July 2024. As shown in the Table below, our main findings remain the same: GPT-4.1 achieves 77.3% accuracy (compared to 77.0% on the 1,585 ideas after June).\n\n| Model         | Accuracy |\n| ------------- | -------- |\n| GPT-3.5-turbo | 44.8     |\n| + FT          | 53.9     |\n| GPT-4.1       | 51.3     |\n| + FT          | 77.3     |\n\n\n> Concern about the shortcut in the proposed method: simply selecting newer methods, which  often outperform older ones. Although I consider the experiments in Section 6 essential for addressing this concern, the results are not compared with baseline methods or expert human performance.\n\nWe agree that simply selecting newer ideas is a very likely shortcut. We are happy to include the following result in our revision. Here we report the best possible performance achievable by simply selecting newer ideas. Our fine-tuned GPT-4.1 substantially outperforms this baseline (77.3% v.s. 53.0%), suggesting that it’s not overly reliant on idea newness in its predictions.   \n\n| Model         | Accuracy |\n| ------------- | -------- |\n| Fine-tuned GPT-4.1       | 77.3     |\n| Always selecting newer ideas |   53.0  |\n\n\n> The paper lacks detailed statistics, example prompts, samples from the datasets, and raw outputs from the evaluated methods.\n\nCurrently, dataset statistics are presented  in Table 2, and an excerpt of dataset example in Figure 2. We will include more detailed dataset examples and example prompts (e.g. prompts for query generation, summarization in Section 3) in the final revision.\nThe raw outputs of our fine-tuned models only include “Idea A or B is better”, while the prompted models also generate CoTs before making final predictions; we are happy to include CoT examples in the appendix although they are not used in our method.\n\n> Missing references\n\nWe will add the following papers about peer-review style research evaluation in our final revision. As discussed in the second paragraph of related work, these papers are based on fully written papers, so still requiring implementing ideas. Moreover, these papers focus on critiques of idea novelty, excitement, and paper presentation, which are often subjective and independent of the ideas actual empirical effectiveness. In particular, a recent paper [5] shows that the ideas with higher LM-judged novelty scores in fact lead to worse empirical outcomes.\n\n[1]  The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv 2024.\n\n[2] The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic\nTree Search. arXiv 2025\n\n[3] GRAPHEVAL: A Lightweight Graph-based LLM Framework For Idea Evaluation. ICLR 2025\n\n[4] Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents. ICLR2025\n\n[5] The Ideation–Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas. arXiv 2025.\n\n>  I recommend pairing ideas with a larger gap.\n\nWe’d like to clarify that by “gap” the reviewer is referring to the gap in win rates of two ideas on the set of benchmarks, for example, by only labeling one idea as the winner if it beats the competitor on 80% of the benchmarks.\n\nWe use different gaps to filter our test set, and report the data size and the performance of our fine-tuned GPT-4.1 in the table below. As expected, increasing the gap leads to higher accuracy, probably because it reduces label noise or picks easier idea pairs.\n\n| Gap             | \\|Test Set\\| | Accuracy |\n| --------------- | ------------ | -------- |\n| > 50% (default) | 1,585        | 77.0     |\n| > 60%           | 1,585        | 77.0     |\n| > 70%           | 1,439        | 78.7     |\n| > 80%           | 1,281        | 80.8     |\n| > 90%           | 1,190        | 81.2     |\n\n\nWe agree that the gap in determining labels is an important hyperparameter. In our data collection and training pipeline, that hyperparameter can be easily adjusted. We will clarify in our revision that we don’t impose any choice of this hyperparameter."}]

Author rebuttal:

Meta-review: This paper introduces a benchmark for predicting which research ideas will succeed, comparing LM performance with human experts. A retrieval-augmented GPT-4.1 system outperforms both experts and frontier LMs, with robustness checks and evaluations on unpublished ideas confirming its promise. Overall, it presents a novel and impactful direction for using LMs to accelerate AI research.
During the review process, all reviewers gave this paper a positive score. Meanwhile, they also pointed out some possible improvements, such as the risk of data contamination, missing references and data curation details, and lack of clarity in some sections. The authors resolved most of these concerns during the rebuttal.

By evaluating the novelty of research idea prediction, I recommend acceptance of this paper.
Decision: Accept (poster)