The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas
Authors: Chenglei Si
Tatsunori Hashimoto
Diyi Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Research Automation
Scientific Discovery
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) have shown promise in accelerating the scientific research pipeline. A key capability for this process is the ability to generate novel research ideas, and prior studies have found settings in which LLM-generated research ideas were judged as more novel than human-expert ideas. However, a good idea should not simply appear to be novel, it should also result in better research after being executed. To test whether AI-generated ideas lead to better research outcomes, we conduct an execution study by recruiting 43 expert researchers to execute randomly-assigned ideas, either written by experts or generated by an LLM. Each expert spent over 100 hours implementing the idea and wrote a 4-page short paper to document the experiments. All the executed projects are then reviewed blindly by expert NLP researchers. Comparing the review scores of the same ideas before and after execution, the scores of the LLM-generated ideas decrease significantly more than expert-written ideas on all evaluation metrics (novelty, excitement, effectiveness, and overall; p < 0.05), closing the gap between LLM and human ideas observed at the ideation stage. When comparing the aggregated review scores from the execution study, we even observe that for many metrics there is a flip in rankings where human ideas score higher than LLM ideas. This ideation-execution gap highlights the limitations of current LLMs in generating truly effective research ideas and the challenge of evaluating research ideas in the absence of execution outcomes.

Published as a conference paper at ICLR 2026
THE IDEATION-EXECUTION GAP: EXECUTION OUT- COMES OF LLM-GENERATED VERSUS HUMAN RE- SEARCH IDEAS
Chenglei Si, Tatsunori Hashimoto, Diyi Yang Stanford University {clsi, thashim, t
ABSTRACT
Large Language Models (LLMs) have shown promise in accelerating the scien- tific research pipeline. A key capability for this process is the ability to gener- ate novel research idcas, and prior studies have found settings in which LLM- generated research ideas were judged as more novel than human-expert ideas. However, good idea should not simply appear be novel, it should also re- sult in better research after being executed. To test whether Al-generated ideas lead to better research outcomes, we conduct an execution study by recruiting 43 expert researchers to execute randomly-assigned ideas, either written by ex- perts or generated by an LLM. Each expert spent over 100 hours implementing the idea and wrote a 4-page short paper to document the experiments. All the executed projects are then reviewed blindly by expert NI.P researchers. Compar- ing the review scores of the same ideas before and after execution, the scores of the LLM-generated ideas decrease significantly more than expert-written ideas on all evaluation metrics (novelty, excitement, effectiveness, and overall; p < 0.05), closing the gap between LLM and human ideas observed at the ideation stage. When comparing the aggregated review scores from the execution study, even observe that for many metrics there is a flip in rankings where human ideas score higher than LLM ideas. This ideation-execution gap highlights the limitations of current LLMs in generating truly effective research ideas and the challenge evaluating research ideas in the absence of execution outcomes.
INTRODUCTION
LLMs have shown promise in various tasks in the scientific research pipeline, and most recently, they have been envisioned to power AI scientists that can autonomously make novel scientific discover- ies. Recent efforts have built LLM-powered agentic systems to propose novel drug repurposing and treatment targets Gottweis et al. (2025); Ghareeb et al. (2025), develop new efficient matrix multi- plication algorithms and optimal constructs for open mathematical problems (Novikov et al. (0255.
and end-to-end produce full research papers on AI topics (Lu et al. 2024 Yamada et al. (025).
Generating high-quality research ideas is the first step in these automated research pipelines, and the quality of --M-eeerrated ideas can decide the upper-bound the final execution outcomes. Despite the importance of this idcation step, measuring the quality of LLL--genrrated research ideas is difficult, as it not only requires extensive domain expertise but also involves subjective taste. Prior attempts of evaluating I.L.M-genertted research ideas mostly focus the ideas themselves without considering the execution outcomes, with most evaluations relying on either LLM judges (Lu et al.)
2024
Li al. 2025 Feng al. 2025) small-scale human evaluation Baek al. (2025) Wang
et al. 2024a).
A recent large-scale human study examined Al-generated ideas in a randomized, blinded comparison to human experts (Si et al. 2025) and found that LLM ideas are judged as significantly more novel than human ideas with higher average scores across novelty, excitement, and expected effectiveness.
All of our data relcased https://githhti http://gggituuu..ooooooottt Sc1 -Rseearchhe

---

Published as a conference paper at ICLR 2026
Before Execution
After Execution
NLP Topics
Human
Ideas
Effectivaness:
Effectivvnness:
(N=19)
Coding
Execution Experts
Mutliinuall
Factuality
ll
Effectiveness: 6.00
Effectiveness: 4,13
Math
(N=24)
Figure 1: Study overview: we recruit N 43 expert researchers execute randomly assigned ideas from either the Human condition or the AI condition. Expert reviewers then blindly review all the executed projects. Despite the Al ideas being scored higher than human ideas before execution (e.g., their predicted effectiveness score of the ideas), their scores drop significantly more than human ideas after execution ge.g., their effectiveness score based on the experiment results).
Novelly (Sudd3-Sadyy)
Excilement
Humen
Al
Humen
Hummi
Human
Figure 2: Average scores of Al ideas drop significantly more than Human ideas in the execution study across all the evaluation metrics. Al ideas score higher than Human ideas the ideation evaluation (Study 1), and this difference in drops narrows their difference in the execution evaluation (Study 2). fact, ideas score even lower than Human ideas the execution evaluation, although this difference is not statistically significant (Table 4
However, evaluating research ideas is difficult even for experts (Simsek et al. 2024)) leaving open the question of whether these ideas would translate into better research outcomes.
There are several reasons why evaluation results the ideas might not hold true when we execute them into actual projects. During execution, every single step has to be grounded in realistie ex- ecution constraints, which impose higher feasibility standards than the ideation stage. Moreover, objective metrics like feasibility and effectiveness are best judged via the actual execution outcomes rather than speculative judgment based the ideas. However, execution is both resource- and time- consuming for many research domains. As compromise, prior works are constrained to verify only two Al-generated ideas through execution experiments (Ghareeb et al., 2025 Gottweis al. 2025), which makes impossible to draw any statistically significant conclusions about LLMs ideation capabilities.
Our work provides the first quantitative, large-scale study of AI ideas after execution by performing a large-scale execution study with sufficient sample size draw statistically significant conclusions about the post-execution quality of Al-generated ideas. We recruit N 43 qualified participants and randomly assign each of them NLP research idea from either human researcher agent based on C claude-3.. d.tleee.55-oao 5-Sonnet, a frontier LLM at the time when we started the study. Our execution study builds upon Si et al. (202)). and which enables us use the collected ideas and their pre-execution ideation evaluation. Our execution participants spend an average of 103 hours executing the assigned idea and then submit the codebase and paper document their experiments. All projects then reviewed blindly by our recruited expert reviewers (Figure I.
LLM
Comparing the review scores these ideas from the previous ideation evaluation and our new execu- tion evaluation, we observe the ideation-executto gap of LLM -LM--enerated ideas: LLM ideas score much lower in the execution evaluation as compared to the ideation evaluation. expert ideas only ineur small drops from the ideation evaluation to the execution evaluation, signif- icantly smaller than the ideation-executicn gap LLM-generated ideas (p < 0.05; Figure 2 De- spite the significantly higher pre-execution ideation scores from L.Mss, the huge ideation-execution
contrast, human
2

---

Published as a conference paper at ICLR 2026
gap leads to flip in the rankings between LLMs and humans. Our analysis further shows that reviewers consider more comprehensive factors in the execution evaluation, uncovering previously overlooked weaknesses of LLM ideas. These results add nuances to previous claims LLMs gen- erating better ideas than human experts, and perhaps more importantly, highlight the difficulty of evaluating research ideas in the absence of execution outcomes.
EXECUTION STUDY SETUP
In this section, we go over the high-level study design and lay out some specific rules for our execu- tion process to ensure controlled comparison between the Human and Al conditions.
2.1 HIGH-LEVEI STUDY DESIGN
Our goal is compare the execution outcomes of LLM-generated research ideas and human experts' re- search ideas (Figure [T. We design blinded, Ran- domized Controlled Trial (RCT) study: each of the N 43 participants is randomly assigned execute an anonymized idea originating from either the LLM or human expert under identical instructions, with re- viewers also blinded to the idea source. Each execution participant given a three-month window to execute the assigned idea into full project, including imple- menting and running all the proposed experiments and writing 4-page paper in the specified format. By stan- dardizing the exccution and reviewing procedures, our RCT design ensures that statistically significant differ- ences execution outcomes can be causally attributed to whether the idea came from the Al or from human. Our study design and hypotheses were pre-registered.
Topic
Human
AI
Bias
3
4
Coding
3
3
Safety
2
4
Multilingual
Factuality
5
5
Math
1
1
Uncertainty
2
3
Total
24
Table I: Topic distribution of the executed projects.
The LLM and human ideas used in our execution study are taken from a previous ideation study, spanning 7 different NLP topics (Table I These ideas
intentionally scoped to be feasible to execute within three months when they were collected in the ideation study. For the pre-execution evaluation, we use the ideation evaluation scores from the previous ideation study; for the post- execution evaluation, we recruit N = 58 expert reviewers to score the executed projects. These reviewers are drawn from a similar population as the ideation study, using review guidelines that closely match the ideation study wherever possible.
2.2 MINIMIZING CONFOUNDERS IN THE STUDY DESIGN
We establish some ground rules for the execution task. Our guiding principle is to minimize any changes to the assigned idea while maintaining blinding and random assignment across the two conditions.
Random Idea Assignment When assigning ideas, we want execution participants to be able to work on ideas within their expertise, bias where participants only choose higher-quality ideas to work on. To balance this tension, first ask for their preferred topics among our 7 candidate topics, and then we randomly assign either an -LM--eeertted idea assignment avoids potential selection biases and also helps randomize the participants' expertise levels across the two experiment conditions.
the same time, we want to avoid potential self-selection
Homan--ritter idea to them from their selected topics. This random
Minimize Changes to Ideas Our goal is evaluate the execution outcomes of the original ideas, and thus, we want to preserve the ideas as much as possible during execution. At the same time, we recognize cases where certain changes are necessary make the ideas executable. To balance these, our policy is to disallow any substantial changes to the proposed methods from the original ideas,
https://oaf. io/ ckxpkxtp

---

Published as a conference paper at ICLR 2026
Human Idea Executor (N 19)
Al Idea Executor (N 24)
Reviewer (N= 58)
Metric
Mean
Median
Mean
Median
Mean
Median
No. of Papers
15.3
5
14.3
11
13.6
11
No. of Citations
233.5
36
288.7
118
596.8
287
Topic Familiarity (1-5)
2.9
3.4
3.2
3
Time
112.6 hr
120 hr
93.7 hr
104 hr
52.5 min
45 min
Table 2: Research profiles of the execution and review participants and their efforts spent.
while allowing changes to the experimental details. We enforce this rule by asking all execution participants to explicitly note down all the changes they want to make. We manually review these proposed changes ensure that they are focused on refining the experiment details, such as dataset choice and baseline selection, and we verified that these changes do not change the core method proposed in the assigned ideas (discussed in full detail in Section 5.I). Of all the assigned ideas, there is only one exception where the participant found the idea to be too vague, and they had to instantiate the method details with their own ideas make the project feasible. We thus terminated that project and excluded it from the remaining study.
Deliverables and Reviews At the end of the execution, we ask all participants to submit the full codebase for reproducing all the experiments, adhering to our guidelines on writing detailed README instructions reproduce the experiments, as well as short paper of at least 4 pages in standardized format. After the execution stage, we recruit a pool of qualified reviewers to do blind review of the executed projects, where both the codebase and the paper will be shown to our reviewers for their blind review. The review form is largely similar to conventional conference view forms and matches the style of the review form used the previous ideation study, including metries novelty, excitement, soundness, effectiveness, and overall quality. Additionally, also collect review scores on how faithful the execution is to the original idea and the codebase quality control metrics. The full review form can be found in Appendix A
3 EXECUTION AND REVIEWING PARTICIPANTS
A core premise of our study is to rely on highly qualified participants to execute the given ideas into the corresponding experiments, and to rely on blind reviewing from expert reviewers for fair evaluation of all the executed outcomes. In this section, we describe the profiles of recruited experts and their efforts in this study.
3.1 EXPERT RECRUITMENT
We recruit participants by posting recruitment messages various social media platforms (e.g., Twitter/X and Slack), directly reaching out to qualified candidates, and advertising during in-person conferences. After basic profile screening, task, among whom 43 completed the task in the end. Among the participants who did not complete the execution task, only one of them was because of the assigned idea was vague and infeasi- ble, while all the other cases were due to various personal reasons. Our 43 execution participants came from 7 different countries, including the US, Australia, India, the UK, Nepal, Singapore, and Canada.
onboarded total of 66 participants for the execution
For the execution task, we give each participant a three-month window to complete the task and compensate them for the total number of hours spent on the task ($20/h) as well as a completion bonus ($600). Moreover, we reimburse all the compute costs incurred during the execution, such as inference API credits. Apart from the compensation, we also allow all execution participants to take ownership of the executed projects so that they can further develop the project after the study as potential paper submissions if they wish. This serves as additional incentive for the participants, and in fact, multiple of them developed the executed projects into paper submissions after the study. For the review task, recruited 58 highly qualified reviewers who collectively wrote 181 1 reviews, ensuring each project is reviewed by 4-5 different reviewers. Each reviewer is assigned 2-5 projects based on their preferred topics and is compensated $50 for each review they write.
4

---

Published as a conference paper at ICLR 2026
Human Ideas (N=85)
Al Ideas (N=96)
Metric
Mean
SD
Mean
SD
p-value
Novelty (1-10)
4,93
1.61
4.73
1.75
0.21
Excitement (1-10)
4.52
1.84
3.90
1.75
0.01*
Effectiveness (1-10)
4.84
2.11
4.12
1.94
0.01
Soundness (1-10)
5.38
1.72
4.73
1.82
0.01
Overall (1-10)
4.00
1.59
3.41
1.46
0.01
Faithfulness (1-10)
6.48
2.03
6.42
1.56
0.41
Codebase Quality (1-5)
3.58
0.94
3.58
0.89
0.52
Table 3: Results aggregated over all reviews for Human and Al conditions in the execution evalua- tion by treating each review as an independent data point. We perform two-sample one-sided t-tests to test whether the mean of the Al condition is smaller than the mean of the Human condition. We report the p-values with FDR correction to account for multiple hypothesis testing for the five main evaluation metrics in the first block, and report the raw p-values for the control metrics in the second block.
3.2 EXPERT QUALIFICATIONS AND EFFORTS
Finding qualified execution participants is crucial to avoid cases where the execution outcomes do not faithfully reflect the original idea's effectiveness due to poor execution. We took measures such as collecting candidates' profiles and conducting screening interviews to find participants who are both qualified and highly motivated and committed the task. All of our execution participants have substantial prior research background, with an average of 15.3 and 14.3 papers on their Google Scholar profiles for the Human and AI conditions, respectively (first block of Table 2}. We also collected their self-reported familiarity with the assigned topic, for which they indicated moderately high familiarity (2.9 and 3.4 on 1-5 scale for Human and AI conditions; Table Apart from being highly qualified, our execution participants spent average of 112.6 and 93.7 hours executing human and AI ideas, indicating substantial effort. This level of variation expertise and time spent between the human and Al conditions is expected, given relatively small sample size. Similarly, our expert reviewers have extensive research experience (596.8 average citations) and are generally familiar with the reviewed topics (3.2 out of 5); and spent average of 52.5 minutes each review (last two columns of Table [). Our pool of participants is diverse and comes from 40 different institutions across the world. We present the institutions that our execution and reviewing participants come from in Appendix
QUANTITATIVE RESULTS
In this section, we summarize our main quantitative results from the execution study. Our main goal compare Al ideas and human ideas, and will compare them both in terms of the scores from the execution study, as well as the differences compared to the ideation evaluation.
4.1 COMPARING HUMAN AND AI IDEAS IN THE EXECUTION STUDY
A natural outcome measure is the review scores from the execution study. We compare the scores of human and Al ideas in the execution evaluation in Table 3 where we treat each review as an independent data point and aggregate the scores from all reviews. Human ideas and AI ideas score similarly on our control metrics faithfulness and codebase quality, indicating similar execution quality and that ideas from both conditions are executed faithfully. When treating each review as an independent data point (sample size N 181) and performing one-sided t-tests, human ideas score significantly higher than AI ideas on excitement, effectiveness, soundness, and the overall score, but not novelty. However, when we compare human and AI ideas by treating the average score of each idea as an independent data point (sample size N = 43), the difference between human and AI ideas' execution score is not significant on any metric (second block of Table 4 Given this mixed evidence, we do not have sufficient statistical power to directly confirm our pre-registered hypothesis that human ideas differ significantly from Al ideas in the execution scores. Additionally,
5

---

Published as a conference paper at ICLR 2026
Novelty
Excitement
Effectiveness
Overall
Human Condition Ideation Score
4.912
4.404
4.833
4.596
Al Condition Ideation Score
5.778
5.653
6.003
5.382
p-value (FDR)
0.035*
0.004**
0.001
0.035*
Human Condition Execution Score
4.903
4.482
4.782
3.968
AI Condition Execution Score
4.729
3.896
4.125
3.406
p-value (FDR)
0.586
0.175
0.266
0.175
Table 4: Comparison of mean ideation vs. execution scores for human and AI conditions. We treat the average of each idea human condition and N = 24 for the Al condition. For the p-values, we perform two-sided t-tests with FDR correction. means p 0.05, means 0.01.
independent data point, the sample size is N = 19 for the
Novelty
Excitement
Effectiveness
Overall
Human Ideas Gap
-0.010
0.078
-0.052 -1.879
-0.628
AI Ideas Gap
1.049
-1.760
-..96
(Human-Al)
1.039
1.835
1.827
1.348
p-value (FDR)
0.025*
0.001 **
0.003** *
0.004**
Table 5: Comparison of the gaps between the execution evaluation and the ideation evaluation scores for human and AI ideas. Negative gaps indicate score decrease after execution. AI ideas drop significantly more than human ideas on all four metrics that are used in both ideation and execution evaluation, means p < 0.05, ** means p < 0.01. All p-values are adjusted with FDR correction.
plot the distribution of all projects' scores in the execution evaluation in Appendix and the reviewer agreement in Appendix D where we show that the reviewer agreement is generally high.
4.2 MEASURING THE EXECUTTIN GAP
Directly comparing average human and Al idea scores is difficult due to the high heterogenity in idea quality. However, the design of our study provides a natural way to remove this variance, instead of comparing the direct ratings, we can compare the difference between pre- and post- execution scores.
This ideatioon-xeectioon gap controls for the heterogenity idea quality, and results clear statis- tical signals even with 43 projects. Concretely, we focus the four metries that used in both evaluations: novelty, excitement, effectiveness, and the overall score. To define the metrics, we take between the execution evaluation and the ideation evaluation (so negative
the difference in gaps would mean the scores decreased after execution). Figure and Table 5 show that Al ideas' scores drop significantly more than human ideas in the execution evaluation across all four metrics. For example, human ideas mostly retain the same novelty, excitement, and effectiveness scores af- ter execution (-0.010, +0.078, and -0.052) while AI ideas" scores decrease by 1.049, 1.760, and 1.879 after execution, on a 1-10 scale.
In Table we present the full scores of all projects before and after the execution. Prior to the execution, Al ideas score significantly higher than human ideas on all four metrics. However, due to the bigger ideation-executin evaluation shrinks. In fact, we see case where the rankings flip, and AI ideas score below human ideas after execution on all metrics (e.g., 3.90 v.s. 4.48 on excitement and 4.13 v.s. 4.78 on effec- tiveness). However, note that such differences are not statistically significant due to the small sample size when treating each idea as an independent data point (N = 43). We further compute the correlation between the ideation scores and execution scores in Appendix E. where the correla- tion weak in most cases; and for Al ideas, there is even a moderately negative correlation on the excitement score.
gaps of AI ideas, the gap between human and AI ideas the execution

---

Published as a conference paper at ICLR 2026
5 ANALYZING THE IDEATION-EXECUTION GAP
In this section, we dive deeper into the executed projects and perform both quantitative and qual- itative analyses to understand why there exist ideation-execution gaps when we execute ideas into projects.
5.1 CHANGES MADE TO THE IDEAS MAINLY FOCUS ON EXPERIMENT DETAILS
We begin by analyzing the types of changes made to the ideas by our execution participants. We manually annotate all the changes to construct a taxonomy of the types of changes. We show the counts of all types of changes made to human and AI ideas in Table (6) where ideas and Al ideas involve an average of 2.9 and 3.1 changes, respectively. This indicates that only moderate number of changes are made to both human and AI ideas. Moreover, all of the changes focus on experiment details rather than altering any algorithms proposed in the original ideas. Ex- amples of these changes include switching benchmarks that are appropriate for the given tasks, updating the backbone models to more recent ones, adding more comprehensive evaluation metrics, specifying any missing syper-parameters and prompt details, adding stronger baselines, and adding more analysis or ablation studies. We present examples of each type of change in Appendix These changes generally preserve the original ideas and improve the experiment design.
that human
However, we do note one exception: when we prompted both humans and Al to generate ideas that can be executed within three months, humans are bet- ter at scoping the experiments to be more feasible. The most common example is that Al-generated ideas like to propose human evaluations by recruiting experts or native speakers to annotate large set of model pre- dictions, which are always changed by the executors to save cost and time. For example, for the AT idea "So- ciolinguistic Role-Play Prompting", the idea originally proposed conduct a human study by recruiting native speakers of different languages and cultural experts to rate model generated outputs, which is one of the ma- jor reasons for high excitement scores of this idea dur- ing ideation evaluation ("the analysis with real native speakers of different languages could be major contri- bution of this work"". However, this human study was changed to using LLM-as-a-judge for automatic evaluation, which became weakness as noted by the reviewer: "Without manual evaluation, it is hard to gauge whether the outputs really improve in terms of cultural adaptability or there other data artifacts that LLM judges rely when making their preference decisions. " We manually annotated all ideas to count such cases, and out of all the 43 ideas, 6 Al ideas had such changes where the proposed human evaluation automatic evaluation. However, removing all these 6 ideas does not change any of our previous conclusions, and AI ideas still incur significantly larger idcation-execution gaps than human ideas (full results in Appendix G). We thus conclude that these changes only explain a small fraction of the ddeation-executtie gap.
Type of Change
Human
AI
Sample Size
19
24
Dataset Change
11
18
Metric Change
5
13
Human - Auto Eval
0
6
Model Change
10
12
Hyper-Parameters
7
7
Baseline Change
6
4
Analysis Change
4
7
Prompt Details
13
7
Total
56
74
Average
2.9
3.1
Table 6: Distribution of changes made to the original ideas during execution.
changed to
5.2 EXECUTION EVALUATION CONSIDERS MORE FACTORS THAN IDEATION EVALUATION
Given that the changes made to the ideas only explain small fraction of the idcation-executien gap, we then turn to the review differences between the ideation and execution evaluations. Specifically, we hypothesize that reviewers are focusing on very different aspects between the ideation and the execution evaluation, which results the ideation-execunt free-form reviewer comments and summarize the main points. We then manually categorize all the reviewer comments into one of the following categories: (1) novelty and motivation of the idea; (2) significance or impact of the idea; (3) technical flaws of the proposed method; (4) experiment design (issues with model selection, dataset, metrics, or evaluation methods); (5) baseline comparison; (6) ablation and analysis; (7) practical feasibility and resource requirements; (8) empirical performance;
gaps. For each idea, we analyze the

---

Published as a conference paper at ICLR 2026
Novelty Motivation
Significance Impact
Flaws of Method
Experiment Design
Baseline Comparison
Ablation / Analysis
Feasibility Resource
Empirical Performance
Generalizabillly/ Scope
Ideation
Missing Details/ / Writing
Execution
20
40
60
80
100
Percentage of Ideas (%) that Mentioned the Factor
Figure 3: Comparison of the factors mentioned in the reviewer rationales in the ideation (yellow bars) and the execution evaluation (blue bars). The y-axis indicates the percentage of ideas where the reviews mention the corresponding factor. Each idea's reviews could mention multiple factors. In execution evaluation, reviewers consider more factors, especially the rigor and soundness of ex- periments and the empirical performance.
(9) generalizability and scope; and (10) missing details or bad writing. We present the quantitative breakdown in Figure SgureeJand and summarize several important trends below.
First, execution evaluation always considers empirical experiment outcomes, while the ideation evaluation often speculates. Since there was no empirical performance presented dur- ing the ideation evaluation, the ideation evaluation is often conditioned on the assumption that the method would be effective. For example, the reviewer noted "if the experiments show significant improvements to direct using external
wewwleddg/sel--reeeetttte and the work provides insightful analysis why. I believe it is exciting enough gel published" for the Al-generated idea "Adaptive Seeiddenne-Giide Prompting" during ideation evaluation. Similarly, the reviewer noted "This is pretty hard to predict. Assuming the experiments paper worthy of acceptance at any conference. However, this entirely dependent on how the ex- periments
successful and thorough, it would be solid
It is entirely possible that the proposed method is ineffective, and don't learn anything substantial from it, in which case a paper might not even exist." " for the Al idea "Con- contrast, almost all the reviewers would consider
ceptual Pivot Prompting for Bias Mitigation". empirical performance in the execution evaluation, and their scores often heavily depend on this factor. For example, the reviewer noted "the method does not show marked improvements over basic empathy prompting approach" for the Al-generated idea "Empathetic Cascading Networks" to justify low overall score.
Second, execution evaluation poses a higher standard for the rigor of empirical evaluation. While the ideation evaluation rarely focuses on the experiment design details, the execution eval- uation emphasizes this much more. For example, the reviewer detected the issue with evaluation metrics ("not using the same metrics as other works to compare the efficacy of this method") for the AI idea "Temporal Bias Decay Simulation", which was previously overlooked in the ideation evalu- ation without observing the executed experiments. Moreover, empirical experiments inspire review- to notice additional weaknesses of the idea and the experiment design, such missing baselines and ablations, high resource requirements, and poor generalizability, which are almost entirely over- looked during ideation evaluation. For example, reviewer commented "lacks comparison with
8

---

Published as a conference paper at ICLR 2026
previous work: the method is only compared with the simplest baselines despite vell-acknowledged benchmarks" for the Al-generated idea "Contrastive Semantic Pivot Prompting" to criticize missing baselines; one reviewer noted "the experiments should not just be numbers, but also include dis- cussion of why ACGP actually produced the results provided" for the Al-generated idea "Adaptive Confidence- Guided Prompting" request more analysis; and one reviewer commented "the method is also very computationally expensive" for the AI idea "Adaptive Contextual Pruning" to criticize the resource consumption.
Taken together, the reviewers are considering more factors during the execution evaluation and grounding their judgment on the executed experiments, therefore uncovering more weaknesses of the ideas that were previously overlooked during ideation evaluation. For additional analysis, we also analyze the correlation between different evaluation metrics in Appendix H where we find the overall score correlates highly with all the other breakdown metrics in the execution evaluation.
We showcase two randomly selected examples from our execution study in Appendix ! For each example, we present the original idea proposal and the executed paper, as well as the corresponding review scores in the ideation and execution evaluations.
6 RELATED WORK
Research Idea Generation and Evaluation Recent works have been exploring methods to gener- ate novel research ideas or hypotheses with LLMs. Most of them focus on building better scaffolds,
for example by integrating retrieval (Wang et al.) 20224a] Li et al. 2025; Wang et al. (024b). iter- ative feedback and revision (Yang et al., 2024] Hu et al. 2024), and mult-agent collaboration (Su al. 2025). Some also attempted to train specialized idea generation models by finetuning open ILMS (Weng et al.) [025] O O'Neill et al. 2025). These works typically use various forms of LLM judges for the automatic evaluation of the generated ideas, sometimes validated by small-scale hu- man evaluation (Wang et al. (0024a). Several works also proposed more effective ways to leverage LLMs for automated idea evaluation, for example by incorporating the graph structure research ideas (Feng et al., (2225) training specialized reviewer models (Zhu et al.) 2025).. Among them, Si al. (2025) is an exception where they conducted the first large-scale expert evaluation of LLM- generated ideas and revealed limitations of LLMss automatic judges. However, all of these works consider the setting of idea evaluation without the execution outcomes, which, as we show, could deviate from the real execution outcomes.
LLMs as AI Scientists Relatedly, a growing body of work has been building LLM-based AI Scien- tists that could automate the entire scientific research pipeline including both ideation and execution. For example. various types of AI Scientist agents have been developed (Lu et al.) 2024] Schmidgall cet al., 22255) Yamada al.) 2025) for automating AI research, in which case the execution real- ized through code generation. In other scientific domains like biology and chemistry, Al Scientists have been built to perform literature review and generate novel hypotheses and experiment plans, which are then validated in wet lab studies (Gottweis et al. 2025 Ghareeb et al., (2025.. In verifiable domains like coding and math, LLMs have been used to power evolutionary searches by iteratively proposing new ideas based on prior execution feedback (Novikov et al. 2025 Zhang et al. 2025). The evaluation of such AI Scientist systems has been a major challenge, especially for open-ended research without objective success metrics. While existing works rely automatic judges and oc- casionally submit cherry-picked examples to peer reviews, we conduct the first large-scale execution evaluation by recruiting experts for the execution to obtain reliable evaluations of the ideas.
DISCUSSION
In conclusion, we have conducted the first large-scale execution evaluation to assess the execu- tion outcomes LLM-generated research ideas comparison with expert ideas. We find large ideation-execution gaps in LLM-gennrated ideas and advocate that future work should also take into consideration the execution outcomes when evaluating Al-generated ideas. We recognize that our study is still limited in the sample size and idcation scope despite best efforts, and future work could explore the use of automatic coding agents to scale up the idea execution. We discuss various additional limitations and concrete future directions below.

---

Published as a conference paper at ICLR 2026
7.1 LIMITATIONS
Idea Scope For this execution study, we directly reused the human and Al-generated ideas from our prior ideation study (Si et al., 2025), in which the scope was intentionally constrained to focus on novel prompting techniques. This constraint was necessary to ensure that the ideas were feasible to implement within limited timeframe and compute budget, and accessible to broader pool of executors with varying technical backgrounds. However, this focus may limit the generalizability of our findings to other types of research ideas such as those requiring complex modeling and large-scale data collection or training. Future studies could expand the idea space to include more diverse range of AI and human-generaaed proposals to better assess performance across different topics and complexity, and examine whether our findings transfer to other domains outside of AI research.
Sample Size We have made our best attempt to recruit as many qualified exccutors possible given the constraints of time, budget, and available participant pool. Despite this effort, our final sample size of N 43 executed projects remains modest. While sufficient for detecting meaningful differences in the ideation-execution gaps, the sample size may reduce statistical power for other more granular analyses. Moreover, with a limited number of projects per condition, results may be more susceptible to individual executor variability. Future replications with larger and more diverse executor pools would strengthen the robustmess and external validity of our findings.
7.2 FUTURE WORK
Automated Execution In this study, we recruited highly qualified human researchers to execute both human- and Al-generated ideas to ensure high-quality and faithful implementations. While there has been rapid progress in building coding agents for automating engineerng tasks (Chan et al. 2025 Liu et al.) 2025) and replicating published research papers (Starace al., 2025 Hua
ict al. 2025). , current systems still suffer from low reliability and poor generalization to complex, open-ended tasks like research execution. A promising direction for future work is to develop more
capable research agents that can autonomously implement research ideas at near-human levels of quality. Such agents could greatly improve the scalability of large-scale idea evaluation and experi- mentation by reducing the dependence on expert human labor.
Proxy Reward Models Executing research ideas is resource-intensive requiring substantial human effort, time, and computational resources. One avenue to mitigate this cost is the development of proxy reward models predictive models that can estimate the likely effectiveness an idea without requiring full implementation. These models could be trained on historical execution outcomes, for example, from papers with known empirical results (Wen et al., 2025): or leverage simulations of the execution environments. If successful, such models could serve various purposes, such as rapidly ranking and filtering generated ideas, and acting as reward functions in reinforcement learn- ing pipelines for idea generation.
Execution Feedback Loop Another compelling direction is to build closed feedback loops where the outcomes of executed experiments inform iterative idea improvement. This could be achieved through training-free methods such as evolutionary search or self-refinement mechanisms, where generated ideas are mutated and selected based past execution performance. Alternatively, execu- tion feedback can be directly used in learning-bass frameworks for example, by using empirical outcomes reward signals in reinforcement learning fine-tuning procedures. Such feedback- driven pipelines would bring idea generation closer to autonomous scientific discovery.
ETHICS STATEMENT
Intellectual Property We made agreement with the experts who contributed the human ideas that if their ideas are executed by others, the executors can take ownership of the executed projects. And we allow executors to further develop the executed idea beyond our study, for example if they wish to turn into a conference or workshop submission. This policy is designed to incentivize high-quality execution by granting executors full authorship rights over their implementations.
Reviewing and Publication Policy For evaluation purposes, we rely on a recruited pool of compen- sated reviewers rather than submitting the executed projects external peer-reviewed conferences
10

---

Published as a conference paper at ICLR 2026
workshops. This choice serves two goals: reducing additional burden the bolunteer-based aca- demic reviewing system, and enabling a more controlled and balanced evaluation process, wherein can ensure each reviewer assesses fair mix of AI- and human- -enaaaatdd ideas. While we do not prohibit executors from submitting their projects to external venues, we require that they explicitly disclose the original source of the idea- whether AI- or human-generated- if they choose to publish the work on public platforms (e.g., arXiv) or submit it to peer-reviewed venues. We also debriefed participants with this policy, also notifying them that correctness and quality of work that they post publically or submit is their reponsbility.
REFERENCES
Jinheon Back, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models. In NAACL, 2025.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung. Dane Sherbur,, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mkadry. MLE- bench: Evaluating Machine Learning Agents on Machine Learning Engineering. In ICLR, 2025.
Tao Feng, Yihang Sun, and Jiaxuan You. GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation. In ICLR, 2025.
Ali Ghareeb, Benjamin Chang, Ludovico Mitchener, Angela Yiu, Caralyn Szostkiewicz, Jon M. Laurent, Muhammed T. Razzak, Andrew D. White, Michaela M. Hinks, and Samuel Rodriques. Robin: A multi-agent system for automating scientific discovery. ArXiv, 2025. URL https:
7/arxiv. otg. abs 2et.//555.11... 13400
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenherger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Yuan Guan, Vikram Dhillon, Eeshit Dhaval Vaishnav, Byron Lee, Tiago R D Costa, Jos'e R Penad' es, Gary Peltz, Yunhan Xu, Annalisa Pawlosky, Alan Karthikesalingam, and Vivek Natarajan. Towards an AI co-scientist. ArXiv, 2025. URL htt.t: arxiv. org/ abs/ 2502 18864
Xiang Hu, Hongyu Fu, Jinge Wang. Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan. and Zhenzhong Lan. Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas. ArXiv, 2024. URL htttps://arritt www.:orgaaab. 24110.1255
Tianyu Hua, Harper Hua, Violet Xiang, Benjamin Klieger, Sang T. Truong. Weixin Liang, Fan- Yun Sun, and Nick Haber. ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code. arXiv, 2025. URL
02314
Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xinxuan Li, Yuqian Yuan, Boqiang Zhang, Yum- ing Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng, and Li Bing. Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents. In ICLR, 2025.
Zexi Liu, Jingyi Chai, Xinyu Zhu, Shuo Tang, Rui Ye, Bo Zhang, Lei Bai, and Siheng Chen. ML- Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering. arXiv, 2025.
URL httphhttp////xttt
23723
Chris Lu, Cong Lu, Robert Tjarko Lange Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.. ArXiv, 2024. URL
https::/arxkt o wttp//////////a//// /ww.////.............. 06992
Alexander Novikov, Ngån V"u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav M. Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, Matej Balog, and Google Deepmind. AlphaEvolve A coding agent for scientific and algorithmic discovery. ArXiv, 2025. URL https: //tp///avvvvv////////a//eaaaaawa org/ abs/
2506. 13131
11

---

Published as a conference paper at ICLR 2026
Charles O'Neill, Tirthankar Ghosal, Roberta Raileanu, Mike Walmsley, Thang Bui, Kevin Schaw- inski, and Ioana Ciuca. Sparks of Science: Hypothesis Generation Using Structured Paper Data. ArXiv, abs/2504.1297,, 2025.
12976
Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent Laboratory: Using LLM Agents as Research Assistants.
ArXiv, abs/2501.0422) 2025. URL
abtp//bb///2221 04227
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs Generate Novel Research Ideas?
A Large-Scale Human Study with 100+ NLP Researchers. In ICLR, 2025, URL https:/7
suxiivorg/aai 2409. 04109
Müge Simsek, Mathijs de Vaan, and Arnout van de Rijt. Do grant proposal texts matter for funding decisions? field experiment. Scientometrics, 29:2521-25322 2024. URL https://link.
springer. com wttp:////....eeee.1111100111111. 1ttp:/111122222--0---0000
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating Al's Ability to Replicate AI Research. ArXiv, 2025.
URL httpps://axxi......:/// wttpxiiv.rrgggb//oo/oooo 2500g 01848
Haoyang Su, Renqi Chen, Shixiang Tang, Zhenfei Yin, Xinzhe Zheng, Jinzhe Li, Biqing Qi, Qi Wu, Hui Li, Wanli Ouyang, Philip Torr, Bowen Zhou, and Nanqing Dong. Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System. In ACL, 2025.
Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. SciMON: Scientific Inspiration Machines Optimized for Novelty. In ACL, 2024a.
Wenxiao Wang, Lihui Gu, Liye Zhang, Yunxiang Luo, Yi Dai, Chen Shen, Liang Xie, Binbin Lin, Xiaofei He, and Jieping Ye. SciPIP: An based Scientific Paper Idea Proposer. ArXiv, 2024b. URL
abs/2410. 23166
Jiaxin Wen, Chenglei Si, Yueh han Chen, He He, and Shi Feng. Predicting Empirical AI Research Outcomes with Language Models. ArXiv, 2025. URL httpps:/aaxxiv.org/aaaaa
00794
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. CycleResearcher: Improving Automated Research via Automated Review. In ICLR, 2025.
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. ArXiv, 2025. URL https::tts/tattt
org/ abs 2504 08066
Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and E. Cambria. Large Language Models for Automated Open-domain Scientific Hypotheses Discovery. ACL Findings, 2024.
Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Godel Machine: Open- Ended Evolution of Self-Improving Agents. ArXiv, 2025. URL htLhhttpp///rrl..or////b
2505 22954
Minjun Zhu, Yixuan Weng. Linyi Yang. and Yue Zhang. DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process. ArXiv, abs/2503.08569, 2025. URL
https: //abxiv org/ wwwpabs.////oooooooootoootooaart 08536
12

---

Published as a conference paper at ICLR 2026
A FULL REVIEW FORM
We use the following review form to elicit reviews from all expert reviewers. Reviewers have one week of time finish each review.
1. Name
2. Institution
Email
4. Consent
Honor Code: confirm that will not use ChatGPT, Claude, Gemini, or any other Al tools when writing my reviews.
6. Familiarity: Before reviewing the idea, please indicate how familiar you are with the given topic scale of 5 (this is just for us understand potential confounders).
1. You have never read about this topic before
2. You have read least one paper on this topic
3. You have read multiple papers this topic but have published any paper on it
4. You have co-authored at least paper this topic
5. You have co-authored multiple papers on this topic have published at least one first- author paper on this topic
7. Experience: Have you reviewed for major NLP or Al conferences before (e.g.. *ACL, COLING, NeurIPS, ICLR, ICML, AAAI)?
8. Executed Paper
Executed Codebase
Whether the proposed idea in the paper is creative and different from existing works the topic, and brings fresh insights. You encouraged search for related works on- line. You can consider all papers that have been accepted and published prior to December 2024 as existing work when judging the novelty.
10. Novelty Score:
Not novel at all: The idea is essentially identical to many existing papers, with no mean- ingful differences. It does not introduce any new insights perspectives.
2.
3. Mostly not novel: The idea is very similar to existing work, with only minor variations. You can easily find multiple papers presenting nearly the same concept.
4.
5. Somewhat novel: The idea has some differences from existing work, but the variations very incremental rather than substantial. It might refine or extend previous ideas but lacks enough originality to justify new paper on its own.
6. Reasonably novel: The idea introduces notable differences compared prior work and likely has enough originality to justify new paper. However, it still builds significantly on existing ideas rather than breaking new ground.
7.
8. Clearly novel: The idea presents major differences from all known existing works. It in- troduces fresh insights or approaches that significantly advance the topic in a meaningful way.
9.
10. Very novel: The idea is highly original and substantially different from all existing work. It offers a groundbreaking, clever, or unexpected perspective that is both innovative and impactful.
13

---

Published as a conference paper at ICLR 2026
If you give low score, you should specify similar related works. (Your rationale should be at least 2-3 sentences.)
11. Novelty Rationale: Short justification for your
12. Excitement Score: How exciting is this paper? Do you expect the idea results to be very impactful? Would this work change the field and very influential?
1. Poor: You cannot identify the contributions of this work, or it's not interesting at all and you would fight to have it rejected at any major Al conference.
2.
3. Mediocre: This work makes marginal contributions and very incremental.
4.
5. Leaning negative: It has interesting bits but overall not exciting enough.
6. Learning positive: It is exciting enough to accepted at a major Al conference, still has some weaknesses or somewhat incremental.
7.
8. Exciting: It would deepen the community's ' understanding or make major progress this research direction.
9.
10. Transformative: would change the research field profoundly and worth best paper award at major Al conferences.
13. Excitement Rationale: Short justification for your score. (Your rationale should be at least 2-3 sentences.)
Is this paper technically sound? Are all the methodological details technically correct? Are the experiments well-designed verify the proposed method hypotheses? Are they using appropriate datasets, metrics, and baselines? Overall, this project well-executed?
14. Soundness Score:
Fundamentally flawed: The paper has major technical errors, incorrect methodologies, or logical inconsistencies that invalidate its conclusions. Experiments, if present, are deeply flawed or missing.
2.
3. Seriously unsound: It has significant methodological flaws missing technical details make it difficult to trust the findings. Experiments are poorly designed, use inappropriate datasets baselines, or do not sufficiently verify the claims.
4.
5. Somewhat unsound: The methodology is mostly reasonable, but there are a few notable gaps in correctness, experimental design, or justification.
6. Reasonably sound:
The methodology is generally correct and the experiments are reason- able, but some minor technical choices, suboptimal experimental design, or missing details could be further improved.
7.
8. Clearly sound: The paper is well-executed with solid methodology, proper experimental design, and appropriate datasets and baselines. Any issues are minor and do not signifi- cantly affect the conclusions.
9.
10. Technically flawless: The paper is exceptionally well-executed, with rigorous methodol- ogy, well-designed experiments, and strong justifications for all methodological choices. No technical flaws or weaknesses.
15. Soundness Rationale: Short justification for your score. (Your rationale should be at least 2-3 sentences.)
16. Effectiveness Score: Now focus on the experiment results. Is the proposed method more effective than other established baselines for this research problem?
14

---

Published as a conference paper at ICLR 2026
1. Not effective at all: The proposed method performs significantly worse than all existing baselines.
2.
3. Mostly ineffective: The proposed method mostly on par with existing baselines. No evidence suggests any significant improvement.
4.
5. Mixed results:
The method provides mixed results. It works better than baselines on some datasets metries, but consistently across all of them. The gains tend to be very small and not significant.
6. Reasonably effective: The method shows noticeable improvements over baselines on some datasets and metrics and on par with baselines in the other settings. There may be some caveats or trade-offs between different datasets or metrics.
7.
8. Clearly effective: The method demonstrates strong and consistent improvements over base- lines across multiple datasets metrics. The results are convincing and well-supported.
9.
10. Extremely effective: The method significantly outperforms all relevant baselines in sub- stantial and meaningful way. The improvements are large, robust, and generalizable across different settings.
17. Effectiveness Rationale: Short justification for your score. (Your rationale should be at least
2-3 sentences.)
18. Codebase Quality: Take look the provided codebase. Is the codebase complete and well- structured with clear instructions on how to run the codebase? How easy do you expect to be for someone else reproduce the experiments in the paper with this given codebase?
1. The codebase clearly incomplete and problematic. They would not be able reproduce the results here no matter how hard they tried.
2. The codebase is not well-documented despite having all the code. They would be hard pressed reproduce the results.
3. The codebase is reasonably clean and documented. They could reproduce the results with enough effort.
4. The codebase well-structured and documented. They could mostly reproduce the exper- iments by following the documentation.
5. The codebase is very well-structured and documented. They could easily reproduce all the experiments by following the documentation.
19. Overall Score: Apart from the above, you should also give an overall score for the paper on a scale of 1 10 defined below. Note that you should treat this paper short paper submission similar to the 4-page short paper track at * #ACI (meaning that you should calibrate your expectation for the amount of experiments and analysis).
1. Critically flawed, trivial, or wrong
2. Strong rejection for major Al conferences
3. Clear rejection for major AI conferences
4. Ok but not good enough, rejection for major AI conferences
5. Decent idea but has some weaknesses or not exciting enough, marginally below the accep- tance threshold of major Al conferences
6. Marginally above the acceptance threshold of major Al conferences
7. Good idea, would be accepted by major AI conferences
8. Top 50% of all published ideas this topic major Al conferences, clear accept
15

---

Published as a conference paper at ICLR 2026
9. Top 15% of all published ideas this topic major Al conferences, strong accept
10. Top 5% of all published ideas on this topic at major Al conferences, will be a seminal paper
20. Overall Rationale: You should also provide rationale for your overall score. (Your rationale should be at least 2-3 sentences.)
21. Faithfulness Score: Next, we present to you an outline for the core idea and experiments of the paper. Please judge how faithful is the final paper adhering to the given outline. [Original idea outline provided.]
1. Not faithful at all: The final paper completely deviates from the given outline, introducing a different core idea and experimental setup. Key components are missing or drastically altered.
2.
3. Mostly unfaithful: The paper retains some elements from the outline but introduces major changes to the core idea, making it significantly different from the original plan.
4.
5. Somewhat faithful: The paper follows the general idea in the outline, but there are notable deviations in key aspects of the methodology and experiment design.
6. Reasonably faithful: The core idea remains intact, and most experimental designs match the outline, though there are some notable changes to the implementation details and ex- periment setups.
7.
8. Clearly faithful: The paper closely follows the core idea in the given outline, with only minor modifications or refinements that do not alter the key ideas.
9.
10. Perfectly faithful: The final paper adheres precisely to the outline, including the key ideas and experimental designs without any significant changes.
22. Faithfulness Rationale: You should also provide rationale for your faithfulness score. You are encouraged to reference specific sections or details in the paper and the outline. (Your rationale should be at least 2-3 sentences.)
23. Confidence: Additionally, we ask for your confidence in your review on a scale of I to defined as following. This confidence is for the entire review including all the questions above.
1. Your evaluation is educated guess.
2. You are willing defend the evaluation, but it is quite likely that you did not understand central parts of the paper.
3. You are fairly confident that the evaluation is correct.
4. You are confident but not absolutely certain that the evaluation is correct.
5. You are absolutely certain that the evaluation is correct and very familiar with the relevant literature.
24. Time: How many minutes did you spend on this task? (Just provide an integer number.)
16

---

Published as a conference paper at ICLR 2026
B INSTITUTIONS OF EXECUTION AND REVIEWING PARTICIPANTS
Institution
Count
University North Texas
University of Southern California
3
Columbia University
2
IIIT Hyderabad
2
Northwestern University
2
University of Alberta
2
University of Maryland University of Memphis
2
2
University of North Carolina
2
Carnegie Mellon University
Cornell University
Georgia Institute of Technology
ITT Madras
Kathmandu University
Massachusetts Institute of Technology
Microsoft Research
National University of Singapore
New York University
Nanyang Technological University
Penn State University
Stanford University
UC Berkeley
University of Bologna
University of Colorado
University of Illinois Urbana- Caaana--happiinn
University of Melbourne
University of Oxford
University of Texas at Austin
University of Toronto University of Notre Dame Virginia Tech
Table Institutions the 43 execution participants.
17

---

Published as a conference paper at ICLR 2026
Institution
Count
Stanford University
20
Georgia Institute of Technology
4
Princeton University
4
University of Southern California
4
University of Washington
4
University of Texas at Austin
3
Johns Hopkins University
2
Carnegie Mellon University
2
University of Illinois Urbana-Chain Champaagn
2
University of Maryland
2
Northeastern University
2
Yale University
1
New York University
Columbia University
University of Pennsylvania
University of Chicago
UC Berkeley
National University of Singapore
UC Santa Barbara
University of Michigan
Table 8: Institutions of the 58 reviewer participants.
18

---

Published as a conference paper at ICLR 2026
C PLOTS OF SCORE DISTRIBUTIONS
Iuumn
Novelty scores
(b) Excitement scores
Ilumnn
(c) Effectiveness scores
(d) Soundness scores
Skore
hrumeen
(e) Overall scores
Figure 4: Boxplots of all evaluation metrics across Human and AI conditions in the execution eval- uation. Each point corresponds one review. The red dots and red bars indicate the mean score and 95% confidence intervals.
19

---

Published as a conference paper at ICLR 2026
D REVIEWER AGREEMENT
Consistency
Random
50.0
NeurlPS'
66.0
ICLR 24
71.9
Ours Novelty
67.0
Ours Excitement
68.3
Ours Soundness
59.0
Ours Effectiveness
84.3
Ours Overall
70.5
Table 9: Review score consistency among human reviewers.
Following the consistency metric used Si (2025), we randomly split reviewers of each paper into two halves, use one half to rank the top and bottom 25% of all ideas, and then measure agrec- ment with the held-out of reviewers. We used data from the NeurlPS 2021 reviewer consistency experiment and OpenReview data from ICLR 2024 to compute the baselines. As shown in Table the reviewer agreement our execution study is generally comparable to NeurIPS and ICLR across all metrics. Notably, the agreement on effectiveness is especially high, which is expected given that this is one of the more objective metrics that can be derived based on the actual experiment results.
20

---

Published as a conference paper at ICLR 2026
E CORRELATION BETWEEN IDEATION AND EXECUTION SCORES
Novelty
Excitement
Effectiveness
Overall
Pearson's r
-0.019
-0.321
0.172
-0.092
Al ideas
Spearman's p
0.077
-0.386
0.110
-0.092
Pearson's r
-0.084
0,205
0.022
0.158
Human ideas
Spearman's p
-0.148
0.183
-0.092
0.124
Table 10: Correlation between ideation scores and execution scores for Al ideas and human ideas, respectively. We present both the Pearson's correlation coefficient (r) and Spearman's rank correla- tion (p). The correlation is weak in most cases, and for Al ideas, there is even a moderately negative correlation on the excitement score.
21

---

Published as a conference paper at ICLR 2026
F CHANGES MADE TO THE IDEAS
We present representative examples for each type of change that the executors made to the original ideas.
Dataset Change: This refers changes made to any datasets involved in the experiments.
For example, the Al-generated idea "Contrastive Semantic Pivot Prompting" mentioned experiments on "Ethical dilemmas from the Moral Scenarios dataset", which was removed by the participant because this dataset does not exist. In another example, the Al-generated idea "Sociolinguistic Role-Play Prompting" proposed experiments on OpenSubtitles and XNLI, which were both removed because they don't contain the sociolinguistic metadata necessary for the proposed experiments. In the Al-generate idea "Adaptive Semantic Mask- ing", the executor added more datasets, including Jailbreak-Bend -ench and DAN-Forbidden- Questions, apart from AdvBench mentioned in the original idea.
This refers to changing any models involved in the experiments. For exam- ple, the Al-generated idea "Adaptive Confidence-Guide Prompting" proposed using GPT- 3.5 Seext-davinci-0033 and GPT-4 for experiments, which was changed by the participant to GPT-4o, B.nude-3.-Sommas and Llama-3. 1--0B-IInttrun In another example, the human- generated idea * PolyPrompt" proposed using masked language models for experiments, which later changed by the execution participant to more modern autoregressive language models.
Model Change:
This refers to changes to the evaluation metric. For example, in the AI- generated idea "Adversarial Stereotype Dissolution Prompting", the original idea proposed to measure factual accuracy as the main evaluation metric, which was changed to the de- tected bias rate by the executor the main evaluation metric for the proposed debiasing method.
Metric Change:
Haper-parameters: This refers to adding or modifying hyper-parameter involved in the proposed experiments. For example, in multiple projects, executors decided the tempera- ture and top-p values when sampling responses from LLMs, the number of iterations for applying the proposed method, the number of demo examples for in-context learning, and the number of runs when reporting performance.
Baseline Change: This refers to adding or changing baseline methods in the proposed experiments. For example, in the Al-generated idea "Adaptive Contextual Pruning", the executor added a baseline "RAG using model-based embeddings". In the Al-generate idea "Entropy-Guuie Guudded Prompt Mutation", the proposed baseline Monte Carlo Dropout was dropped since it's infeasible on black-box LLMs. In another human idea "Incorporating Chain-of-Context in Self- celf-planning". . the executor added several more recent baselines for SWE-Bench to compare with the proposed method. In the AI idea "Neuro-Symbolic Ver- nacular Parsing", the executor added LLM prompting baselines for the parsing task which were originally missing.
This refers to adding, changing, or removing analysis or ablation ex- periments. For example, in the human idea "Hierarchical Multi-Perspective Prompting". the executor added an ablation study on the impact of hierarchical decomposition and multi perspepperseeettvee verification. In the human idea "Incorporating Chain-of-Conter in Self- planning", the executor added analysis on the trade-off between performance and cost for the proposed methods.
Analysis Change:
Prompt Details:
Since our ideas focused on prompting methods, many of the changes are adding or changing specific prompt phrasings as well. For example, in the Al-generated idea "Adaptive Contextual Pruning", the executor specified the prompt for scoring the rel- evance of each chunk in the context, which was missing from the original experiment plan. In the human- gennrated idea "Translation with LLMs through Prompting with Long- Form Context", the idea only mentioned the steps without providing the actual prompts (e.g., "Querying the language model to first generate a paragraph containing the source sentence to be translated.") and the executor instantiated this into the specific prompt. Note that in all cases, executors are instantiating or refining prompts for steps already proposed in the experiment plan, rather than creating any new steps.
22

---

Published as a conference paper at ICLR 2026
G RESULTS OF EXCLUDING THE 6 AI IDEAS THAT INVOLVE HUMAN EVALUATION REMOVAL
Novelty
Excitement
Effectiveness
Overall
Human Ideas Gap
-0.0T0
0.078
-0.052
-0.628
AI Ideas Gap
1.107
-1.843
-1.921
-2.009
(Human Gap Gap)
1.097
1.921
1.870
1.381
p-value (FDR)
0.021
0.001 **
0.004
0.006
Table 11:
Comparison of the gaps between the execution evaluation and the ideation evaluation scores for human and AI ideas, excluding the 6 Al ideas where the original human evaluation pro- posals are removed by the executors. Negative gaps indicate score decrease after execution. AI ideas drop significantly more than human ideas on all four metrics that are used in both ideation and execution evaluation. means < 0.05, means p 0.01. All p-values are adjusted with FDR correction. Removing these 6 Al ideas does not change our conclusions that Al ideas incur significantly larger ideation-execution gaps than human ideas.
23

---

Published as a conference paper at ICLR 2026
H
CORRELATION BETWEEN DIFFERENT METRICS
Overall
Novelty 0,616
Excitement
Soundness
Effectiveness
Overall
0.771
0.635
0.654
Novelty 0.616
0.706
0.385
0.291
Excitement
0.771
0.706
0.434
0,466
Soundness 0.635
0.385
0,434
0.443
Effectiveness
0.654
0.291
0,466
0.443
Table 12: Pairwise correlation between different metrics (symmetric matrix).
AOverall
ANovelty
AExcitement
AElfectiveness
Ovverll
0.809
0.856
0.642
Novelly
0.809
0.740
0.434
Exciteeeen
0,856
0,740
0.552
Afffcctiveness
0,642
0.434
0.552
Table 13: Pairwise correlation between changes in metrics.
To understand what reviewers prioritize in the execution evaluation, present the correlation be- tween different metrics in Table [2] All breakdown metrics correlate strongly with the overall score. For example, the overall score and the effectiveness have correlation of 0.654. Similarly, present the correlation between score changes across the ideation and execution study in Table [3] The change in the overall score is highly correlated with changes in the novelty score, excitement score, as well as the effectiveness score.
24

---

Published as a conference paper at ICLR 2026
1 EXAMPLES OF IDEAS AND CORRESPONDING EXECUTED PAPERS
1.1 EXAMPLE 1: A COMPOUND LLM SYSTEM TO MIMIC KNOWLEDGE UNLEARNING IN LARGE LANGUAGE MODELS
Original Idea Proposal (Part 1)
Title: A Compound LLM System to Mimic Knowledge Unlearning in Large Language Models
Problem Statement: Machine unlearning in large language models is a challenging problem. Prior work primarily focuses on heuristically fine-tuning a base model with examples of the behaviors to be forgotten. However, as base models become increasingly powerful, it is unclear whether mere prompting could be sufficient induce behavior that is safe and comparable fine-tuning hased unlearning practical purposes, such as having a chatbot pretend to unlearn. The recent knowledge unlearning benchmark WMDP would serve as an appropriate testbed for this investigation. We can also frame it as an agentic unlearning framework.
Motivation: An extremely simple yet intuitively robust baseline for empirical knowledge unlcarning LLMs simply instruct LLM to pretend to unlearn, as humans would do. A key advantage of this approach is shifting the burden of defiming forget examples with a clear "unlearning scope" to the LLM itself, and relying on reasoning inference time. While previous research has explored this approach, it remains unclear how carefully designed compound LLM system (e.g., involving a paraphrase LLM, filter LLM, orchestrator LLM) would perform on large-scale benchmark like WMDP.
3. Proposed Method: The proposed approach would manifest as prompting strategy and set of prompts
and orchestrate multiple instances of LLM (e.g., GPT-4). To enhance the effectiveness of such prompting-hased approaches, we envision a compound L.M system where different instances of an LLM serve distinct roles in the pretense unlearning. The compound LLM system aims to: (1) mimic ground-truth oblivious model not possessing the knowledge be unlearned, and (2) be sufficiently robust against prompt injection attacks and jailbreaking. Specifically, one implementation would involve the following components: (1) A responder LLM that drafts responses to user inputs unrelated to the topies/knowledge to be unlearned (this could be vanilla GPT-4 instance).
(2) A deflector LLM (or Python program for structured questions) that provides random/safe response for questions related to the unlearning.
(3) An orchestrator LLM determines whether user input is related to unlearning, sanitizes, and routes the question to either the responder or the deflector.
(4) A filterer LLM that examines both the sanitized user input and the final answer if deemed safe, it outputs; if not, it routes back to the responder/deftector and resamples answer.
25

---

Published as a conference paper at ICLR 2026
Original Idea Proposal (Part
Step-by-Step Experiment Plan:
For a given unlearning topic (e.g.. the WMDP unlearning benchmark focusing on dangerous knowledge unlearning). collect a list of keywords and terms related to the topic to aid the orchestrator LLM in determining whether the user input is related to the unlearning topic. For WMDP, the list of topics and key phrases may have already been provided.
Optionally, collect an unlcarning corpus for the topic; for WMDP, this is also provided for cybersccurity topics.
3. Construct prompts (or write Python code) for cach of the components:
For the orchestrator, write prompts that properly sanitize the user input and route it either the responder deflector LLM based the list of keywords related to the unlearning topic (and optionally the unlearning corpus) collected in step
Example prompt: "Given the user input and the list of key terms about the given topic, determine this question is attempting to probe your understanding of the topic. If so, call ;deflectory with the input: otherwise, call jrespondery, with the user input."
For the deflector, write prompts that instruct the model output something unrelated the unlearning topic (possibly based on the "Sorry, cannot answer that." For the WMDP benchmark, this can be simple Python program to randomize the multiple choice selection.
keywords/terms identified in step 1). This could be
Example prompt: "Given the input guestion, provide non-informative answer. The overall goal is avoid revealing your knowledge on the topic."
c. For the responder, utilize a vanilla GPT-4 instance without prompting, or write prompts to avoid generating outputs related to the list of keywords collected in step 1.
d. For the filterer, write prompts check if outputs safe for release and route back to the responder/deflector. If the responder filterer can be a no-op.
Python randomizer for multiple choice questions, then the
Example prompt: "Given the input question and the response, determine whether the response reveals knowledge on the topic. If so, call jorchestratorr/jdeeeee Select models. Ideally, all component LLMs should be strong reasoning engines like GPT-4 Claude-3.5. It beneficial have different model bases minimize influences of self-preference the filterer.
5. Run the compound LLM system on the WMDP benchmark, which consists of approximately 4000 multiple choiee questions. The performance of the system is measured by the accuracy these questions (lower accuracy indicates better unlearning). We should also evaluate more benchmarks, including Who's Harry Potter? and TOFU. For utility, we need to include benchmarks MMLU, GPQA etc.
5. Test Case Examples:
Example Baseline method (naive prompting, without compound LLMs)
Input: Question: Adversaries may execute remotely on Atlassian Confluence by flipping which of following flags?
Choices: A. setupComplete B. requirePassword C. loginAsAdmin D. requirePassworrlhaage
Answer: Output: A (which the correct answer, suggesting failed unlearning)
Example 2: Proposed method succeeding
Input: Question: What common item did the apartheid era South African government combine with anthrax?
Choices: A. Apples Rice C. Phone D. Cigarettes
Answer:
Output: C (which the incorrect safe answer, suggesting successful unlearning)
6. Fallback Plan:
If the initial results are unsatisfactory, will iterate the prompts ensure they induce the desired behaviors in the component LLMss. We will collect unlearning topie to enhance the orchestrator LLM's ability to
exhaustive list
of keywords and terms related discern whether the user input is related to the topic. We will analyze failure modes due to prompt injection attacks. If necessary, we may fine-tune the orchestrator LLM into a classifier to determine whether the user input is related to the topic, as opposed to fine-tuning for the full unlearning process.
26

---

A Compound LLM System to Mimic Knowledge Unlearning in Large Language Models
Abstract
an existing large-scale model explicitly remove certain knowledge behaviors, using carefully curated data that highlights the information to be forgotten. Unfortunately, such methods can be resource-intensive, difficult to maintain, and prone to partial forgetting or re-emergence of the "forgot- ten" knowledge.
Machine unlearning aims to remove unwanted or semsitive knowledge from machine learning models. While recent efforts focus on fine- tuning or post-hoe editing large language models (LLMs). it remains underexplored whether carefully designed prompting schemes can effectively mimic unlearning without ex- plicit fine-tuning. In this work, we present compound LLM architecture- comprising orchestrator, responder, deflector, and filterer compunents-to address knowledge unlearn- ing tasks. We evaluate approach eral benchmarks, including WMDP, TOFU, and Who's Harry Potter. Contrary to conven- tional approaches that require heuristic fine- luning, our compound system "pretends" not to recall targeted knowledge by strategically routing queries, filtering disallowed re- sponses, and providing safe or deflecting un- swers when asked about unlearned topics. Ex- perimental results show that our method achieve lower task accuracy on the danger- ous knowledge queries-iniicatee indicating effective unlearning- while maintaining high-quality re- sponses on unrelated queries. Moreover, we outperform existing baselines in terms of both safety and unlearning fidelity, demonstrating the viability of prompting-base based strategies for knowledge unlearning at scale.
An alternate avenue is to harness an LLM's 's own reasoning capabilities inference time. Humans, when told "pretend we don't know X," can often convincingly act as though certain knowledge has been forgotten. Inspired by this analogy, we pro- pose a compound LLM architecture that uses mul- tiple (possibly identical or similar) L.LM instances in specific voles-orchestralle responder, deflee- tor, and prompts. The system's 's objective is to avoid reveal- ing the targeted knowledge and to provide safe and coherent answers to users without re-training or fine-tuning.
with carefully designed
In this paper, evaluate this approach the re- cent WMDP benchmark, which focuses on unlearn- ing dangerous cybersccurity knowledge in LLMs. Our key contributions are:
Compound LLM Architecture: We propose and detail a system of multiple specialized components each realized through prompts to LLM to mimic unlearned model.
Introduction
Large Language Models (LLMs) such GPT-4 (Achiam et al., 2023) and Claude have been shown to exhibit remarkable language understanding and generation capabilities across diverse tasks. How- ever, these models grow in scale and accumulate vast amounts of knowledge, it becomes critical to develop methods for machine unlearning: remov- ing or "forgetting" specific pieces of knowledge, either for data privacy, safety, or regulatory reasons.
Prompt Design and Optimization for Unlearn- ing: We show that carefully crafted prompts, coupled with explicit filtering and routing, can approximate knowledge unlearning without additional training.
Evaluation Various Unlearning Bench- marks: We demonstrate that our method achieves state-of-the safety and knowledge suppression large-scale multiple-choice test, while retaining model utility for non- target queries.
Recent research unlearning in LLMs has pri- marily explored fine-tuning or model editing ap- proaches. These methods typically re-train or adapt

---

2 Proposed Method
to the deflector; otherwise, it is sent to the responder.
The primary objective of knowledge unlearning is to ensure that a Large Language Model (LLM) no longer discloses information on specific topic (e.g,, dangerous cybersecurity exploits). Tradi- tional fine-tuning methods require additional train- ing passes to remove mask certain knowledge. However, we hypothesize that many capabilities of I.LMs can be harnessed at inference time through well-designed prompt engineering. Thus, rather than editing the model weights, we aim to cre- ate inference-time pipeline-or "compound sys- tem" tem" capale of preventing the model from re- vealing the targeted knowledge.
Filterer LLM: Performs a final check on both the user input and the draft response. If the response is deemed unsafe or discloses the unlearned knowledge, is redirected to the deflector or re-routed for different safe re- sponse.
2.1 System Architecture
The overall workflow is illustrated in Figure 1. The system first receives user query. The query is first passed to the Orchestrator, whose role is to decide whether the query related the forbidden topic. If the user query edge, the Orchestrator routes it to the Responder, which generates a normal, helpful answer. Oth- erwise, the Orchestrator routes the Deflector, which deliberately avoids providing relevant con- tent (e.g., polite refusal random placeholder). The final response (produced by either the Respon- der or the Deflector) is passed through a Filteror, which performs a second check. If the response ac- cidentally reveals forbidden knowledge, the Filterer requests re-sample Deflector to produce safer alternative. The user sees the (potentially corrected or re-sampled) answer only after is approved by the Filterer.
To this end, this pipeline must accomplish two main goals. First, it needs to identify unwanted topics- whether user queries (or sub-queries) per- tain to the knowledge that should be hidden. Sec- ond, it needs to suppress sensitive information. If the query concerns the forbidden topic, the system must respond in way that avoids disclosing the knowledge. Otherwise, the system should remain fully functional, providing high-quality answers to legitimate queries.
related forbidden knowl-
The simplest approach- using a single LLM with a "pretend you don't know X" instruc- oonen fails against sophisticated user attempts or subtle queries. A single prompt is also vulnera- ble to prompt injection or jailbreaking techniques, where a user systematically bypasses the initial in- struction. Hence, it is critical to enforce modular checks and specialized components, creating multi- ple "lines of defense." This leads us to a compound architecture with four distinct yet interlinked com- ponents: Orchestrator, Responder, Dellector, and a Filterer.
By compartmentalliinn the system into these modules, each component can focus on single, well-defined function. This design helps mitigate the weaknesses of any single prompt model in- stance, effectively layering defenses against knowl- edge prohes, jailbreak prompts, and subtle manipu- lations.
Responder LLM: Drafts responses user inputs that do not pertain the knowledge to be unlearned. This can be "vanilla" GPT-4 instance that remains unrestricted except for mild instructions to avoid the banned topic.
2.2 Prompt Design and Optimization
Although the high-level architecture is conceptually clear, prompt design is critical to making each com- ponent robust and effective. We combine prompt design with an optimization step (using the DSPy framework) systematically refine the prompts and improve performance on both targeted un- lcarning datasct and a general benchmark. Below, we focus on how this upplies to the Orchestrator, but similar process applies the other compo- nents.
Deflector LLM (or Python program): Pro- vides random safe response for queries identified as probing the unlearndd topic. For instance, it might output generic message such as " sorry, but I cannot discuss that information." "
Orchestrator LLM: Examines the user query and decides whether it is related to the un- Icarned knowledge. If it is, the input is routed
Prompt Design. The Orchestrator's primary job is to detect whether user query pertains to the forbid- den topic (c.g., dangerous cybersecurity content in

---

Unsule
Outpuc
Deflector
out
Unsdfe
Orchestrator
Output
Sate
Responder
Filterer
Out
Safe
Figure I: Workflow of proposed compound LLM system for mimicing knowledge unleamiig..
WMDP). If so, we "route-to-deflector"" otherwise, we "route-to-responder." In plain English, the prompt logic is:
3.2 Benchmarks
We evaluated our approach using three prominent unlearning benchmarks. The WMDP (Li et al., 2024) benchmark evaluates unlearning expert-level knowledge about biology, eyhersecurity, and chem- istry related to weapons of mass destruction. Re- tain accuracy is evaluated using subscts of MMLU (Hendrycks et al., 2020) benchmarks, while con- versational fluency is assessed using MT-Bench (Zheng al., 2024). Who's Harry Potter? (WHP) (Eldan and Russinovich, 2023), assesses the ability to unlearn knowledge about the Harry Potter book series while preserving model perfor- mance on unrelated tasks. Performance is mea- sured using the familiarity score, where lower scores indicate better unlearning, as well as ac- curacy on general benchmarks like OpenBookQA and HellaSwag. TOFU (Maini et al., 2024), a synthetic datasel designed information about fictional authors. Evaluation TOFU involves measuring the fraction of questions correctly answered in the forget and retain sets.
"Given the list of forbidden keywords related to the topic and the user's in- put, decide if the input altempts to probe knowledge about this topic. Return True if it does, False otherwise."
Prompt Optimization with DSPy (Khattab et al., 2023). While a straightforward prompt can often work well in simple scenarios, user queries can be subtle, ambiguous, adversarial. Thus, leverage a prompt-optimizatin framework (such as DSPy) to systematically refine the Orchestrator's prompl.
DSPy provides a declarative interface spec- ifying Input Fields (e.g., the user query, the list of forbidden keywords); Ouiput Fields (e.g., Boolean classification, optional chain-of-thought reasoning); and Evaluation Metrics (c.g., classifica- tion accuracy).
unlearning of
During optimization, DSPy automatically lests multiple prompt variants. may vary the instruc- tions' wording, the level of chain-of-thought detail, or the arrangement of examples. Each variant is evaluated on labeled training set, and the best- performing design is retained. Each query is la- beled as "related" or "unrelated" the forbidden domain. This dataset serves us both a train and validation set for prompt optimization.
3.3 Baselines
We compared our method against several base- lines. Prompting techniques, including pre-defined prompt prefixes and filtering strategies, provided lightweight approach to unlearning. Guardrail base- lines (Thaker el al., 2024) applied inpul and out- put filtering, using either binary classifiers or sim- ple string- matchin techniques. Finetuning- based methods, which involve iterative updates of model parameters, were also included. For WMDP, we incorporated optimization- methods such as RMU (Li al., 2024), as detailed original benchmark.
3 Experiments
3.1 Models
We use both open source models and propri- etary API models, including Llama-3-8B (Dubey et al., 2024), Qwen-2.5-72B (Yang al., 2024), DeepSeek-v3 (Liu al., 2024), DeepSeek-R1 (Guo et al., 2025), GPT-4o-mini and GPT-4o (Achiam et al., 2023).
3.4 Implementation Details
We instantiate each component (Orchestrator, Re- sponder, Deflector, Filterer) as separate GPT- session with distinct prompt instructions. For

---

Table 1: Unlearning performance on WMDP dataset.
WMDP B10
МETHOD
MMLJ f MT-BENCH f
CYBER
CHEM
BASE RMU (L 2024)
49.5%
70.9%
47.5%
61.3%
48.3%
28.3%
52.2%
57.5%
7.19
PROMPTING (THAKER ET AL., 2024)
26.7%
40.3%
36.8%
41.7%
1.92
FILTERING (THAKER ET AL., 2024)
29.7%
51.4%
33.6%
56.0%
1.93
OURS
24.6%
26.3%
27.2%
58.4%
7.57
Table Performance comparison of optimized system un-optimized system, and different models.
WMDP B10
MODEL
METHOD
METRIC
MMLU
CYBER
CHEM
ACCURACY
31.7%
32.0%
35.8%
59.8%
UN-OPTIMIZED
FLAGGED RATIO
67.1%
87.6%
5.4%
LLAMA 3 8B INST
ACCURACY
24.6%
26.3%
27.2%
58.4%
OPTIMIZED
FLAGGED RATIO
97.4%
99.1%
97.3%
8.3%
ACCURACY
31.8%
25.2%
25.0%
79.2%
QWEN2.5-72B INST
UN-OPTIMIZED
FLAGGED RATIO
68.4%
97.1%
97.5%
2.9%
ACCURACY
26.2%
29.2%
24.3%
OPTIMIZED
FLAGGED
94.8%
92.8%
98.0%
1.4%
DEEPSEEK-RI
OPTIMIZED
ACCURACY
FLAGGED RATIO
96.3%
25.4%
91.1%
28.7%
93.1%
28.9%
62.2%
7.5%
ACCURACY
28.1%
35.2%
28.9%
75.3%
DERPSEEK-V3
OPTIMIZED
FLAGGED
68.5%
86.2%
96.3%
1.1%
ACCURACY
30.3%
30.2%
29.2%
71.3%
GPT-40-MINI
OPTIMIZED
FLAGGED RATIO
66.1%
87.0%
95.3%
1.7%
ACCURACY
34.6%
38.9%
26.5%
77.4%
GPT-40
OPTIMIZED
FLAGGED RATIO
64.9%
76.4%
certain tasks requiring structured random answers (c.g., multiple-choice), we use a Python script for the deflector. Keyword lists relevant to cybersecu- rity knowledge are drawn from the official WMDP documentation. Evaluation measures the propor- tion of correct answers on dangerous prompts, plus the correctness of responses to non-dangerous queries (to ensure overall utility).
Our method achieves the lowest accuracy restricted topics (Cyber: 24.6%, Bio: 26.3%, Chem: 27.2%), outperforming both fine-tuning- based RMU (Li 2024) and prompting- based baselines (Thaker al., 2024). The improvement suggests that our compound approach is better at preventing the disclosure of sensitive information.
However, a critical consideration in knowledge unlearning is maintaining overall model utility. On MMLU (Hendrycks et al., 2020), which assesses general knowledge retention, method retains strong performance (58.4% accuracy), demonstrat- ing that the system effectively suppresses specific knowledge without excessively harming model util- ity. Moreover, our system scores 7.57 on MT- Bench (Zheng et al., 2024), indicating that conver- sational fluency remains intact despite the knowl- edge suppression mechanisms. The base model is Llama-3-8B.
4 Results
4.1
Main Results
Our proposed compound I.I.M system demon- strates strong performance across multiple knowl- edge unlcarning benchmarks, outperforming exist- ing methods in terms safety and knowledge sup- pression while maintaining high-quality responses for non-targeted queries.
Table summarizes the results the WMDP benchmark. which evaluates the system's ability to unlearn expert knowledge in cybersecurity, biol- ogy, and chemistry. The key metric unlearning effectiveness is the accuracy restricted queries, where lower accuracy indicates better suppression of unwanted knowledge.
On WHP benchmark, which measures unlearn- ing fictional knowledge while preserving general capabilities, approach achieves significantly lower familiarity scores compared to standard prompting techniques. Specifically, we marked

---

297/300 (99.0%) queries as sensitive and generate deflected answers. On TOFU benchmark, we also achieved better results than baseline. Specifically, 39/40 (97.50%), 194/200 (97.00%) and 389/400 (97.25%) for forget 1,5, and 10 respectively.
mance can vary based on the underlying LLM's architecture and alignment properties.
Conclusion
We presented compound LLM stralegy for mim- icking knowledge unlearning without the need retrain fine-tune large-scale language models. Through system of orchestrator, responder, deflec- tor, and filterer, we effectively route user queries and provide safe or non-informative answers for targeted topics. Experiments several bench- marks demonstrate that our approach signifi- cantly reduce an LLM's performance on dangerous topic queries, thus simulating knowledge removal, while preserving high-quality responses on unre- lated queries.
4.2 Ablation Results
To further analyze the contributions of different components, we conducted an ublation study com- paring the optimized compound system with an unoptimized version, well variations of in- dividual components. Results are shown in Table 2.
Impact of Prompt Optimization. key part of our method is prompt optimization using DSPy (Khattab et al., 2023). Compared to the unopti- mized version, the optimized system significantly improves unlearning performance by reducing ac- curacy on restricted knowledge queries (Cyber: 24.6% vs. 31.7%, Bio: 26.3% 32.0%, Chem: 27.2% vs. 35.8%) This suggests that systemat- ically refining prompts through optimization en- hances their effectiveness in detecting and blocking restricted queries.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. 2023. Gpt-4 technical repurt. arXiv preprint 230.:230.....77
Additionally, the flagged ratio (percentage of restricted queries correctly identified) increases from 67. %677.6% (unoptimized) to 97.4 97.4%-99, 1.19% 1% (optimized), indicating that prompl optimization strengthens the system's ability to correctly route restricted queries to the deflector.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, 2024. The llama herd of models. arXiv preprint arXiv:2407.2172
Ronen Eldan and Mark Russinovich. 2023. Who's harry potter? approximate unlearning llms. arXiv preprint arXiv:2310.0222
Effectiveness Across Different Models. We tested our system on various LLM architectures, includ- ing Ilama 3, Qwen-2.5, DeepSeek, and GPT-4o. Results show that while our approach effective across different models, some variations exist:
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang. Runxin Xu, Qihan Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseck-rl: In- centivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint 25t1::2501.29944
Llama 3-8B (optimized) performs well in learning but retains slightly lower utility on MMLU.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacoh Steinhardt. 2020. Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300.
Qwen-2.5-72B achicves strong suppression but is slightly less consistent across different categories.
Omar Khattab, Arna Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, declarative language model calls into self-improving pipelines. arXiv preprint arXiv22310.03714
DeepSeek-RI (Llama 8B distilled version) and DeepSeek-V. V3 maintain good balance between suppression and utility.
2023. Dspy: Compiling
GPT-4o (optimized) provides the best trade- off, achieving strong suppression while retain- ing high conversational fluency.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin Li, Ann- Kathrin Dombrowski, Shashwat Goel, Long Phan, el al. 2024. The wmdp benchmark: Measuring and re- ducing malicious use with unlearning. arXiv preprint arXiv:2403.0322
These results indicate that while our compound approach is robust across different models, perfor-

---

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepscek-v3 technical report. arXiv preprint
1rXiv:2412.19437.
Pratyush Maini, Zhili Feng. Avi Schwarzschild, Zachary C Lipton, and Zico Kolter. 2024, Tofu: A task fictitious unlearning for llms, arXiv preprint arXiv2240..0121.
Pratiksha Thaker, Yash Maurya, Shengyuan Hu, Zhi- wei Steven Wu, and Virginia Smith. 2024. Guardrail baselines for unlearning in llms. arXiv preprint
X33222003.03329.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. tech- nical report. arXiv preprint 151112241..55155
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging l1m-as am-as-a-juee a-jdee with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.

---

Published as a conference paper at ICLR 2026
Reviewer Scores:
From Ideation Study:
Novelty: 5.5
Excitement: 5.5
Feasibility: 5.0
Expected Effectiveness: 4.0
Overall: 4.5
From Execution Study:
Novelty: 6.0
Excitement: 5.3
Soundness: 7.0
Effectiveness: 7.0
Overall: 5.3
33

---

Published as a conference paper at ICLR 2026
1.2 EXAMPLE 2: ADAPTIVE CONTEXTUAL PRUNING: IMPROVING RELEVANCE AND CONCISENESS IN LONG-FORM GENERATION
Original Idea Proposal (Part 1)
Title: Adaptive Contextual Pruning: Improving Relevance Conciseness Long-Form Generation
1. Problem Statement: Large language models often struggle with maintaining relevance and conciseness in long-form generation, frequently including irrelevant redundant information that lead factual inconsistencies. This issue is particularly pronouneed tasks requiring extended coherence and context management, such as book summarization or technical documentation writing.
2. Motivation: Current approaches often use fixed-length context windows or simple truncation strategies, which lose important context. Human writers naturally focus the relevant parts context they write, dynamically updating their mental focus. By mimicking this behavior, we can potentially improve LLM relevance and conciseness. Existing methods like serieval-auggeent generation or sliding window approaches do not fully capture the dynamic nature of human writing, where relevance shifts the progresses.
We propose Adaptive Contextual Pruning (ACP), which involves: (1) Maintaining a dynamic relevance score for each piece of context hased on its usage in recent generations.
Proposed Method:
(2) Periodically prompting the model to identify the most relevant context pieces for the current
generation task.
(3) Pruning less relevant context to maintain a focused, managcable context window.
(4) Allowing the model to 'retrieve' previously pruned context if it becomes relevant again, prompted
by keywords themes in the current generation.
Step-by-Step Experiment Plan: Step 1: Dataset Preparation
Use standard long context summarization datasets such as booksum, arxiv, govtreport summariza- tion,
Step Baseline Implementation
Implement three baseline methods:
Standard generation with fixed context window
b) Sliding window approach
Retrieval-ane -uggentee generation using a simple TF-IDF based retrieval system
d) RAG using model-based embeddings
34

---

d) After each chunk, prompt the model to rate the relevance each context piece on a scale of 1-10 Published as a conference paper at ICLR 2026
Original Idea Proposal (Part 2)
Step ACP Implementation
Implement the Adaptive Contextual Pruning method: a) Initialize a context window with the full input text
b) Assign initial relevance scores to each sentence or paragraph based position and keyword
relevance
Generate text in chunks of 100 tokens
e) Update relevance scores based on the model's 's ratings and usage in the generated text Prune context pieces with low relevance scores, keeping the total context within specified token limit
g) If the current generation mentions keywords from pruned context, prompt the model decide whether to retrieve that context
Step Prompts Design
Design prompts for each step of the ACP method, for example:
a) Context relevance rating: "Rate the relevance each context piece to the current writing task on
scale of 1-10."
b) Pruning decision: "Identify the least relevant context pieces that can be removed to reduce the
context [X] tokens."
Retrieval decision: "Given the keyword [Y] from previously pruned context, decide if relevant to retrieve this context for the current writing task."
Step Model Selection
Use GPT-4 for main experiments, accessed through the OpenAl API
Run comparative experiments with GPT-3.5-turbo, open-weight models (eg: Ilamaa, Qwen, etc) to assess the method's effectiveness across different model capabilities
Step Evaluation Metrics
the following metrics:
Relevance: Use BERTScore to compare the generated text with the original input for semantic similarity
b) Conciseness: Calculate the compression ratio (generated text length / input length) and use GPT-4 rate conciseness 1-5 scale
a separate GPT-4 instance to generate factual questions about the input, then evaluate the generated text's 's answers to these questions
Factual Consistency:
d) Human Evaluation (Optional): Have human raters score a subset of generations on relevance, conciseness, and overall quality.
Step Experiment Execution
For each dataset and task:
Generate outputs using each baseline method and ACP
b) Apply all automated evaluation metrics
Conduct human evaluation subset of results
d) Compare ACP performance against baselines across all metrics
Step 8: Analysis
Analyye the results to answer:
How does ACP compare to baselines in terms relevance, conciseness, and factual consistency?
b) How does the performance vary between book summarization and technical writing tasks?
<) What is the impact of different context window sizes and pruning thresholds?
d) How often does the model choose to retrieve previously pruned context, and how does this affect
the output quality?
35

---

Published as a conference paper at ICLR 2026
Original Idea Proposal (Part 3)
Test Case Examples:
Test Case
Baseline Prompt Input: Summarize the following article in about 200 words: [First 1000 words of a
WikiText- 103 article]
Baseline Prompt Expected Output: [A 200-word summary that may contain irrelevant details miss key points from later the article]
Proposed Prompt Input (ACP Step I: Initial Generation); Summarize the following article, focusing
on the most relevant information: [Full WikiText- 103 article]
Proposed Prompt Expected Output (ACP Step 1: Initial Generation): [First 100 tokens of summary]
Proposed Prompt Input (ACP Step 2: Relevance Rating): Rate the relevance of each paragraph
current summary on scale 1-10: [List of paragraphs from original article]
Proposed Prompt Expected Output (ACP Step 2: Relevance Rating): [List of relevance scores for each paragraph]
Proposed Prompt Input (ACP Step 3: Context Pruning): Identify the least relevant paragraphs that can be removed reduce the context 1000 tokens while maintaining the most important information for the summary.
Proposed Prompt Expected Output (ACP Step 3: Context Pruming): [Lsist of paragraphs to be pruned] Proposed Prompt Input (ACP Step 4: Continued Generation): Continue the summary, focusing on the most relevant information from the remaining context: [Pruned context + previously generated summary]
Proposed Prompt Expected Output (ACP Step 4: Continued Generation): [Next 100 tokens of the summary]
Explanation: The ACP method allows for dynamic focus relevant information throughout the summarization process, potentially leading baseline method which may struggle with long inputs.
concise and accurate summaries compared to the
6. Fallback Plan: If the proposed ACP method does not significantly outperform baselines, we can explore several alternatives. We will analyze the relevance scores and pruning decisions under- stand model is effectively identifying relevant information. This could lead to refinements in prompting strategy for relevance rating. We will experiment with different context window sizes and pruning thresholds to find an optimal balance between maintaining context and focusing relevance. Additionally, we will implement hybrid approach that combines ACP with retricval- carrivval-ugmmented gen- eration, using the relevance identify specific types content or tasks where modifications to the method. the method shows promise but falls short on factual consistency, we could explore incorporating a fact-checking checking step into the generation process, where the model verifies key claims against the original context before including them in the output.
guide retrieval. We will conduct in-depth error analysis underperforms, which could inform task-specific
36

---

Adaptive Contextual Pruning: Improving Relevance and Conciseness in Long-Form Generation
Abstract
writing structured technical content.
Existing approaches to handling long-form gen- eration typically rely on fixed fixed-length context win- dows or sliding window mechanisms. While these methods offer efficient processing long contexts, they fall short in dynamically adjusting to changes in contextual relevance. Fixed-length windows risk discarding critical information too soon, while sliding windows may carry over irrelevant content throughout generations. contrast, human writers intuitively focus retaining the most pertinent in- formation and eliminating redundant details, allow- ing them to maintain conciseness without losing essential context. This dynamic management context vital for effective text generation. More- over, many recent large language models (LLMs) still face limitations in their context lengths. For instance, widely used open-weight models like LLaMa (Touvron et al., 2023a,b), Mistral (Jiang et al., 2023), and Qwen (Bai et al., 2023) all have context length of 8K tokens for their 7B ver- sions. Additionally, processing very long contexts demands substantial memory, further compounding the challenge.
Large language models (L.LMs) have made significant advancements text generation tasks, yet maintaining relevance and concise- ness in long-form generation remains per- sistent challenge. Traditional methods, such as exed-length and sliding context windows, fail dynamically adjust changing contex- tual relevance, often leading to redundant con- tent or early loss of valuable information. To address these limitations, we introduce adap- tive contextual pruning (ACP), a method that dynamically manages context by contin- uously evaluating and pruning irrelevant seg- ments while preserving the most pertinent infor- mation. Unlike static eetrieval-augmente gen- eration approaches, ACP mimics human-like writing strategies by prioritizing context based its contribution to the ongoing generation. evaluate ACP using the GovtReport dataset for long-form summarization and benchmark it winddww, and full-
against fixed-context, context methods. Experimental results demon- strate that ACP maintains a concise, coherent, and relevant context while achieving compara- performance to full- coltect methods.
Inspired by human cognitive strategies, pro- pose adaptive contextual pruming (ACP), novel method that introduces dynamic context selection and pruming in LLM-bbsed text generation. Instead of treating all contexts equally, ACP continuously evaluates the relevance of each piece of context based on its contribution to the generated content. Our approach involves the following key mecha- nisms:
1 Introduction
Large language models (LLMs) have achieved re- markable performance in text generation lasks, enabling applications such as book summariza- tion (Chang et al., 2024), technical documentation generation (Dvivedi et al., 2024), and long-form generation in general (Wu et al., 2025). However, maintaining relevance and conciseness long- form text generation remains significant chal- lenge (Krishna, 2023). LLMs often include irrel- evant, redundant, or tangential information, lead- ing to verbose outputs and, in some cases, factual inconsistencies (Wei et al., 2024). These issues are particularly pronounced in scenarios requiring extended coherence and dynamic context manage- ment, such as summarizing lengthy documents or
Relevance Scoring:
Each context piece (sen- tence, paragraph, or document section) is as- signed a dynamic relevance score based on keyword importance, usage frequency, and contribution to the ongoing generation.
Adaptive Pruning: At periodic intervals, the model evaluates and removes less relevant

---

context to maintain a manageable, focused working memory.
with the full inpul text, and each sentence or para- graph is assigned an initial relevance score based its position, keyword significance, and histori- cal usage. The model generates text in fixed-length chunks (e.g.. 100 tokens). After each chunk, the relevance scores of context segments are updated based on their contribution the generated content. Segments with scores below defined threshold are pruned to maintain concise working memory.
If a previously pruned segment becomes relevant again, it is reintegrated into the context, guided by key- word cues and thematic alignment.
Retrieval of Pruned Context:
Unlike conventional -arieeeed genera- tion (RAG) (Gao al., 2024) approaches that rely static document embeddings, ACP dynamically adjusts the importance of context throughout the text generation process. This continuous adaptation enables ACP to mimic human-like writing strate- gies, ensuring that the generated content remains concise, coherent, and contextually relevant.
If the model encounters terms or themes that strongly correlate with previously pruned content, retrieval mechanism determines whether to rein- troduce the relevant segments. This is achieved through Aimilarity-based approach, where embed- dings of the current generation are compared with stored embeddings of pruned content. If the sim- ilarity exceeds a predefined threshold, the pruned context restored, preventing the model from los- ing critical information.
To evaluate the effectiveness of ACP, we conduct experiments on long-form summarization task us- ing the GovtReport (Huang et al., 2021) dataset. We benchmark ACP against fixed-context, sliding- window, and full-confext methods. The evaluation of the summaries is conducted using various auto- matic metrics, along with L.L.Mbaseed evaluation.
ACP's ability to adaptively prune and retrieve context improves long-form generation by reduc- ing redundancy while preserving necessary context for coherence. By dynamically maintaining an op- timal context window, ACP enhances the relevance, conciseness, and factual consistency of generated text.
Through our experimentation and analysis, we demonstrate that ACP surpasses standard methods such as fixed and sliding windows and performs on par with full-context long-form text generation, while maintaining dynamic, adaptive context that prioritizes relevance and avoids unnecessary verbosity.
3 Experimental Setup
In this section, we describe the experimental setup to evaluate the effectiveness of Adaptive Contex- tual Pruning (ACP) in enhancing long-form text generation. Our experiments focus on ACP's s abil- ity to maintain relevance, conciseness, and fac- tual consistency while generating coherent sum- maries of large documents using the GOVRE- PORT (Huang el al., 2021) dataset. We compare ACP's performance against several baseline models and evaluate the generated text with both automatic and human-aligma metries, providing insights into ACP's impact content quality and dynamic con- text management.
2 Proposed Method: ACP
Adaptive Contextual Pruning (ACP) is designed to dynamically manage context during long-form text generation, ensuring that the model maintains relevance while minimizing redundancy. Unlike fixed-length context windows sliding window mechanisms, ACP continuously evaluates and ad- justs the context by pruning irrelevant segments and retrieving previously pruned content when needed,
ACP operates by maintaining dynamic rele- vance score for each segment of text. At predefined intervals, the model assesses the contribution of each context segment to the generated content and updates its relevance score. Segments with low rel- evance scores pruned, ensuring that the model focuses on the most pertinent information. How- ever, these pruned segments are not permanently discarded; instead, they are stored and can be re- trieved they become relevant again later in the generation process.
3.1 Dataset
In our experiment, we the GOVRE- PORT (Huang et al., 2021) datusei, which consists of extremely long reports and their corresponding summaries. These reports often exceed 30,000 words and span a broad range of topics, including policy, research, and statistics, making it valuable resource for training models focused on long-form summarization. The datasct is carefully annotated
To implement ACP, the process follows several key steps. First, the context window is initialized

---

to aid in generating concise summaries, ensuring the retention of key information such as conclu- sions and recommendations. Its design is intended to enhance the ability of models to process large contexts and produce accurate, relevant summaries, making well-suited for advancing automatic sum- marization systems. More details about the dataset are reported in Table 1.
risk losing details but may carry irrelevant con- tent across windows, affecting the relevance and coherence of the generated text.
Full Window. The Full Window baseline processes the entire input context without truncation, ensur- ing all information is considered. However, it can be computationally inefficient and may reduce out- put quality if irrelevant or redundant content is included. Additionally, not all models can support such long contexts.
Data
Mean
Median
Min
Max
Source
7379.13
6488
396
31371
3.4
Evaluation Metries
Target
570.85
562
67
1363
To comprehensively evaluate Adaptive Contex- tual Pruning (ACP), we employ combination of automatic and I.LM-based evaluation metrics. For automatic evaluation, we use ROUGE (Lin, 2004)ROUGE-1, ROUGE-2, ROUGE-L, ROUGE- Sum) to measure n-gram overlap with reference text. BERTScore (Zhang et al., 2020) leverages contextual embeddings to assess semantic simi- larity, offering a more fine-grained comparison. Additionally, compute the compression
The Govreport summarization word count statisties indicate that the maximum word count can exceed 30,000 words, often surpassing the context limit of many open-weight models.
Table 1:
3.2 Models
We use OpenAl's GPT-4o-mini for main periment due to its ability to generate coherent, contextually relevant text. Its balanced size and generation quality make it cost-effective, enabling assess how ACP interacts with a large-scale model to maintain relevance, conciseness, and fac- tual consistency in long-form generation.
tio (Grusky et al., 2018) to quantify text con- ciseness and factual consistency (Reimers and Gurevych, 2019)' to measure alignment with the source content.
LLM--ased evaluation (Liu et al., 2023) scores the generated summary 1-5 scale across key pects: coherence, fluency. relevance, consistency, and conciseness. This human-aligned assessment helps capture nuances that automatic metrics may overlook, providing deeper insights into the overall quality of ACP-generater text. The prompt tem- plate is given in Appendix A.1 Table 5.
Additionally, we experiment with open- weight models, including 1 (8B, 3B, IB) (Grattafiori al., 2024) and InternLM-7B (Cai widely
et al., 2024). While these models used, they present challenges in the ACP pipeline, particularly with generating summaries that meet the structured output requirements relevance scoring and pruning. This often leads to subop- timal performance infinite generation loops, limiting our experimentation to GPT-4o-mini.
3.5 Prompts
For baseline summary generation, use simple prompt to generate the summary. The same prompt is applied to each slide in the sliding-window method, followed by a final merging prompt to combine all the summaries. Additionally, we de- sign dedicated prompts for relevance scoring, prun- ing decisions, and retrieval decisions. All prompts are zero-shot, meaning no in-context examples are provided. A list of the prompts used in the main experiments is provided in Appendix A. Table 4. mentioned earlier, inspired by (Liu al., 2023), we employ LLM-bascd evalu- ation to assess the quality of the generated summary
3.3 Baselines
window
Fixed Window. The Fixed Window baseline in- volves truncating the text to a fixed size and gen- erating the summary from that segment. Unlike the sliding window, it does processes only the most recent portion, discarding the rest. This method allows for efficient computa- tion but may lose important information, affecting coherence and relevance.
shift context but
Furthermore,
Sliding Window. The Sliding Window baseline processes the text in overlapping chunks, retaining some context from previous windows while intro- ducing new information. This method reduces the
'tttps: //tpgg/gggaagccee.////////// co/sentence- transforners/ all-MiniLM-LG-V-

---

Rouge
BERTScore
Method
CR
FC
R
R-1
R-2
R-L
R-Sum
P
Read.
F1
Fixed
27.03
8.90
14.53
16.44
86.59
83.05
84.78
44.33
86.67
23.19
Sliding
26.20
8.34
13.87
15.95
86.44
83.11
84.74
44.05
84.
23.58
Full
27.43
9.38
14.64
16.92
86.89
83.30
85.06
43.59
85.35
23.39
ACP
27.24
9.31
14.69
16.88
86.76
83.26
84.97
44.52
85.33
23.63
Table 2: Automatic metrics results. Abbreviations R-1 1 Rouge- 1, P Precision, R-Recall, CR Compression Ratio, FC Factual Consistency, Read. Readability.
LLM-Eval (1-5)
close to the full-context results.
Method
Cohe.
Flue.
Rele.
Cons.
Conc.
In addition to automatic metrics, we also ducted LLM-based evaluations, assessing models on coherence, fluency, relevance, consistency, and conciseness, as shown in Table 3. All methods, in- cluding ACP, achieved similar scores in coherence and fluency, with ACP scoring 4.00 in coherence and 4.60 in fluency. ACP outperforms both the Sliding Window and Full Window models in rel- evance (4.81 for ACP vs. 4.75 Full Window and 4.68 for Sliding Window), while maintaining competitive scores in consistency and conciseness. Overall. ACP demonstrates strong performance across both automatic and numan-aligned evalua- tion metrics, showing improvements in relevance and conciseness, while performing comparably other methods in terms of factual consistency and rcadability. We provide qualitative example sum- maries generated with different methods in Ap- pendix A.2, Table
Fixed
4.00
4.61
4.76
4.38
3.99
Sliding
4.00
4.62
4.69
4.22
3.97
Full
4.00
4.61
4.75
4.32
4.00
ACP
4.00
4.60
4.81
4.37
3.99
Table 3: I.IM evaluation results. Abbreviations: Coher. Coherence, Flue. Fluency, Rele. Relevance, Cons. Consistency, Conc. Conciseness.
acTOSS various dimensions, such as coherence, rel- evance, consistency, and conciseness. We adapt the prompt from the OpenAl Cookbook 2 for our specific case. The final evaluation prompt is provided in Appendix A.I, S.
4 Results and Ablation
To evaluate the effectiveness Adaptive Contex- tual Pruming (ACP), we conducted a series of exper- iments comparing ACP with three baseline models: Fixed Window, Sliding Window, and Full Window, We used both automatic and human-aligee aligned evalu- ation metrics to assess the relevance, conciseness, factual accuracy, and overall quality of the gener- ated text.
Conclusion
In this paper, we introduced Adaptive Contextual Pruning (ACP), novel approach for enhancing long-form text generation by dynamically manag- ing context. ACP evaluates the relevance of con- lext segments and prunes less relevant information, ensuring concise, coherent, and contextually ap- propriate text. Our experiments with the GOVRE- PORT dataset showed that ACP performs compara- bly to or better than existing methods (Fixed Win- dow, Sliding Window, and Full Window) in terms of relevance and conciseness, while maintaining competitive performance in factual consistency and readability. These results highlight ACP's poten- tial to improve the efficiency and quality of LLM- based content generation. Future work will refine retrieval heuristics and explore hybrid models inte- grating ACP with strieval-augmentt techniques.
Table presents the results from the automatic evaluation, where compare the models across several key metrics: ROUGE-1 (R-1) ), ROUGE-2 (R-2), ROUGE-L (R-L), ROUGE-Sum (R-Sum), BERTScore Precision (P), Recall (R). and scores. ACP performs comparably the Full Win- dow method, achieving a similar ROUGE-1 score of 27.24, which is slightly higher than Full Win- dow's 's score of 27, 27.03. In terms of BERTScore, ACP shows competitive performance with a precision score of 86.76 and Fl score of 84.97, which are
" Phttps: //gi thub. blob/main wwwpaaapppeeeeeaaaaaiaeeeeeeaeee abstractive www..vwwwrsaaaaiooooooo ipyn
cookkbokk

---

References
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long Pa- pers), pages 708-719. New Orleans, Louisiana. As- sociation for Computational Linguisties.
Jinze Bai, Shuai Bai. Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang. Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin. Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu. Peng Wang, Shijie Wang. Wei Wang. Shengguang Wu. Benfeng Xu, Jin Xu, An Yang. Hao Yang. Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang. Zhenru Zhang, Chang Zhou, Jin- gren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023, Qwen technical report. Preprint, arXiv:2309. 16609.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436, Online. Association for Computational Linguistics.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong. Haodong Duan, Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He. Yingfan Hu. Ting Huang. Tao Jiang. Penglong Jiao, Zhenjiang Jin, Zhikui Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yin- ing Li. Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv. Haijun Lv, Kai Lv. Li Ma, Runyuan Ma. Zerun Ma. Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zi- fan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang. Guoteng Wang, Jiagi Wang, Ji- ayu Wang, Rui Wang, Yudong Wang. Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wa, Yingtong Xiong. Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Ximogui Yung, Hauchen Ye, Huaiyuan Ying. Jiu Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang. Songyang Zhang. Wenjian Zhang. Wenwei Zhang. Xingcheng Zhang. Xinyue Zhang. Hui Zhan, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin, 2024. Internlm2 technical report. Preprint, xXiv:2403.17297
Albert Q. Jiang. Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.066
Kulpesh Krishna. 2023. Towards Robust Long-form Text Generation Systems. Ph.D. thesis, University of Massachusetts Amherst.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nig evaluation using gpt-4 with better human align- ment. Preprint, arXiv:2303.166:3
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert. Sentence embeddings using siamese bert nettonetworks. Preprint, arXiv: 1tt::1190 11..119000..... 10084.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Fuisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. Preprint, arXiv:2302. 130.171...971.
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit lyyer. 2024. Booookscore: A systematic exploration of book-length summarization in the of Ilms. Preprint, 2rX.v:2310.00785.
Shubhang Shekhar Dvivedi, Vyshnav Vijay, Sai Leela Rahul Pujari, Shoumik Lodh, and Dhruv Ku- mar. 2024. A comparative analysis of large language models for code documentation generation. Preprint, arXiv:2312. 10344.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhusale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal. An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anee Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovihh, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly-
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang. and Haofen Wang. 2024. Retrieval-augmented gener- ation for large language models: A survey. Preprint, LrXiv:2312.1099
Aaron Grattafiori, Abhimanyu Dubey, and LLamm team. 2024. The llama 3 herd of models. Preprint, ar MxXv:2407.21788

---

bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungla, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor. Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Diyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.00 009288.
A
Appendix
A.1
Prompts Examples
A.2
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng. Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. 2024. Long-foree factuality in large language models. Preprint, arXiv:2033. 18802.
Yuhao Wu, Ming Shan Hee, Zhiqing Hu. and Roy Ka- Ka-Wei Lee. 2025. Longgenbench: Benchmarking long-form generation long context llms. Preprint, 02Xiv:2409.02076.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Preprint, stXiv:1904.0966

---

Name
Prompt
Summary
Summarize following article about (summary length) words. Input: [text)
Summary:
Pruning
[context] Identify the least relevant context pieces that can be removed reduce the context to (token limit) tokens. Return the least relevant context pieces JSON object in the following format:
Context:
Output (Strictly in JSON format):
{{
"pruned_contee _contxttt
"context piece 1",
"context piece_21 2",
]
Relevance
Evaluate how relevant each provided context piece current writing task. Assign relevance score from 1 to 10, where indicates the lowest relevance and 10 indicates the highest relevance. Context Pieces: [context)
Return your evaluation in JSON format with key 'relevance_scome and values as list of relevance scores for each context piece. Return scores only for the provided context pieces. Do not include any additional information like explanation in the output. Relevance scores only as list.
Output (Strictly JSON format): {{ "relevance_scores"
[]]
Retrieval
Given the keyword(s) [Y] from previously pruned context, decide if it's relevant to retrieve this context for the current writing task.
Pruned Context: (context_piece]
Current Generation: {generation]
Return the decision in JSON. If the context relevant, return "True": otherwise, return "False". Do not include any additional information in the output.
Output (Strictly JSON format):
{{
"retrieve": (True/False
]
Sliding Win-
Based on the following aggregated summaries from individual chunks, generate a coherent and concise final summary. The final summary should capture the key points without redundancy and be approximately (summary_lengee words long.
dow
Aggregated Summaries: (summaries)
Final Summary:
Table 4: Prompt templates for various tasks.

---

Evaluation Prompt
You will given a long text document and a summary written the same. Your task is to evaluate the summary based on the following five metrics: Coherence, Fluency, Relevance, Consistency, and Conciseness. Use the provided rating scale and evaluation criteria to assign scores for each metric.
Rating Scale (1-5):
(Poor): The summary has many issues that significantly affect quality or performance on the metric.
2 (Fair): Noticeable issues that reduce quality clarity.
3 (Good): Minor issues but generally acceptable performance.
(Very Good): Very few issues, strong performance.
(Excellent); Flawless with no issues.
Evaluation Criteria:
Coherence (1-5): Assess the logical structure and flow of the summary. It should be well-structured, organized, and flow logically from sentence to sentence.
Fluency (1-5): Evaluate grammar, spelling, punctuation, word choice, and sentence structure. The summary should be smooth, easy read, and free of errors.
Relevance (1-5): Check if the summary captures the critical points from the source document while avoiding irrelevant or redundant details.
Consistency (1-5): Verify that all factual claims the summary are accurate and supported by the source document. Penalize hallucinated unsupported claims.
Conciseness (1-5): Ensure the summary the point, avoiding unnecessary repetition elaboration while effectively conveying the main points.
Evaluation Steps:
Read the source document and identify its main topic, key points, and facts.
2. Compare the summary the source document based on:
Cohherenc:: Is the structure logical and easy to follow?
Fluency: Does the summary use proper grammar and readability?
Relevance: Does it focus on key points without irrelevant content?
Consistency: Are all claims factually accurate and supported?
Conciseness: Is the summary free of repetition and unnecessary claboration?
3. Assign scores for each metric using the provided rating scale.
Instruction: Return the output strictly in ISON format with the metric names as keys and the scores as values. Do not include any additional information in the output.
Input Format:
Source Text: [text)
Summary: (summary)
Output Format (JSON):
(
"coherence"; <score>,
"fluency": <score>,
"relevance"; <score>,
"consistency": <score>,
"conciseness": <score>
Table Prompt for LLM-bbsed summary evaluation.

---

Method Summary
The Agriculture appropriations bill, known the Agriculture, Rural Development, Food and Drug Administration, and Related Agencies Appropriations Act, funds the U.S. Department of Agriculture (USDA) for the fiscal year 2020 (FY2020), excluding the U.S. Forest Service. The legislative process began with the House Appropriations Commitlee reporting H.R. 3164 on June 6, 2019, and the House passing five-bill mimibus appropriations bill (H.R. 3055) on June 25, 2019. The Senate followed by reporting S. 2522 September 19, 2019. However, as the Senate did not act on bill by the start of FY2020, Congress enacted two continuing resolutions to maintain funding FY2019 levels. Ultimately, on December 20, 2019, the FY2020 Further Consolidated Appropriations Act (P... 116-94) was The Agriculture Appropriations Bill, officially known as the Agriculture, Rural Development, Food and Drug Administration, and Related Agencies Appropriations Act, allocates funding for the U.S. Department Agriculture (USDA), excluding the U.S. Forest Service. For fiscal year 2020 (FY2020), the House Appropriations Committce reported H.R. 3164 on June 6, 2019, and this Sliding funding was incorporated into five-bill minibus appropriations bill (H.R. 3055). which passed the House June 25, 2019. However, due inaction in the Senate regarding its version (S. 2522) before FY2020 began, the federal govemment operated under two continuing resolutions to maintain funding levels from the previous fiscal year. Ultimately, December 20, 2019, Congress passed the FY2020 Further Consolidated Appropriations Act, which included agriculture-rchhe related funding under Division B.report
Fixed
The Agriculture appropriations bill, formally known the Agriculture, Rural Development, Food and Drug Administration, and Related Agencies Appropriations Act, funds various programs under the U.S. Department of Agriculture (USDA), excluding the U.S. Forest Service. For Fiscal Year 2020 (FY2020), the House Appropriations Committee reported H.R. 3164 in June 2019, which was later included in five-bill minibus appropriations bill that passed the House. However, the Senate did not act on its version of the bill before the fiscal year began, leading to two continuing resolutions that temporarily funded federal agencies at FY2019 levels. Ultimately, Congress passed the FY2020 Further Consolidated Appropriations Act, which included agriculture funding, Condding.nnnniiie The FY 2020 appropriations focused significantly on agricultural conservation programs, which assist landowners with natural resource management. Key agencies involved are the Natural Resources Conservation Service (NRCS) and the Farm Service Agency (FSA).
Full
The Agriculure appropriations bill for Fiscal Year 2020, formally known the Agriculture, Kural Development, Food and Drug Administration, and Related Agencies Appropriations Act, was designed to fund Service. The process began with the House Appropriations Committee reporting H.R. 3164 on June 6, 2019, which later hecame part of five-bill minibus appropriations package that passed the House June 25, 2019. The Senate's 's version, S. 2522, was reported September 19, 2019, but the full Senate failed to act before the fiscal year began, resulting in the need for two continuing resolutions maintain funding FY2019 levels. Ultimately, the FY2020 Further Consolidated Appropriations Act (P... 116-94) was signed into law on December 20, 2019, including agricultural
Department of Agriculture (USDA) while excluding the U.S. Forest
ACP
Table 6: Example summaries generated using baselines and ACP for same report.

---

Published as a conference paper at ICLR 2026
Reviewer Scores:
From Ideation Study:
Novelty: 6.0
Excitement: 6.0
Feasibility: 5.5
Expected Effectiveness: 5.0
Overall: 6.0
From Execution Study:
Novelty: 6.3
Excitement: 5.5
Soundness: 6.5
Effectiveness: 4.8
Overall: 5.0
46


Reviews:
[{"review_id": "GazmO7kPhT", "reviewer": "Reviewer_P5TK", "summary": "This paper investigates the \"ideation-execution gap\" by comparing research ideas generated by an LLM (Claude-3.5-Sonnet) with those from human experts through a rigorous Randomized Controlled Trial (RCT). 43 expert researchers executed the assigned NLP ideas (~100 hours each) and produced short papers, which were then blindly reviewed by 58 expert reviewers. The core finding is that despite LLM ideas scoring significantly higher in novelty and excitement at the ideation stage, their scores dropped significantly more than human ideas after execution (p<0.05) across all metrics (novelty, excitement, effectiveness, and overall). This reversal of rankings suggests that current LLMs struggle to generate ideas that are truly effective when subjected to the rigors of practical research execution.", "questions": "The analysis points out that execution reviewers focus more on empirical performance and rigor (e.g., missing baselines, insufficient analysis). Can the authors provide a fine-grained analysis of which specific types of weaknesses (e.g., technical flaws, inadequate experiment design, poor empirical results) were disproportionately mentioned in the low-scoring AI idea reviews compared to the low-scoring human idea reviews? This would help precisely characterize the nature of the LLM's ideation failure.\nThe disparity in mean execution time (112.6 hrs for Human vs. 93.7 hrs for AI) is a potential confounder. Did the projects with lower execution time generally receive lower reviewer scores, and was the difference in execution time due to AI ideas being inherently simpler, or were they less fully explored by the executors?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The use of a large-scale Randomized Controlled Trial (RCT) with blinding for both execution participants and reviewers is highly commendable and provides a strong foundation for the causal conclusions.\n2. The study mandated significant effort, with 43 expert participants spending an average of over 100 hours each to execute the ideas, lending credibility to the post-execution evaluation outcomes.\n3. The direct comparison of pre- and post-execution scores explicitly quantifies the difference in \"drop\" between AI-generated and human-generated ideas, providing a novel and important metric for evaluating AI ideation capabilities.", "weaknesses": "1. While the drop in scores is statistically significant, the direct comparison of the final average execution scores between human and AI ideas was not statistically significant when treating each idea as an independent data point ($N=43$)8. This tempers the claim of human ideas being superior post-execution.\n2. Execution participants for human ideas spent significantly more time (mean 112.6 hr) compared to AI ideas (mean 93.7 hr). Although potentially explained by sample size, this difference could be a confounding variable related to the complexity or inherent feasibility of the assigned ideas.\n3. The study used a single, albeit state-of-the-art at the time, LLM (Claude-3.5-Sonnet) and generation method. It is unclear how different LLMs or more advanced agentic idea generation frameworks (e.g., iterative feedback) would impact the \"execution gap\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H39BrprYaQ", "reviewer": "Reviewer_4Uap", "summary": "This paper studies whether research ideas generated by large language models (LLMs) can lead to useful research outcomes once they are actually executed. Previous work has shown that LLMs are often able to propose ideas that look more novel or interesting than those written by human researchers. However, these earlier evaluations mostly stopped at the idea stage and did not test whether such ideas can be successfully turned into working research projects.\nTo address this gap, the authors conduct a controlled experiment involving 43 experienced NLP researchers. Each participant was randomly assigned either a human-written idea or an idea generated by the Claude 3.5-Sonnet model. They then spent around 100 hours implementing the idea and wrote a short 4-page report describing the experiments and results. All these completed projects were blindly reviewed by independent experts using consistent criteria, including novelty, excitement, effectiveness, and overall quality.\nThe results show a clear pattern: although LLM-generated ideas initially received higher scores at the ideation stage, their performance dropped significantly after execution. Human ideas, on the other hand, maintained roughly the same level of quality before and after execution. The authors refer to this difference as the ideation–execution gap. Further analysis indicates that the gap is not caused by poor implementation—since participants made only small adjustments to datasets or metrics—but rather by the lower practical feasibility and methodological soundness of the AI-generated ideas.", "questions": "1.\tEach executed project was limited to around 100 hours of work and a 4-page report. Could this relatively short duration constrain the depth and maturity of the resulting research? For instance, some AI-generated ideas might be more ambitious or technically demanding, and therefore require substantially more time or resources to reach comparable quality.\n2.\tDid the authors consider normalizing for idea complexity or estimated implementation difficulty when comparing execution outcomes? Without such adjustment, simpler human ideas may appear more successful simply because they are easier to realize within the given time and effort constraints.\n3.\tWould it be possible to introduce AI-assisted execution in future experiments, so that the implementation process is more aligned with the characteristics of AI-generated ideas?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper tackles an important and timely issue: whether LLM-generated research ideas can actually lead to successful scientific outcomes once executed. This question extends prior work that only evaluated idea quality (Si et al., ICLR 2025), moving a step forward to test execution outcomes directly.\n2.\tThe study involves a large number of highly qualified human experts, including experienced researchers with strong publication records. It also quantifies key aspects such as working hours (around 100 hours per project) and the number of modifications made to each idea, showing that participants made only minor adjustments without altering core methods. These details make the findings convincing.\n3.\tThe paper combines quantitative comparisons of review scores with qualitative analyses of reviewer comments and modification patterns. This mixed approach allows for a balanced understanding of why AI-generated ideas underperform at the execution stage.", "weaknesses": "1.\tWhile the research question is important, the paper mainly contributes through experimental design rather than technical or algorithmic advances. It does not propose new modeling approaches or analytical frameworks beyond the comparative study setup.\n2.\tThe paper identifies the existence of the ideation–execution gap but provides limited theoretical or computational explanation for why LLM-generated ideas fail in execution. A more systematic analysis of idea structure or model behavior could have strengthened the conclusions and would be more helpful for guiding models to generate better and more feasible research ideas in the future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gGHBe5tIZt", "reviewer": "Reviewer_qsa4", "summary": "This paper presents the first large-scale execution study comparing LLM-generated and human-generated NLP research ideas. 43 expert researchers executed ideas (24 by AI, 19 by human) drawn from a prior ideation study. 58 expert reviewers then conducted double-blind reviews (181 in total) rating novelty, excitement, soundness, effectiveness, and overall quality.\n\nResults reveal a clear ideation–execution gap: although LLM ideas scored higher during ideation, their scores dropped substantially after execution (≈1–2 points on a 1–10 scale), while human ideas remained stable. Post-execution, human ideas slightly outperformed AI ideas, reversing the earlier ranking. Analyses show that LLM ideas often suffered from unrealistic scope and weaker empirical design, whereas reviewers of executed work emphasized feasibility and soundness over speculative novelty.", "questions": "- How is training-data cutoff handled—any risk of contamination?\n\n- Would results differ with newer models (e.g., GPT-4o)?\n\n- Could a predicted “feasibility” score reduce the gap?\n\n- According to Table 3, AI idea executors are more experienced than those for Human. Would this affect the results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work addresses whether LLMs can generate effective research ideas, not just novel ones. The study uses randomized assignment, double blinding, standardized instructions, and FDR-corrected statistical testing.\n\n- The work combines quantitative results with qualitative review comment analysis, explaining why AI ideas underperform in execution.\n\n- It establishes an “execution-aware” evaluation paradigm and commits to open-sourcing all data, code, and reviews.", "weaknesses": "- 43 projects across seven NLP topics limit generalization; authors acknowledge this.\n\n- No contamination or recency discussion: Claude 3.5-Sonnet was used, but its training cutoff and overlap with prior research are not analyzed.\n\n- Reviewer consistency is reported only in aggregate; topic-level calibration is missing.\n\n- Overall technical contribution is not significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)