RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback
Authors: Chunyu Miao
Henry Peng Zou
Yangning Li
Yankai Chen
Yibo Wang
Fangxin Wang
Yifan Li
Wooseong Yang
Bowei He
Xinni Zhang
Dianzhi Yu
Hanchen Yang
Hoang H Nguyen
Yue Zhou
Jie Yang
Jizhou Guo
Wenzhe Fan
Chin-Yuan Yeh
Panpan Meng
Liancheng Fang
Jinhu Qi
Wei-Chieh Huang
Zhengyao Gu
Yuwei Han
Langzhou He
Yuyao Yang
Xue Liu
Irwin King
Philip S. Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Code generation
Primary area: datasets and benchmarks
TL;DR: A multi-turn interactive research code generation benchmark.

Abstract: Large language models (LLMs) show the promise in supporting scientific research implementation, yet their ability to generate correct and executable code remains limited. Existing works largely adopt one-shot settings, ignoring the iterative and feedback-driven nature of realistic workflows of scientific research development. To address this gap, we present RECODE-H, a benchmark of 102 tasks from research papers and repositories that evaluates LLMs through multi-turn interactions with human feedback. It includes structured instructions, unit tests, and a five-level feedback hierarchy to reflect realistic researcher–agent collaboration. We further present ReCodeAgent, a framework that integrates feedback into iterative code generation. Experimentswith leading LLMs, including GPT-5, Claude-Sonnet-4, DeepSeek-V3.1, and Gemini 2.5, show substantial performance gains with richer feedback, while also highlighting ongoing challenges in the generation of complex research code. RECODE-H establishes a foundation for developing adaptive, feedback-driven LLM agents in scientific research implementation.

Published as a conference paper at ICLR 2026
RECODE-H: A BENCHMARK FOR RESEARCH CODE DEVELOPMENT WITH INTERACTIVE HUMAN FEED- BACK
Chunyu Miao', Henry Peng Zou Yangning Li Yankai C.en1 Yibo Wangh Fangxin Wang', Yifan Li,, Wooseong Yang', Bowei He4.6 Xinni Zhang?, Dianzhi Yu Hanchen Yang', Hoang H Nguyen', Yue Zhou', Jie Yang', Jizhou Guo', Wenzhe Fan Chin-Yuan Yeh Panpan Meng", Liancheng Fang', Jinhu QF, Wei-Chieh Huang', Zhengyao Gu', Yuwei Han', Langzhou He", Yuyao Yang', Yinghui Li², Hai-Tao Zheng", Xue Liu Irwin King?, Philip S. Yu-
l University of Illinois Chicago TTinghua University 'MeGill University *MBZUAI
Thhe Chinese University Hong Kong "City University of Hong Kong "Shanghai Jiao Tong University NNaional Taiwan University 9Xi an Jiaotong University
(cmiao8, pzou3, ychen588, (wang633J@uic.ede
ABSTRACT
Large language models (LLMs) show promise for supporting scientific research implementation, yet their ability to generate correct, executable code remains limited. Existing works largely adopt one-shot settings, ignoring the iterative and feedback-driven nature of realistic scientific research workflows. To address this gap, present RECODE-H, benchmark of 102 tasks from research pa- pers and repositories that evaluates LLMs through multi-turn interactions with human feedback. includes structured instructions, unit tests, and a five-level feedback hierarchy to reflect realistic researcher- collaboration. We further present ReCodeAgent, framework that integrates feedback into an iterative code- generation process. Enperimmentsxxprrieeeen GPT-5, "laude-Sonnets DeepSeek-VSs V3.1, and Gemini 2.5, show substantial per- formance gains with richer feedback, while also highlighting ongoing challenges in generating complex research code. RECODE-H establishes a foundation for developing adaptive, (eedback-drivens LLM agents in scientifie research imple- mentation.
with leading I.MMs, including
INTRODUCTION
Large language models (LLMs) have been increasingly adopted across the scientific research pipeline, assisting tasks from ideation to writing (Zhang et al. 2025b Si et al. 2024 Chen et al.) 20256, Li et al., 2025b; Wu al. 2025b Zou al.. 2025b] Zhang ating correct and executable research code remains a difficult problem, not only because it requires long-range reasoning and robust verification (Padigela et al. 2025 Starace et 2025 Zhu al.) 2025c), but also because the input contexts in research settings are often complex, indirect, and noisy. Research papers describe methods through high-level narratives, mathematical formulas, and domain-specific conventions, while leaving many implementation details implicit. As a result, trans- lating these fragmented and underspecified descriptions into functional code remains a fundamental challenge for current LLMs (Li et al. 2025c
2025a) However, gener-
Existing benchmarks for research code generation (Zheng et al. 2023 Sun al. 2023 Toledo al.) 2025 Hua et al. 2025) primarily evaluate models in non-interactive setting, where they pected to produce correct code in a single response. This design neglects the crucial role human feedback in realistic workflows: on the one hand, users often cannot fully specify their requirements
We will open-source benchmark, and code
O GitHub

---

Published as a conference paper at ICLR 2026
Domain
Benchmark
Repo
Unittest
Feedback
Task
BigCooeBench
x
x Verbal Feedback
Function Code Generation
General Code
ConyCode World
x
Function Code Generation
Generation
X
Lazy User Feedhack
Function Code Generation
InterCodee
V
Execution Feedbaack
Function Code Generation
MI. A..E-hench
X
X
Machine Learning Engineering
MI AgentBench
Machine Learning Engineering
Research Code
PaperBench
Reproduce ICML Papers
Generation
SciReplicate- Rench
Code generation for reseanch
RECODE-H
Hierarchical Researcher Feedback
Code generation for research
Table 1: Overview of benchmarks and their characteristics. Categories are abbreviated for compact- ness: Our benchmark introduces structured feedback hierarchy (Section (32) to support systematic evaluation of interactive research code generation under progressively richer forms of guidance.
in one shot. On the other hand. LLMs rarely generate perfectly aligned code on the first attempt (Zou al.) 20256). In practice, effective implementation relies on multi-turn interactions, where models must accurately interpret and leverage iterative user feedback (Li et al. 2024d). However, current benchmarks focus solely on end-to-end correctness, without evaluating models' capabilities in- teractive refinement.
To fill this gap, we introduce RECODE-H (Research COde DEvelopment,, benchmark designed to evaluate how LLMs generate and refine research code through interactive feedback with human researchers. The benchmark consists of 102 tasks drawn from real research papers across machine learning, natural language processing, computer vision, and computational science, each paired with its original codebase. Unlike simple function-completion tasks, these tasks focus on repository-leve code generation, requiring models implement classes, functions, modules that correspond to methodological descriptions in real research codebases. Building RECODE-H is challenging (Cao al., 2025). as it requires aligning paper descriptions with code implementations, selecting repre- sentative tasks, and maintaining expert-level quality at scale. We address these challenges through a hybrid LLMMassisted and numan-curated pipeline. RECODE-H has three key features:
PhD-level difficulty and expert annotation. All tasks are drawn from real research projects and manually curated to ensure clarity, realism, and high annotation quality.
Feedback-level controlled difficulty, Task difficulty is systematically controlled through the structured feedback hierarchy introduced in Section 3.2 enabling fine-grained evaluation of mod- els' ability to leverage multi-turn feedback guidance.
Research method focus. Tasks center on the faithful implementation of research methods, typically requiring the development of several functions up to entire classes, rather than isolated function completion.
In our experiments, we evaluate seven leading LLMs, including GPT-5, DeepSeck- V3.1, Claude- Sonnet-4, and the Gemini family, RECODE-H. We also introduce ReCode Agent, a stronger baseline that leverages human feedback through structured multi-turn interactions. ReCode Agent progressively integrates diagnostic signals, refinement instructions, and correction feedback guide model revisions, providing a more faithful simulation of real research workflows. Experimental results show that all models substantially benefit from interactive feedback, with even minimal di- agnostic signals nearly doubling pass rates and recall compared to the no-feedback setting. For example, GPT-5's recall improves from 29.4% without feedback 71.6% with the most detailed feedback, while DeepSeek-V3. V3.1 shows a similar jump from 10.8% to 70.6%. Larger models, such as GPT-5 and DeepSeek-V3. demonstrate stronger adaptation to progressively richer feedback, while Claude-Sonnete and Gemini lag behind. Increasing feedback granularity not only improves success rates but also accelerates convergence, enabling stronger models to solve tasks in fewer turns. Error analysis further reveals that failures dominated by misinterpretation of paper in- structions and gaps in domain knowledge, rather than syntax or integration issues. These findings highlight the effectiveness of ReCode Agent and structured feedback, establishing a solid baseline for future work on research code generation.
Overall, our work makes the following contributions:
2

---

Published as a conference paper at ICLR 2026
We introduce RECODE-H, the first benchmark for multi-turn interactive research code gen- eration, providing a high-quality dataset that evaluates how LLMs generate and refine code under iterative human feedback.
We propose ReCode Agent, a strong baseline that effectively leverages structured human feedback through iterative interaction to implement and refine research code in realistic repository settings.
Through extensive experiments on seven leading LLMs, we reveal key limitations of cur- rent models and demonstrate the effectiveness of our method, offering insights for building future feedback-driven research agents.
2 RELATED WORK
LLMs for Research Development. L.LM-based agents have been applied across the scientific work-
flow, from survey writing and idea generation to paper review (Wang et al. 2024h Si et al., [0055
Weng et 2024 He et al. 2026; Zhang et al. 2025a). Early benchmarks focused on end-to-end
workflows (Toledo et al. 2025 Kon et al 2025
Chan et al. 2024] Chen et al. 2024c Starace et al.,
2025 Edwards
2025). but implementation remains major bottleneck: current models struggle to translate textual descriptions into executable code. Recent datasets such as SciReplicate (Xiang al. 2025), ResearchCode (Hua et al., (2025). and
(Yan al. 2025) shift toward code-level evaluation. Our work builds on this direction by introducing qualitatively new evalua- tion paradigm by incorporating multi-turn, feedback-driven interactions. RECODE-H the first benchmark to systematically assess agent's 's ability to iteratively revise and refine research code through a structured hierarchy of feedback, reflecting the iterative cycles of diagnosis, correction, and alignment that characterize real scientific practice.
Interactive Code Generation. In software engineering (SWE), benchmarks span a wide spectrum, from function-level code generation tasks (Chen et al. 202 Austin al. [222 Chen et al. (202aa Hendrycks et al.
Zhuo et al. 2024
2021
Athiwaratkun et al. 2022) to repository-level genera- Ding | 2023). Most evaluations adopt one-shot
tion (Jimenez et al.
2023 Li et al.
2024a
setting, where models produce sol lutiions without further interaction with humans. More recently, as
summarized in Table [ Han et al. (2025); Wang al.) (2023b); Yang et al. (2023) have explored
multi-turn interactive code generation by introducing various forms of feedback. However, these tasks remain limited to relatively simple function-level generation within the SWE domain. con- trast, our benchmark targets research code generation at the repository level, where the nature and complexity of feedback differ substantially from those existing SWE-oriented benchmarks.
3 BENCHMARK
RECODE-H is designed to evaluate the ability of large language models generate and refine research code in realistic, feedback-driven workflows. consists 102 tasks drawn from research papers and their corresponding repositories, each paired with structured instructions, explanatory comments, and unit tests to ensure reproducibility and reliability. The detailed benchmark statistics provided in Appendix B
3.1 BENCHMARK CONSTRUCTION
We developed a collaborative framework for constructing RECODE-H. To support this process, we assembled an annotation team of 26 annotators. All annotators are Ph.D.-level researchers with least one publication at top-tier computer science conferences and are familiar with the target methods or algorithms, multi-step pipeline: (1) paper and code selection, (2) annotation of explanatory comments for code, (3) construction of code generation instructions, and (4) development of unit tests. This design ensures both the reliability and reproducibility of the benchmark. An annotated example could be found in Appendix N
well their implementation code. The annotation process follows
Paper and Code Selection. All papers and repositories in RECODE-H were manually curated by expert annotators across diverse Al research domains, including machine learning algorithms, NL.P,
3

---

Published as a conference paper at ICLR 2026
ReCode Agent
Action
Agent
Feedback Feedback:
Researcher
FILE)
syess evnor
condip.ysel
implementation Aligement
[WRITE FILE]
Feadback:
Code Review
NXTTA PISK 570/0/ss.2y
shoreto eyetse ernor
(OOMEWAT)
[SEARCH]
Knowledge Contect Feedback:
Action
Reflect
The otandard Elerary
Paper
(Cunction descriptions
Fonecios
[SUBMIT]
Repasitory Indegration Feedback:
ARAD FiLe config.ysnt
Observe
calmucet alceady as2ada cn calcuiate.g .27
Log Analysis
RECODE
Instruction
Code Base
Unit Tests
Execution Result
Functionally Descritiion
Repo Context
Code File Context
Teef cases
Funotion
clacs Ar2
beet, procoss bench forward
Execution Log
Oulput
to teplerace
JM
Execution Serpet
Fatlore tenctions procuzss batch back toaces
Avoeuu
Mathod Deacription:
Conda activace cest pysout
Figure I: Illustration of the RECODE-H workflow, where LLM agents iteratively generate, test, and
refine research code through structured researcher feedback.
computer vision, graph learning, time-series modeling, diffusion models, and recommendation sys- tems (dctailed distribution in Appendix B). The paper selection process follows strict curation protocol to ensure realism, reproducibility, and alignment between research descriptions and imple- mentation details.
Specifically, we include only papers that satisfy the following criteria: (1) the paper was published in 2023 or later; (2) the venue is top-tier computer science conference (CVPR, ICML, NeurIPS, or ICLR)(3) the authors provide an official open-source repository that is correct and executable; and (4) the paper and repository exhibit a clear correspondence between methodological descrip- tions and code structure. Annotators verify each repository by running the official scripts to ensure the targeted components exccute successfully within standard hardware constraints (less than 24 GB GPU memory). Only repositories meeting all criteria are included in RECODE-H, ensuring that the benchmark reflects realistic, well-specified, and executable research code tasks. The full paper list revealed in Appendix K
Annotation of Explanatory Comments for Code. The correspondence between code and paper descriptions is often distributed across multiple functions classes, and the mapping is not always explicit. To address this, we add explanatory comments that clarify the relationship between the code and its associated paper. These comments will help produce consistent feedback, which is crucial for the reproducible evaluation process of RECODE-H.
To reduce annotation workload and ensure consistent comment content, we adopt a human-LLM collaborative approach. We employ Gemini-2 1212-.2.0 10......P.. to generate explanatory comments, taking as input both the paper text and the identified code segments. The comments primarily focus on three aspects: (1) the correspondence between the code and the paper description, (2) any discrepancies between the paper and the actual implementation, and (3) implementation details present in the code but absent from the paper. To ensure reliability, all generated comments are subsequently reviewed and validated for correctness.
Construction of Code Generation Instructions. Once the correspondence between the code and the paper description is established, we construct detailed code generation instructions. Each in- struction specifies the target function name, its intended functionality, and the input and output pa- rameters, including their names, data types, and semantic roles. For functionality descriptions, we mainly focus illustrating the relationship between the function and the paper description and add
4

---

Published as a conference paper at ICLR 2026
necessary explanations to the functionality, ensuring the instructions remain faithful to the original research context.
To promote consistency, employ Gemini- 1emiini-2.5-PP0 5-PPo to generate initial draft of the instruc- tion a standardized format. These drafts are then manually reviewed and refined by annotators to guarantee accuracy, clarity, and alignment with the source paper and code.
Development of Unit Tests. Unit tests are the core mechanism for evaluating functional correctness in RECODE-H. While LLMs (Gemini-22 5-Pro) were used only draft initial candidate test cases; all tests were subsequently reviewed, corrected, and finalized by expert annotators. Annota- tors verified every test for correctness and completeness and enforced strict 80% code coverage requirement for the canonical implementation. This ensures that the tests exercise the majority of logic branches rather than relying on superficial or syntactic checks. To accommodate complex research code functions, including those that internally perform complex computations, such as training updates, optimization routines, or data preprocessing workflows, we adopt general dif- ferential testing strategy. Rather than evaluating full end-to-end system behavior, each unit test executes both the canonical implementation and the model-generated implementation under identi- cal, fixed-seed conditions and compares their intermediate or final outputs. This comparison ensures functional equivalence within appropriate numerical tolerances, allowing us to verify the correctness of individual computational steps even when the overall task involves inherently dynamic or stateful processes. This approach provides a reliable and scalable mechanism for validating algorithmic cor- rectness across broad range of complex research tasks. These differential tests provide reliable and reproducible evaluation framework across all task types in RECODE-H.
3.2 FEEDBACK HIERARCHY
In real-world scenarios, the feedback provided to an LLM agent after code execution can vary sub- stantially. Factors such as the execution environment, the expertise of the feedback provider, and the effort invested in analyzing and writing feedback to the code all influence the form and quality of the feedback. Among these, the provider's expertise and the depth of analysis play the most significant roles in determining how informative the feedback is. To systematically evaluate LLM agents under varying feedback conditions, we design five-level feedback hierarchy, where each level provides progressively more guidance:
Level 0: Minimal feedback. The agent is only informed that code execution failed and that the execution result log is available.
Level I: Execution result plus high-level error description that briefly characterizes the failure.
Level 2: In addition Level 1, explanation of why the error occurred, offering diagnostic insight.
Level 3: Extends Level by providing natural language guidance that specifies how the imple- mentation should be corrected. The feedback outlines the appropriate algorithmic modifications or design choices needed to align the code with the canonical implementation, without supplying any executable code.
Level 4: The most detailed setting. Beyond Level 3, the exact ground-truth code snippet is provided. This level isolates the model's ability accurately integrate correct solution into the existing repository, serving as an upper- bound condition for evaluating instruction adherence and repository- integration.
Feedback Generation. We employ GPT-o4-mini to simulate an expert and generate feedback to ensure the benchmark's 's reproducibility and scalability. After each code-generation round, the feed- back model provides concise, actionable feedback. The feedback is conditioned on code execution results, including test outcomes and error logs. It also leverages the canonical code and annotated comments to clarify functionality and verify alignment with the paper's 'ss description. Using LLM to produce feedback in this loop preserves the itcrative nature of real research workflows while ensuring that feedback is standardized and reproducible across runs. We chose GPT-o4-mini because of its cost efficiency and high-quality feedback, as discussed in Appendix
Evaluation Metrics. We assess LLM agents RECODE-H H by functional correctness and code similarity. Functional correctness is measured with test cases using Mean Reciprocal Rank (MRR), which assigns score of & based on the first turn & where correct code appears, and Recall @n, which
5

---

Published as a conference paper at ICLR 2026
is the proportion of tasks solved correctly within n turns. We also report the average proportion of test cases that pass to capture partial correctness. Code similarity is evaluated against canonical implementations using CodeBLEU (Ren et. 2020), which combines lexical and structural signals, and CodeBERTSScor (Zhou et al. 2023), which measures semantic similarity via embeddings.
4 RECODEAGENT
To evaluate our benchmark, we introduce a feedback-driven L.MM code agent, ReCode Agent. The agent iteratively incorporates researcher feedback to generate modify code, aiming to produce fully correct and executable implementations. ReCode Agent engages in multi-turn interactions with researcher, refining its output until all tests pass predefined interaction limit is reached.
Agent Strategy. Our agent strategy follows the ReAct framework (Yao et al. 2023 and orga- nized into four stages. (1) Observation. It gathers the current repository state, execution logs from previously submitted code, and researcher feedback. (2) Reflection. It analyzes failures and gaps against the task specification, integrates feedhack into actionable insights, and ensures consistency with repository constraints. (3) Planning. It formulates a concise, structured plan for the next step, specifying the goal, target files spans, and the intended effect. (4) Action. It executes one op- eration from the predefined action space according to the plan. It should be noticed that, in our agent design, the use of "one-shot" refers to the human-agent interaction pattern rather than the agent's 's internal ReAct steps. Existing benchmarks support only single human prompt, after which the agent operates autonomously. In RECODE-H, the agent engages in multi-turn human- agent collaboration, even though its internal reasoning-acting loop remains the same as standard ReAct.
Memory Management. To keep context length bounded under multi-round interaction, the agent maintains memory similar to Reflexion (Shinn et al. (023). We enforce a threshold on the number of recent memories to keep. When the memory exceeds the threshold, the agent compacts prior observations and actions into concise summary that preserves the unresolved failures, design deci- sions, and generation context information. This memory compression promotes consistency across rounds while avoiding context bloat.
5 EXPERIMENTS
Experimental Setup. We evaluate ReCode Agent on RECODE-H using seven mainstream LLMs, spanning both reasoning and not-reasoning models. Specifically. we evaluated the GPT OpenAA 2025) family (GPT-5, GPT-5-min:: GPT-5-nanon the (eminiiTTeam et al. 2023) fam- ily (Gemini-2.55 5-pro and Gemini-2. 5-flash),
2025)
from Anthropic, and DeepSeek- feedback conditions a multi-turn setting to examine performance across varying levels human interaction. For code generation, we fix the decoding temperature to 0 and top-p to 1, ensuring deter- ministic outputs. The evaluation for each task proceeds for up to 10 rounds of feedback generation interaction. Within a single interaction limit the agent to have at most 3 actions before automatic submission. And the threshold number of memory the LLM agent keeps is set to 5.
(DeepSeek, 2025). Each model is assessed under multiple
between the LLM agent and the simulated human, we
5.1 OVERALL RESULTS AFTER 10 INTERACTION TURNS
Overview. As shown in the Table 2. the richer feedback consistently improves LLM performance. With minimal feedback, models achieve low pass rates and limited correctness. As feedback be- comes more detailed, from Levels to 3, both success rates and efficiency improve, reflected in higher MRR, Recall, and test case pass rates. the most detailed level, all models show substantial gains, with GPT-5 and Deepseek- -. benefiting the most.
Model size and capability play a clear role in performance. Within the GPT family, larger models consistently achieve higher scores across all feedback levels, showing that scaling up enhances the ability to utilize feedback effectively. However, this trend is less evident in Gemini and Claude than in small-sizee GPT models. Despite their relatively large sizes, their improvements from richer feed- back are modest compared to GPT-5 or Deepseek-Ve -asseeeke...... -3. 1. We attribute this to a lower feedback adoption rate these models appear less efficient at incorporating feedback signals into subsequent

---

Published as a conference paper at ICLR 2026
Pass Rate
Similarity CodeBERT
Model
Feedback Level
MRR %)
Recall(%)
Test Case( %)
CodeBLEU
OpenAI GPT
Level
1.4
5.9
23.8
0.252
0.854
Level
10.8(+4.9)
31.9(+8.1)
0.258
0.821
GPT-5-nano
Level
4.1
16.7(55.9)
39.4(+7.5)
0.262
0.843
Level
4.6
19.6(+2.9)
42.2(42.8)
0,266
0.817
Level 4
9.1
35.3(+15.7)
54.7(+12.5)
0.290
0.857
Level
1.3
19.6
42.3
0.262
0.870
Level
38.2(+18.6)
62.8(420.5
0.299
GPT-5-mini
Level 2
7.8
47.1(+8.9)
70.2(47.4)
0.303
0.887
Level
52.9(45.8)
73.2(43.0)
0.303
0.888
Level
11.1
66.7(+13.8)
81,0(+788
0.309
0,889
Level 6.0
29.4
53.7
0,287
0.879
Level
7.5
45.1(+15.7)
1)).()
0.294
0.884
GPT-5
Level
9.3
55.9(+10.8)
77.4(48.6)
0,302
0.885
Level
63.7(+7.8)
83.7(+6.3)
0.304
0. 886
Level 4
11.9
71.6(+7.9)
84.3(+0.6)
0.317
0,888
Anthropic Claude
Level
5.2
14.7
35.1
0.287
0.88.5
Level
6.0
24.5(+9.8)
48.5(+13.4)
0,303
0.881
Claude-sonnet-4
Level
7.5
33.3(+8.8)
51.8(43.3)
0.300
0.888
Level
7.7
33.3(+0.0)
53.5(+1.7) 64.4(+1.7
0.312
0.890
Level
11.4
48.0(+14.7)
0.335
0.892
Deepseek
Level 5.1
10.8
30.7
0.292
0.886
Level
8.9
26.5(+15.7)
47.2(+16.5)
0.292
0.890
DeepSeck-V3.1
Level
12.1
43.1(+16.6)
60.6(+134)
0.301
0.891
Level
14.1
49.0(+5.9)
57.6(-3.0)
0.308
0.892
Level
21.0
70.6(421.6)
77.3(+19.7)
0.341
0.897
Google Gemini
Level
3.9
8.8
32.2
0.294
0.889
Level
7.4
27.5(+18.7)
52.6(+20.4)
0.300
0.891
Gemini-2.5-fan
Level
9.6
34.3(+6.8)
58.7(+6.1)
0.309
0.893
Level
12.0
47.1(+12.8)
70.7(+12.0)
0.311
0.893
Level 4
14.2
58.8(+11.7)
77.6(+6.9)
0.348
0.899
Level 0
4.3
12.7
35.5
0.265
0.876
Level
6.1
16.7(+4.0)
40.1(+4.6)
0.278
0.880
Gemini-2.55 2.5-pr
Level 2
9.6
37.3(420.6)
57.6(+17.5)
0.285
0.883
Level
10.4
37.3(+0.0)
58.0(40.4)
0.291
0.874
Level 4
17.6
58.8(+21.5)
70.6(+12.6)
0.331
0.888
This table reports performance of diverse LLMs on the RECODE-H benchmark over 10 rounds of interaction. Metrics include Mean Reciprocal Rank (MRR). Recall, average test pass rate, and code similarity scores (CodeBLEU and CodeBERTScore)) Stronger feedback con- sistently improves functional correctness and alignment with ground-trutt implementations. with higher feedback levels leading to more effective error correction and refinement
Table 2:
generations, as discussed in Section 5.4 As result, their final performance lags behind other large models that adapt more quickly and fully iterative guidance.
Non-linear gains across feedback levels. We observe the improvements across feedback levels in Table are non-linear, with the largest performance boost often observed from Level Level 1. For example, most models nearly double their Recall and test case pass rates once minimal di- agnostic information provided, indicating that even shallow guidance strongly accelerates error correction. In contrast, the gains from Level 2 to Level 3 and from Level to Level 4 are more mod- erate, suggesting diminishing returns as feedback becomes increasingly detailed. Nevertheless, the highest level of feedback, which provides explicit code snippets, still delivers substantial improve- ments in functional correctness, especially for stronger models like GPT-5 5 and Deepseek- 1. This pattern demonstrates that while structured feedback is highly valuable, the marginal benefits taper off as approaches full supervision.
Differences across model families. The results also reveal notable differences across model families. The GPT family shows consistent and significant improvements as feedback becomes richer, with GPT-5 and GPT-5-mi n maintaining high scores across all metrics. In contrast,

---

Published as a conference paper at ICLR 2026
gpt-S-mini
0.7
Frediascteve
Fenebadk
Senttack Leael
Foedsece Lovel
Feudbace Level
Needtack Lovel2
Frediacetedg (evel
feedhand Level
leadtack (mael3
05-
Prodsccs Level
Pouebucc Levdl
Neadbock Jcecl4
0.4
P53z
0.1
0.0
5
13
10
10
Tam
0emini-2.5-Tlal
despseel-v3.1
Gludde-anned-4
0.7
Frodseot
Rocdbuuc
Poalleck
0.8
FeediaccLLve
Fenttack Lewel
Feodzace Lovel
Feuebacc Level
feodbuck Lovel4
Niw PM
0.4
0.3
0.2
0.5
7um 10
Tur
Tum
Figure 2: Pass rate trajectories across interaction turns under varying feedback levels. Richer feedback consistently boosts model performance, with the largest gains appearing in early turns. Stronger models like GPT-5 and Deepseek 1eepseek-V3.1 adapt more effectively, while Gemini 1aiini-2.55 -- and Claude Sonnet plateau earlier.
Claude-Sonnet-- shows only moderate gains, plateauing earlier and failing to fully utilize the detailed feedback. The Gemini family presents a split. Gemini-22 2--faassh perform bet- ter than Gemini- - 5-pro across all levels. while Gemini-22 S-pro shows competitive sults at higher feedback levels but still lags behind GPT-5 and Deepseek-V3 Deepseek-V2
Notably, demonstrates the largest relative improvement from Level 0 to Level 4, indi- cating a high sensitivity to feedback and strong adaptability in multi-turn interactions. These differ- ences suggest that, beyond model size, the architecture and training approach of each family play a crucial role in determining how effectively models can incorporate iterative feedback into code generation.
Recall and MRR improvements align. A consistent trend across Table i that Recall and MRR improve as feedback levels increase. 5'ss Recall rises steadily from 0.294 at Level 0 to 0.716 at Level 4, and its MRR improves from 0.060 to 0.1 over the same range.
5.2 PERFORMANCE DYNAMICS ACROSS INTERACTION TURNS
Figure 2 illustrates the test case pass rate with the increase of interactive turns. The trajectories clearly show that richer feedback only improves the final success rate but also accelerates con- vergence in early turns. Both GPT 5 and DeepSeek VV. rapidly increase their pass rates within the first 3-4 turns when provided with Level Level feedback, whereas with Level 0 feedback their improvement remains gradual and plateaus at much lower levels. In addition, the first-turn re- sults shown Figure 2 also serve as our single-turn baseline. In this initial turn, the model receives only the task prompt without any interactive feedback, corresponding to the highlighted single-turn setting. As the figure indicates, subsequent turns that incorporate richer feedback yield substantial improvements over this baseline. After 10 turns, all models demonstrate markedly higher pass rates, confirming that multi-turn interaction provides a significant uplift over a single-turn scenario.
In contrast, smaller weaker models exhibit slower gains and limited sensitivity to feedback rich- ness, resulting in lower overall performance. Gemini-2 5-pro and hibit unstable performance gaps between feedback levels as interaction turns increase, which aligns with the inconsistent feedback adoption rates discussed Section5...aad Appendix Among them, cl - -4 shows the weakest performance trajectory across turns: the differences be-
ex-
8

---

Published as a conference paper at ICLR 2026
tween feedback Levels 1-3 are far less pronounced than in other models, indicating difficulty in effectively leveraging moderate feedback.
5.3
ERROR ANALYSIS
To better understand the role of feedback in guiding multi-turn code generation, we conducted fine-grained analysis of the errors encountered during the benchmark and the corrective signals associated with them. Our analysis revealed that errors can be systematically categorized into four types based on their root causes, each reflecting different source of failure in the generation process.
Code Syntax and Runtime Errors. This category encompasses fundamental programming issues that are independent of high-level algorithm design. Typical cases include syntax violations, runtime failures such as type mismatches or out-of-bounds indices, and basic logic flaws such as incorrect loop termination or variable misuse.
Type 1:
Type 2:Paper and Instruction Misunderstanding Errors. These errors arise when the generated code diverges from the intended method fails meet the benchmark requirements. Common manifestations include incorrect implementations mathematical formulas, omission or misappli- cation of algorithmic steps, and mismatches in input/output formats tensor shapes. They reflect misalignment between the model's output and the research description it expected to reproduce.
Type 3: Missing Knowledge and Context Errors. Some failures result from incomplete domain knowledge or implicit assumptions that are not explicitly stated in the task description. Examples include neglecting to standard library functions, misunderstanding technical terminology, overlooking sepesitoy--specific conventions. These errors highlight the model's difficulty in bridg- ing gaps between textual instructions and domain practices.
Type 4: Repository Integration Errors. Finally, certain errors are caused not by the algorithm itself, but by poor integration with the surrounding codebase. These include reimplementing ex- isting helper functions, misusing predefined modules or classes, or disregarding established coding conventions. Such errors limit the usability of generated code within the broader repository and emphasize the importance of context-aware generation.
LLM
Typel(% %
Type2 %)
Type3(%)
Type4 (%)
Gpt-5
11.35
34.04
50.25
4.36
Gpt-5-mini
11.53
27.06
55.22
6.19
Gpt-5-nano
20.82
37.32
34.95
6.90
Gemini-2.5-pp
14.94
39.91
37.34
7.82
Gemini-2.5-flas
16.16
26.15
49.41
8.28
Deepseek-chie
20.60
31.64
40.30
7.46
-lauee-sonnet-4
26.45
32.26
33.75
7.54
Table 3: Error type distribution across models the RECODE-H benchmark.
In this experiment, employ GPT-5 classify the reasons for errors. To ensure the reliability of the classification,, we randomly sampled 100 cases and verified them with human annotators, achiev- ing 98% agreement with GPT-5's predictions, providing strong evidence of classification accuracy. As shown in Table 3 the majority of failures are attributed to higher-level semantic issues rather than low-level coding mistakes. Specifically, paper and instruction misunderstanding eryors(Type 2), missing knowledge and context errors 3) dominate across all models, whereas syntax and runtime errors(Type 1) occur less frequently, and repository integration errors(Type 4) are the least common. This distribution shows that modern LLMs have largely overcome basic coding chal- lenges. However, they still struggle to align implementations faithfully with research descriptions and to bridge implicit domain knowledge. In addition, occasional but impactful gaps in reposi- tory awareness remain. A more detailed discussion of error type patterns and their implications is provided in Appendix

---

Published as a conference paper at ICLR 2026
5.4 FEEDBACK ADOPTION
We analyze how often models adopt the provided feedback and whether adoption results in a correct fix. GPT-5 is employed as a classifier to determine whether the feedback is incorporated into the revised code and whether the targeted eπror is resolved. Our evaluation shows that both adoption and fix rates vary substantially across models, guidance levels, and feedback types. Detailed Table TO and Table 11 supporting these findings are provided in Appendix
Adoption as a necessary pathway. Across all models and settings, nearly every successfully cor- rected error is one where the model explicitly adopted the provided feedback. Cases where errors were fixed without adoption are exceedingly rare, underscoring that feedback-driven improvement is the main driver of repair.
Effeet of guidance level on adoption. Stronger feedback guidance increases the likelihood of adop- tion overall, but the magnitude of this increase differs substantially across models. Models that ex- hibit significant improvement in pass rates as guidance level rises, such as GPT-5, 5, GPT-5 5-mini, and DeepSeek-V3. 1, also show clear gains in feedback adoption. For instance, GPT-5 adoption rises from 80.2% at Level-1 to 90.1% % at Level-4, with DeepSeek-V3. 1 and GPT-5-mint fol- lowing similar upward trajectories. This pattern suggests that the models most capable of leveraging feedback effectively are increasingly receptive to guidance as it becomes more explicit.
Declining fluctuating adoption among weaker improvers. By contrast, models that show low pass rate under stronger guidance, such as Gemini- 5-ppr and Claude Saalee-Sonnt-tte -4, exhibit different pattern of adoption. For instance, the adoption rate of Claude-Sonnet decreases as the feedback guidance level increases, while the adoption rate of Gemini 12-2.... 5-pro fluctuates around 70%. These feedback adoption patterns align with their weaker performance in the benchmark.
Simple and highly-specific feedback is adopted most. Models exhibit the highest adoption rates for feedback targeting Code Correctness and Repository Integration errors. For example, DeepSeek-cher adopts nearly 80% of syntax-related feedback. In contrast, adoption rates of feedback for implementation alignment error (Type 2) are consistently lower across models, with GPT-5 nano showing the lowest rate at 56. 1%. This pattern suggests that models struggle to address subtle logical errors that require deeper understanding of the research method's intent.
5.5 DISCUSSION AND INSIGHTS
Our results highlight that the main barrier in research code generation is semantic fidelity. Models most often fail by misinterpreting methodological intent or missing domain knowledge, rather than through syntactic errors. Feedback effects also highly non-linear. The minimal diagnostic signals yield the largest improvements, while richer natural-language guidance yields diminishing returns until explicit code snippets are supplied. Furthermore, feedback utilization varies sharply across models, indicating that effective incorporation of iterative guidance is model-dependent capability. These observations suggest that future research should prioritize stronger semantic grounding and more stable multi-turn refinement. In practice, concise execution-level feedback is most effective, while human effort is best spent addressing conceptual misalignments.
6 CONCLUSION
We introduce RECODE-H, a benchmark for evaluating LLM-based research agents in realistic sci- entific workflows with feedback-driven code implementation. Unlike prior work focused on end-to- end pipelines or one-shot generation, RECODE-H emphasizes interactive evaluation, reflecting core aspects of real research practice. Our experiments reveal three findings: (1) scientifie grounding, not coding, is the primary bottleneck, as models struggle with methodological interpretation and implicit domain knowledge; (2) feedback yields strongly non-linear gains, with minimal signals un- locking substantial improvements; and (3) feedback receptivity is distinct capability, with stronger models benefiting from richer guidance while weaker ones plateau. Future work includes extending the benchmark to broader research stages, multi-agent settings, and iemaann-n-the-lee feedback. Overall, effective research agents will require not only coding ability but also adaptive reasoning and sustained interaction with complex research environments.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported in part by the National Science Foundation (NSF) under Grants III-2106758 and POSE-2346158.
Yangning is supported by the Basic Research JCYJ202408 331 22090113).
Fund of Shenzhen
City
(Grant
No.

---

Published as a conference paper at ICLR 2026
REFERENCES
Silas Alberti, Kenan Hasanaliyev, Manav Shah, and Stefano Ermon. Data unlcarning in diffusion models. arXiv preprint arXiv:2503.01034 2025.
Anthropic. Introducing claude 4. 22 2025. Accessed: 2025-09-20.
aude- May
Abolfazl Asudeh, Nima Shahbazi, and Stavros Sintos. fair and memoryttine-effiieet hashmap. arXiv preprini arXiv:2307. 1355, 2023.
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Farahani, Siddharth Jain. Robert Giaquinto, Haifeng Qian, M. Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, D. Roth, and Bing Xiang. Multi-lingua evaluation of code generation models. ArXiv, abs/2210. 14868, 2022. URL /tttps://arxiiii .rg. wwtg/pd..... httppppf/221122 112110118855 14868. pdf
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski. David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, et al. Program synthesis with large language models. arXiv preprint urXiv:2108.07720 2021.
Hao Bai, Yifci Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training -n-lhe-wild device-contro, agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37:12461-12495, 2024.
I. Lao Beyer, Tianhong Li, Xinlei Chen, Sertac Karaman, and Kaiming He. Highly compressed tokenizer can generate without training. arXiv preprint arXiv:2506.088557 2025.
Xuheng Cai, Chao Huang, Lianghao Xia, and Xubin Ren. Lightgel: Simple yet effective graph contrastive learning for recommendation. arXiv preprint arXiv:2302.0819), 2023.
Jialun Cao, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuging Li, Mingwei Liu, Ruixi Qiao, Yuting Han, Chaozheng Wang, Boxi Yu, al. How should we build benchmark? revisiting 274 code-related benchmarks for llms. arXiv preprint arXiv:2501. /071 2025.
Zhiwei Cao, Qian Cao, Yu Lu, Ningxin Peng, Luyang Huang, Shanbo Cheng, and Jinsong Su. Retaining key information under high compression ratios: Query-guidee compressor for llms. arXiv preprint arXiv:2440..02376 2024.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung. Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learn- ing agents on machine learning engineering. arXiv preprint arXiv:2410.07005 2024.
Chen Chen, Ruizhe Li, Yuchen Hu, Sabato Marco Siniscalchi, Pin-Ye Yu Chen, Ensiong Chng. and Chao-Han Huck Yang. It's never too late: Fusing acoustic information into large language models for automatic speech recognition. arXiv preprint arXiv:2402.05457, 2024a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.0337. 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprini arXiv:2304.055/2, 2023a.
Yankai Chen, Yixiang Fang, Yifei Zhang, and Irwin King. Bipartite graph convolutional hashing for effective and efficient top-n search in hamming space. In Proceedings of the ACM Web Conference 2023, pp. 3174-3172, 2023b.
Yankai Chen, Yixiang Fang, Yifei Zhang, Chenhao Ma, Yang Hong, and Irwin King. Towards effective top-n hamming search via bipartite graph contrastive hashing. IEEE Trans. Knowl. and Data Eng., 36(12):9418 11329944-- December 2024b. ISSN 1041-4347. doi: 10. 109TTDE.. 2024.3425891. URL https: //doi http///tooooooo/ooo//o/oooooooot 10. 1109/ TKDE, 2024 .4358891
12

---

Published as a conference paper at ICLR 2026
Yankai Chen, Taotao Wang, Yixiang Fang, and Yunyu Xiao. Semi-supervised node importance estimation with informative distribution modeling for uncertainty regularization. In Proceedings of the ACM on Web Conference 2025, WWW '25, pp. 3108-3118 New York. NY. USA, 2025a. Association for Computing Machinery. ISBN 9798400712746. doi: 10. (0.1145/66694103774455))
org/ 110...145/36661111 371 3714591
Yankai Chen, Xinni Zhang, Yifei Zhang, Yangning Li, Henry Peng Zou, Chunyu Miao, Weizhi Zhang, Xue Liu, and Philip S Yu. Embracing trustworthy brain- agent collaboration as paradigm extension for intelligent assistive technologies. arXiv preprint arXiv: 10. 2025b.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for driven scientific discovery. arXiv preprint arXiv:2410.05080, 2024c.
Ziqiang Cui, Xing Tang, Yang Qiao, Bowei He, Liang Chen, Xiuqiang He, and Chen Ma. Treatment- aware hyperbolic representation learning for causal effect estimation with social networks. In Proceedings of the 2024 SIAM International Conference Data Mining (SDM), pp. 289-297.
SIAM, 2024a.
Ziqiang Cui, Haolun Wu, Bowei He, Ji Cheng, and Chen Ma. Diffusion- based contrastive learning for sequential recommendation. arXiv preprint 49999924,..09369, 2024b.
Yizhou Dang, Jiahui Zhang, Yuting Liu, Enneng Yang, Yuliang Liang, Guibing Guo, Jianzhe Zhao, and Xingwei Wang. Augmenting sequential recommendation with balanced relevance and diver- sity. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 11 11563- 11571, 2025.
Inc. DeepSeek.
Deepseek-väs v3.1 release.
news250821
August 21 2025. Accessed: 2025-09-20.
Aaron Defazio, Xingyu Yang. Harsh Mchta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky, The road less scheduled. Advances in Neural Information Processing Systems, 37: 9974-10007, 2024.
Mohsen Dehghankar, Mahdi Erfanian, and Abolfazl Asudeh. An efficient matrix multiplication algorithm for accelerating inference in binary and ternary neural networks. arXiv preprint arXiv:2411.06366 2024.
Mohsen Dehghankar, Rahul Raychaudhury, Stavros Sintos, and Abolfazl Asudeh. Fair set cover. In Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 189-200, 2025.
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakr- ishna Vedantam. Hyperbolic image-text representations. In Andreas Krause, Emma Brun- skill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Pro- ceedings of the 40th International Conference on Machine Learning. volume 202 of Proceed- ings of Machine Learning Research, pp. 7694-7731. PMLR, 23-29 Jul 2023. URL https: //proceedings. mlr wwwpppeeess.e.. htt.aaeeees//22222222222222
atml
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, M. K. Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross- file code completion. ArXiv, abs/2310.11248, 2023. URL wwwiv.iiiii....a wwt. pttp///pf//31122 1 11248. pdf.
Xiangyu Dong, Xingyi Zhang, Yanni Sun, Lei Chen, Mingxuan Yuan, and Sibo Wang. Smoothgnn: Smoothing-aware gnn for unsupervised node anomaly detection. arXiv preprint arXiv:2405.17525 2024.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et Loramoe: Alleviate world knowledge forgetting large language models via moe-style plugin. arXiv preprint arXiv:2312.09999, 2023.
Tianyu Du Luca Melis, and Ting Wang. Remasker: Imputing tabular data with masked autoencod- ing. arXiv preprint arXiv:2309. 13793, 2023.
13

---

Published as a conference paper at ICLR 2026
Benoit Dufumier, Javiera
Devis Tuia, and Jean Phinlppi Thiran. What align in multimodal contrastive learning? arXiv preprint arXiv:2409.07402, 2024.
Nicholas Edwards, Yukyung Lee, Yujun Audrey Mao, Yulu Qin, Sebastian Schuster, and Najoung Kim. Rexbench: Can coding agents autonomously implement ai research extensions? arXiv preprint arXiv:2506.225500 22598, 2025.
Lan Feng, Fan Nie, Yuejiang Liu, and Alexandre Alahi. Tarot: Targeted data selection via optimal transport. arXiv preprint arXiv:2412.00422 2024.
Soham Gadgil, Ian Covert, and Su-In Lec. Estimating conditional mutual information for dynamic feature selection. arXiv preprint (rXiv:2306.03301 2023.
Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform. arXiv preprint 2000000000000003 2024.
Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew G Wilson. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36: 36:19622-19635, 2023.
Jialong Guo, Xinghao Chen, Ychui Tang. and Yunhe Wang. Slab: Efficient transformers with simplified linear attention and progressive re- re-parameterized batch normalization. arXiv preprint arXiv:2405. 15582, 2024.
Hojae Han, Seung-won Hwang. Rajhans Samdani, and Yuxiong He. Convcodeworld: Bench- marking conversational code generation in reproducible feedback environments. arXiv preprint arXiv:2502. / /9852, 2025.
Bowei He, Yankai Chen, Xiaokun Zhang, Linghe Kong. Philip S. Yu, Xue Liu, and Chen Ma. Ped- agogically inspired data synthesis for language model knowledge distillation. In The Fourteenth International Conference on Learning Representations (ICI.), 2026.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, D. Song, and Steinhardt. Measuring coding challenge com- petence with apps. ArXiv, abs/2105.099) 105.09938, 2021. URL 09938. pdf]
Guoqing Hu, An Zhang, Shuo Liu, Zhibo Cai, Xun Yang, and Xiang Wang. Alphafuse: Learn em- beddings for sequential recommendation in null space of language embeddings. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1614-1623, 2025.
Jiaxi Hu, Yuehong Hu, Wei Chen, Ming Jin, Shirui Pan, Qingsong Wen, and Yuxuan Liang. At- tractor memory for long-term time series forecasting: A chaos perspective. Advances in Neural Information Processing Systems, 37:20786-20815, 2024.
Tianyu Hua, Harper Hua, Violet Xiang, Benjamin Klieger, Sang T. Truong, Weixin Liang, Fan-Yun Sun, and Nick Haber. Researchcodebench: Benchmarking 1lms on implementing novel machine learning research code. ArXiv, abs/2506.02314, 2025. URL lnttps://ag) semant escholar wttp/oorgg/ooaoo.ooo
99933
Baizhou Huang, Shuai Lu, Xiaojun Wan, and Nan Duan. Enhancing large language models in coding through multi perspectiepeeee self-consisteney. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1429-1450, 2024.
Wei-Chieh Huang and Coreelaa Caragea. Madiave: Multi-agent debate for implicit attribute value extraction. arXiv preprint 2xX.::5510000111.. 2025.
Wei-Chieh Huang, Henry Peng Zou, Yaozu Wu, Dongyuan Li, Yankai Chen, Weizhi Zhang, Yangn- ing Li, Angelo Zangari, Jizhou Guo, Chunyu Miao, et al. Deepresearchguard. Deep research with open-domain evaluation and multi-stage guardrails for safety. arXiv preprint 25X.14550..0099 2025.
14

---

Published as a conference paper at ICLR 2026
Wei-Chieh Huang, Weizhi Zhang, Yueqing Liang, Yuanchen Bei, Yankai Chen, Tao Feng, Xinyu Pan. Zhen Tan, Yu Wang. Tianxin Wei, et al. Rethinking memory mechanisms of foundation agents in the second half. arXiv preprint NXXx:22002.00552, 2026.
Guillaume Jaume, Anurag Vaidya, Richard J Chen, Drew FK Williamson, Paul Pu Liang, and Faisal Mahmood. Modeling dense multimodal interactions between biological pathways and histology for survival prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11579- 11599, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint vrXiv:2310.067700 2023.
Tommie Kerssies, Niccolo Cavagnero, Alexander Hermans, Narges Norouzi, Giuseppe Averta, Bas- tian Leibe, Gijs Dubbelman, and Daan de Geus. Your vit is secretly an image segmentation model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25303-25313, 2025.
Patrick Tser Jern Kon, Jiachen Liu, Xinyi Zhu, Qiuyi Ding, Jingjia Peng, Jiarong Xing, Yibo Huang, Yiming Qiu, Jayanth Srinivasa, Myungjin Lee, et al. Exp-bench: Can conduct ai research experiments? arXiv preprint 25Kiv:2505.2478: 2025.
Dilfira Kudrat, Zongxia Xie, Yanru Sun, Tianyu Jia, and Qinghua Hu. Patch-wise structural loss for time series forecasting. arXiv preprint arXiv:2503.00 200.07777..0 2025.
Chengpeng Li, Zhengyang Tang, Ziniu Li, Mingfeng Xue, Keqin Bao, Tian Ding, Ruoyu Sun, Benyou Wang, Xiang Wang. Junyang Lin, and Dayiheng Liu. Cort: Code- iote-integrated reasoning within thinking. ArXiv, (ts//2506009820, 2025a. URL https://api. htt///ppsssmanncsccchhhoo.oo.s
Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. Evocodebench: An evolving code gen- cration benchmark aligned with real-world code repositories. arXiv preprint arXiv:2404.00599, 2024a.
Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, et al. Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories. arXiv preprint arXiv:2405. 19856. 2024b.
Jiangmeng Li, Yifan Jin, Hang Gao, Wenwen Qiang, Changwen Zheng, and Fuchun Sun. Hierar- chical topology isomorphism expertise embedded graph contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13518-13527, 2024c.
Ryan Li, Yanzhe Zhang, and Diyi Yang. Sketch2code: Evaluating vision-languee
models for interactive web design prototyping. In North American Chapter of the Association for Compu- tational Linguistics, 2024d. URL inttps://agg semant escho ar. worg/ hrgg/Corpusdd:
273507760
Wen Li, Chang- Dong Wang, Hui Xiong, and Hlang Lai. Homogel: Rethinking homophily in graph contrastive learning. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, pp.
2023.
Yangning Li, Weizhi Zhang, Yuyao Yang. Wei-Chich Huang, Yaozu Wu, Junyu Luo, Yuanchen Bei, Henry Peng Zou, Xiao Luo, Yusheng Zhao, et al. A survey of rag-reasoning systems in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025,
pp. 12120-12145, 2025b.
Zhixun Li. Yushun Dong, Qiang Liu, and Jeffrey Xu Yu. Rethinking fair graph neural networks from re-balancing. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pp. 1736-1745, 2024e.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liam Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang. Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong. Zhijiang Guo, Le Song, and Cheng- Lin Liu. From system 1 to system 2: A survey of reasoning large language models. ArXiv, abs/2502.117199 2025c. URL //api. semanticscholar.
brre/Corrpus1d2275555522
15

---

Published as a conference paper at ICLR 2026
Zongxia Li, Ishani Mondal, Huy Nghiem, Yijun Liang, and Jordan Lee Boyd- Gryd-Gr Pedants: Cheap but effective and interpretable answer equivalence. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 9373-9398, 2024f.
Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua. Data- efficient fine-tuning for llm-based recommendation. In Proceedings of the 47th international ACM SIGIR conference research and development in information retrieval, pp- 365-374, 2024a.
Yiqing Lin, Jianheng Tang. Chenyi Zi, H Vicky Zhao, Yuan Yao, and Jia Li. Unigad: Unifying multi-level graph anomaly detection. Advances in neural information processing systems, 37: 136120-1361 136148,,, 2024b.
Chongming Liu, Songting Li, and Douglas Zhou. Dendritic integration based quadratic neural networks outperform traditional aritificial ones. arXiv preprint arXiv2230..36609 2023a.
Meng Liu, Yue Liu, Ke Liang, Wenxuan Tu, Siwei Wang. Sihang Zhou, and Xinwang Liu. Deep temporal graph clustering. arXiv preprint Ttt::2355.00333 2023b.
Pei Liu, Luping Ji, Jiaxiang Gou, Bo Fu, and Mao Ye. Interpretable vision- lisoon-lngguag survival analysis with ordinal inductive bias for computational pathology. arXiv preprint arXiv:2409.09 2024a.
Qidong Liu, Xian Wu, Yejing Wang, Zijian Zhang, Feng Tian, Yefeng Zheng, and Xiangyu Zhao.
Llm-esr: Large language models enhancement for long-tailed sequential recommendation. Ad- vances in Neural Information Processing Systems, 37:26701-26721, 2024b.
Shuo Liu, An Zhang, Guoqing Hu, Hong Qian, and Tat-seng Chua. Preference diffusion for recom- mendation. arXiv preprint 10.1::2400.31111 2024c.
Yixin Liu, Shiyuan Li, Yu Zheng. Qingfeng Chen, Chengqi Zhang, and Shirui Pan. Are: A generalist graph anomaly detector with in-context learning. Advances in Neural Information Processing Systems, 37:50772-50804, 2024d.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long- itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 2023c.
Yue Liu, Shihao Zhu, Jun Xia, Yingwei Ma, Jian Ma, Xinwang Liu, Shengju Yu, Kejun Zhang, and Wenliang Zhong. End-to-end learnable clustering for intent learning in recommendation. Advances in Neural Information Processing Systems, 37:5913-5949, 2024e.
Jiecheng Lu, Yan Sun, and Shihao Yang. In-context time series predictor. arXiv:2405. 14982, 2024.
arXiv preprint
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: language models. Advances in neural information processing systems, 36:21702-21720, 2023.
On the structural pruning of large
Odelia Melamed, Gilad Yehudai, and Adi Shamir. Malt powers up adversarial attacks. Advances in Neural Information Processing Systems, 37:44366-44399 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198-12493 2024.
Jiacheng Miao and Qiongshi Lu. Task-agnostic machine-learing-astiie asssited inference. Advances in Neural Information Processing Systems, 37:106162-106188 2024.
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guaran- tees. arXiv preprint 24:2 2024.
Minh Nhat Nguyen, Andrew Baker, Clement Neo. Allen Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. Turning up the heat: Min-p sampling for creative and coherent Imm outputs. arXiv preprint ar XrXiv:2407.01088 2024.
Nie. A time series is worth 64words: Long-term forecasting with transformers. arXiv preprint 1.Xv:22111117300 2022.
16

---

Published as a conference paper at ICLR 2026
OpenAl. Gpt-5 system card. Technical report, OpenAI, 2025. URL htttps://coe openai....pp.......... Eonm/ppt-S---sstem--aaa-- carddd pdf Accessed: 2025-09-20.
Harshith Padigela, Chintan Shah, and Dinkar Juyal. MI-dev-bench: Comparative analysis of ai
agents on ml development workflows, arXiv preprint arXiv:2502.00066, 2025.
Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng. Zhaopan Xu, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, al. Infobatch: Lossless training speed up by unbiased 2023.
dynamic data pruning. arXiv preprint arXiv:2303.0453
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009. 2000:2000000297,,, 2020.
Maxon Rubin-Toles, Maya Gambhir, Keshav Ramji, Aaron Roth, and Surbhi Goel. Conformal language model reasoning with coherent factuality. arXiv preprint arXiv:2505.17126, 2025.
Philipp Seidl, Andreu Vall, Sepp Hochreiter, and Günter Klambauer. Enhancing activity predic- tion models in drug discovery with the ability to understand human language. In International Conference on Machine Learning, pp. 30458 304900 PMLR, 2023.
Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, and Xipeng Qiu. Balanced data sampling for language model training with clustering. arXiv preprint arXiv:2402.14526, 2024.
Juntong Shi, Minkai Xu, Harper Hua, Hengrui Zhang, Stefano Ermon, and Jure Leskovec. Tabdiff:: mixed-type diffusion model for tabular data generation. arXiv preprint arXiv:2410.20626, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8652, 2023.
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can Ilms generate novel research ideas? large- scale human study with 100+ nlp researchers. arXiv preprint WrXiv:2409.041000 2024.
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https://////..... httposss///pereevew...o. net forum?ids
M2 M23dTGWCZy
Andrew H Song, Richard J Chen, Guillaume Jaume, Anurag J Vaidya, Alexander S Baras, and Faisal Mahmood. Multimodal prototyping for cancer survival prediction. arXiv preprint arXiv:2407.00224, 2024.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung. Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint ar 25Xiv:2504.0/883 2025.
Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Rhe-Wei Shen, Baocai Chen, Lu Chen, and Kai Yu. Scieval: A multi-level large language model evaluation benchmark for scientific re- search. ArXiv, abs/2308. 13149, 2023. URL https: www.stecsccoooolaataaoeomm
Ctc. .org/// orgg
Corpusid:261244633
Sophia Sun and Rose Yu. Copula conformal prediction for multi-step time series forecasting. arXiv preprint arXiv:2212.0328) 2022.
Yao Tang, Zhihui Xie, Zichuan Lin, Deheng Ye, and Shuai Li. Learning versatile skills with cur- riculum masking. Advances in Neural Information Processing Systems, 37:65562-65582, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. /1805 2023.
17

---

Published as a conference paper at ICLR 2026
Edan Toledo, Karen Hambardzumyan, Martin Josifoski, Rishi Hazra, N. Baldwin, Alexis Audran- Reiss, Michael Kuchnik, Despoina Magka, Minqi Jiang, A. Lupidi, Andrei Lupu, R. Raileanu, Kelvin Niu, Tatiana Shavrina, Sean-Christophae Gagnon- Michael Shvartsman, Shagun Sod- hani, Alexander H. Miller, Abhishek Charnalia, Derek Dunfield, Carole-Jean Wu, Pontus Stene- torp, Nicola Cancedda, Foerster, and Yoram Bachrach. Aí research agents for machine learn- ing: Search, exploration, and generalization in mle-bench. ArXiv, 0ass/200.002554. 2025. URL https::/appi www.nticscco.cooc....e.. ar. org. httg//orpssaa::: 2001::22018887
Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and Jun Zhou. Timemixer: Decomposable multiscale mixing for time series forecasting, arXiv preprint arXiv:2405. 24li:22655.1666,, 2024a.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offiine multi-agent reinforcement learning with implicit global-to-local value regularization. Advances in Neural Information Pro- cessing Systems, 36:52413-52429, 2023a.
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating ilms in multi-turn interaction with tools and language feedback. arXiv preprint arXiv:2309. 10691, 2023b.
Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Qingsong Wen, Wei Ye, et al. Autosurvey: Large language models can automatically write surveys. Advances in neural information processing systems, 37:115119- 115145, 2024b. 2024b.
Jeremy Wayland, Corinna Coupette, and Bastian Rieck. Mapping the multiverse of latent represen- tations. arXiv preprint 2402012402001114,, 2024.
Tiansheng Wen, Yifei Wang, Zequn Zeng, Zhong Peng. Yudi Su, Xinyang Liu, Chen, Hong- wei Liu, Stefanie Jegelka, and Chenyu You. Beyond matryoshka: Revisiting sparse coding for adaptive representation. arXiv preprint arXiv:2503.01777, 2025.
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. Cycleresearcher: Improving automated research via automated review. ArXiv, 0os/2411.00816 2024. 273811997
URL https::/apt: semanticschol ar. oomtam/ci.cccoo....ooooo/ooooooo
Qitian Wu, Fan Nie, Chenxiao Yang, and Junchi Yan. Learning divergence fields for shift-robust graph representations. arXiv preprint arXiv:2406. 04963, 2024a.
Qitian Wu, Chenxiao Yang, Kaipeng Zeng, and Michael M. Bronstein. Supercharging graph trans- formers with advective diffusion. In Forty-second Intermational Conference on Machine Learn- net Maa.rruumndMMMMM3133PP3233
ing. 2025a.
Shirley Wu, Kaidi Cao, Bruno Ribeiro, James Zou, and Jure Leskovee. Graphmetro: Mit- igating complex graph distribution shifts via mixture of aligned experts. arXiv preprint 12X.::2312.04693, 2023.
Yaozu Wu, Dongyuan Li, Yankai Chen, Renhe Jiang, Henry Peng Zou, Wei-Chieh Huang, Yangning Li, Liancheng Fang, Zhen Wang, and Philip S Yu. Multi-agent autonomous driving systems with large language models: A survey of recent advances, resources, and future directions. Findings of the Association for Computational Linguistics: EMNLP. 2025, 2025b.
Zhaomin Wu, Junyi Hou, Yiqun Diao, and Bingsheng He. Federated transformer: Multi-party verti- cal federated learning practical fuzzily linked data. Advances in Neural Information Processing Systems, 37:45791-4581 18, 2024b.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333 2024.
Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang. Lin Gui, and Yulan He. Scireplicate-benchh Bench- marking Ilms in agent-driven algorithmic reproduction from research papers. arXiv preprint arXiv:2504.00 00255, 2025.
18

---

Published as a conference paper at ICLR 2026
Teng Xiao, Huaisheng Zhu, Zhengyu Chen, and Suhang Wang. Simple and asymmetric graph contrastive learning without augmentations. Advances in neural information processing systems, 36:16129-16133 16152, 2023.
Conghao Xiong, Hao Chen, Hao Zheng, Dong Wei, Yefeng Zheng, Joseph JY Sung, and Irwin King. Mome: Mixture of multimodal experts for cancer survival prediction. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 318-328. Springer, 2024.
Yanchen Xu, Siqi Huang, Hongyuan Zhang, and Xuelong Li. Why does dropping edges usually outperform adding edges in graph contrastive learning? In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 21824-21832, 2025.
Yingxue Xu and Hao Chen. Multimodal optimal transport-based based co-attention transformer with global structure consistency for survival prediction. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 21241-21251, 2023.
Zhijian Xu, Ailing Zeng, and Qiang Xu. Fits: Modeling time series with 10k parameters. arXiv
preprint arXiv:2307 003556, 2023.
Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, George Michalopoulos, Yue Zhang, et al. Lmr-bench: Evaluating IIm agent's ability on reproducing language modeling research. arXiv preprint arXiv:2506. 17335, 2025.
Hanchen Yang. Jiaqi Wang, Jiannong Cao, Wengen Li, Jialun Zheng, Yangning Li, Chunyu Miao, Jihong Guan, Shuigeng Zhou, and Philip S Yu. Okg-llm: Aligning ocean knowledge graph with observation data via Ilms for global sea surface temperature prediction. arXiv preprint
arXiv:2508.00 00933, 2025.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36:23826-2388 23854, 2023.
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction generation: Taming optimization dilemma in latent diffusion models. In Proceedings of the Computer Vision and Pattern Recogni- fion Conference, pp. 55703-15712 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Representations (ICLR), 2023.
Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. arXiv preprint arXiv:2410.05258, 2024.
Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Long- bing Cao, and Zhendong Niu. Frequency- domain mlps are more effective learners in time series forecasting. Advances in Neural Information Processing Systems, 36:76656-7666 76679, 2023.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang. Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.069440 2024.
Yiwen Yuan, Zecheng Zhang, Xinwei He, Akihiro Nitta, Weihua Hu, Dong Wang, Manan Shah, Shenyang Huang, Blaž Stojanovič, Alan Krumholz, et al. Contextgnn: Beyond two-tower recom- mendation systems. arXiv preprint 1. 19513,, 2024.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference artificial intelligence, volume 37, pp- 11121-11128, 2023.
Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li. Yuchen Hao, Yantao Yao, et al. Wukong: Towards a scaling law for large-scale recommendation. arXiv preprint arXiv:2403.025.2 2024a.
19

---

Published as a conference paper at ICLR 2026
Hanlei Zhang, Hua Xu, Fei Long, Xin Wang, and Kai Gao. Unsupervised multimodal clustering for semantics discovery in multimodal utterances. arXiv preprint arXiv:2405.12775, 2024b.
Hengrui Zhang, Jiani Zhang, Balasubramaniam Srinivasan, Zhengyuan Shen, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-type tabular data synthesis with score- based diffusion in latent space. arXiv preprint arXiv:2310.09663 2023a.
Hengrui Zhang, Liancheng Fang, and Philip S Yu. Unleashing the potential of diffusion models for incomplete data imputation. arXiv preprint arXiv:2405.20690 2024c.
Shichang Zhang. Jiani Zhang. Xiang Song, Soji Adeshina, Da Zheng, Christos Faloutsos, and
Yizhou Sun. Page-link: Path-based graph neural network explanation for heterogeneous link prediction. In Proceedings of the ACM web conference 2023, pp. 3784-3793, 2023b.
Weizhi Zhang, Yangning Li, Yuanchen Bei, Junyu Luo, Guancheng Wan, Liangwei Yang, Chenxuan Xie, Yuyao Yang, Wei-Chieh Huang, Chunyu Miao, et al. From web search towards agentic deep research: Incentivizing search with reasoning agents. arXiv preprint arXiv:2506. 18959, 2025a.
Yanbo Zhang, Sumeer Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, et al. Advancing the scientific method with large language models: From hypothesis to discovery. arXiv preprint arXiv:2505.16477 2025b.
Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. Prototypical informa- tion bottlenecking and disentangling for multimodal cancer survival prediction. arXiv preprint 3xXi::4411001446 2024d.
Mingyang Zhao, Gaofeng Meng, and Dong-Ming Yan. Occlusion-aware non-rigid point cloud reg- istration via unsupervised neural deformation correntropy. In The Thirteenth International Con- ference on Learning Representations, 2025.
Kaixiang Zheng and En-Hui Yang. Knowledge distillation based on transformed teacher matching. arXiv preprint arXiv:2402.1 / 148, 2024.
Zibin Zheng, Kaiwen Ning. Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Ji- achi Chen. survey large language models for code: Evolution, benchmarking, and fu- ture trends. ArXiv, abs/2311. 2023, URL URLnttps//pppp..eeccc...oooooo
CorpusId:265281383
Allan Zhou, Chelsea Finn, and James Harrison. Universal neural functionals. Advances in neural information processing systems, 37:104754-10477, 2024.
Junjie Zhou, Jiao Tang, Yingli Zuo, Peng Wan, Daoqiang Zhang, and Wei Shao. Robust multi- modal survival prediction with the latent differentiation conditional variational autoencoder. arXiv preprint arXiv:2503.0944 2025.
Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. Codebertscore: Evaluating code generation with pretrained models of code. arXiv preprint WrXiv:2302.05522 2023.
Chaoyi Zhu, Jiayi Tang, Jeroen M. Galjaard, Pin- Yu Chen, Robert Birke, Comeelis Bos, and Lydia Y.
Chen. Tabwak: A watermark for tabular diffusion models. In The Thirteenth International Con- ference on Learning Representations, 2025a. URL https: http:////////eeee...omoo net /wt./fffo.oooo.o. 1d=7lpur4y8qs
wtw.:/iwiiiiteeeoruuum forum?
Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without nor- malization. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14901-14911, 2025b.
Minjun Zhu, Qiujic Xie, Yixuan Weng, Jian Wu, Zhen Lin, Linyi Yang, and Yue Zhang. Ai scientists fail without strong implementation capability. arXiv preprint arXiv:2506.01::1 2025c.
Xiangyang Zhu, Renrui Zhang. Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin. Shanghang Zhang, and Peng Gao. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2639-22551 2023.
20

---

Published as a conference paper at ICLR 2026
Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchan- dran. Embedllm: Learning compact representations of large language models. arXiv preprint arXiv:2410.02223, 2024.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint 1xXv:22406.15877, 2024.
Henry Peng Zou, Wei-Chieh Huang, Yaozu Wu, Yankai Chen, Chunyu Miao, Hoang Nguyen, Yue Zhou, Weizhi Zhang, Liancheng Fang, Langzhou He, et al. Llm-based human-agent collaboration and interaction systems: A survey. arXiv preprint arXiv:2505.0077 2025a.
Henry Peng Zou, Wei-Chieh Huang, Yaozu Wu, Yankai Chen, Chunyu Miao, Hoang Nguyen, Yue Zhou, Weizhi Zhang, Liancheng Fang, Langzhou He, et al. A survey on large language model based human-agent systems. arXiv preprint 250:::2550000533, 2025b.
Henry Peng Zou, Wei-Chieh Huang, Yaozu Wu, Chunyu Miao, Dongyuan Li, Aiwei Liu, Yue Zhou, Yankai Chen, Weizhi Zhang. Yangning Li, et al. A call for collaborative intelligence: Why human-agent systems should precede autonomy, arXiv preprint arXiv:2506.09400, 2025c.
21

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
In this work, LLMs were used only to aid or polish writing. Specifically, initial drafts of the manuscript were written by the authors, after which LLMss were employed enhance clarity and richness of expression. All content was subsequently reviewed and revised by the authors, and the final version reflects the authors' own corrections and approval. No parts of the research design, experiments, analysis, or results relied on LLMs.
B BENCHMARK STATISTICS
8.8%
20.6%
8.8%
10.8%
18.6%
14.7%
17.6%
Machine Learning Algorithms Efficiency
Computer Vision Multimodal
Diffusion Generative Model Recommendation Systems
Large Language Models
Time Series Forecasting Sequence Mode ling
Graph Learning & GNN
Figure 3: The domain of the tasks within RECODE-H.
Figure 3 illustrates the distribution of task domains RECODE-H. The benchmark covers diverse range of research areas. This balanced coverage highlights RECODE-H emphasis on capturing the breadth of modern AI research.
C DATA CONTAMINATION ANALYSIS
common concern benchmarks is that the evaluated LLMS may have been exposed to the source papers or repositories during pre-training, particularly for works published before late 2023. To ex- amine whether such contamination materially affects model performance, we analyze results strat- ified by the publication year of the underlying research papers. Our dataset includes tasks from 2023, 2024, and 2025, enabling a coarse but meaningful assessment of year dependent performance trends.
Table 4 reports evaluation metrics across years for seven LLMs. If pre-training contamination were dominant factor, models would be expected to show substantially higher performance 2023 tasks, which are the most likely to appear in pre-training corpora. Instead, we observe the opposite pattern across nearly all models: performance on 2023 tasks is consistently the lowest, 2024 tasks improve markedly. and 2025 tasks often match or exceed 2024 levels despite being too recent to be included any model's training data.
These findings suggest that contamination is unlikely to be primary driver of model performance. Instead, differences across years more plausibly reflect (1) evolving research trends, (2) variation in methodological complexity, and (3) stronger alignment between newer tasks and modern model capabilities. While eliminating contamination entirely is infeasible for frontier models trained on broad web corpora, this analysis indicates that leakage does not explain the performance patterns observed in RECODE-H.
22

---

Published as a conference paper at ICLR 2026
Model
Year
MRR
Recall
Test Case
CodeBLEU
CodeBERT
GPT-5
2023
0.116
0.588
0.806
0.297
0.884
2024
0.230
0.685
0.847
0.322
0.890
2025
0.277
0.833
0.853
0.318
0.888
GPT-S-mini
2023
0.065
0.294
0.659
0.288
0.890
2024
0.187
0.741
0.856
0.306
0.890
2025
0.216
0.733
0.806
0.329
0.889
GPT-5-nano
2023
0.051
0.176
0.490
0.275
0.885
2024
0.097
0.352
0.562
0.301
0.853
2025
0.106
0.433
0.536
0.283
0.849
DeepSeck
2023
0.111
0.588
0.802
0.342
0.901
2024
0.212
0.704
0.817
0.342
0.897
2025
0.234
0.767
0.830
0.343
0.896
Gemini-2.5 Pro
2023
0.127
0.588
0.723
0.332
0.897
2024
0.202
0.574
0.676
0.336
0.891
2025
0.154
0.600
0.740
0.330
0.879
Gemini-2.5 2.5 Flash
2023
0.090
0.471
0.727
0.331
0.903
2024
0.161
0.630
0.782
0.355
0.900
2025
0.128
0.567
0.784
0.354
0.896
Claude
2023
0.087
0.412
0.561
0.336
0.891
2024
0,109
0,500
0.675
0.334
0.894
2025
0.130
0.467
0.622
0.337
0.892
Table 4: Performance of LLMs across paper publication years.
D RECODEAGENT TOOL DESIGN
Recent LLM agent research has rapidly expanded to include advanced agent strategies (Yao et al. 2023 Huang & Caragea 2025| Zou 2025a|c). memory mechanisms(Huung al.) 2026). enhanced reasoning paradigms(Zhang et al. 2025a Yang et al. 2025), and safety controls Huang jet al., 2025). While these directions improve capability, they also introduce architectural complexity and evaluation confounds. In contrast, ReCodeAgen follows minimal design principle, restricting agent to a small, well-defined set of repository interaction tools. This lightweight design reduces hidden state and emergent behaviors, improves reproducibility, and isolates the core problem of iter- ative code refinement under structured tool constraints. ReCodeAgent interacts with the repository through a fixed set of tools that enable file inspection, code retrieval. rewriting, and execution based validation. These tools define the agent's action space and support reproducible multi-turn code refinement.
The READ tool returns the full contents of any repository file, allowing the agent to recover context and understand existing implementations before making changes.
READ.
RETRIEVE.
The RETRIEVE tool retrieves the most relevant functions code regions given natural-language query, enabling the agent to locate where specific functionality is implemented across the repository.
The REPLACE tool rewrites the entire contents of the target file with newly gener- ated code. Although earlier versions of the system also included span-based EDIT tool for detailed modifications, empirical evaluation showed that models frequently misidentified edit boundaries or produced inconsistent patches, resulting in structural corruption. Whole file rewriting via REP LACE consistently yielded more reliable results. Therefore, REP LACE is retained as the sole editing mech- anism.
REPLACE.
SUBMIT. The SUBMIT tool executes the current implementation and returns unit test outcomes and execution logs, providing the diagnostic signals that guide iterative refinement.
The BRONSE tool lists all files and directories in the repository, allowing the agent to inspect project structure and locate relevant modules. TTheee tools support ReCodeAgent's iterative workflow: inspecting repository context, locating rel-
BROWSE,
23

---

Published as a conference paper at ICLR 2026
evant code, applying revisions, and validating correctness. This structured tool interface ensures transparency and reproducibility in evaluating research code development.
E EVALUATION OF GENERATED FEEDBACK
We additionally conduct a small-scale study using real human feedback. We collect 50 intermedi- human and LLM interaction samples in which annotators provide natural language feedback code that fails unit tests, following the same iterative protocol used for LLM augmented feedback. shows that human feedback yields measurable but consistently smaller performance gains compared with LLM-aus ILM-augmented expert feedback. Human annotations are typically partial, noisy and identifying only surface level issues and lacking the deeper diagnostic reasoning or prescriptive correction that LLM augmented feedback provides. This qualitative gap is consistent with prior ob- servations from MINT(Wang et al. 2023b) and Although our benchmark uses simulated expert feedback for reproducibility, we acknowledge that does not capture the full variability of real human feedback, including imprecision, underspec- ification, and heterogeneity in expertise communication style. Our framework in princi- ple incorporate these phenomena by introducing controlled perturbations-sucll as probabilistic emror injection, incomplete messages, stylistic variation, expertise-condiitone feedback distri- autions-within the existing feedhack-sampling pipeline.
et al. 2025).
We restrict human annotation to this analysis due to the high cost and variability of real feedback, which limits scalable and reproducible evaluation. LLM-augmented feedback therefore serves as standardized and repeatable reference for the main benchmark, while extending the framework with more realistic noise remains an important direction for future work.
8.30
LEN
0.20
E.C.
5.324
2.20
02
8.00
8.300
8.2
40
08.5
13
Figure 5: Average pass rate of GPI-5-mini when guided by different feedback models across feedback levels. GPT-5 feedback yields the strongest improvements, particularly at Level
8.c8
Lone
Figure 4: The rise of average pass rate of GPT-5 when guided by GPT-o4-mini and real human feedback.
F ABLATION STUDY ON FEEDBACK MODEL
F.1 FEEDBACK QUALITY
We conduct an ablation study to evaluate how different reasoning models used for feedback gener- ation influence the overall interactive code generation process. Specifically, we test GPT-5-mini as code generator while varying the feedback model among GPT-5, 5, GPT-03, GPT-o3 -rro, and GPT-o4 Gp.-o4-mini.
repporss the average test case pass rates across these settings. Overall,GPTT55mmi achieves pass rates around 30% when guided by external feedback. Among the feedback models, GPT- provides the strongest improvements, yielding the highest pass rate 32%. GPT-o3-pr p-rro and GPT-o4-mini produce comparable results around 30%, while GPT-o3 feedback leads to the lowest performance of 27%.
Figure 5 further examines how pass rates evolve across different feedback guidance levels. The trends for GPT-5, GPT-o3, and GPT-o4 BeT-o4-mini align with those observed in Table 2 Notably, GPT-5 demonstrates clear advantage at Level 4 feedback, achieving significant improvement over
24

---

Published as a conference paper at ICLR 2026
Feedback Model
Pass Rate
GPT-5
0.32
GPT--3
0.27
GPT-o3-pro GPT-o4-min
0.30
0.30
Table 5: Ablation study of average pass rate using different feedback models.
other models. This highlights GPT-5's stronger ability provide precise and corrective code-level feedback that accelerates convergence to correct implementations.
F.2
FEEDBACK COST
Model
Avg. Input Tokens
Avg. Output Tokens
Avg. API Cost(S)
GPT-5
20 098.62 62
7781.97
0.102
GPT-o3
20 099.38
2997.10
0.064
GPT--33-rr
20 089.31
3267 83
0.663
GPT-o4-mini
20 103.95
5060.34
0.044
Table 6: Comparison of average token usage and API cost across different feedback models.
In this section we compare different feedback model in the aspect of cost. T shows that when evaluated on the same tasks, all feedback models process a similar number input tokens, but differ in output size and API cost. GPT-5 provides the most effective feedback but at moderate cost (0.102& 102 per feedback), while 03-pro is disproportionately expensive (0.663 $). In contrast, o3 and especially 04-mini deliver much lower costs, with o4-mini offering the best cost-efficiency.
G DATA LEAKAGE ANALYSIS
In generating feedback, provided the LLMs with canonical code annotated with explicit com- ments to ensure correctness. While necessary for accurate feedback, this setup raises concerns about potential code leakage, particularly at feedback Levels 1-3. these levels, the LLMs are expected to offer natural language guidance for correcting errors rather than reproducing ground truth im- plementations. assess the risk, we sampled 1,000 feedback instances and evaluated leakage by detecting occurrences of ground truth code snippets within the feedback.
Our analysis shows that code leakage at Levels 1-3 is negligible, remaining below 2%. Results across models are summarized in Table
Model
Level
Level 2
Level
Level 4
GPT-5
0
0
0.00
0.020
GPT-o3
0
0
0.01
0,305
GPT-o4-mini
0
0.00
0.195
GPT-o3-pro
0
0.02
0.330
Table Code leakage rates across models. Leakage remains negligible 1%) for Levels 1-3, but
is more pronounced at Level 4 where ground-truth snippets are explicitly provided.
We further analyzed the extent of coverage- that is, the proportion of canonical code revealed in leaked snippets. These results are reported in Table Again, leakage is effectively absent at Levels 1-2, minimal at Level 3, and substantially higher at Level 4.
H EXTENDED ANALYSIS UNDER INCREASED INTERACTION BUDGETS
To further characterize model performance under varying feedback conditions, we extend the maxi- mum number of researcher LLM interaction rounds from 10 15. This extended setting allows us to examine whether remaining errors stem from insufficient feedback or from iterative integration dynamics.
25

---

Published as a conference paper at ICLR 2026
Model
Level
Level 2
Level
Level 4
GPT-5
0
0
0.00
0.245
GPT-o3
0
0.02
0.837
GPT-o4-mini
0
0.00
0.478
GPT-o3-pro
0
0.01
0.676
Table 8: Proportion of canonical code covered by leaked snippets. Leakage coverage is near-zero for Levels 1-3, but increases significantly at Level 4.
Table reportt Recall 15 for GPT-5 across all feedback levels. Performance consistently improves with additional turns, indicating that multi-turn refinement continues to contribute meaningfully beyond the default horizon.
Feedback Level
Recall 15 (GPT-5)
Level 0
0.549
Level
0.779
Level 2
0,880
Level 3
0.963
Level
0.972
Table 9: Recall@ 15 for GPT-5 under different feedback levels.
Lower feedback levels exhibit the largest marginal gains. Level 0, which provides only execution logs, improves substantially but remains constrained by weak error localization. Level and Level benefit from the extended horizon as well, reflecting that coarse and mid-level diagnostic information often requires multiple refinement cycles before errors are resolved.
Higher feedback levels demonstrate accelerated convergence. Level 3 approaches saturation by 15 turns, driven by the effectiveness of targeted natural-language correction. Level 4 reaches near- complete correctness, with remaining discrepancies attributable to multi-step repository integration, such as interface alignment and dependency resolution, rather than missing supervisory information.
The extended turn evaluation indicates that richer feedback enables faster and more reliable conver- gence, while lower levels remain limited by error localization and conceptual understanding chal- lenges.
I ERROR TYPE ANALYSIS
The Table reveals that while all models share common categories of mistakes, the distribution of these errors varies significantly across LLM families, producing distinet patterns of weakness. Broadly, low-level syntax and runtime errors are relatively uncommon, showing that most modern models have surpassed the stage of struggling with surface level coding rules. Instead, the majority of failures stem from higher level challenges, such as faithfully interpreting research descriptions or bridging implicit domain knowledge.
Within the GPT family, model size plays a decisive role. The largest variants, GPT-5 5 and GPT-5-mins, produce relatively few syntax mistakes, with their errors dominated by missing knowledge and instruction misalignment. This shows that scaling tends to reduce shallow failures while pushing the challenge into semantic fidelity. By contrast, GPT-5-nano displays very dif- ferent profile: it is far more prone to low-level bugs and misunderstandings, indicating that smaller models lack the stability and consistency to translate complex descriptions into working code.
The Gemini models demonstrate another interesting split. Gemini-2.5 5-flash resembles larger GPT models, with its main failures concentrated in missing knowledge and context, reflect- ing tendency overlook implicit assumptions repository-specific conventions. In contrast, Gemini-2. is far more vulnerable to instruction misunderstandings, producing errors that stem from misreading misapplying the methodological steps. This divergence between two variants of the same family highlights that architectural or training differences can lead to funda- mentally distinct error patterns.
26

---

Published as a conference paper at ICLR 2026
DeepSeek and Claude each illustrate contrasting limitations. DeepSeek-V3. shows relatively balanced distributions misunderstandinns and knowledge gaps. Llaudd-Sonnee on the other hand, stands out for having the highest rate of syntax and runtime errors among all mod- els. While it shares semantic weaknesses with other L.Ms, its disproportionate low-level fragility undermines reliability and signals weaker baseline robustness in code execution.
Taken together, these patterns emphasize that while the frontier of error types has shifted away from syntax toward semantic fidelity, different model families exhibit characteristic signatures of failure. Some models, like GPT-5, excel at suppressing shallow mistakes yet still falter on implicit domain knowledge, while others, like Gemini- 1i-2. 5-pro or C1 Lauud--Sonnee struggles with faithfully grounding research instructions or ensuring execution stability.
expose deeper
J FEEDBACK ADOPTION ANALYSIS TABLE
Tablc10 provides statistics on feedback adoption and error resolve reate across different feedback levels. TT provides statisties on feedback adoption and error resolve reate across different error types.
Model
Feedback Level
A
NA
AS
AP
ANS
NAS
NASP
NANS
(%)
(%)
(%)
(%)
(%)
(%)
(%)
(%)
GPT-5 GPT-5
80.2
19.8
50.9
5
24.3
0.5
0.5
18.9
93
7
65.4
3,7
23.8
0
0
7
GPT-5
91
66.3
18.6
0.5
GPT-5
91.8
8.2
65
5.5
21.4
0
8.2
GPT-5-mini
68.8
29.4
0.9
30.3
min-5-mini
87.1
12.9
55.4
26.7
0
12.9
GPT-5-mini
96.3
3.7
61.9
10.2
24.2
0
3.7
GPT-5-mini
97.6
2.4
73.8
4.8
19
2.4
Gan-5-nano
45.2
54.8
4.6
10.6
0
0
54.8
Gano
68.1
31.9
38.9
7.4
21.8
0
31.9
GPT-5-nano
68.7
31.3
45.7
5,3
17.7
1.2
30
GPT-5-nano nano
75.6
24.4
45.9
22.7
0
24.4
Gemini-2.5-pro pro
51.5
48.5
36.1
3
12.4
0
0
48.5
Gemini-2.5.4 2.5-pro
65.8
34.2
46.5
3.9
15.4
0
34.2
Demini-2.5-pp
58.5
41.5
35.1
8.3
15.1
O
41.5
Ge5ini-2.5-pro
58.9
41.1
51.3
1.5
6.1
0.5
0
40.6
Gemini-2.5-ffa
80.7
19.3
40.6
7.5
32.5
0
0
19.3
Se5-ai-2
88.4
11.6
46.9
6.3
35.3
0
0
11.6
Demini-2.5--flas)
94.1
65.1
24.7
0.5
95.6
4.4
62.1
5.5
28
0
0
4.4
DeepSeck-V3
75.3
24.7
47.1
7.3
20.8
0
0
24.7
DeepSeek-V2 V3.
87.9
12.1
51.8
7.3
28.7
0
12.1
DeepSeek- V3.1
88.1
11.9
65.2
6.3
16.7
0
0
11.9
DeepSeck-Ve V
86.8
13.2
68.8
12.9
0
13.2
Claude-sonnet-4
64.6
9.7
0.5
0.5
Claude-sonnet-4
71.4
28.6
$7.6
3.3
10.5
0.5
0
28.1
Claude-sonnet-4
67.8
32.2
55.1
3,4
9.3
0.5
0,5
31.2
Claude-sonnet-4
77.6
22.4
68.3
7.3
0.5
22
Table 10: Adoption and error-resolution outcomes across models and guidance levels. Columns
denote: A = adopted (%), NA non-addpted (%), AS adopted and solved (%), AP = adopted and
partially solved (%), ANS adopted and not solved (%), NAS non-adopted and solved (%), NASP
mono-ddoo mon-addpped and partially solved (%), and NANS = non-adopted and not solved (%)
27

---

Published as a conference paper at ICLR 2026
Model
Error Type
A
NA
AS
AP
ANS (%)
NAS
NASP
NANS
(%)
(%)
(%)
(%)
(%)
(%)
(%)
GPT-5
TI
96.7
3.3
72.5
2.2
22
0
3.3
GPT-5
T2
87.9
12.1
52.4
6.6
28.9
0.4
0.4
11.4
GPT-5
89.1
10.9
66.3
10.7
GPT-5
T4
85.7
14.3
68.6
5.7
11.4
0
0
14.3
GPT-5-mini
T1
94.7
5.3
66.3
3.2
25.3
5.3
GPT-5-mini
T2
78.5
21.5
50.2
5.8
22.4
0.9
0
20.6
GPT-5-mini
T3
89
11
56.7
6.2
26.2
0
11
GPT-5-mini
96.1
3,9
9.8
35.3
3.9
GPn-5-nano
69.9
30.1
53.9
0.5
0
29.5
GPT-5-namo
T2
56.1
43.9
30.3
6.4
19.4
0.3
0
43.6
GPT-5-nano
T3
66.7
33.3
40.1
6.2
20.4
0
33.3
GPT-5-nano
65.6
34.4
45.3
1.6
34,4
Gemini-2.5-pra
TI
72.7
27.3
54.7
3.9
14.1
0
27.3
Gemini-2.54 p.5-ppro
44.7
55.3
32.7
55.3
Gemini-2.5-pro
T3
63.7
36.2
44.7
4.4
14.7
0.3
0
35.9
2e5-ni-2.5-pro
T4
77.6
22.4
52.2
7.5
17.9
0
22.4
Gemini-2.54 f.5-llash
TI
93.5
6.5
69.1
2.4
22
0.8
0
5.7
Gemini f.5-
T2
90.5
9.5
51.8
10.6
28.1
0
9.5
Gemini -.5-fllah
T3
88.3
11.7
48.4
4.3
35.6
11.7
Gemini-2.55flas
T4
87.3
12.7
54
6.3
27
0
12.7
SeepSeck-V3.
T1
87.4
12.6
70.5
2.4
14.5
0
12.6
DeepSeck- V3.1
T2
83.3
16.7
54.7
23.6
0
16.7
DeepSeek-Ve 1
83.5
16.5
56.3
19.3
0
16.5
DeepSeek-V3.
13.1
T4
85.3
14.7
48
14.7
22.7
0
14.7
Claude-sonnet-4
79.4
20.6
62.6
13.6
20.1
Claude-sonnet-4
T2
68.2
31.8
55.9
3.1
9.2
0
0
31.8
Claude-sonnet-4
T3
78.8
21.2
67.8
6.2
4.8
1.1
0.4
19.8
Claude-sonnet-4
67.2
32.8
52.5
1.6
31.1
Table 11: Adoption and error-resolution outcomes across models and error type/ Columns denote:
adopted (%) NA = non-adopted (%), AS adopted and solved (%), AP adopted and partially solved (%), ANS adopted and not solved (%), NAS non-adopted and solved (%), NASP non- adopted and partially solved (%), and NANS = non-adopted and not solved (%).
28

---

Published as a conference paper at ICLR 2026
K
PAPER LIST
The paper list are in Tables 12
13
I4 and 15
Paper Title
Venue
1. PEDANTS: Cheap but Effective and Interpretable Answer Equivalence 20244)
EMNLP 2024
al.
2. Differential Transformer (Ye et al. 2024)
ICLR 2025
Mapping the Multiverse of Latent Representations (Wayland et al.
2024)
ICML 2024
4. TABDIFF: A Mixed-Type Diffusion Model for Tabular Data Generation (Shi
ICLR 2025
ct al.
2024)
5. Mapping the Multiverse of Latent Representations (Wayland et al.
2024)
ICML 2024
Task-Agnostic
Machinne-Lamming--Assseee Inference (Miao & Lu 2024
NIPS 2024
Universal Neural Functionals (Zhou al.
2024
NeurIPS 2024
8. Dendritic Integration Inspired Artificial Neural Networks Capture Data Correlation (Liu al. 2023a)
NeurIPS 2024
Transformers without Normalization (Zhu et al.
2025b)
CVPR 2025
SimPO: Simple Preference Optimization with a Reference-Fres
NeurIPS 2024
Reward (Meng et al. (2024)
11. Knowledge Distillation Based on Transformed Teacher Matching (Zheng & Yang, (024)
ICLR 2024
12. Data Unlearning in Diffusion Models (Alberti et al.
2025
ICLR 2025
13. It's Never Too Late: Fusing Acoustic Information into Large Language
ICLR 2024
Models for Automatic Speech Recognition (Chen et al. 2024a
14. GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Experts (Wu et al. 2023)
NeurIPS 2024
15. Balanced Data Sampling for Language Model Training with Clustering (Shao
ACL, 2024
et al. 2024)
16. LESS: Selecting Influential Data for Targeted Instruction Tuning Xia et al.
ICML 2024
2024
17.
Copula Conformal Prediction for Multi-step Time Series Forecasting (Sun &
ICLR 2024
Yu 2022)
18. Conformal Language Model Reasoning with Coherent
ICLR 2025
Factuality (Rubin-Toles et al., 2025)
19. Data-efficient Fine-tuning for LLM-based Recommendation (Lin et al.
SIGIR 2024
2024a)
20. Language Models with Conformal Factuality Guarantees (Mohri & Hashimoto 2024)
ICML2024
21. Universal Neural Functionals (Zhou et al. 2024)
NIPS 2024
22. Frequency-domain MLPs are More Effective Learners in Time Series Forecasting (Yi et al. 2023)
NIPS 2023
23. Time Series is Worth 64 Words: Long-term Forecasting with
ICLR 2023
Transformers NNe
2022)
24. Are Transformers Effective for Time Series Forecasting? ZZng et al. 2023)
AAAI 2023
25. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (Liu et al. 2023c)
ICLR 2024
26. TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (Wang et al. 2024a)
ICLR 2024
Table 12: Paper list part (a)
29

---

Published as a conference paper at ICLR 2026
Paper Title
Venue
27. TimeXer: Empowering Transformers for Time Series Forecasting with
NeurIPS 2024
Exogenous Variables (Wang et al. 2024a)
28. Large Language Models Are Zero-Shar Shot Time Series Forecasters (Gruver et al. 2023)
NIPS 2023
29. ARC: A Generalist Graph Anomaly Detector with In-Context Learning (Liu
NIPS 2024
et al. 2024d)
30. An Efficient Matrix Multiplication Algorithm for Accelerating Inference in
ICML 2025
Binary and Ternary Neural Networks (Dehghankar et al. 2024)
31. FairHash: A Fair and Memory/Time--ffiienn Hashmap (Asudeh et al. 2023)
VLDB 2024
32. MALT Powers Up Adversarial Attacks (Melamed al.
2024)
NeurIPS 2024
33. End-to-end Learnable Clustering for Intent Learning in Recommendation (Liu
NeurIPS 2024
al.
2024e)
34. ContextGNN: Beyond Two-Tower Recommendation Systems Yuan et al.
ICLR 2025
2024)
35. Wukong: Towards Scaling Law for Large-Scale Recommendation Zhang
ICML 2024
cct al. 2024a)
36. Preference Diffusion for Recommendation (Liu et al. 2024c)
ICLR 2025
37. Augmenting Sequential Recommendation with Balanced Relevance and Diversity (Dang et al.
AAAI 2025
2025)
38. Why Does Dropping Edges Usually Outperform Adding Edges in Graph Contrastive Learning? (Xu et al.
AAAI 2025
2025)
39. Simple and Asymmetric Graph Contrastive Learning without
NeurIPS 2023
Augmentations (iaao et al. 2023)
40. Hierarchical Topology Isomorphism Expertise Embedded Graph Contrastive Learning (Li et al. 2024c)
AAAI 2024
41. Simple and Asymmetric Graph Contrastive Learning without
NeurIPS 2023
Augmentations (Xiao et al. 2023)
42. HomoGCL: Rethinking Homophily in Graph Contrastive Learning (Li et al.
KDD 2023
2023
43. SmoothGNN: Smoothing-aware GNN for Unsupervised Node Anomaly Detection (Dong et al. 2024)
WWW 2025
44. Representation Alignment for Generation: Training Diffusion Transformers Is
ICLR 2025
Easier Than You Think (Yu et al. 2024)
45. LoRAMoE: Revolutionizing Mixture of Experts for Maintaining World
ACL 2024
Knowledge in Language Model Alignment (Dou et al.) 2023)
46. Reconstruction vs. Generation: Taming Optimization Dilemma in Latent
CVPR 2025
Diffusion Models (ao et al. 2025)
47, Patch-wise Structural Loss for Time Series Forecasting! (Kudrat et al. 2025)
ICML 2025
48. Parfmeter-Efficient Fine- Tuning with Discrete Fourier Transform (Gao et al.
ICML 2024
2024)
49. Attractor Memory for Long-TTrm Time Series Forecasting: A Chaos Perspective (Hu et al. 2024)
NeurlPS 2024
50. FITS: Modeling Time Series with 10k Parameters (Xu al. 2023)
ICLR 2024
51. Learning Versatile Skills with Curriculum Masking (Tang et al. 2024)
NeurIPS 2024
52. Federated Transformer: Multi-Party Vertical Federated Learning on Practical
NeurIPS 2024
Fuzzily Linked Data (Wu et al. 2024b)
Table 13: Paper list part (b)
30

---

Published as a conference paper at ICLR 2026
Paper Title
Venue
53. In-context Time Series Predictor (Lu et al. 2024)
ICLR 2025
54. Retaining Key Information under High Compression Ratios: Query- -eidee
ACL 2024
Compressor for LLMs (Cao et al. 2024)
55. Estimating Conditional Mutual Information for Dynamic Feature
ICLR 2024
Selection (Gadgil et al. 2023)
56. Deep Temporal Graph Clustering (Liu al. 2023b)
ICLR 2024
57. DigiRL: Training In- Whe- Device-Contro Agents with Autonomous
NeurIPS 2024
Reinforcement Learning (Bai et 2024)
58. Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal
ACI. 2024
Utterances (Zhang et al. 2024b)
59. Occlusion- aware Non-Rigia Point Cloud Registration via Unsupervised
ICLR 2025
Neural Deformation Correntropy (Zhao et al. 2025)
60. Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent
ICLR 2024
Space (Zhang et al. 2023a)
61. ReMasker: Imputing Tabular Data with Masked Autoencoding (Du et al.
ICLR 2024
2023)
62. TabWak: A Watermark for Tabular Diffusion Models (Zhu et al.
2025a)
ICLR 2025
63. TAROT: Targeted Data Selection via Optimal Transport (Feng et al.
2024)
CVPR 2024
Learning Divergence Fields for Shift-Robust Graph Representations et al. 2024a)
(Wu
ICML 2024
65. Supercharging Graph Transformers with Advective Diffusion Wu et al. 2025a
ICML 2025
66. DiffPuter: Empowering Diffusion Models for Missing Data Imputation (Zhang al. 2024c)
ICLR 2025
67. Fair Set Cover (Dehghankar et al. 2025)
KDD 2025
68. The Road Less Scheduled (Defazio et 2024)
Neurips 2024
69. SLAB: Efficient Transformers with Simplified Linear Attention and
ICML 2024
Progressive Re te-parameterized Batch Normalization (Guo et al.) (024)
70. Turning up the Heat: MIN-p Sampling for Creative and Coherent LLM
ICLR 2025
Outputs (Nguyen al. 2024)
71. Data Unlearning in Diffusion Models (Alberti al. 2025)
ICLR 2025
72. Your VIT Secretly an Image Segmentation Model Kerssies et al. 2025)
CVPR 2025
73. TABDIFF: A Mixed-Type Diffusion Model for Tabular Data Generation (Shi
ICLR 2025
et al.
2024)
74. Bipartite Graph Convolutional Hashing for Effective and Efficient Top-N
WWW 2023
Search in Hamming Space (Chen et al. 2023b)
75. Semi-supervised Node Importance Estimation with Informative Distribution
WWW 2025
Modeling for Uncertainty Regularization (Chen et al. (025a)
76. Towards Effective Top-N Hamming Search via Bipartite Graph Contrastive
TKDE 2024
Hashing (Chen et al. 2024b)
77. Hyperbolic Image- Text Representations (Desai et al. 2023)
ICML 2023
78. Modeling Dense Multimodal Interactions Between Biological Pathways and
CVPR 2024
Histology for Survival Prediction (Jaume et al. 2024)
Table 14: Paper list part (c)
31

---

Published as a conference paper at ICLR 2026
Paper Title
Venue
79. Multimodal optimal transport-based co-attention transformer with global
ICCV 2023
structure consistency for survival prediction (Xu & Chen, 2023
80. MoME:
Mixture of Multimodal Experts for Cancer Survival
MICCAI 2024
Prediction (Xiong al. 2024)
81. Prototypical Information Bottlenecking and Disentangling for Multimodal
ICLR 2024
Cancer Survival Prediction (Zhang et al. 2024d)
82. Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathology (Liu et al. 2024a)
ICLR 2025
83. Multimodal Prototyping for Cancer Survival Prediction (Song al.
2024)
ICML 2024
84. Robust Multimodal Survival Prediction with the Latent Differentiation Conditional Variational AutoEncoder (Zhou al. 2025)
CVPR 2025
85. InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning (Qin et al.) 2023)
ICLR 2024
86. UniGAD: Unifying Multi-level Graph Anomaly Detection (Lin et al. 2024b)
NeurlPS 2024
PaGE-LLink: Path-based Graph Neural Network Explanation for Heterogencous Link Prediction (Zhang et al. 2023b)
WWW 2023
88. Rethinking Fair Graph Neural Networks from Ralaninng (Li et al. 2024e)
KDD 2024
89. LightGCL: Simple Yet Effective Graph Contrastive Learning for Recommendation (Cai et al. 2023)
ICLR 2023
90. Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language (Seidl et al. 2023)
ICML 2023
91. What align in multimodal contrastive learning? (Dulumier et al.) 2024)
ICLR 2025
92. AlphaFuse: Learn ID Embeddings for Sequential Recommendation in Null
SIGIR 2025
Space of Language Embeddings (Hu et al. 2025)
93. Beyond Matryoshka: Revisiting Sparse Coding for Adaptive
ICML 2025
Representation (Wen et al.) 2025)
94. EmbedLLM: Learning Compact Representations of Large Language
ICLR 2025
Models (Zhuang et al. (0224)
95. Highly Compressed Tokenizer Can Generate Without Training (Beyer al.
ICML 2025
2025)
96. LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (Liu et al. 2024b)
NeurIPS 2024
97. Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Loce
NeurlPS 2023
Value Regularization (Wang et al. 2023a)
98. Enhancing Large Language Models Coding Through Multi-Perspective Self-Consistency (Huang et al. (0244
ACI. 2024
99. Context Matters: Enhancing Sequential Recommendation with Context-aware Diffusion- based Contrastive Learning (Cui et al.) 2024b)
CIKM 2024
100. Treatment- Aware Hyperbolic Representation Learning for Causal Effect Estimation with Social Networks (Cui al., 2024a)
SDM 2024
101. PoI V2: Prompting CLIP and GPT for Powerful 3D Open-world
ICCV 2023
Learning (Zhu et al. 2023)
102. LLM-Pruner: On the Structural Pruning of Large Language Models (Ma
NeurlPS 2023
al.
2023
Table 15: Paper list part (d)
32

---

Published as a conference paper at ICLR 2026
L CASE STUDY
In this section, we provide a step-by-step demonstration of how feedback progressively guides model-generated code toward the correct canonical implementation. We begin with the formal pa- per description (Fig. 6). which specifies the underlying algorithm and serves as the authoritative reference. From this description, detailed instruction is constructed, translating the theoretical quirements into a precise programming task. The model's initial output, shown as generated code (Fig. 8 captures the general intent but diverges from the canonical code (Fig. 7 in subtle yet im- portant ways, such as the handling of min tokens to keep. To close this gap, the feedback (Fig. produced, which diagnoses the deviation, explains its consequences, and prescribes targeted correction using a scatter-based enforcement strategy. Finally, we present the revised code (Fig. 10 where the integration of feedback yields an implementation that fully aligns with the canonical method.
Paper Description
Formally, at each time step sts, Let #lmachoal (V:5 denote the vocabulary, and SXIX Amid (1:t 1115 represent the conditional probability distribution over the vocabulary for the zoken $x_t8. Min-M sampling works fellows:
Nhegin/crumarsta
Vitem
Maximun Probabfi'ty:1 Tdentify In the distribution, denoted as \mex) (Max_ev Vin \mathcal(VH) Piv Anid x_llit
probabi
-1115.
Vitem (Defte the Truncation Thsesholds) Set base probability threshold, $p_( text (base)) Vin (0, 118, and scale it by (p_(\max]8 to desermine the actusl truncation threshold:
(aggtn(]quation))
P_ (acaled) (Ateet [pase] Vtimes P_i\max)
(end[equation)
This threshold enaures that tokens with sufficiently high relative propabliitiea are considezed while filtering out less probable tokens in context dependent
Vitem (texthf[Define the Sampling Pool:; Construct the sampling pool #lnatheal (VI
min] consisting of takens whesa probabilities text (scsle]]]))
greater than equel $p_1%
[begin[equatLon)
11
(Nveee (min)) = My Vin \mathcal ( Amid x_11t-111) Ageg (Atext
scaled)] \I
12
(endteguaiion)
13
Vitem (textif from the Fool:) sample the next taken $x_t from the reduced set 8% (V) (minx according to their normalized probabilities:
14
Nbeginlequationn
15
p' (x) (2 tv. Amid xill:t 1)) ('eun Vin \mathcal(V) (itext (min] P(v Amid x_
[1:t-1)]] Aquad \text Vin (matheal (V) Ntext (min)
Nenc(eguation]
17
(enumerae)))
Instruction
*nInttruction 1: Implement Weighted PooLing Utilitye
*APanciioaaltty Description:: Create utility function that performs weighted pooling of hidden states within fixed-size windows using Pyrerch tensor operations. the function should aggregate token representations by using "acatter_add" operation efficiently ca calculate a weighted sum for each window. It muat also generate a new attention mask for pooled position considered one tcken in its criginal window was welid. This function directly implements the surmacion part of the weightted pooling operation described in Equation of the paper. Expected Punctien NamezA wix_window_ssee_pooo
the resulting pooled sequence,
++ Imput:*.
"hiddan_statee
Tenaor/e The input tokar representationss with ahape of "( batch_aige, num nin_windows, vindow_size, hidden_size
"attention_mast "ttoroh.Tenso")
indicating valid in original,
windowed sequence.
"weights" "terch.Tensoo") The weighte to apply to each soken zepresentation before summing. Must have the same shape 'tatden_states".
"pooled_iy aad__ddddeen _iddee_atttes"
atates" ("torch.Tensor"1 Tensor"): The reauiting sequence pooled ixpresentations with a shape of (batcch_szee num_wiadows, hidden_size)" ('tooeh.
new attention mask corsesponding
pooled sequence.
Figure A example of paper description and code generation instruction
33

---

Published as a conference paper at ICLR 2026
CANONICAL CODE
Loall (self, input_ids: torch, Toreh.oongTenaor,, scorea: torch, Ploat Tenaor] torch.
PloatTensor:
probs torch (scores,,, (scores, dim--1)
top_probs, grobs.maxjcims--) kaepdineTrua)
aca acaled_nin_s self.min_p top_prabs
Lokena_to_remove probs acaled_minn_
sorted_indicas = torch.argsort (scores, descending-Trus, dima-1)
Reorder the "tokens_to_remeee
maak align with the sorced tokens.
(tokeeettteeesstoo_eeoo self. tow.snittotssttkeeeeeeo indicos_to_rraeee sorted_indicos__t statte.. sorted_indicas
bottee_inddree_tt_rreei
dim=-1, 1, Lndexerorte__iniices
False
10
11
pcoces_ptoceseed scores meorrs. fill (indices remove, sell.fiiler__vluee
12
return acorea_prooesser
Figure 7: Example of generated code
Generated Code
def _call__a________________________ input_ids: BorcheLongeesser, scores: torch.Flost Tensort >> torch. gloattensoo::
if not forch.is_tonann (
raise TypeError ("scores must be torch. Densor")
if (oocesdin() !- 2:
raise ValueEsror (f* (f"sce must be a 20 tenser of shape (batchusize, voceb_size), get shape (scozes.shape)
device scores.device
dtype scores.dtype
bateh_size, vocab acores.ehape
10
probs tordh p_zax, probs max probs.max (dim--1,, kespdime/Izue)
dim--1) shape (batch_eize, vocab_size)
11
P_scaled p_mex float imel..mipppp shape (batch, cokens_te_remore probs p_scaled (batch, vocab), boolean min taaaoo..minooonsttt__oeeooo it k
13
vacab_aize)
15
16
kept_counts ("Lokens_See samee__o__remeee remove) sumidiin (batch,) reed_fix kept_counts
17
(batch,)
18
need fix.__ix.
20
topk_vels, topk_indices = torch. torch.topkiproea. kek, dim--1) + (batch, k)
keep_topk_mes maak torch. tike (tokennnak (wwoeee_toooeaeeeeeeeeeee dtypestorch. dev' ca= device)
22
keep_topk_mssk.saatte
indi_in True) (batch, vocab)
23
24
need idx idx = need_ tog. aqueeze (-1) if fied_fix_idx.nee idx ftxed__i__xxxxxixe Un-remove top-k poaitions for this aequence
1:
26
27
28
Lokens_te_remeve [1xx] s- "oeep_Lopk_makk/aae
29
else:
30
If multiple sequences need fixing, advanced indexing
31
tokena tasena_tt__tt te_reooveaaaaeeeeeee [need_ffx_idiee_eeeeeee_e_e ş= "keep_topk_maak (neaak(need__ie faae__fi__dd____
32
33
processad_acore acores scores.alone if self.filter_wa. value float ("=in(";; fili_val float ("-inE") else:
35
36
37
fil__wal float (self. filter_value)
38
39
processed_ese
processed scorescccccooooooooo Eall
fill_val)
40
41
retura processed_scoree
Figure 8: Example of generated code
34

---

Published as a conference paper at ICLR 2026
Feedback
Feedback Report
#4 Peedback
Feedback
wiitteffaee:** _crll
#4 Description
Lon_toxees___keeee logic deviales
sort/gather/scallee approach conditional top-k mask, which can lead to discrepancies in which tokens are preserved
10 #4 Anslysis
11
The canonical code enforces the "min_tokens_to keep' safeguard by sorting the logits, unmarking the top-k pesiticns, and seattaring baok to the
gatharing remeval original order. This approact. is quaranteed to mirror the paper'a requi in all tie ba:ch cases. generaled code's manual "Lopk"
may select different tokens when probsbilities tie or in edge cases, causing mismatches (as seen in "test_min_tokens_____ooo
iadexing
12
13
#+ Actionable Fredback
14
Replace the existing "min_tokena__ to_keep" 'o' cok with the canonical pattern: sort logits, gather the semoval mask, reset the first "min_tokena_te_ keekeepp" entries, and scatter back the criginal vecabulary
15
Direct Gode Feadback teetx
17
18
In "_aal__', lacste the section starting with the comment 'I Enforce min min_tokeng__to kocpeenn_t__oop""""" replace entirely with:
19
ppythen
implementation Decails: ensure sested_indice
baa__tokes_tt_kee tokens kept iscores, descendingstrue, dims-1)
21
corted_mast toreh.gather (tokens_to, sarran__t____reeeeeee to_reeove, dime-1, index=so (nc)
23
sorted_maek tat_mmii__keea_t___eeo tokens_ta_ remove = satpod_maa.ssatttteeeeeeeeeeeeee sorted_indicas sorted_maak)
False
24
25
26
This aligns the implementstion with the canonicai scatter based method.
Figure 9; Feedback to the generated code.
Result Code
def Loall (self, input_lds: torch, torch, LongTensor, scorea: torch, torch,Float FloatTensort -> torch. FloatTensor:
if not torch. torch..e_tenseerseeeee (sccress:
raise TypeRrror
torch, "ensor")
if (iares.dim() I= 2:
Val waFrror (f*scores must tensor shape (bstch_aize, vocab_size), got
shape (acores.ahape)
device devies.device batch_size, vocab_size scores.shape probs
device (scorea, dim=-1) shape (patch_aize, vocab_size) P_max, arohs.max (dim=-l,, kespdim=Truc) + (batch, 1)
11
P_acaled p_max float isels.min_pp # shape Ibatch, 1)
13
tokena_to_renove probs p_scaled (bstch, vocab), boolean
14
min_keep min
vocsb_size)
15
if min_keep 0:
16
carted_indices torch.argsort iscores, desoendingsTrue, dime-1) $ Ibatch, vocah)
17
sozted_mask Lorch.gather (lokens_too_eemeeeeee dim--1, index=so:ted_ (batch, vocat)
sorted_maskls, smin_keep] Palse
19
rew_tokena_to_remivee sorch.ones_ like (tokeas_to_reee dtypertorch. device- devicel
20
pww_tokens_to_rmmee
sorted_ indiass, sorted_mask]
21
tokens_ta_remant = poe_cokenattorrnoon
22
processed_scoze scores.clone
23
processed_scores scoress wwweee.masseeeed..c....o..o..... masked__ fill (ttoeeesttooreeeeeeeeee float (self, filter_valse
24
25
return proressed_acores
Figure 10: Example of revised code.
35

---

Published as a conference paper at ICLR 2026
M PROMPTS
M.1 CODE GENERATION PROMPT
System Prompt
You
machine learning researcher.
Task description: Your task is to generate high-quelity, vell-documenteé, and contextually relevant code impleatntiiion based the provided LaTeX veraion of paper and a specific user instruction. You will also have access to the exiating code repository enaure your generated
integrates seemlessly.
Commanda actions: You also have
CODE READ TOOL
code reading tool.
This tool allows you to read the centent of any file in the current repesitory. Et helps you before making changes generating new You can zead file using the follewing command: READ READ is the word READ, and <Iile path to be reads is the relative pach of the file want to inspect. This command will return the full content the specified file. this conmend when you need examine the contents of any file in the zepository, including the current target file.
understand the
path
10
11
12
CODE RETRIEVE TOOL
access This tool allows you zetrieve the most selevant code functions for given naturel language query, It helps you understend where and how specific fuactionslity is implemented in the codebase.
retrieval
14
15
You can retrieve code using the *allowing command:
16
RETRIEVE
17
QUERY
RETRIEVE
RETRIEVE, QUERY your request natural language. response will return the relevant functiona, including their code and location in the repository. this will be the primary wey to locate and explcre code related to specific
functionality concepts.
19
Use this command when you want to investigate or modity code related to particular feature, action, or
20
REARITE
22
You also have accesa to code replacing tool.
23
This tool allows you to entirely re-w:ite/roplas :ll of the current coda and erase ail existing code.
24
You can use this Lool via the following command: ""REIIACE
25
26
where REPLACE is the word REPLACE and <cade heres will be the new code that is zeplazing tool useful if you want to make very signi ficant changes, such aa entirely changing the code file content. Try limiting the use of rewrizing and oim for aditing the code more.
the antira set
code.
27
SUBMET 700L,
28
You also have access to code submission tool. executes cusrent code execuzion, including
target and returns results unit tast outcomas and feedback related to the cada' is behavior. can suonit cade vaing the following command:
31
SUBMIT
32
SURMTT is the word SURMET, and it will ron the code surrent ly presens in the target fle. execulion, you receive results any unit tests, along with diagnestie messages or errers thst occurred during cuntime.
33
34
anssesasaaas REPO BRONSE TOOL. c*ssaseaaaa..
35
You also have access to repository browsing tool.
36
This tool allows
browse the entire code zepository associated with the curreat task. It helpa you uncerstand the overall structure, locste files, and explore code across different modules components.
37
You can browze the repository using the following command:
39
#ROMST: is the word BRowSF, and it will return "ist of fi and rectoriiss in the repositozy, This will be useful for understanding how different parts of the codebase are organized and where specific functionslity is implemented.
40
should
access
reterieve
repositozy
information. Before the actions you should relect about the context and make sure the conmand fallowing carrect systex.
42
You should follow this fornat:
43
seflect: [Your reflection on the centext and the action you are going to zake] action:
44
45
[The action you are going to take] reflect should cover aspects:
47
fon Results
36

---

Published as a conference paper at ICLR 2026
49
50
Ciagnose compilatioon or runtime errors (e-g-, ayntax errors, miasing dopendenctes).
52
Inspect Leet outeomes, focusing which tests failed and the cocresponding ezror messages.
53
54
Consider performence-relater aignals if available (e.g., timeouts, memory overuse).
56
2 Code Cons! stency
57
58
Check slignment between the generated code and the method description in the paper.
59
60
Fnoure compatibility with the existing repository (function signatures, class struatures, module dependencies).
62
Maintain coherent structure and style consistent with the project.
63
64
Feedback integration
65
66
rxtract actionable guildnnce human simulated) feedback,
67
68
Identify Logical flaws, missing components, oz suggested improvements.
69
70
Trenslate nstural language feedback into concrete modification strategies.
71
72
4. Fistory Awsreness
73
74
Review previcus attempts to avoid repesting failed solutions.
76
Ideatify patterns in mistakes and refine strategy accordingly,
77
78
5. Next-Step Planning
79
80
Ideatify
stacus.
81
Lecide en pricrity actions (e.g., reading files, searching repasitory, editing
code:.
83
84
cetermine acope changes iminor patch major refactor).
85
86
Identify additional information
geseration.
User prompt
1+f Research Generstion Request
sv]. Relevant LaTeX Contentier
Below you will find the necessary information to generate the requeszed code. Please process all sections carefully.
(latex code;
2. Code Generation Instruction:
10
11
(cede generation instruction)
12
13
Conversation History
15
---
16
(conversation history)
17
18
19
Current Code tmp Itpeettaatiion
20
laurrent code content)
22
23
24
5. Feedback Previous Submition
25
26
[Teedback content]
27
28
29
6. Action Execution Result
30
31
faction execution result)
32
33
37

---

Published as a conference paper at ICLR 2026
M.2 FEEDBACK GENERATION PROMPT
System Prompt
is genezaLe detailed reedback on piece of genarated code. This readback will used to improve future code generation attempte.
expect Analysis &gent.
You will be provided with the follewing information to perform your analysis:
1. +*laTeX Code from Research Paper: This decument for anippet) desoribes the intended target code should implement.
mathemetical sigorithmic funetionality to understand the core logie, equations, and thecretical underpirnings.
*Aeee instruction/Per Prom/t/ This is the original instruction given to the code generation model that produced the 'denerated Code'. Evaluste the generated code aligned with this instruction.
ideal imp emennaatiion desized funct innlloosllyy It contains specific comments highlighting key aspects, logic flows, potential pitfalls. vxGenerated Codesva This is the code produced by another LLM that need to analyse.
ccCaronics
with Commenta:
as
for feedback.
5. AAEroo Information: This is the execution ezror information of the generated information.
10
6.
Sereration Guidance & Feedback Specification:= This document outlines: generation format this Peedbbckk Categories: The predefined categories your feedback should address. v*eeedbeck Granularity:
11
veGnneration
12
level
required for your feedback.
14
Based on the provided information, you must generate your feedback based on the user 15
specification.
16
17
vxYour
18
Carefully compare SiLlN--eeeratee todexv vxcanonical
specified interface, functior, mothoday as instructed in the vaCode Generation Instructiona.
20
21
Your
follows:
22
23
siIdentify differences:Av
24
Explicitly list every relevant difference between the veLLB--enerated Codes and the Cononical CodavA within instructed interface, furcticn, nethod. For each diffferenc,, clearly specify what la present in one verston and absent or different in the other.
25
26
2. AvAnalyze each difference:*s
27
Clearly explain impact of difference why zo an exror context the instruction and the canonical solution (expected code).
28
29
2. A*nnalyee each difference:**
30
For each difference, analyze its significance and datarmina whether it offects correctness completeness, style, performance. Select appropriate reedback category (TO 14) for each difference. Clearly explain the impact of the difference and why it error context axpected code).
canonical
31
32
33
4. rect code-leve feedbsck:*>
34
actionsble iten, provide detalled description of exactly modily LIM Generated Code to zesolve the difference. The code should be identical with the code snippet
35
Outbutt Format:.A
37
Preduce JSON object with ""differences" erray. For esch difference, include the following fields:
intezface, function, methed the diffecence eccuze "cateeslace ""sateozy""; the feedbsck cazegery (I0-14)
38
" "interface"';
39
"description""; brief description
difference
41
"analysia""s Row current inplementation ead to error, why it is correct
""actionable. "etiiiaabblleeeeaaa".".."""""" Clear, concrete, and actionable quidance for cozrection
43
Censistent with the actionable feesback, detailed description of now zo modity the code to resoive the difference. Use the canonical code aniopet as the guidance feedback.
44
*=xample Ou
46
11
47
"diffesences":
48
49
"interfzee"; *caloulata__otal"
"cstegozy"; "11*
51
*description": "The L1M-Generaved code does not initialize the variable 'var' before use
38

---

Published as a conference paper at ICLR 2026
52
"analysis"; generated omits initialization of 'var', which con cause rustime arror incorract resulta.",
cettionable_eeebbokee "Ensure that a.l variables are properly initialized betore they are used.",
54
feedback"; "Add the line 'var [I" before the iteration to initislize the variable shown in the Cononicsl Dodo."
55
56
Add more differences needad
57
58
1:
59
Instructions:
60
Focus all fferencees the instructed code region, not just thoge leading major 61
ecrora,
62
63
Cifferences Include changes in logie, mlasing or additional statements, variable initialization, seturn values, control flow, structure, fanction signatuses, etc.
64
65
Be explicit and systematic for each abserved di difference.
User Prompt
TTaak Information:*
Please generate code analysis feedback based on the following information:
1. *APaper Description (AV >>>lstex
(latex code]
10
2.
Code Seneration Instruction:**
12
(cade generation fon instruction]
13
14
15
3.
+*Caronical Code (Ground Truth/Ban Bathh///xappeeeeeee :**
16
('((]
17
(canoncial code)
19
20 4.
Ge-l--Gaeeatted Code:*.
21
.'([
22
(generated code costent
23
...
24
5.
*sError information:
[exeoution error log]
26
27
28
**Generation Guidance Feedback speciticctionn:*
29
ject: Understanding the 5 Peedback Categories for Code Generationss LiK-generated
ensure
consistently, we This system helps pirpoint the exact nature of an error.
feedback system.
31
32
Our system la based two simple questions:
33
1. *+ahat kind of error ta It?+. (Thia la the +*Gstegory**, ST ST_8-T_49 help
34
provide?>s (This
vxLevels, SL_U-L_49)
35
sastegoriess. of your first step in to identify which of these five categories the smost significant. ecror falla into.
document introduces
anslyzing piece code,
37
38
39
40
*** The 5 Feedback Categeries (ST_0 1_48)
Here is a guide to each category, deal baai to he's quickly classity any error 42
encounter.
43
44
#44 xx$T_3S: Code Structure (Planning) Feedbackvs
45
Mutshell:s. The overall architectural plan is wrong.
46
.*tore Cuestion:s* is the aode's high-level organization structure fundamentally
different the isterded desigr, aven some the internal logic is corract?
47
*4oook for:**
48
single, monelithic Sunction when it should have been broken down into smalles helper zethods.
49
A class that minsing ensential methods required for its core purpose. or high-level implementation strazegy.
Incorrect
51
#4## ++ST_18: Code Correctness Feadback+s
39

---

Published as a conference paper at ICLR 2026
53
Watshell:s* code fundamentally broken won't cun. ++Core Quastion:*s Doed the code fail due to bosic syntax mistake, a type, Fundamental Python error?
54
55
rAoook for:**
56
"'yataxtrroo'' e.g., an uncicsed parenthesis or unterminazed string.
57
"KaneError's 6.g., uning variable before it has been assigned.
58
"TypeError"s e.g., trying to add a atring to an integer (when not part of the core algerithn's lagic).
59
#44 AxST_28; Implementatio: Alignment Feedback
61
raln Nutshell:a The code ignores misinterprets the provided peper instructions.
62
..ror Question:*e can point to a specific sentence, formula, or requizement in the provided that this code directly contradicts?
63
++ookk for:**
Implementing Equation
paper Equation (3) was specified.
65
Using the wrong pärameter namea in function call compared to the instructions.
66
Producing an cutput with the wrong shape or data type described in the paper.
67
68
# +- 35: Knowledge context Foedback*>
69
Nutahell The code fafla because mlasing orveial knowledge provided.
generai domain exper.ise, or la it implicit "secret" of the original codebase that wasn't written down?
Question:**
something
would need
71
Look for:**
72
issing placeholder logic for a complex but atandard operation (e.g., "7000: implament topology calculation here*),
73
Needing a specific, unstated hyperpazameter le.g., learning rate of '0.001"), specific, somein-standard library (e.g-r 'gudhi', the obvious choice.
74
Using generis library
buggingface")
75
76
# ++$T_48: Repository Integration Foedback..
77
TT Nutahell :*. The code reinvents the whee or fails to use existing tools from the
codebase.
78
.+rore Question:*s Did the code rewzite a helper function or class that was already awailable
project's existing files?
79
Loook for:**
80
writing a new 'normalizeteext function when nww.meeleee (1 already exists. Incorreatly uaing provided class from anather module in the repository.
81
82
Ignoring the establianed coding style or conventions of the repository.
83
84
86
#+ How Choose the Right Category: Decision Guide
87
88
ways
the of this Hat. The retst "Yas" determines error's category. Thia helps distinguiah between similar issues (especially $T_2$ and $7_3$),
89
90
1. *xis it basic systax/muntime erzoriex
91
vaYosPAx +> Te's 1
92
93
2.
++oo? okay, doen it ignore or re-inplement existing code from the repo?** **$7_45**,
94
<*Yos?**
95
96
overall code architesture or structure vaYos?xx +> It's sxST_05xx.
problem?*s
97
99
++No? Con find the fix explicitty written- in the oo ++Yos?** +> Tt'a **ST_28** (The TMM fat led to read careful ly).
100
102
5. *all? Is the fix based on knowledge souteidey the paper/ing /is:::iuttttinss
.. >> **$e_3g*e, (The lacked nesessary consext).
104
Please generate the feedback with the specified format.
M.3 CATEGORY CLASSIFICATION PROMPT
System Prompt
You are expert code reviewer and research enginees.
Your
Classify Teedback about code into ecror categories T41-
Judge whether the next version of the code (v2) adopted the feedback and whether the issue resolved.
Follow definitions and disambiquation strictly.
Provide evidence (<m3 items) location references.
If unsure, use "#uncertain"".
40

---

Published as a conference paper at ICLR 2026
ingut specification
10 The user prompt will provide the following blocka of content (each delimized clearly):
12
1. x* (PAER)) Excerpts from the research peper Imay include formulas, equations, section reEs).
13
2. [INSTRUCTION] The instruction given generate cade based
papes.
14 3. ** [GROUNn__RRTEE Baa____RR_____________ (optionaal
first generated reference or canonical code implemmntationn version.
[Vi__coE]]
16
5. ** [V1 [V1_RON_TGG]]
execution results and error loga for vt.
17
6. [FEEDSACK] xv Feedback on vi (may contain multiple feedback enicies).
18
[V2_CODB] The generated code (after applying feedback).
19 8. ** [[22_KJ__L00]
executicn results and error logs for v2.
20
21
14 Brcor Types
error because the high-level design, architeature data flow, modularization, atrategy not aligned with the canonical intended design.
Code Structure |Planning)
23
+1i1: Code Correctness The
because of general syntex, runtime, basic
programming login errors.
24
because code implemented with misalignmeat Lo the paper alçoritha, Cormulas, or inatruction I/0 requirements. ezror because missing author preference.
ementation AligmmentAv
25
Knowledge
knowledge, conventions,
implicit assumptions, *AI4: Repository Integrationas The ezror is because not using the same function defined within this repo (other sile), misuse of existing codebase (from other file), or reimplementing pers.
26
27
28
#4 Disambiguation Guidelines #ill 12+.:
30
If the issue would arise in any code regarcless of the paper T1. stens from not following the paper's elgerithm or 170 specs >> 12.
31
32
33
+* vs. T2++:
34
Arractural/acchileettaal guidance, modularization.. 12 algorittam.cc/eehhhaalooog.. alignment with the paper.
35
37
.I T0Ax:
38
repository-specific (reuse of helpers, placement, convencions from other file). general dasign/arcchttacuurs not tied repo aasets.
39
40
41
(Key Rule)**;
42
Assign v*13.v when the feedback relies on rexternal implicit knowledge, author instruction, repo, paper. Examples: demain-stenãard library usage, interpreting ambiguous applying auther praffecnnce/nniiihe or common domain practices, filling in paper onissions. feedoack ba
praforencs/inst inniighht/////ttt in cadav not fully contained
43
44
solely tixing syntax/zudt imee >> Tl, aligning with explicit paper restructuring mndules/classes -> I0,
45
46
or instruction specification 12,
47
revaing repo asseta -> otherwise, if it depends Avdomain expertise implicit assumptiensuv, classify 13.
49
50
51
Evaluation Pipeline Specification
52
When analyzing inputs, fellow these ordered steps:
53
54
1. <*Porse Inputvi
Collect paper excerpta, code generation Instruction, ground trush (I. anyl, v1 code lags, feedback text, and v2 code logs.
56
Split multi-point feedback into atomic itema.
57
58
2. s+Classify Error Type A
59
the follswing
rules:
60
AVTO Strmuttrr//Paanntie kx: Feedback is about shou code is organized or architecteds, e. 9., "this logic should be a class method, *aplit into functiona," "refactor data flow
61
x+T1 (Correctness) Feedback is about agenrrio programming bugse like syntax errors, misuse, bed Loop conditions,
Lype mismatches, paper content.
of boands. relation
++
(implementation Alignment)**: Feedback about enot matching the paper wrong formula, wrong output shape va. paper, incorrect gorithm step. T5 (Knowledge Context) Peedback requires +domain knowledge, implicit assunptions, sibraar..unetten conventioas- to understand, e.g., "use library. functien for NLP task," " "by atcention" they mean Scaled Doo-Froduct," "paper onita activation X.*
apec
63
64
AvI (Repesitozy Integration) Feedback e.g., "use utils.vatsiodie instead of reinglementing," "place logic in Rodel. forward, "use config object."
rusing ez misusing existing zepo
65
66
3. Judge Adoption (adopted) vl and v2.
67
68
II v2 reflects meaningiul changes zelated to the feedback "YES". change '80".
41

---

Published as a conference paper at ICLR 2026
partielly implemented "PARTLAL".
71
72
**Judge Resolution (resolved)
73
Check
outputs, and aliganent with papes/instruetions. if the probiem is fully fixed -> "YES".
74
75
still prasent "NO"
76
Tf partially Improved -> "PARTTAL".
78
5. **xxplain afener
Write a concise explanation f1-3 sentences) citing evidence te.g., code diIr, log 1ine, paper refereace).
80
81
CaConfidance
82
Assign float 0-1 reflecting ceztainty in claasification and judgments. certoin.
Nigher more certain,
84
85
7. v=Qutputv
86
Return results the specified output format aeror_type: Errortype Fieldl.... cescription= adopted: AdoptStatus Field(..., desc "MT0://PTT00 resolved: AdopLölatus Tieldl.... wxplsin_esror__yee str Fieldf.... mingtength=,, tex_Leng:h=000, deccrippionn
87
88
89
the reason the errer is calegorized to this ercor type.")
91
sggleinnddoptee_sslveee stz Field(.... min_length-l, lexgthhtt--000, description-"
Describe the the feedback adopted not and the error resolvad") confidence_soone score: conflost (ge=0.0, le=1.0) Field(..., desoription="0-11)
92
93
94
y5
96
#4 Important Notes:
97
*he canonical code meaning in feedbacks mentioned in the feedback is the expcted
generated code (is the TROUNE_TRUTR___ in the prempt) for the task, but veis the repo code... You need to anslyata the roct cause of the error and catacory to the calegory.
98
Output content..:
the errer runtime if the errer is in category 2, the explain type should explain how the minstruction paper+* ia specified and the code miaaligned with the paper description apeci
category 1, explain_error__ype should explain syntex errer
100
category 3, explain kype should explain implemented and what information is not specified in the **papes instructienv.. the errer category 4, the explain_error_tiie should explain function that defined the repository misused used it expected.
errer
is
User Prompt
given following
Please analyze and cutput structured JBON according to the schema above.
[PAPER]
6
[papes content)
[INSTRUCTION]
(cede generation instruction]
11
(CROUND_TRUUR_._C_ [RUTH__OOE]]
Iground truth iontalll)
13
14 [M]_00DB]
15
(genecaled code vl)
16
[vi_RUN_LCC)
18
[v] code execution fon log)
20 [FEEDBACK]
21
[Teddaakk contenta)
22
23 [v2_coua]
24
(revi code v2)
25
26
[V2 12__RNLLOC
27
[v2 code execution
28
Important Notes:
30
++The canonical aode moaning in faedbacks mentioned in the foodback in the expcted
generated code (1s the CROUND_TRUTEE CoDR. the user prompt) for the task, but not some
42

---

Published as a conference paper at ICLR 2026
exisits repo. You need enalysis cause eætecory to the correct category.
N ANNOTATION EXAMPLE
N. PAPER DESCRIPTION
ssec (Probble Formulstions, sabel:sec:baccgroumnc)
Ysubsect lon (Secting)
Se rocus en statistical inference problems for the parameter Vitheta" lequiv Utheta"/x) mPthbb(P)) vin (RKKKA) defined the joint distributien of ((XX, Aain \nathbb(P)M) where VY vin \mstheal/YJV is scalar outaoma and V/X Vin [X]] be a (XV-- dimenatonal
ing features. We using labeled data (icalL - MBXI, Y_i,, 1 1, (odots,aN) lequiv (AXL, MYL),, unlabeled data \ MMcald MNXL, ML model (Swh [(odott) (matheal(X) Arightarrow (matheal Heer,, (I is black-bos box function with unknown operating characteristice and can be mis- salsssppeeieddd We also require algorithm (leAM that
interested eat MAtheta"
ati, acdo lequiv AXV,,
pre-trained
the labeled data V(calLA) and constatent and asymptaticallyy normally distributed astimator V(lwddehat (1theta/ for VA theta"*\), There three (beginiitemize)
waya
estimate \(thetta*>))
apply algorithm (Acal) only labeled \(lealL (AX,, W,),, and returns the estimator and its estimated variance V/oah bi_NeslL, WwhaayVer Corvar//Ww /whhbe_icelll the asymptotic distribation of the estimator.
\texthf stacisticsl
Velid confidence intervels and hypothesis tests
then be construeted it ignorea the unlabel data and MT. prediction.
Vitem (Ipouaatoon
metheds) treat HL predicticn Vowh EV) in the unlabeled data observed outcoma, and apply algorithm V\aM)) to ('ealu (uxu, \aul V. Ne denote the estinator and estimated variance as M[WWhhtbmeta_/aa Aoalt, \ah (aa)) hmeta_/caluj] IV). This has been shown to give invalid inference results and Selse scientific findings Veitop wawgg000mmtthod,,ooooooooo wotp./ooooooooooo2220aeeeeoe miac2025aegumption ion, miao2t24valid:
Vitem \textbī assisted
Milaall and M\cal1V) as input. These approaches add debiasing term in the loss function ler estimating equation) M-estimation problams, this remeving bias from imputarion-besed estimators and producing results that are statiatically valld and universal ly more powerful compared to classical mathoda miso2023asaumpt mlaa2024val
methods;
10
Nend Vend{ilemize) [emiize)
we example previde intuition aL-assisted
protoco..
12
Yaubseat (Roilding the intuition mean matior)
14
Me consider the nean estimation problen, where \itheta" \E[YAA Vequly \argmin Atheta (roc(11(2) (211lett tittttttttaaarighhe " The classical method only takes the labeled data conalatent estimator VAtheta">>y
V(VYA) Input and yields unbiased
15
11
Wahttheta_teall targmin_lshet Atheta U. The imposetion-baaed mathed only takes the unlcbeled data ((MfU)) as input and returns
Ameann \
17
\1
19
Wehleta_leaau Vargmin_Atheta MIaa([I: \eun_(i~ai1)" (niN) \[rac(11:2) (1uh E_A Atheta) \frac(1] comm_(1ent)) (" [n-K]] \wh f_t
20
model Vowh TV) is -specified, To address this, Ml-assisted estimator takes both labeled data ((YL, \ELEM debiasing
blased inconsietent estimator
\(AB(Y_11
unlabeled data MAEUU) as imput
function
rectify the bias caused by Mr impotatioo cattep/angeloooi 222023redd
miao2023asaurption http:/assuurtiooooaaooe22202 wiw.2024aa22aaaaammmm (an.:0/2wvaad.ccnn2222dddeeeee. can2023reddeiiion
21
\begin;smell)
22
\beginial
23
Mitttheta_//Lextttt IMLA)
24
&=
Nargmin_lther \frac[1) (2)
26
27
Wollomege__0aaae O'meenN (1wh t_i Atheta)
28
29
Aunderbraceff:
wh whllomega_0 V'nesan (Awh =_1 (thetah
31
Amaann (y-hNchhttorrihht)
33
(xxtt term)) MAY
34
35
36
Wwhlomega_ Nahoomga_0lmeznNii OveeznWieh f_i
37
kunderbrece;
38
twheoomgg__0 O'mmaan \wh f_i -imeann y_f.
I. Ntext (neblaaing term]],
43

---

Published as a conference paper at ICLR 2026
41
\end small]
where the modifled loas enaures the consistency of the Mirassisted eatimacor and the weight <(Whhoomege_ "frect (W (1531////aaaaaar (h(f)//a (Eh ensures that Mi-assisted estimater is no less efficient than the classicat estimstor with arbierory ML predictions:
43
1
\myVar htwaaaaataataaaeeeeeeeeaaee (MLA]
45
46
VnyVaz hwwithheaaaaaalll
47
48
\frec ffacc/cpperatooname/Coo [X, (widehat Voperatername (Ver) (\widehst (f)] ==)) \eperatorname
Var (Wrldahst(f]] N)
49
Yleq
\myVar wwwhhhhttatttaall V).
51
52
Our proposed method is motivated by the observation that the \texthr/sufficien: statistics) of the Nu-assisted estimator Vishttheta_ (MLA) and its estinated veriance Vowh) myVar (wwhhhhhtta (MGA)) (M are the following summory statistios:
53
54
\wh Abt_/ttexiLLees
55
56
('nean y_i, \meann \wh I_i, AmeanN Ash (_1) / and
57
58
\whlmyvar (tt_(())oxtt((ss
59
60
\oegin(bmateelt
61
Whivari [X)/n (M, / 5 OV)
Wwhccovi \/b/E)1/n (atwaal/WW((E)//n o (wtwaru(Wwb/33// Vend(bmatrix:
63
64
65
66
67
Moreover, Lhey can essily obcained by applying the [LexCb[[same algorithm) MAAA))
estimation) to
(begin(itemizen [irensep=lpt, parsep=0pt]
69
Vitem labeled data with observed outcome \( \CA(/YL)
70
71
Nighharroww (tthitheta_/eaall
72
73
l'zesnn y_i, Wallaarl
74
V)
75
Aitem laboled (CA(VEL)
predicted autcome V
76
77
Nrightarrow
78
[Nwnteta (allmytar (Wahteta (Yoalt)))
79
Iineann Swh [i,, (whlvacl[Nva:[]]/ VI
81
litem unlebeled date with predicted outcomel (ACA()EU) \rightarrow [twhhees_//esit)) AoEA)
myVar (aatrtaaaaaaaae
82
83
(Amann bah fLi, lah)voru (ch)voro[\ww]/]]] (Lwh()))/R) V)
84
85
Vitem bootstrap of labeled data MACAI/VY, \IL) q- Adots, @IV for estimation of V(whhmyyor (ww.yyooorwweeeeeeee (twblthhta_ (thleta__(call)) \whiecvl [2, Woh/E)1/aN), sere, MON YL, (fL) LMM represents the ViqU/tt hootstrap of labeled data.
\end (temize)
87
88
Combi ng these aummary atatiatica for one-atep deblaaing V/lhloorgga Swuhiommga_hhhittta.ee Mwhhataa (eal1)) (weh) omega_ WWiieetaaeaaaa Wwhttheta_/eeal//
89
summarize, algorithm
resempling, sufficient assiazed mean astimatior. This abservation inspired to generalize this protocol for broad range taaks, our protecol illustrated in Verefttigsworktlok only requires three steps: 1) using
mean estimation, coupled
trained ML model to predict outcomes for labeled and unlabeled data, 2) applying existing aralyais routines to generate summary statistica, and 3) using statistics deblasing proceduze produce statistically valid assiszed infereace.
in ML
92
\hegin/rigurel
93
Noentering
94
inccudeeraghhic [width 0.8511inewidsh; |Pigures/PSPS. wwtp:/ssSSSS.........o.o
95
(oork of \PSPs" for Task Agnoatio ML Aseisted Inference. (fig:wwkkkoo
97
\end figure;
99
ssect (Kethods:
102
\subsect Leneettionneeee General protecol for \PSeS)
44

---

Published as a conference paper at ICLR 2026
suilding on vereefpactibachesoond we formalized our protocol \eref(figrworkkflo)
assiazed inference:
tbeginvalgorithmi [R]
105
(eeption( For XL assisted inference)
106
boggtn(algorsttmicc/l
108
Require pre-trained Mr model M/wh rV, lane led data Vlcaat (ATT, XMT Mr, unlabeled
M/eall (X01)
109
Ascate Use the Mr. model V/lh "y to predict t the outcome in both labeled and unlabe ed data. ASLate Apply the algorithm V(CA in the analysis routine to
111
lbegin
112
litem lebeled date A(AX, AXL/V and chtain Mila be_icslt, luhlmyvar (luh (et_lcslu)
1V
113
Vitem labeled data A(AX, MLDEM and obtain M//Mh Ameet__caal, lah ('wh \ cal1)1M
114
Vitem unlabeled data with (1(129 ArDF)) and cacain Wil lealt, tunkyyVar (Awt V lcala
115
Vitem VIQU beotstrap of labeled data MI(LL AYL, (EL)_g, q (dots, and obtain (haakyytov keslt,,,, lah Vameta
116
/endy itemt \State Employ one step debiasing to the summazy statistics in step2: \begin(center!
117
119
120
\btpsp - 1oh Vachemeega__ Oitrans wit (baeeta ('wh Wbbbebameaaa_tteeee \ L'cealL i/h/ (btaaa)))
V
122
centecenter)
123
whece (lahhbonegga_ - (WhhlayVar \bmeta_lcalI) 1 WallaVarr (wwh tbmeta_ (caiC) ] 1) Awh)
mylov ibsli la \bmeta_lcalL/ end 1 Aohhayyaa (bbppsps) Vahlmyvar \bt_\ call) WhhlayCov (wwh lcall,, \eh temeta
Vaa (1uh lometa_loall) \b=_)call, lwh \bmeta \eall)
11- Vensure Mi-zanisted point estimator (Abtpaps standard error MAggrt (wwhlmyaar ((trpppa (Abtpeps) JM, (halphal)-laws confidence interval for the \IkA]-th coardinate A Amathoa (C) Apa a_11 Valpha 2) Vagrt (Wwbkayaa Natpssps [_kk 111)), and sided) p-value V 11- 1Phitl VEzaci Waal (PPPPSPP__KK H Hagrrt (oggr((Ve) ('wh)) 1M, where (iPhiiM
WahlyyVar (luh
124
alpha, (MPPP))
(bopppppp dissribution. lendlelgorihhmic) (and
the standard
125
126
127
only requirements our protecol are: i) sigorithm \0cMM applied Labeled \((bb\/aaaL, M,, returns a consistent and asymptotically normally distributed estimater ( if) labaled unlsbeled data are independent and identically distributed. Under these assumptions, the summary statistics have the following asymptotic properties:
129
\hegin(align]
130
Mlabeliegreste
11 2/\1eft (abes)n(arrss))
132
\hh (Mh(a)__/caal)) - Vot*v V\
(Neaetaa__/ \breta W
134
Sh [Neeta) ibmeta V.
Nand(array/\rightt \ooD \mothcal Meftt|
136
Aleft (bbggnn/arrayy
137
10 AA
139
Lend(array//rrigne)
141
142
llett ( (bbegin/arrayl [coc]
Wriwh AbC_Acaltie Whwh Aot_AcaiL, twh Vaneta Vameta_lealti lealt: \OVA
144
Wi/M \ee_NcelL, \wh (bmeta & (wuh Lcalta & 10 A1
10
\rholvrlsh (bmeta_lcalu) (arrayyy
146
\r,
148
(aiggn)
149
Vin (RRTE)) defined on VXX, D) Aain /mathbb/Pt fIV, M/W/tcdst: denotes the asysptetic variance and coveriance of estimater, and Virho \érae(n) (M) esymptatic approximation gives
((bmeta Vequiv Abneta (aaathb
151
(wwh \ot_lealT) Napprox mlmytar/1ah \bt tealt,,, Wrlwh \bt lealt, \wh Abmeta_loall \ approx olayCov (Awh Abt_lealL, leh Vaseta laseta_lcalbh, tealti, WWWh Vaneta \ealL) \approx nlayVaz Net Nemesa_lcalL) V and ((W/W (bmeta_icalu) Napprex BlnyVar (Iwa (bmeta_lcalU) V., Bere, do not require Arwa Memetallcalty and A/lwh \bmeta_YoalUS) to be consistent for \Abb "A)), sllows arbitrary ML model.
152
With the sunmory statistics following multivariate normal discribucion asymptoticalisg the (Wwh
debisaed satimator ABbbpppa Vatmega Laboomgga_Xttrnns// Oxtranswwhi Abmeta)_[\cain) arens Nbmeta) (eaal) asympsotically normally distributed (Xcref (tmrof[thm:assyoor)) conslatent estimator for its asymptotic variance 1/W/\btppps \approx alayVaz (bbtpaps) M valid confideace interval and hypothesis testing can achieved.
is consistent for \(itheta"e) and
Therefore, by plugging in
45

---

is Published as a conference paper at ICLR 2026
155
\hegin/renark]
\PSP8"La more "Laak-agnostic" than existing methods in three aspects:
157
tbegintenumeratee
158
Vitem for K-estimaticn tesks, cursently, only mean and quancile estimation, as well as linear, logistic, and Peisson regression, have been implemented in softwara tools and are ready for inmediate application for other M-estimat fon tasks, task-specific functions and saymptetie veriance I limit theorem are necessary. After that, researchers still need to develop software packages and ept eptimization. algorithms to earry out real applications In contrast, \ PSPS only requires already implemented elgorithms and seftware designed for classical inference based on labeled data.
derivation the Mirassistad
the central
159
Vitem For prablema that
fall under M-astimation but have asymprotioal normo lly distributed est imators, only \PSPS"can be applied, and all current methods would fail The principles behind Mi-assisted M-estimation de not extend to these tacks.
160
Vitem Even for K-estimation taska that have already been implementec, \PSPS offers the additional advantage of zelying solely on summary statistics. The "Lask specific desivaziens" refer not only statistical tasks but also to scientific tasks. Real- world dsta analysia in any scientific discipling often involves nonventions and example, our work partly that require careful consideration. Siatistically, GMAS is a lineaz notivated by EMAS regressicn that regresses an outcome on many genetic variants. While based atatiatical foundation is simple, conducting valid GKAS requires account for numerous technical issues, such as sample relstedaess (L.e., study participants may be genetically related) and populatien the some ancestry are both gereticslly and phenotypicallly similar, creating
regression-
(i.e., unrelated individusls of
contounded associations in developed to address these complex issues Stttepppasscchu2222eomuaaaat reimplemented
Sophisticated algorithma and software have been
It
will very challenging
all these important features need researchers can utilize
ML assisted 3MAB framework, With our 1PS25"protoce existing tools that are highly optimized for genetic applications to perform ML- assisted GMAS, This adaptability is just lizited CHAS, but is major feature researchera to conduct Mi-
of our approach acroas actentific domaing. assisted inference using established data analysis roatines.
161
(endeenume
Land(remarks
163
164
bneginiremaris
165
The "faderated data analysis' feature of rases'rafors the fact that only require aurmary statistica as input for inference, rather than individua-leee raw data ( SYS:. For example, consider scenario where lsbeled
features and in one center and unlabeled data is in another, yet researchers cannet access individual-leval data from both centars similtaneuaal. Under such conditions, current Mi-assisted inference, which relies accessing both labeled and unlabeled deta to minimize a joint lass function, is not feasible. However, \PSES sircumvents this isaue by aggregating summary statiatics from multiple centers, thereby performing statistical inference while upholding the privacy of individual data.
-evell level
166
\end semark)
168
\subsection i (heeretical guarantees)
169
In this section, we examina the theoretical properties of \(PPSSS). In what follows, M) denozes convergence probability and \ \ooob)) denotes convergence in distribution. All procfs are deferred to the \creefapp:procf).
171
The first rasult shews that our preposad estimator is consistens, acymptocically normally ristributed, and unt formly better in terma of element-wise atymptotie vartance compared with the classical estimstor based labaled data only.
172
(hegtn(theormm
173
(abell
174
Assuming equation" "guatiin"Negreeeeeeee "egreffes:estee: holds, then (ibtpaps YeaP \bt" and
M
176
n1 / 2) Notppsss - (be"v) Voon (M)ttca
177
178
11
179
where ((Wr(btpppp) = Wriun Aot_leatii - Wrlwh Vot_lcalt, \wh Vbmeca_ loals Aerans Wi/Wh
ameta_Aeall)) WehoWiila (cal)) 1/W/Wh tht leall, \bmeta_NeaiL)) Assume the (0VV-th columa of (/W Vat_ lua (cmet/ is not a zero vector at least one MAV/lh \bmeta_lcaltii) positive definite, then \cW//aatppps (kk) Aleq Wrlwh [ok]5)_(KK]V) With V/Wwhvr bipspa) \ooP W(bbtpsps), Alim (Pathbb(P) "Mthetak"" (leetar) Min Imatheal::O__Naaphhaa -11P5P9))=I-ialphel)
V/Wriish \bmata_Vealu) are
180
tend(theores)
182
( V can be obtained by applying the algebraic form of \/W V using the bootstrap astimators for MWW/Wh Vet_icall), Wrlch thmeta_\ealls Wrluh \ht_l \uh poreta_lcalti,. A) and \AW(W. Aometallcaln V. The regularity conditiona for consistent bootstrap varience estimation are outlined in Theoren 3.10 (i) of \eitep! shac2012jackknife) also refer readers to bootatrap based variance provides valid but potentially conservative infereace.
which showed that
46

---

Published as a conference paper at ICLR 2026
This result indicates that greater reduction variance associated with larger values of (/Veluu iet_lea,t, Wih ammetallols//1 and smaller valuea of VAWAW \bmeta_ M, 1/W/iwh lbmeta_loalt)y and MArtoo), The variance reduction term (Awh \bt_leall, Wi (imeta_lcall) Atranst WWWh \bmeta_Veal)) W(ls Vaneta_ ) W-1 let_lcole, \wh (tmera_lcall)) (_ikk) can also serve as metric for selesting the optimal ML model in Mi-asaisted inforenes.
Mi-assisted estimator
Arho
185
three existing methoda, i.0., (textt=(PEI), Stexttt (PPI++), and textt: (PSPAI, are asymptoticall equiva snrivalent lent to \PSPS"with different wet weighting matrices. A broader class for consistent estinator of (AbE">A is ((Wh)(b (ibomega: Abonega V ( (Nell)) (bbenega Virans wWb(tbt]//csslu where (Vbomegga is (KXtimes RM) matrix. The consistency of (owh(bbt) (Abomega) for VB bt"xl] only requires \(boomega (ahh [tttmmma__///alli \wh [tmesta mathbr(0])). Since \/Wwh (tmetta assigning
next result
187
\oop
\wh/ MMtAAmeta)_[/aalll \coP (mathb
188
Arbitrarily fixed weights for will satisfy the condition. Rowever, the choice of weights influences the efficiency of the estimator as illustrated in Veref(prop:efE) later.
189
190
pueeiniproppsitiinn
abel [propsequal]
192
Asauming equation "egre""eeeeeeeeeeeeee and reqularity condition for the asymplotic normallty of current ML-assisted estimater holds. For any M-estination problem,
193
have
194
195
(Errac(1) Mbb//bb//leettt (dagg//oooggg
(oveighe) lth//btis(ttexxtaomm
|itexstt (PSPAA))) NooD
196
197
a" (erac(1):2 (owh) (eee) (alaag//bomeen (twemega (1cex)tt(22-+))) \eou V0,
Nt//bbii//ttxxtnermaal (Mt. YeeD
198
a (M /twp////////e/.e..ee Meett (Ndiag (11) (oVeight)
199
10.
200
AI
201
Here, \Aboonega_M \Lextnormallleee
Aeettoocaeett 1), Adols, Vonega Mtxxttrraaa elel, X}] Strans Vin ABR"XA] and /texseccceeeetmma/eeeeeeee kJ minimizing the Sk8-th diagonel slement
seslar used minimize the trace of MW WWWWhh (AbE) (\bomega)) and ((CW)) is matrix associated with the second derivatives of the loss function in M-estimation, with further details deferred (apeproo))....
(Nwh(Nat) (Nbomega) [Nmmega_(tr)
203
propossttit
205
This demonstrates that for M-essinatiion problems, our method is asymptoticelly equivaient to \taxtet with the respective waights M(dizgg)) /twpe/egaa
cextt=(PPPA] (PFI+)) Domega_\textnermallee (ele))
ACV, and /Advay Therefara, \FSPS" can be viewed as a generalization of these existing methods.
206
207
Our third zeault shows that the weights used in the
Veref (prop: are not optimal. Instead, our choice V\bomega_0V represents optimal smoath combination of X /heelll whh//tmera)_ (Ncall)), Whh [bneta) terma minimising the asymptotia variance, while still preserving conststancy.
209
210
/heetnipropasttionn
211
1 (progsetf]
212
Suppose (a (1/2) (Ve Neaill,, W |_ (Ncall), MaeNbbetal_//al100 (bL**) AcoD
(nathcal \bSigma_g) Nigh) is smooth function, then (AbSigma_igg_ Ageg
(NPPs__(ssPPP___ troposttoo)
215
216
Together with Veref (prop:equal), our rssulta demonstrate that our protocol provides a more esceiination problems. Purthermaze, the applicability of cur protocel is not limited to M-estimation and only requires summary statistics as imput. alse indicates that in setting federated data analysia leitepi Bordda2211oommmmooooi Bcommunication) where individnal-leeel data are not available, provee to be the optimal approach for combining shared summary statistics.
efficient estinater compared existing aethoda
217
\beginiremar?
219
(texctt (citep/angriogeuuoz2Ze... Miodaaiiooooooo222pppi employs a pewer-tuning scalar for variance reduction Mi-easisted inference. This sezlar obtained by minimizing the trece or possibly other scalarization of the iattorr'''s varlance-cavarlane asymp=otic variance of \PSPS"1s always equal to or smaller than that (texttL (PPI isrespective of the scalerization chosen researchers. This adventage arises because PSPs'utilizes SK \times KS power tuning matrix, Sibonegas, for varlance reduction, parameters. This matrix facilitates information sharing across diffarent perameter cacrdinates, thereby enhancing astimaticn precision. The cheice of waighting matrix \PSPS alse allows for element-wica variance reduction, recucing each diaçonal element of the variance-oovarilanee matrix. In contraat, the single scalar in (2exI cen only target overall trace reduction or variance reduction of specific element. deteiled example provided in \cref(app:ooi++). Only in one dimensional paramele: estination Lasks, such as mesn estimation, (exILL and Aeses exhibit
natrix. Rowever, the
where S8S zepresents
dimensionality
asymprotic variance.
47

---

K-dimensional point Published as a conference paper at ICLR 2026
Yend remark;
N.2 CANONICAL CODE WITH COMMENTS
inport nungy 48 up import pendas from scipy.stets import momm
def papa last_ eat_ est_unlab_yhat, Sigma, alpha-0.05) .n*
Performa Post-Prediction Sumeary-taatisties-eeee (PSPS) inference baaed on the paper
"Task Agnostic Kachine Learning Assisted Inferense" (Miao s Lu, a:XIv:2105.2003333)
10
This function implements Algorithm 1 from the paper to combine astimates derived Mi-pprediated outcomes unlabeled data (using ML predicted outcomes That) Lo produce a more efficient estimator
11
labeled data (using true outcomes
12
13
X-dimensicnal paramete: thetar. It leverages summary statistics (point eatimates and their variance covariince matrix) input.
14
15
16
Functionality and Correspendence Paper:
17
Inputa correspond to summary statistias descrisad in Algorithm 1, Scep 2: Shhat(Athetal (tteeaaaaaitttttllllllllll (b) estinate using observed Y in labeled date.
18
"eat_lab_y"s Corresponds
"est_lab_yhat" corresponds to (aaatt//eee___/ttteesastt
k-dimensionsl point
estinate using predicted $ (EJS in labeled data.
20
"est_unled" "est_unleb_yhat" Cerresponds to
the K-dimensional point astimore using pradictad (hhat(ff8 in unlabeled data.
21
"Sigma": Corresponds to the 3x 38 nt. varlance-covarlange matrix of the estinators
22
"T, \hat/ (U])" The structure assumes independence between labeled and unlabeled data sets (covariance between es:inatora from different sets is zerol,
( Mhat
23
24
consistent with the asymptotic variance structure in comment notes
the furction
25
asymptotic variance scaled by n. "alpha": significance level for confidence interwels (e.g., 0.05 for 954 ci).
"Sigma" the
variance,
26
27
calculations implement Algorithm 1, Step 3: "K": Dimension of the parameter theta.
28
29
'v_est" Extracta 5Var
(L)))S.
30
SCow (aeat)
[t]],, (hat/ (TT)) (Ahat.
31
"v_eta_lab's Extracts 8Vaz (aaaat/larra___aaattcaaall
32
"v_eta_unlab': Extracts 9Vaz
33
"V": caleulates svar (Nhat [Xetaa
Varr (Nar (
34
"omega_0": Computes the optinal weight matrix. Solves $V \omega_B r&, ylelding
35
$tomaga_0"r a (Var Laaat///ee____atttaaa 11)1 Cov Netat
Vahat /e (t
Aast (ttthttt___Ammattttatt (T)
36
This "omega_0" computed by "ap.linelg.soiveit =1") is transpose of 6ihat (S (Vamega)__ defined in Algorithm Step 3 (Shhat (omegaa 0 [Var
Var (Vhat/ (hat()e
(U]: 1) (or.ilaet(Nletaaalllaaaaeee
(
38
"eat": Calculates 28PS
estimate $that [tthetaa_ (the machcal[L]] tomega_0 (hhat/eetai_i (Amathcaalaal)
where "omega_0"
variable computed
39
Substituting "omaga_9" (the coda's variable) = $ (emeog)lmm paper) this becones
40
$ (hatt/thetaa_
(hat('omegaa_O0 paper] "T which matches
the formula Algorithm step
41
sstendard_errors Calculates the standard errers for each element of Sthat//theta// [PSPS]S.
42
Vomega computes $1sgrt ag (Var Athhtai__ Amachoal (machhal (T:])
"I 18, where "omega_D" is the variable computed in the code
43
This calcuiation cocresponds to taking the square root of the diagonal elemeats
44
of the PSPS variance matrix $1hat [var] (Ahat [csheta as defined in Algorithm Step
Theoren 1.
45
ISee Discrepancy section below!.
46
"Lower_ei" "upper_ci", "p_values"; Calculate Wald type confidence intervels end two sided p-values
47
based on the asymptotic normality of $1hat theta)_ established
48
in Theorem 1.
48

---

Published as a conference paper at ICLR 2026
Discrepancy between
Papez:
51
Variance Calculation Detail: The code colauletes the variance reduction cerm as 'np.dot omega_9.3, x) watch correspords to Shomega_D'T r Vaatt 110 [hetaa)_(mathhalll
Cov 'TS. The variance formuls in the peper (Algorithm 1, Step 3, theorem 1) involves 60ev (that(tttheaaa (tthetai (1a "I") (aat (1 Cov"TS and SCov"T Vhat (_ pa,ee_,, are generally equal only if K-t lacalar case) or under specific symeetry conditicns on the covariance matrix sCov (Atheta I_ L'Imahhca [T)) \ htl.mmmmaaaaaaaaaaaaa calculated atandazd ezrors night differ slightly Irom a direct implementation of the (E>1).
Ahat /eaa//eeaaa/aaaaaalalaaae
Mhat (eeta) terms, @hbat / (omeggg paper)
which may not hold for x>1, This, the
paper's varience fermala for multivariate
52
53
Tmp ommentatioon Betaill Explicit in Paper's Algorithm 1: Raynototic Normality Assumption: The calcuiation of confidence intervals I'norm.ppte ppt") p-valuas ("norm.af") explicitly ralies on assumption that ESPS estimater la asymptotically normally distributed. This is justicled by Theorem 1 in the Input Metrix Structure: The code assunes the input "Sigma" correctly zepresents the matrix with covarience between estimators derived from independent labeled and unlabelad datasets. Matrix Invartibl
54
55
block veriance-covarien
56
natrix (8Var ('hat (etaa) is lavertible, which is necessacy Lo solve for "omega_0". Theorem also requires corcesponding azynptotis veriance natrix to be invertible.
inplic (1matheaa[1)H] Vaz (tmaaeaaaammtthhalaaaaa
57
Output Pornat: The function conveniently zeturns results in pendas DataFrame, common practice not spenified in the paper's elgorithm description.
58
59
Parameters:
60
est_lab_y: & dimensional array of point estimates using Y in labeled data (9Khat (Maaal0(Mmhhhh (L)]8).
61
est_iab_yhat: & dimensionsl acray point estimates using That in labeled data (9Ahat (Netai_
62
aat_unlcb_yhet: k-dimensional arrey of point astimates using that in unlabeled data (hatt'leta__i [mmaa (U)))).
63
Signas
variance covariance
stacked vector above three level alpha confidence intervals.
estimazors.
64
alpha: Specifies
65
66
Returns:
67
pandas DatcFrame containing the PaPs summary statiatics: "Escimate' (6Vhat (\theta|_ PSPSI5),
68
'Std.Erroz' ($1aqrt (diag ('hat (1iig//attt (Var) (Nat (XL 'Lowez.CI', 'Uppez.CI', and 'g.velue'.
70
len ars. sigms_matteeee = np.array (Sigma)
71
72
components from the varlance-oovarlane Corresponds to Vaz (theta_I) in paper notation
$1 gma
73
74
v_sst 1ilgma_matrix(0::5 Corresponds to Coy (ata_L, theta_a) cov (theta__,, cta_L)"r D:K)
75
Sigma Sigma_matrix wtt.mmaaattiiiaaaaaaaaa Corresponds to Var (eta_4)
77
78
>_ata_lab = Sigma 1asga___mtrii(:2--1,,, K:2+K)
79
Corresponds to (eta_t)
80
v_eta_unleb Sigma_natceix22455
2x8:348]
82
calculate v = Varleta_1) (araeta
v_sta_lab v_eta_unlab
84
85
Caloul ate the optinal weight matrix (omega C in coda = omega_0egap.o pTpppr"")) omega_0"T
86
emega_0"T [-[-1)
87
omaga_0 np. www.iiigg.soveeil
89
calculate PSPS point estimate: est theta_t. + omaga omaga____ppen poper'T (ata_ eta_ Here, omega_D is K&K (omega_0_ p_aperr" (eta_ D eta_ I) Rxl vector"
90
91
est est_lab_y dot temega_0, lest_unled__hatt est_lab__yat!!
92
93
Calcuiate
of
estimate: (ar(theta_PPPS) (Shelaa onega_0
Note: see discrepency discussion
regarding paper's
(ar(thets_L) Covl
theta_T, eta_t) 'T omega_9_papes
95
var ww__est_ssps_mtted
v_est np.dot (omega_ 2!
96
Ensure the veriance is non-negative due to potential aumerical issues
97
Taking diagonal before clipping to hancle the matrix vs vector case correctly variances np, diag lvar_est__pe_eeeeeee sear___ss_sss_______ mapss_mtriik] standard_errors sp. (variances)
98
99
101
Calculate confidence intervals using Normal approximation (Juscified by Theorem 1) z_score po slpha / 2)
lowes_ci est z_score standasdeerrors upper_ci est s_score elcode
104
49

---

Published as a conference paper at ICLR 2026
Calculate p-values thets_k using approximation Avoid division by zaro standard erro"
107
PL p_values np,full np.nan) + Initialize with Nall
109
je__ceco_se__ndiee stendard_erzoos (fffffafffll .eps Indices where SE is effectively nen-zero
110
if ipp.anyinon_erro____aaao
2_stat rр.abs lest [non_ zero_se_______ (e__sse_nddccs) standard_errors errors [non__ero_se___eeeee____________ (erc_se_tnctcee P_values (oo__
norm.sf(z_stat)
113
Create output Lable
115
output_table pd.Dataframe
"Estimate': est,
117
'Std.Brrar'; atandard_arrars,
118
'Lower.0''; lower_ck,
'Usper.ct'; upper_ci,
120
'P.value:; P_values
1)
122
Add matedata attribute to the BataFrane output_table.att
124
'papa summary' retuzn output_Lable
125
N.3
UNIT TEST CODE
class
def cetupisele)
**"Set up test fixtures.
(scalar pazameter)
self.est_leb_y..e - [0.5]
self.est_lab__hess
[0.6]
selffest_wm wtt.e__iiaa_hhht_______ = [0.55]
self.sigma.ki I[1.0, 0.2, 0.0],
[0.2, 1.1, 0.0],
11
[0.0, 0.0, 1.2]]
self.elpha.kl
13
Data (vector poramster)
15
self.eat_lab_y__ [0.3, 0.3] self.est_leke wtt.eert_ttb_ht__kkeek. [9.6, 0.25] celf.eat_innabb_hhtt_t (0.35,
16
17
18
v_est_k2 I[1.0, 0.11, [0.1, 0.911
20
cov_thi_stei_ke [[0.2, 0.05], [0.03, 0.15]] cov thw_atat__tht_ke [[0.2, 0.03], (0.05, 0.15]] v_eta_lab_k2 [11.1, 0.15), [0.15, 1.01: v_eta_unlab_ke = ((1.2, 3.12), [0.12, 1.21]
22
23
24
25
self, Signa x2
26
[v_est_ka k2 [0] (_e]t, cov_thi_etal_k2211100, cov_thi_etai (1),
0.0, 0.0],
[v_eat_k2 k2 [0], v_est_k2([][[)) cov_tht cov_tht_etat_ (1),
0.0, 0.0], (oov nov_ettaf_tht_kka [3], cov_etat_thi_k2100000 k_ata_lah_k2( v_eta_'ab__
28
[0] J.0, J.0], @hy_etta__th__2[111
29
cor_etsi_thh_ 1_2___a________________000 v_ete_iab_k2
[1] [1], 3.0, 0.01, [0.0, 0.0, 0.0, D.C. [ats__nnab_k2(0]]]] v_ata_unlah__211 [1] [0.0, 0.0, 0.0, D.C. un__a___aaa_a2 _2
30
32
33
self.alpha_k2 0.05
34
35
self.est _sl____e_y_kll_ee_e_
[0.1]
helf.est_leb._o. wtwplleb_______eeeeece _i_at_k [9.2]
37
hat__jinab__hha__ae = (G.15)
38
self.Bigna 110.01, 01, 0.5, 0.0],
39
[0.5, 1.0, 0.01.
40
10.0, 0.0, 1.0]]
self.eloha_kleede
42
44
def _rwu__and_ommorretselll est__ eet_lab_yhat, est_unlab_yhat, Signa, alpha,
45
46
"*"Hepper method est jrt_lbb___err = np.array
both functions and compare their Datarrame outputs. an
47
pat_leb_ybat_arr = ap.arraylest__eb_ee
49
est_unlab_phat_arr -..np.arr sigma_arr ap..rrayySiee (81gma)
50

---

Published as a conference paper at ICLR 2026
51
52
cononical_ outpptt = Bacpnicco__________
e est est_lab__hat__rr, eat_inlab_yhat_aree Bigma_art, siphe
54
55
56
oontrt genorated
57
est [___lab_y_arg est est_lah_yhat_ars, est_inlab_ynst_ar sigma_ers, alpha
59
61
if expe:t_nan__e:
aasf.aaserriiua (canenical_output I' Stror Arror").isnuli .isnull .)ll(),
63
*Canoni cal Std. Std.Error Frror should MaN for this adge case.") aelI.asaertTrue (generated_ "canerated Std.Error should
64
Btzorrror' J.innll ().l1) this adge case.")
66
67
Frame, Erame__uuaeeeeellea__a (generated_ouuputt cenerical_output, check_dcyperirue,
68
69
stcl=le-7,
70
atol=1e-7]
71
def test_kl_ese Bec__kl_eeqivvleeeeeeee (self):
73
"*"Test ionalional equivalence for K=1 (scalar parameter)... an. "**
74
75
lab_y_ki, wel.f.es__lab.hattto self.est_.mlla__hor_....
76
self selff5igma_kl, self.alpho_kt
77
78
def test_k2_ "ett:KLLoqiivalenncttel....
80
"*"Test functional equivalence for K=2 (vector pazameter)... a*.
81
82
solf.est_lab_y lab_y_k2, self.est_las yhat self.est..mlab_hhot... salf.51gma k2, self.al pha_k2
83
84
85
def test gee__kl_differennn.aphaaae
87
n*nlest functional equivalence for K-1 with different alpha. ""
88
89
self.est_lab._ lab_y_k1, laif self.ast..ml:bb_hht__ aelf.5igma_k1, 3.1
90
92
93
test test_kl_ppd _i__dgee_case cadge_case_negativve wegatiiw_.orranneeeeee
variance (self):
94
a*nSest K=1 case where Var (theta_PPPSPP) might negative, leading NaN Std.Error. "*" with (innrld "inalid==iggooe') ltt___uunann__onnappee _on_porr self._ jell.eet_lbb_y__L_eee
95
(invalldd'
96
97
98
Saff.sigma_kl_ede self.allph__K__dgg.
expect_nan, sxpeat_nan_se--rran
100
def wtw.smodeeeeeeeeooooooeoooeeo lemmondeee_deiieeeeee wet_teed_dppds____ee_ee (self))
103
nenzest K-1 with inputs Frem statsmodels GLMs and boots:cap, zeading data from C3V."* "**
105
semiings.simllfiii (' ('nnner'', convergencewarssing!
sarnings. Iter iinore'', Perfect Separatt Use-Narning for Farfectsoparation varnings.atmpleflli ('ignore', Runt imeMarnin))
107
109
* 1. Load data from CBV files fil__ihe_peth "..//../estt_data/ anlab_fill__ate
111
113
114
lab_at rd. read_eav Lav Ilab_ hat__tiee_paa_aee plleeppthh) unlab_df pd.read_esv (ualab, File__ot_______
116
118
2. Pit logistic regreaston models and extract the 'x' coefricient try:
120
modei_lab_y glm('y dasamleb_dd, family=sm. families Lemiliies.Bioniill .Et (disp-0)
121
est est_iab___ecclla: model_ Lab__llbbyy..aaal y.p.aaaa...eee (1X', 0.0)
123
modei_ Modei_labb_hatt gim(" x data=lab_df, familymsm.famii dammlies..innmiiie.. sinomial ()) ()).. ffit
disp=0)
124
est set_iab_yhat_scalaat model Laa_yhatt parama (1X', 0.01
126
modei_unabb_yhat glm('yhat datemunlab_ds, family lis.iiiissi
fit (disp=0)
ae._ualeb-vha_eccale
unleb unlee_yhth_ htw.hiab_hhaaappaaa.ggroe params. (I2", 0.0)
51

---

Published as a conference paper at ICLR 2026
ver_unlab,_o yhat model_unnab. wht.uunnaabbaaaaaooeeoaea wet.gatt...gte.te..eeeeeooe...ee ('x', 0.1).+2
129
except (np. wwwallin.gga.a. L.ndlgrrrro,,,, Porfactseparationeers
Exception) as Catching a broader range exceptions that might occur during ritting self.ekiplest (f (f"GIM fitting failed for initial models: (e). Skipping teat.")
131
return
133
134
eat lab_y @est_la__y_aclll est est_leb_yhes [est_lab__hat sealaa est_unlah_yhat
136
138
# 3. Bootstrap for coveriance calculation = 50 1_lab (enblab_df, r.p.radon.
140
143
eat est_lab_y__oa np. nр.nan)
est_jab__hat__ yhat np. ap.nan)
145
successful_Joor_Eits
147
:
148
ioo__indiees - ap..andea. cheice (lab_df dr, index, sisera_lab, replave=T:ue)
boot_lsb_di lab (boot_ wtt.://eeeeeeee....eeeeeeeeeeeee
150
try:
152
fit_y glm('y x', dataeboot_ family=sm. stnmoii sinomial .i).fit (disp =0)
eat_lab_y_boot [i] tit_y.paraas.get est_lab_y_ecalln
134
fit_ glm( x', lata-boot_lab_de
families.mm......mm Eamilies..innmiall Binomial ()).
fit (diap=3)
est Bi__iaa_yhat_boottie
fit_ghht..parmm..eee. get ('X', est est_labLyhat_sceiar
157
srrceessful_bctt__ee
except (op. ttp.l
Sepreet Exception):
eat_Iab_y_boot[]] est__aa____calle
160
est_lab_yhat_boott.3 = ess__ab._yattsscllee
162
if succeseful_boct_Hee < /:
print (E*AnNrrnin ttest_statsmcdaa_dee derived inrpvee Only
succesaful satpll_oot__ffaallaaaaaaaa boatstrop fits auccseded. Signa matrix might be leas reliable.")
165
valid_ bata np.vstack boby est ast_lab laatthattt__haaa boot]] valid garid_boot_ddas. valid_boct_dott
aay
167
valid_boct sot.t_ccdtt.caapee11
169
pov_matrix_ie np.array(! [0.1, 0.0" [0.01, 0.1]]I
170
print ([aMaaaiig krest_atatsmodell letessmmoeedeeeeeeeeeeeeeeeeeeee der deriiee__ar indddieed__pt__alll Not enough valid bootstrap
samples Hlalid shape (11)) compute reliable covariance. Using fallback.")
172
cov_natrix_lat lab (o.coo/vaaid__boooddaaa
174
Signa matrix
175
Signa - up. zeros ( (3,, 3)
coy_matrix shape (2,2): sigma [0:2, [0:2, 0:2] = nov_macsix_lab
177
else:
179
sigma [0:2, 0:2] mp. arroy 0.01], [0.01, 0.111) Failback print ("Anaarning bootstrap not Using fallback.")
180
wed__eputa__apteeeeee inputa_kl)s k1) Covarlance matrix from
181
Signa (2, var_unlab_yhha
183
algha 0.05
185
186
# 5. Run and compaee suli.__ru__ed__coppprr
188
est_lably, est eat_lab_yhat, est_uniab_yhat, Sigma, alpha
190
warnings. d'ffault', ConvergenceMarninng) warnings awwpingg.eimiletiiii inplefilter ('default', Per SerfeelSepaca LonBeror) warnings.einilefiltee default', imnKarnin))
193
195
197
it name
main
IE running the file dizecsly:
Ensure canonical and the CSV data files are at d unittest.main()
and generated__pps.p in the python path
200
52

---

Published as a conference paper at ICLR 2026
N.4
CODE GENERATION INSTRUCTION
have instruction for code generation tack:
*vFunction Name:v*
peps
A-Fuctionaliit Description:
implement the Post-Pradiction tumarryatarlartice-bbaaa (PSPS) inferance procedure detailed in Algorithm
paper "Task-Agrostic Machine-tearminggkkkii Inference" acXiv:2405.200380 This furctien combines statistical estimates derived from labeled (using true predieted outcomes) outcomea) to calculate more statistically efficien: escimate for a k-dinensioncl perameter, $1theta"+8. The function operates on summary ststistics (pcint estinates and their joint varianco-sovarianee matrix) provided input.
(using predisted
*<Input/Output Specificationss
11
vaimput:ay
13
"est_lab_y" (floatt): sist representing the K-dimanstonal point estimate vector
abtained using true outcomes in the labeled dataset (9that (naathal
14
"est_lab_yhat" (liat
list representing the K-dimensional point estimate vector obtained using Mi-predicted cutcomes in the labeled dataset (5%hat (VetalL
m2)hcal([))3).
15
"est_ rest_unlb__yhat (fist(fleat)): list representing the x-dimensicnal point estimate
vector obtained using Mi-pradictad outcomes the unlabeled dataset (Sthat/
(
"Sigma" (list [Lat(oiiatf 3K x 3K List lists zepresenting the joint variance of the atacked estimatora S[Nhat[\eheta]] ( (\matheal Abat Neta [_(\mscheall() hhat:/eeaa__Naaathaallll (U) "T)" It assumes independence between labeled unlabeled (zero covarience between estimators
covarlance
different sets). "alpha" (flozt): the significance level used for calculating confidence intervels (e.g.r interval). Refaulta to 0.05.
17
0.05
18
19
Cutput:..
20
following columna: A ""sttinate" The PSPS point estinates (#Ahat (Mthataaa_aaaaaaPe "rsdd.
"pandas. DataFrame"
21
element (tbatt\thhetal staadard ecrors ((1-\alpha) confidence intervais.
23
"Lover,CI" "I'": The lower bounds for the
24
"ropper,CT The upper bounds for the $11-Valpha)s confidence interva 8.
25
"P.vvauu!'ss The twe aided values for testing the null that each element of theta"+$ + is zero.
53


Reviews:
[{"review_id": "QjwWJltmt7", "reviewer": "Reviewer_6h4u", "summary": "This paper presents RECODE, a benchmark for 102 code-generation tasks, based on LaTeX snippets from recent ML papers. They introduce a ReAct-style multi-turn ReCodeAgent that interacts with a code repo and receives feedback from a simulated expert (similar to LLM as a judge). The authors find that richer feedback substantially improves performance, though with diminishing returns. The paper also analyzes error types and how effectively models adopt feedback.", "questions": "See above weaknesses.\nI am curious if you had results with a single-turn format. It would be interesting to see if the multi-turn format gives a big lift.\nMinor typos/issues: Fig 7 and 8 have same captions", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The empirical findings are valuable to broader coding agents: early feedback levels yield large improvements, while higher levels show smaller gains; and that most model failures stem from misunderstanding the paper or repo semantics.\n2. The benchmark targets realistic, interactive research-coding scenarios, which are increasingly relevant as LLM agents enter scientific domains.\n3. Solid engineering effort, clear and intuitive definitions and hierarchies of feedback", "weaknesses": "1. It is not clear how the papers were selected and the list of papers is not disclosed. How do you ensure that evaluated LLMs haven’t already seen the source repositories during pretraining?\n2. There is no code editing tool in the system prompt. The replace tool replaces the entire file content, but the prompt mentions \"aim for editing the code more\" -- it is not clear how this is implemented.\n2. LLM-generated feedback is clean and complete; real human feedback is often noisy or partial. But results with human-generated feedback on these tasks are missing.\n2. Prior benchmarks like SciReplicate-bench and ResearchCodeBench already explore research-code generation; the paper needs a clearer statement of what’s uniquely new here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ckhuPbXbL", "reviewer": "Reviewer_AynR", "summary": "The authors propose RECODE, a new benchmark for generating research code in the novel setting of multi-turn interactions with a (simulated) human, which iteratively provides hints/details/corrections to help steer the system. They show that interactive feedback helps, while also highlighting that models still struggle.", "questions": "See weaknesses. Also:\n - Isn't level 4 feedback basically giving the system the answer? Why doesn't the coding agents score 100% as a result? Adding some discussion around this would be interesting, in particular that \"coding\" requires more than just knowing lines of code. What would you need to include in a \"level 5\" feedback to ensure the coding agents did score 100%?\n - Do your results suggest any advice for what *kinds* of feedback people should be giving to their coding agents, i.e., provide insights that make interactive coding agents more usable/effective?\n  - The Conclusion seems pretty weak, surely there's more to conclude than just that LLMs continue to face coding challenges? What are the big insights you found?\n   - Building/exanding this benchmark looks very labor intensive. Can you think of ways you might reduce the cost / semi-automate the process to extend it further?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Introducing the interactive multi-turn angle is novel and refreshing, and adds some extra realism to the coding challenge\n - novel framing, with levels of feedback\n - benchmark looks like a useful product of a lot of hard work\n - evaluation is thorough", "weaknesses": "- The interactive feedback is (for reproducibility) actually simulated so has a degree of artificality associated with it (even though humans helped create it). In what ways does this setup differ from a (truely) real setup with actual humans, e.g., noise, imprecision, etc. Could you include any of those elements in your framework?\n - The conclusions from the experiments seem obvious (e.g., more feedback helps). What did you learn that was surprising/interesting/informative? It seems to me that should be more nuanced findings. The error analysis is perhaps more informative. If you were to advise future researchers on where to invest their energy in building better coding agents, what would you tell them based on the learnings from your work?\n\nMinor:\n - \"evaluates LLMs through multi-turn interacations with human feedback\" - makes it sound like there's a human in the eval loop. Perhaps \"(pre-collected) human feedback\" or \"(simulated) human feedback\" or something else to indicate the actual eval is fully automated.\n - Figure 2 would be more readable showing % rather than fractionals (e.g., \"1.4\" rather than \"0.014\" etc.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5OLrCy7VId", "reviewer": "Reviewer_61B7", "summary": "This paper presents RECODE, a benchmark of 102 research coding tasks in AI/ML to study the interactions between LLM-based agents and user feedback. The user feedbacks are simulated with LLMs and controlled to have five different granularity levels of information. Experimental results show that LLMs benefit from additional feedback, especially straightforward ones, but still struggle with those requiring deeper understanding of the research tasks.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper adds a nice new dimension around user feedback to the evaluation of agents for research coding, which is helpful and adheres to the real-world use cases of such agents.\n2. The paper presents a reasonable amount of analysis of experimental results. The error analysis and feedback adoption analysis are interesting and may facilitate future research.", "weaknesses": "1. While it is appreciated that the authors assembled a team of 26 annotators, their roles in the entire annotation process are not very clear to me. It seems that LLMs (Gemini 2.5 Pro and GPT-4o-mini) are used to perform many annotations. What are the jobs of the annotators and how their involvement ensures the benchmark’s quality and real-world utility?\n2. Relatedly, the quality of LLM-generated unit tests is unclear and not thoroughly discussed in the paper, which is critical to the reliability of the experimental results. It is also unclear to me how the “unit tests” can be leveraged to evaluate some tasks, such as training machine learning models or analyzing data. \n3. The authors should list and cite all the papers adapted. Meanwhile, discussions and tests of data contamination are also missing.\n4. The validity of “level 4” feedback, which provides ground truth code, may not be a valid setting since it directly “supervises” the code generation process and plays an exceptionally helpful role for most models. The evaluations would be more sound and clean by just stopping at “level 3.”\n5. Some example tasks in the appendix would be appreciated to help the benchmark description be more grounded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3XYF47Se8Q", "reviewer": "Reviewer_oRzg", "summary": "In this paper, the authors present a new benchmark called RECODE, which is a set of 102 code execution tasks from research papers and their respective repositories. Their dataset is built via a clever combination of expert human annotation effort (using PhD-level experts to select the target code and repositories and provide code generation instructions) and LLM boostrapping using LLMs (e.g., for helping to annotate code explanations and construct on-the-fly unit tests). The particular focus of their dataset is on explicitly modeling multi-turn coding (intuitively, they say that the benchmark aims to capture standard programming where developers \"interatively refine implementations through cycles of execution, debugging and feedback\") and on testing directly the effect of different forms of feedback using a novel feedback hierarchy that they define in Section 3.2. This benchmark looks similar to those they cite in the RelatedWork, as well as some they don't cite, notably the work below (where the landmarks annotations appear to be similar in spirit to the types of feedback they collect):\n \n[Bogin et al.] SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories. EMNLP 2024\n\nMost uniquely, however, their problems are annotated with different levels of feedback (generated using LLMs) using the feedback hierarchy from Section 3.2, and a set of concrete unit tests that allow them to measure functional code correctness. \n\nThey couple their benchmark with a new agent design called ReCodeAgent. Based on how this agent is described, both in Figure 1 as well as in the begining of Section 4, is it really unclear how this agent is nothing more than a ReACT agent or a closely related variant such as a Reflexion agent. Indeed, their earlier claim (starting on line 035) that \"existing benchmarks .. for evaluating LLMs in research code generation mainly adopt a one-shot setting, where models are expected to produce final code in a single interaction* simply seems incorrect given that the models in virtually all of the studies they cite involve ReACT agents, much like ReCodeAgent, that engage in precisely the kind of act-observe loop they show visually in Figure 1. This point needs to be directly addressed by the authors and is the main source of my  concern about this paper. \n\nTheir main empirical results are show across seven LLMs (including 4 LLM model familities) in Table 2, where they also carefully report the performance effect of different types of feedback (not surprisingly, the most detailed feedback, level 4, is clearly the most hepful in improving end task performance).  This is coupled with other fairly expected conclusions, e.g., (citing the authors) *Model size and capability play a clear role in performance*. Further error analyis is provided (Table 3).", "questions": "-- In what way is the agent workflow in Figure 1 (top left) not standard ReACT?\n\n-- If it is different from ReACT, did you try to compare against a standard ReACT or relfexion approach?  Or compare your approach on other benchmarks?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-- A new research coding benchmark with explicit feedback annotations that allow for more granular analysis of multi-turn coding. I could imagine this benchmark being used by others working in this area.", "weaknesses": "-- **Misleading motivation and discussion of past work**. As noted above, claims like \"existing benchmarks for evaluating LLMs in research code generation mainly adopt a one-shot setting, where models are expected to produce final code in a single interaction\" and \"the ability of [LLMs] to generate correct and execute code remains limited\" (line 011) seems inconsistent with virtually all of the papers cited. In the latter case, most studies involve REACT agents that are by design multi-turn agents. I would like to see the authors directly address. \n\n--  **Limited to No Novelty of their ReCodeAgent** As noted above, their proposed solution, and their sole modeling approach, seems to be nothing more than a ReACT or Reflexion agent. \n\n-- **Limited empirical validation** Experiments are limited only to their new dataset. Especially if they claim that their coding agent is unique, it would be expected that this approach shows improvements on other tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)