Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
Authors: Minju Seo
Jinheon Baek
Seongyun Lee
Sung Ju Hwang
Venue: ICLR 2026 Poster (2026)
Keywords: Natural Language Processing
Code
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Despite the rapid growth of machine learning research, corresponding code implementations are often unavailable, making it slow and labor-intensive for researchers to reproduce results and build upon prior work. In the meantime, recent Large Language Models (LLMs) excel at understanding scientific documents and generating high-quality code. Inspired by this, we introduce PaperCoder, a multi-agent LLM framework that transforms machine learning papers into operational code repositories. PaperCoder operates in three stages: planning, where it constructs a high-level roadmap, designs the system architecture with diagrams, identifies file dependencies, and generates configuration files; analysis, which focuses on interpreting implementation-specific details; and generation, where modular, dependency-aware code is produced. Moreover, each phase is instantiated through a set of specialized agents designed to collaborate effectively across the pipeline. We then evaluate PaperCoder on generating code implementations from machine learning papers based on both model-based and human evaluations, particularly from the authors of those papers, with author-released repositories as ground truth if available. Our results demonstrate the effectiveness of PaperCoder in creating high-quality, faithful implementations. Furthermore, it consistently shows strengths in the recently released PaperBench benchmark, surpassing strong baselines by substantial margins. Code is available at: https://github.com/going-doer/Paper2Code.

Published as a conference paper at ICLR 2026
PAPER2CODE: AUTOMATING CODE GENERATION FROM SCIENTIFIC PAPERS IN MACHINE LEARNING
Minju Seo Jinheon Back" Seongyun Lee', Sung Ju Hwang
KAIST, DDeepAuto.ai, 'LG AI Research Iminjuseo, jinheon. baek, seongyun, sungju. (eaguu/ttttaakiistt.e.a. (kaist
ABSTRACT
Despite the rapid growth of machine learning research, corresponding code imple- mentations are often unavailable, making slow and labor-intensive for researchers to reproduce results and build upon prior work. In the meantime, recent Large Lan- guage Models (LLMs) excel at understanding scientific documents and generating high-quality code. Inspired by this, we introduce PaperCoder, multi-agent LLM framework that transforms machine learning papers into operational code reposito- ries. PaperCoder operates in three stages: planning, where it constructs high-level roadmap, designs the system architecture with diagrams, identifies file depen- dencies, and generates configuration files; analysis, which focuses interpreting implementation-sppcinn
details; and generation, where modular, dependency-aware code is produced. Moreover, each phase is instantiated through set of specialized agents designed to collaborate effectively across the pipeline. We then evaluate Pa- perCoder on generating code implementations from machine learning papers based on both model-based and human evaluations, particularly from the authors of those papers, with author-released repositories as ground truth if available. Our results demonstrate the effectiveness of PaperCoder in creating high-quality, faithful im- plementations. Furthermore, it consistently shows strengths in the recently released PaperBench benchmark, surpassing strong baselines by substantial margins. Code is available at: hhttts:/ pettps://gituble
dwweaaggeodrrrpaeeeeaaee PPaper2oode
INTRODUCTION
Reproducibility lies at the heart of scientific progress, which enables researchers to validate findings, build upon prior work, and ultimately push the boundaries of knowledge (Collaboration) 20155] Baker 2016 Pineau et al. 2021). However, reproducing scientific results remains an enduring challenge. This often due incomplete documentation, missing experimental details, lack of access data or proprietary tools, and, especially in machine learning research, the absence of corresponding code: for example, only average 19.5% of the papers accepted top-tier machine learning conferences in 2024 provide their code implementations shown in Figure T] As result, researchers frequently invest substantial effort in reverse-engineering methods and experimental results from papers, a process that both time-consuming and labor-intensive, subsequently slowing down the overall pace of science.
Meanwhile, recent Large Language Models (LLMs) have shown outstanding capabilities in under- standing and generating both natural language and programming code (Dubey et al. 2024 OpenAI 2024 Reid et al.) 2024). with performances increasingly approaching or even surpassing that do- main experts in scenarios. In addition, this progress has sparked growing interest in leveraging LLMs to accelerate scientific workflows, particularly in the early stages of idcation for new and valid
research hypotheses (Lu al. 2024 Li et al.) 2024 Yang al. 2024 Si et al. 2024 Yamada et al. 2025; Scchmidall et al. 2025 Baek et af.) 2025). Furthermore, some of these studies, as well as
others focusing on later stages automating experimental validations and improvements (Huang
ct al. 2024; Zhang et al., 2024 Thirat et al. 2024 Chan et al. 2025), demonstrate the potential of
LLMs generate code and even carry out experiments end-to-end however, they typically assume and heavily rely on access to pre-existing implementations, partial code snippets, or well-defined
Equal Advising.

---

Published as a conference paper at ICLR 2026
PaperCoder
Nusber of Pfgerss
4000
Accepted Papec
Code
Paper
Planning
2. Analysia
Coding
Code Repos
-2000
2000
D
(a) PaperCoder overview
(b) Code availability
Figure 1: (a) PaperCoder, which aims to transform given scientific papers into code repositories, consisting of planning, analysis, and coding steps. (b) Code availability, where blue bars indicate the total number of accepted papers and orange regions show those with officially released code (See Appendix B. for calculation details).
APIs. As such, remains questionable whether generating faithful implementations solely from papers (without access to prior code, APIs, or additional supplementary materials) can be achievable.
To answer this question, we introduce PaperCoder, a multi-agent L..M-powered framework, designed to automatically generate faithful code repositories in machine learning directly from and contextual- ized with research papers, which differs from prior work that requires partial implementations from human inputs. Specifically, PaperCoder aims to emulate the typical life cycle of human developers and researchers in writing the repository-level code, by decomposing the task into three structured stages: planning, analysis, and generation. First, during the planning stage, the proposed framework constructs a high-level roadmap to identify core components to implement, draws the overall system architecture with class and sequence diagrams model structural relationships between modules, identifies file dependencies with their execution orders to guide correct build and execution flows, and generates configuration files to enable flexible customization of experimental workflows by human researchers. This is followed by the analysis stage, performing a fine-grained interpretation of each file and function with respect to their intended functionality, such as required inputs and outputs, interactions with other modules, and any algorithmic or architectural constraints derived from the source paper. Finally, in the generation stage, the framework synthesizes the entire code base based on the execution order determined earlier, along with the artifacts produced in the previous stages.
To validate the effectiveness of PaperCoder, we conduct extensive evaluations on subset of recent machine learning papers from ICLR, ICML, and NeurIPS referred to our proposed Poper2Code benchmark (in short, Paper2CodeBenchh, Also, we incorporate the recent benchmark (Starace et al.) 2025) in our evaluation suite, enabling fine-grained evaluations of code implementations. Then, on battery of tests conducted not only with automated model-based evaluations (covering both reference-free free and reference-ber based settings, conditional on the availability of author-releasee ground- truth repositories) but also with expert human evaluations (based on authors of original papers), PaperCoder demonstrates substantial improvements over baselines, generating more valid and faithful code repositories that could meaningfully support human researchers in reproducing prior work. Specifically, 88% of the generated repositories by PaperCoder are rated as the best over baselines. and 92% of human judges report that the generated repositories are indeed helpful. Also, analyses show that each component of PaperCoder (consisting of planning, analysis, and generation) contributes to the performance gains, but also that the generated codebases can be executed, sometimes with only minor modifications (averaging 0.81 1.81% of total code lines) in cases where execution errors occur.
2 RELATED WORK
Large Language Models for Code
LLMs have shown impressive capabilities in text understanding Dubey et al. 2024, Reid al. (2024) and widely utilized for specialized domains (beyond general tasks), such as mathematics, science, and coding (Prabhakar
and generation (OpenAT, 2024
ct al. 2025 Wang et al. 2024b Trinh et al. 2024). Particularly, code-specialized LLMs (Hui et al. 2024 DeepSeck-AI et al. 2024 2025) have received significant attention thanks to remarkable performance on various software engineering tasks (Xia et al. 2024), including software design and development (Qian al. 2024 Hong et al.) 2024). requirements elicitation (Mu al.) (023), and formal specification generation (Luo et al. 2024). Our work aligns closely with this line of research, exploring and expanding upon the capabilities and applications of (code-specialized) L.MMs.
2

---

Published as a conference paper at ICLR 2026
Repository-Leve Coding Early work on code generation typically focuses on single-file tasks, whose objective is to generate short code snippets to solve isolated tasks, such as (algorithmic- leveell programming competition problems (Chen et al. 2021 Austin al. [02T Hendrycks et al. [02T]] Li et al. 2022). However, as L.LMs have advanced in comprehending and generating code with the long-context reasoning ability, recent studies have increasingly shifted their attention toward more challenging repository-level coding tasks, which involve generating multi-fii file repositories that jointly account for architectural design, modular structure, and inter-file dependencies (Liu al.
2024 Jain et al. 2024, Tang et al. 2024). In particular, several recent efforts explore this emerging paradigm (Zhang et al. 2023 Ouyang al. 2025), adopting multi-agent or role-based frameworks to emulate realistic development workflows. For instance, ChatDev instantiates LLMs into role-playing agents that collaborate through structured dialogues (Qian et al. 2024), while MetaGPT implements a waterfall development pipeline with specialized agents (Hong et al. 2024). Beyond prior work, we explore the underexplored task of transforming full, complex papers into repository- code.
LLM-Powered Scientific Research LLMs have been adopted to support the scientific process
from ideation to experimental validation (Popper 1959 Qi et al. 2023 Li et al.] [024] Yang et al.
2024 D' Arcy et al., 2024] Liang et al.) 2024] Back et al. 2025 Weng et al. (025); thereby, helping
rescarchers overcome existing challenges and ultimately accelerate scientific discovery (Lehr et al.
2024 Lu et al. 2024 Yamada et al. 2025). Specifically, in fields such as computer science (where code-basede experimentation is central), LLMs have been used to design, refine, and extend code implementations. However, many recent efforts in this space assume access to and build on top of the original codebase (Huang al. 2024 Thirat et al.. 2024 Xiang et al. 2025 Chan al. 2025), which significantly limits their appl applicability in real-world scenarios since such implementations are oftentimes unavailable (See Figure []. To address this, concurrent to our work, Sttrace et al. (2025) introduces benchmark dataset called PaperBench, evaluating the capability of existing agentic Al systems in reproducing papers with fine-grained metries. Notably, on top of PaperBench (which emphasizes evaluation), we further complement and extend this line by focusing on methodological aspects of how to transform scientific papers into repository-level code implementations.
3 METHOD
In this section, we start with describing the task of repository-level code generation from machine learning papers, and propose PaperCoder, a multi-agent, multi-stage framework designed to tackle
3.1 REPOSITORY-LEVER CODE GENERATION FROM MACHINE LEARNING PAPERS
The goal of our repository-Ievel code generation task is to automatically produce a repository that faithfully implements methods and experiments described in machine learning papers (especially for cases where authors do not release their code), to support reproducibility and accelerate scientific progress (Pineau al. 202T Magnusson al., (023). Formally, define this task as a function (or a model) M that maps a paper R to a corresponding code repository C, as follows: (R) = C. Here, C is composed of multiple files {c), C22 ... Cn each responsible for implementing different components of the methods and experiments in R, but together they should form a cohesive pipeline.
The most straightforward approach instantiating M is to instruct the L.M generate the entire code repository, conditioned on the given paper, as follows: M (RR := LLM( ((R), where T is the prompt template that specifies the intended behavior of the LLM for the target task (including task descriptions, detailed instructions, and any other relevant context). Yet, generating complete, single pass is extremely challenging, even for powerful L.MMs.
modular, and faithful repository due to the inherent complexity of scientific papers and their corresponding implementations, the long- context limitations of current models, and the difficulty in maintaining consistent global structure and cross-file dependencies. Therefore, we propose to decompose the overall task into smaller subtasks, each handled by a specialized agent tailored to a specific aspect of paper-to-code transformation.
3.2 PAPERCOder: HLM--OOEERN MULTI-AGENT FRAMEWORK FOR PAPER-TO-CODE
We now introduce PaperCoder, a structured, multi-agent framework for generating code repositories directly from machine learning papers (without access to pre-existing artifacts or implementations,
3

---

Published as a conference paper at ICLR 2026
Naive
PaperCoder
Planning
Paper
Attention AE You Neod
1.1 Overall Pian
1.2 Arch. Design
DaQ 1.3 Logic Design
1.4 Configuration
Kacading
Opsinieer We caeil
1. disasse,, Jeadorgyy
raald "Sa mobeLibe: "aea" sanean
ACTION DIOS DNTGOMENT SCTUP
Sey MAbiET
3. trainespy: Traisr clasti-
batch Jodn Loount 2owuttt 25000
Cnemrtt the
4. evoluction.ps pat atton
optimbre::
positional encodiing
5. main py itan extra
"Adimm
6. uols py: fanctoon
Code
2. Analysis
3. Coding
No posizional encoding!
intttal cntion
kaid and parso
Aprieteer
Fauc//oweer Network
PyNMA opainiie astiage
and orrere croDect will
bente Betsll,
optimtteer
Pestionnl
(Adom kaii
Figure 2: (Left) The naive approach, which directly generates an entire code repository from a paper. (Right) Our PaperCoder framework, which is operationalized by decomposing the task into three stages: (1) Planning, where a high-level implementation plan is constructed from the paper, including overall plan, architectural design, logic design, and configuration file; (2) Analysis, where the plan translated into detailed file- level specifications; and (3) Coding, where the final codes are generated to implement the methods and experiments of the paper.
such as skeleton code). Specifically, inspired by typical software development workflows, PaperCoder decomposes the task into three coordinated stages: Planning, Analysis, and Coding, each orchestrated by specialized LLM agents. Formally, given a paper R, the overall process can be defined follows:
Planning: P = Mplan R).
Analysis: A = Manatssis R, P), Coding: C = Moode ( R, P. A),
where P, A, and C' represent the high-level implementation plan, the detailed function-level analysis. and the final code repository, respectively. The overall pipeline of PaperCoder is shown in Figure 2
3.2.1 PLANNING
It is worth noting that, in contrast to implementation specifications designed explicitly for software development, papers are written to communicate ideas and findings to humans. As a result, they often contain high-level motivations, persuasive narratives, and auxiliary details that are crucial for human understanding but noisy, loosely specified, or ambiguous from a software engineering perspective. To mitigate this, we introduce a planning phase that transforms unstructured textual content into implementation-leve abstractions. Also, we decompose the planning process into four sequential subcomponents (to simplify the task and reduce cognitive load of LLM LLL--oeeeeed agents at each step): 1) overall plan, 2) architecture design, 3) logic design, and 4) configuration generation. Formally, we define this as: Mplann Rt) - P = (o, d, 1, g). where is the overall plan, is the architecture design, I is the logic design, and is the configuration file, with each stage using the outputs of the previous ones as contextual input. We then describe how each subcomponent is instantiated below.
Overall Plan The first step is extract high-level summary of the core components and function- alities described throughout the paper, to identify the specific methods and experiments to be imple- mented. In other words, this high-level overview includes model components, training objectives, data processing steps, and evaluation protocols (distributed across the entire paper), which can form the foundation for all subsequent steps, formalized as follows: M(L) (R) I.I.M(((11)
plan (R)) o.
4

---

Published as a conference paper at ICLR 2026
Architecture Design Based on the extracted overall plan alongside the input paper, the next step is to define the repository-Iever architecture, which includes identifying files, organizing them into modules, and defining their relationships, to ensure coherent and maintainable structure. Specifically, the -.L.--owered agent is prompted to generate a file list, which outlines the overall file structure of the repository; class diagram, which details static representations of files (such their attributes); and a sequence diagram, which models the dynamic interactions. Formally, similar to overall plan, this process can be defined as follows: plan (R. o) [LM (R. o)) + d.
classes and
Logic Design While the previous architecture design focuses on what to build, the logic design phase specifies how these components should be instantiated in practice by considering their dependencies in terms of overall execution flow. This step is crucial because individual modules often depend on shared utilities, configurations, or data loaders that are defined in other parts of the repository. and without an explicitly defined execution order, the code generation (e.g., generating file B before file when B imports modules from A). To address this, the logic design stage not only produces an ordered file list that dictates the sequence in which the files should be implemented and executed, but also further elaborates on the logic within each file; thereby, providing more fine-grained specifications. Formally, M (R, d) == LLM (3) plan RR o, d,) I.
result in failure inconsistency
plan
Configuration Generation In the last stage of planning, PaperCoder synthesizes configuration file (config. yaml) that includes key hyperparameters, model settings, and other runtime options based prior outputs alongside the given paper. We note that, in addition to grounding the code generation process with the explicit configuration details, enables researchers to easily review and adjust experimental configurations without modifying the source code. Formally, M RR o, d,
LLM(T (R, 1)) g. We provide prompts used to elicit each planning output in Appendix D
3.2.2 ANALYSIS
Following the planning stage, which defines the overall structure and execution flow of the repository, the analysis phase focuses on interpreting and specifying the implementation-leve details for modules within each file. In other words, unlike planning that answers what components build and how they relate, this phase addresses the question of how each component should be operationalized and concretely implemented the file level, which includes the definition of functional goals, input-output behaviors, intra- and inter-file dependencies, and algorithmic specifications derived from the original paper. Specifically, given the input paper R and planning outputs P fo. d, L, g). the analysis agent iteratively processes each file fi (identified during planning) and generates a detailed analysis a; describing what needs to be implemented in that file. Formally, Maantysis (R, P, /=) where Manaysis (R, f() == LLM(Taalysia (R, P. fi) + a,, with F as the set of identified files, e.g., fo E F.
3.2.3 CODING
The final stage is the coding phase, where the complete code repository is produced. In particular, each file is generated based on all the available contextual information accumulated from the previous stages, including the overall plan, architecture design, logic design, configuration file, and file-specific analyses, as well as the original paper. Additionally, to ensure consistency across different files, we generate them sequentially according to the execution order (i.c., the ordered file list determined during the logic design stage). To be formal, for each file ft, the corresponding code c; is generated as follows: Meuse (R,, P, fe, ae (cI,
== LLM((Teee R, P, f., a;, {01, ..., G-1)) -> c, resulting in the { |Fl, We note that this iterative formulation can ensure that i-th code generated with full awareness of its dependencies and the evolving state of the repository.
complete code repository C' = {c:
4 EXPERIMENT
We now describe the experimental setup and the experimental results with reproducibility analyses.
5

---

Published as a conference paper at ICLR 2026
4.1 EXPERIMENTAL SETUP
Datasets To evaluate our PaperCoder, we construct a new benchmark (Paper2CodeBench), Specif- ically, we collect the accepted papers from recent machine learning venues (such as ICLR, ICMI, and NeurIPS 2024) with the OpenReview A and filter them based on the availability of code with its total number of tokens less than 70,000, to ensure the full repository remains within reasonable processing limits of modern LLMs for generation and evaluation. Also, to maintain the quality, perform model- based evaluation (Liu et al.) 2023) with GPT-4o on all the collected repositories and select the top 30 from each venue, resulting in a total of 90 papers listed in Tables 20 21 and 22 Moreover, we additionally consider 21 papers for human evaluation (See Table 23p. In addition to Paper2CodeBench, we also use the recently released PaperBench Code-Dev (Starace et al., (025), which consists of 20 papers from ICML 2024 with paper-specific rubrics annotated by humans. In particular, those rubrics are used to judge the correct implementation based on LLM-based evaluation.
Baselines and Our Model We target the novel problem of Paper2Code, and there are no baselines designed for it enable direct comparison. Nevertheless, we consider several related approaches proposed to implement repository-level code (or the entire software) from natural language inputs (such as software requirements), in addition to the ablated variants of our full PaperCoder framework. as follows: ChatDev (Qian et al. 2024) is multi-agent framework for software development, where several role-specific LLM -owered agents collaborate via structured dialogues; MetaGPT (Hong et al. 2024) similarity adopts a role-based multi-agent paradigm, but its process is organized by the principle of Standardized Operating Procedures (SOPs); Abstract a variant of our PaperCoder, which uses only the paper abstract for implementation: Paper, while using the full paper, performs one-shot code generation; PaperCuder (Ours) is our full framework, structured into three stages of planning, analysis, and code generation. Additionally, for the PaperBench Code- Dev, consider baselines suggested by it: Basic Agent is the agentic architecture that can run a predefined set of tools with the ReAct-style approach Yao et al., 2023), built upon the agent from Inspect A and Iterative Agent that extends Basic Agent, iteratively instructing the model to complete the next subtask.
Evaluation Setup Recall that, as shown in Figure D the official code implementations of many papers are not available; however, manually annotating their corresponding code implementations to evaluate the quality of automatically generated code repositories is highly labor-intensive and challenging. To address this and ultimately perform the evaluation at scale, we design two evaluation protocols: seaeencee-based (when ground-truth code is available) and reference-free (when it is not), following the recent trends in using I.LMs as a judge (Zheng et al., 2023 Fu et al., 2024 [ Liu et al., 2023). In addition this, we also perform human evaluations with the authors of the original papers. to ensure reliable judgments and to assess the quality of our model-basee evaluations by measuring their correlation with human scores. We discuss each evaluation protocol detail below.
Reference-Basee Based Evaluation. We use the official author-released repository as the gold standard only if it is available, since it most accurately reflects the implementations intended by the authors, including the components they consider essential to their main ideas. Specifically, we prompt the model (such as o3-mini-high") to judge the quality of the generated repository with respect to the gold repository, alongside the input paper as context (See Appendix D for the detailed prompt). The model then identifies components (to be implemented), categorizes them into three severity levels (high, medium, and low), and critiques how well each component is implemented. After that, it returns the overall score on a 5-point Likert scale. We note that, to ensure the reliability of the model-based evaluation, we sample multiple outputs (e.g.. 8) and report the average score.
Reference-Free Evaluation. For cases where the official author-released code is not available, introduce the reference-free evaluation protocol that leverages only the paper to assess the quality of its generated repository. Similar the enferencebbased evaluation, the evaluation model is prompted to identify key components, categorize them by severity, and critique their implementations in the generated code, but they are performed solely based on the information provided in the paper. The rest of the evaluation process, such as sampling and score averaging, follows the same setup.
'htpss//inspcct.td ai--aa---aa -sacacyiisttituteee www.iiittt.og..u//gggoom wk/.gents..tsttc htt.///gntts.mm/isscc-oooo -asssccc-aa Unless otherwise stated, we use o3-mini-high due to strong code understanding and reasoning capability.

---

Published as a conference paper at ICLR 2026
Table 1: Results on our Paper2CodeBench, where report average scores and standard deviations (in parenthe- ses) grouped by conferences. Oracle denotes the evaluation results with the official repository released by the paper authors. Also, on the right side, we report statistics on the number of tokens, files, and functions, averaged over all implementations. Bold indicates the best scores, statistically significant than baselines (p S 0.05).
Reference- Based Evaluation
Reference- Free Evaluation
Statistics
ICLR
ICML
NeurlPS
ICLR
ICML
NeurIPS
# of Tokens
# of Files of Funcs
ChatDEV
2.70 (0.63) 2.97 (0.58) 2.96 (0.69) 4.00 (0.65) 4.12 (0.53) 4.01 (0.74)
6150.54
6.99
23.82
MetaGiT
2.48 (0.48) 2.75 (0.70) 2.95 (0.87) 3.52 (0.60) 3.63 (0.75)
3.59 (0.92)
5405.21
3.24
18.08
Abstract
2.28 (0.42) 2.43 (0.49) 2.35 (0.62) 3.03 (0.64) 3.01 (0.60) 2.99 (0.78)
3376.99
1.28
12.62
Paper
3.08 (0.66) 3.28 (0.67) 3.22 (0.80) 4.15 (0.63) 4.30 (0.53) 4.08 (0.84)
3846.33
1.79
14.84
PaperCoder
3.68 (0.52) 3.72 (0.54) 3.83 (0.50) 4,73 (0.32) 4.73 (0.44) 4.77 (0.38)
14343.38
6.97
35,22
Oracle
N/A
N/A
N/A 4.84 (0.26) 4.80 (0.32) 4.83 (0.38) 32149.04
28.00
122.03
Table 2: Results with human evaluation. For model-based evaluations (both reference- based and reference-freed S-point Likert evaluation scores are converted rankings for comparability with human ranking results. Human rankings are also converted to scores of 5 (top repository), 3 (middle repository), and (bottom repository).
Ral-rra Score
0.79
Score (1)
Ranking (
Ref-based
Ref-frce
Human
Ref-based
Ref-free
Human
Ref-baased
2.26 (0.37)
2.94 (0.61)
(0.56)
2,96 (0.20)
2.96 (0:00)
(0.56)
Paper
3.00 (0.54)
3,91 (0.63)
2.76 (1.20)
1.92 (0.41)
1.88 (0.38)
2.09 (0.60)
Figure 3: Correlation be- tween model- based eval- uations: reference-basee and reference-fiee
PaperCoder (Ours)
3,66 (0.43)
4.55 (0.51)
(1 (1.00)
1.08 (0.28)
1.08
1,22
ChaiDEV
2.68 (0.60)
3.82 (0.37)
(1.77)
2.58 (0.50)
2.23 (0.59)
2443 (0.59)
MetaGPT
2.61 (0.54)
3.39 (0.67)
(1.17)
2.38 (0.58)
(0.51)
2.43 (0.59)
PoperCoder (Ours)
(0.43)
4.55 (0.51)
(0.88)
1/44 (0.20)
(0.20)
(0.46)
Human Evaluation. While model-based evaluation offers scalable and automated way of assess- ment, we also conduct human evaluations to validate our PaperCoder based on expert- grounded evaluation. Specifically, to ensure informed and accurate judgment, each participant is assigned paper for which they are the first author. Also, they are presented with multiple implementations generated by different approaches, and asked to rank them. We offer more details in Appendix A.2
Lastly, for evaluation on the PaperBench Code-Dev benchmark (Starace et al. 2025 we follow their evaluation setup, measuring the score over the paper-specific rubrics with LLM M-based evaluation.
4.2 EXPERIMENTAL RESULTS AND ANALYSIS
Main Results Table D presents main results Puper2CodeBench, in which PaperCoder consistently outperforms all baselines. We hypothesize that this performance gap stems from its top-down behavior, analyzing full papers thoughtfully before generation, unlike prior approaches that typically follow bottom-up strategy, which begins with and expands short requiremental descriptions (via role-playing or SOP). In other words, the top-down approach, operationalized through the sequence of planning, analysis, and coding, is effective in handling long-form scientific documents, which are often loosely structured from software engineering perspective. Also, when compared to the non-comparable Oracle setting (which performs evaluations on the author-released repositories), PaperCoder achieves performance that on par, without statistically significant differences, demonstrating its effectiveness in faithfully implementing code whose quality is closer to the implementation by authors.
Correlation between Reference-Bassee and Reference-Free Evaluation Recall that the reference- free evaluation protocol is designed for cases where the ground-truth repository is not available, and to investigate whether it works as a reliable proxy for the reference-basee based evaluation protocol, we measure their rank correlation on all samples from Paper2CodeeBnnch. Then, shown in Figure 3 there is a strong positive correlation between them, achieving a Peanson correlation coefficient of r == 0.79. This result supports that the reference-free evaluation can serve as reliable proxy for the eeferenccebased evaluation, ultimately functioning as a standalone metric to assess the code quality.
7

---

Published as a conference paper at ICLR 2026
Table 3: PaperBench Code-Dey results. We report the averaged performance over three runs with standard deviations.
Table 4: Results based both model-based and human evaluations with varying backbone LLMs for PaperCoder.
DS Coder
Qwen
DS-Dusill-Qwes
high
Replication Score (9)
Score
Ref-hased
(0.46)
1.78 (0.25)
2.05 (0.25)
3,66
Model
o3-mini-high
clusdo-3 Monsl--..-ssumnc 3.5-sunnen
Ref-free
(0.54)
2,09
(0.34)
(0:51)
Human
1.32 (0.55)
2,71 (1.12)
(0.98)
4.68 (0.80)
Rasic Ageie
5.1 + 0.8
35.4 + D.8 27.5 1.0
Renking (1)
Ref- bef-based
3.46 (1.00)
2.92 (0.5)
2:25 (0.65)
1,00 (0.20)
ItenativeAgent
16.4 + 1.4
Ref-free
3.50 (0.00)
2,88 (0.83)
2.12 (0.54)
1.00 (0. (0.25)
PaperCoder
45.14 + 0.3
51.14 1.4
Human
5.74 (0.45)
2.74 (0.86)
(0.70)
1.22 (0.60)
Table 6: Ablation results the subset Paper2CodeBench with scores and standard deviations.
Table 7: Results of the PaperCoder and PuperCoder with Self-Refine, under the reference- hased evaluation protocol.
4.0
Poster Spotlight
3.00
3.5
3.72
Average
3.0
Ret based
Ref tize
PuperCloder
wr Self Reline
2.83
2.87
Paper
(0.67
50 (0.5 5%)
Overall Plare
6.67
4.87 (+0 201
2.5
Ovurall Plan
5.40105571
14 10 583
Auch Design
GPT-4o
03-mini-high
4.09
4.38 (+0
Logic Thesign
3.50:00 (0:575
Coiting Inle
2.45
3.91
Figure 4: Model-based evaluation
Coniig File-
45)
4.45 (0 (3.5%)
Asdyyi
Asulysis (Ours)
(0.54)
Code
3.39
3,89 (+0.50)
results by paper presentation types.
Human Evaluation Results In addition to automatic evalu-
Table 5:
Rank correlation coefficient between human and model- -ased eval- uations (with GPT-4o or o3- -3-miii).
ations, we conduct human evaluations and report the results in Table[ From this, we confirm that PaperCoder achieves the best ranking, consistent with model-based evaluations, which reaf- firms its effectiveness. Also, to ensure whether the model-based evaluations are a reasonable proxy to judge the implementation quality, we measure their correlations with human evaluation scores. As shown in Table(5] we observe strong rank correlations
GPT-4o
03-mini-hig high
Ref-based Ref-free
0.74
0.78
0.71
0.73
across both reference-basee baase and reference-free free settings, which suggests that model-based evaluation can reliably approximate human judgment. Also, based on this result, we use 03-mini-high as the default evaluation model. Lastly, we ensure the quality and reliability of human evaluations by measuring the inter-annotator agreement based on Cohen's 's kappa coefficient, which exhibits a high score of 0.79, indicating strong consistency.
Results on PaperBench Code-Dev In addition to our Paper2CodeBenchh we further validate the effectiveness of PaperCoder on another PaperBench Code-Dev dataset, which enables fine-grained evaluations for code implementations. As Table 3] shows, PaperCoder achieves the highest replication scores across two different LLMs of o3-mini-high and Claude 3.5 Sonnet, substantially outperforming baselines designed for PaperBench Code-Dev. These results further demonstrate the generalizability and robustness of PaperCoder across diverse evaluation benchmarks and models.
Extending the model variations results on PaperBench Code-Dev we CaappSeee-Code--Vee -e---VV---LL--ttuuuee DeepSeek-Al et al., 2024), Qwen-Coder (Qwen2.5-Coder- In-structttt Hui et al. 2024), DS--Distil--Wwa (DeepSeek-RR- Distill-ween- 14B: DeepSeek- Al et al. 2025). and 03-mini-hin high (the high reasoning-effon variant of 03-mini) on Paper2CodeBench. As summarized in Table 4 the proprietary model (o3-mini-high)
Analysis on Different LLMs
conduct an auxiliary analysis with DS-Code
consistently outperforms all other backbones across all evaluation settings. Among other open-source models, DS-Distill- Distill-Qwen performs the best, followed by Qwen-Coder and DS- DS-Coder. These results suggest the importance of selecting a capable backbone to instantiate PaperCoder, particularly one with strong reasoning capabilities. Also, based on this, we primarily use 93-mini-high as the basis.
Ablation Studies To see how much each component of PaperCoder contributes to the performance gain, we conduct ablation studies on the subset of Paper2CodeBench (composed of ICML papers). Specifically, we start with the method that uses only the full paper and incrementally add components in the order they are executed (such as overall plan, architecture design, logic design, configuration generation, and analysis), reported in Table 6 From this, we observe that the performance steadily improves additional components are incorporated. Meanwhile, performance drop occurs when the architecture design module is added; however, while this might seem surprising at first, it is in fact expected: architecture design alone does not specify the execution implementation order of files, which leads to confusion during the code generation (see Figure 77. However, this issue is addressed once the subsequent logic design module explicitly defines file dependencies and establishes clear
8

---

Published as a conference paper at ICLR 2026
Table 8: Replication scores on 10 papers from PaperBench, includ- ing execution and result match.
36
Helpfulnssa
112%
Dekuppling
24.4
Processing
Muthod
30%
80%%
(+7.3)
12.3
1922 11.5
164
20.2
(+4.3)
20.4
Evaluction
79%
Model
Score (%)
5.7
20
40
60
do
160
BasicAgent
2.60
Abstouct
Paper
MenaPP
Chabev PuperCoder
Percentage 1961
lterativeAgent
11.22
Figure 6: Results on the author-written rubric
Figure 5: Fine-grained analyses on code by PaperCoder.
for papers from Paper2Code (human
PaperCoder 28.46
evaluated), with gains in parentheses.
generation order. Overall, integrating all modules yields the highest performance, confirming the effectiveness of our fully structured, multi-stage pipeline with various modules proposed.
Experiment with Refinement We confirm in thhtt the planning and analysis stages play a pivotal role in guiding subsequent analysis and coding, and we further test whether refining earlier outputs can improve downstream performance. Specifically, we augment the planning and analysis phases with verification wwwpaattooonnnnfiiiiaa and-refinement steps (See Figures 23 to 32 for prompts), following Self- Refine (Madaan et al. 2023), and evaluate a total of 30 papers subsampled from Paper2CodeBench (10 from each conference). As shown in Table 7 refinement of planning and analysis improves their own outputs but also leads to measurable gains in the subsequent stages, reducing downstream errors.
Correlation on Paper Type To see whether the acceptance category (or presentation format) of papers correlates with the quality of their corresponding implementations by PaperCoder, we analyze it by separating papers into oral/spotlight and poster categories on Paper2CodeBench (which includes 14 oral or spotlight papers and 76 poster papers). As shown in Figure scores are slightly higher for oral/spotlight papers on model-based evaluations with GPT-4o and 03-mini, suggesting that papers with higher recognition might reflect clearer writing, probably leading faithful code generation. For further analysis on how the completeness of papers impacts the results, please refer to Table 12
Fine-Grained Analysis of Generated Repositories To more thoroughly evaluate the quality and practical utility of the generated code, we conduct set of fine-grained human analyses according to its usability for reproduction and its component-wis wise implementation quality. Specifically, we ask annotators whether the top-ranked repository from PaperCoder would make reproducing the original work easier than starting from scratch, and 92% agree, highlighting its practical value. Also, we conduct a component-level analysis assess which parts of the papers are most effectively translated into code, by asking human annotators to identify key elements for Data Processing, Method, and Evaluation, then measure how many are actually implemented. As shown in Figure 5 the coverage reaches 80% for Method and 79% for Evaluation. Notably, among the errors observed, many of them originate from the Data Processing stage, where papers often under-specify details about data formats, preprocessing steps, or loading procedures. Lastly, to investigate why human annotators prefer PaperCoder over its baselines and ablated variants (with 22 out of 25 selecting the repositories from PaperCoder), ask them to provide the reasons for their choices, and the majority of which are completeness, clean structure, and faithfulness to the original papers, summarized in Table [6
4.3 ADDITIONAL ANALYSIS ON REPRODUCTION FROM IMPLEMENTED CODE REPOSITORY
While our focus is on generating faithful implementations that can aid research, we further examine whether these implementations can fully reproduce the original experimental results end-to-end.
Analysis on Executability It is worth noting that making the repository-level code executable and fully reproducible in one go is extremely challenging (even for humans), as demonstrated by Starace al. (2025). Also, our goal to provide a faithful starting point that meaningfully aids reproduction efforts (Figure 5) rather than aiming for perfect reproduction. Nevertheless, to assess how close our generated repositories are to being directly executable, we perform manual execution evaluations on five papers. Specifically, when execution fails, we manually debug and refine the code and adapt the input data as needed to enable successful runs. We then find that, on average, only 0.8 % of the code lines require minor modification, such updating deprecated API correcting data type mismatches, for successful execution (see examples in Figures(8] [2 with statistics in Table 13). which highlights that generated repositories near-executable with minimal human intervention.

---

Published as a conference paper at ICLR 2026
Analysis on Reproducibility An equally important, though not our primary focus, question is whether the generated repositories can reproduce the results intended by the original authors. To examine this, we sample 10 papers from PaperBench and another 10 from the human evaluation set of Paper2CodeBenchh Also, we automatically invoke L----ssisted debugging (only when execution errors occur), where the model was provided with error messages, source code, and relevant training data (if needed) to resolve issues. First, for PaperBench, we use the full rubric provided, including the aspects of result match as well as code development and execution, with 03-mini serving as the judge. Then, as shown in Table 8] PaperCoder achieves the highest score. Also, for Paper2CodeBench, we adopt the rubric defined by the paper authors, covering Data Processing, Method, and Evaluation, with o4-mini as the judge, and as shown in Figure 6 PaperCoder outperforms all baselines regardless of whether debugging is used. These results show that its repositories are not only executable with minimal (and automatically debuggable) intervention but also more faithfully reproduce the papers.
Case Study We further conduct a manual case study on five repositories, where annotators check whether the returned outputs match the reported results. As described in Table with Appendix AA.5 four reproduce results (at least partially), while one fails due to issues in loss function design.
with
5 CONCLUSION
In this work, we introduced PaperCoder, framework that automatically generates code repositories from research papers in machine learning through a structured, three-stage pipeline. Specifically, we defined a high-level roadmap. system architecture, execution logie, and configuration via the planning stage, which are then enhanced through detailed per-file analysis, followed by the sequential code generation informed by artifacts from prior stages. To validate PaperCoder, we performed evaluations on two benchmarks: our Paper2CodeBench, comprising recent papers from top-tier machine learning venues, and (recently released) PaperBench Code-Dev, providing fine-grained evaluation protocols, on which PaperCoder consistently outperforms existing baselines on both model-based and human evaluations. Furthermore, additional analyses demonstrate its robustness and practicality: remains effective across different LLM backbones, shows practical executability with only 0.81 % of the lines requiring minor fixes, and benefits from each stage in the pipeline. We envision PaperCoder as one important step toward accelerating scientific progress by aiding the reproduction of research papers.
ETHICS STATEMENT
Our work aims generate faithful code repositories from scientific papers in machine learning, and we believe it has substantial positive impact in contributing to open science and facilitating rapid experimentation. However, we also acknowledge potential risks and misuse of our framework. For example, some papers intentionally refrain from releasing implementations due to security concerns, such as those involving jailbreaking or exploitation techniques. Yet, our method could potentially be used to reproduce such sensitive implementations. To address such risks, in real-world production, it would necessary to develop and incorporate safeguards (such as harmful content filters, protective prompting, and secure execution environments) to ensure responsible and safe use of our framework.
REPRODUCIBILITY STATEMENT
We provide the public GitHub link to the code to reproduce our work. Detailed instructions for running the experiments are included in the accompanying README files, and furthermore, all necessary details to reproduce our experiments are described in Section 4.1 and in Appendix A.
ACKNOLEDGEMENTS
This work was supported by the Institute for Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (RS-2019-11190075. Artificial Intelligence Graduate School Program (KAIST)), the National Research Foundation of Korea (NRF) the Institute of Information & Communications Technology Planning & Evaluation (IITP) with a grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research (No. RS-2024-0046948 & 022002270 and the InnoCORE program of the Ministry of Science and ICT (No. N10250156).
grant funded by the Korea government (MSIT) (No. RS-2023-00256255
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL http:://axxvv.rrope wwwp////////oooooo///.o.//eaaeea 2108 07732
Jinheon Back, Sujay Kumar Jaubar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models, 2025. URL https
7/arxiv. wwtp:/s...bbb.... /0t.bb//2244..000
Monya Baker. 1,500 scientists lift the lid on reproducibility, 2016. URL https:://///mmm nature com
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. Mle- bench: Evaluating machine learning agents machine learning engineering, 2025. URL https
l/axxiv.org/// abs / 2410. 07995
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol. Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. http:s////axii...ggggee/ee// wttp://www.rr//ba//1......22 03374
Open Science Collaboration. Estimating the reproducibility of psychologiiaa science. Science, 349 (6251):aac4716, 2015, doi: 10..1126scceneccacc4777777 URL hwtp:///wwwww.inc..e/c. org/
2bt/abs/10..116// wwwp:///eecccnncccce...e aac471
Mike D'Arey, Tom Hope. Larry Birbbau,, and Doug Downey, Marg: Multi-agent review generation for scientific papers, 2024. htt.tttts:///aiiiie
bb////1///221101////// 2401 04259
DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang. Peiyi Wang. Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Deepseck-cooeer-v:: Breaking the barrier of closed-source models in code intelligence, 2024. BRLLhhtss://axx.tt. http://wxxxxv....aaaa/// abs /40..24066.. 1931
DeepSeek-AL, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai. Deli Chen, Dongjie Ji, Brhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li. J. L. Cai, Jiaqi Ni. Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu. Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu. Shengfeng Ye. Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. L.. Xiao, Wei An, Xiaodong Liu, Xiaohan
11

---

Published as a conference paper at ICLR 2026
Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li. Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong. Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao. Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. http:/bbb/250000 12948
URL
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu. Joanna Bitton, Joe Spisak, Jongsoo Park. Joseph Rocca, Joshua Johnstun, Joshua Saxe. Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafilld Kevin Stone, and al. The llama 3 herd of models, 2024. URL https::/aaxx.
htwggabs/2400 21783
Jinlan Fu, See- Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mex- ico, June 16-21, 2024, pp. 6556-6576. Association for Computational Linguistics, 2024. doi: 1188665/11/2244MAALLLL00000 naacl-long. 365
URL https: org 10 11w53811622000
Kanishk Gandhi, Ayush K Chakravarthy, Anikait Singh, Nathan Lile, and Noah Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective STars. In Second Conference on Language Modeling, 2025. URL httttts:///ppereer// eww..wwwww....................ww ew. net. / forum d=
QGJ9tEXITY
Conghui He, Wei Li, Zhenjiang Jin, Chao Xu, Bin Wang, and Dahua Lin. Opendatalab: Empowering general artificial intelligence with open datasets. arXiv preprint 24Xi::2407.13773 2024.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL httppp/s/otttessseeoocccarrrssss http://eeeeeeaeee.e.rooo...ee//p
Sirui Hong. Mingehen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng
12

---

Published as a conference paper at ICLR 2026
Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi- agent collaborative framework. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htttps://cpenreee ew. net forum? d=
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 2/-27, 2024. OpenReview.net, 2024. URL https //openreview net / forum? id 1F 1Fs1LvjYOW
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang. Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024. URL https
org abs/ 2409 2409. 12186
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. httttt::///aatt wwg. h/tp://b//22
07974
Steven A. Lehr, Aylin Caliskan, Suneragiri Liyanage, and Mahzarin R. Banaji. Chatgpt research scientist: Probing gpt's capabilities as a research librarian, research ethicist, data generator, and data predictor. Proceedings of the National Academy of Sciences, 1211 2024. doi: 10. 1.11733paas.244432555 24C4328121
URL https::///// /www//wwww..// http://w.....s............ pnwsp http://pms.orgg/aaa/ab////o..... ppa3/pmsss
Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang. Deli Zhao, Yu Rong, Tian Feng, and Lidong Bing. Chain of ideas: Revolutionizing research via novel idea development with Ilm agents, 2024. URL
https: /
13185
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang. Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition--levl code generation with alphacode. Science, 378(66244 2022. doi: 10. 10.1126//cce 1588. abg1158
www.:ww.iecccccc.... URL http:://wwwwaa..cee.......oooo//
Zhang Li, Yuliang Liu, Qiang Liu, Zhiyin Ma, Ziyang Zhang, Shuo Zhang, Zidun Guo, Jiarui Zhang, Xinyu Wang, and Xiang Bai. Monkeyocr: Document parsing with patuucuuerrcoogiiioo-ellllaeeeee relation triplet paradigm, 2025. URL pattps://artte
org/ abs 2506. 05218
Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. Can large language models provide useful feedback on research papers? large-scale empirical analysis. NEJM AI, 188):1oo2400199 2024. doi: 10.1056/A 10100/11122400196 URL. htttps:/ //atpp:://a...e. ne
org/ ffwllfllll//o/ 1/10. 20556//11224001110
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
net met/forummid-pp"
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10. 2023, pp. 251 1-2522. Association for Computational Linguistics, 2023. doi: 10.18653/V112023 2ttp://112222...L.PP.... WMW..--AILLLLLI.... 7/doi .ww. 00. /tt///10..6653/112000000 emn pp-ma 153
URL https
13

---

Published as a conference paper at ICLR 2026
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 4969-4983, Online, July 2020. Association for Computa- URL htt.h:ttt/:wwwtt ac web. org/
tional Linguistics. doi: 10.
onthology/222233 acl
447
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery, 2024. URL httpp:://axxv...o
abs/2408 2208 06292
Qinyu Luo, Yining Ye, Shihao Liang. Zhong Zhang. Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, Xiaoyin Che. Zhiyuan Liu, and Maosong Sun. Repoagent: An Ilm- powered open-source framework for repository-level code documentation generation, 2024. URL
/ttps://arxiv. owoooogg//....o 220.24422.112 166677
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refie: Iterative refinement with self-feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurlPS 2023, New Orleans. LA, USA, December 10 16. 2023, 2023, htLttt/://ppppete (Jeedf077222tbbb55500eeeeetccccd
wttpssscccppppppppppp Files/paperr 2ttr//22///aaa haahh/ -aee---stttttt Cottctttoooofeeeeee html
Ian Magnusson, Noah A. Smith, and Jesse Dodge. Reproducibility in NLP: what have we learned from the checklist? In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 12789- 12811 Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023. FINDINGS-ACL.80Y URL http::httt////t org 11..1....131/// htt.//20.....iinnngeeeiiiaa findings acl http::////o.....101111111111133
809
Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, Chenxue Wang, Shichao Liu, and Qing Wang. Clarifygpt: Empowering llm-based code generation with intention clarification, 2023. URL httthtt////axxii
abs /2sss2310..om 10996
Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, Linke Ouyang, Zhiyuan Zhao, Tao Chu, Tianyao He, Fan Wu, Qintong Zhang. Zhenjiang Jin, Guang Liang. Rui Zhang, Wenzheng Zhang, Yuan Qu, Zhifei Ren, Yuefeng Sun, Yuanhong Zheng, Dongsheng Ma. Zirui Tang, Boyu Niu, Ziyang Miao, Hejun Dong, Siyi Qian, Junyuan Zhang, Jingzhou Chen, Fangdong Wang. Xiaomeng Zhao, Liqun Wei, Wei Li, Shasha Wang, Ruiliang Xu, Yuanyuan Cao, Lu Chen, Qianqian Wu, Huaiyu Gu, Lindong Lu, Keming Wang, Dechen Lin, Guanlin Shen, Xuanhe Zhou, Linfeng Zhang, Yuhang Zang. Xiaoyi Dong, Jiaqi Wang, Bo Zhang, Lei Bai, Pei Chu, Weijia Li, Jiang Wu, Lijun Wu, Zhenxiang Li, Guangyu Wang, Zhongying Tu, Chao Xu, Kai Chen, Yu Qiao, Bowen Zhou, Dahua Lin, Wentao Zhang, and Conghui He. Mineru2.5; A decoupled vanion-l model for efficient high-resolution document parsing, 2025.
OpenAl. Gpt-4 technical report, 2024.
Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. Repograph: Enhancing ai software engineering with repository- level code graph, 2025. URL
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence A'ché- Emily B. Fox, and Hugo Larochelle. Improving reproducibility in machine learning research(a report from the neurips 2019 reproducibility program). J. Mach. Learn. Res.,
12:16::11164:20, 2021. URI.https. // htt.m/rrrrrggppp////22.... 2ttp////2222223....2. m0m0mmmmml
Karl Raimund Sir Popper. The logic of scientific discovery. Systematic Biol- ogy, 26:361, 1959. URL https://shilceetees int.//nooooee s01p/IM//pcE/
popper-logice wtc.iiniii--d..ioeeeeeeeee
14

---

Published as a conference paper at ICLR 2026
Vignesh Prabhakar, Md Amirul Islam, Adam Atanas, Yao-Ting Wang, Joah Han, Aastha Jhun- jhunwala, Rucha Apte, Robert Clark, Kang Xu, Zihan Wang, and Kai Liu. Omniscience: A domain-speciallzed IIm for scientific reasoning and discovery, 2025. URL https:/:///ax
org/abs/2503 17604
Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng. Zhang-Ren Chen, and Bowen Zhou. Large language models are zero shot hypothesis proposers, 2023. URL http::t/////// //axivv.///iivoa/aa org/
abs 22ss/231..0 05965
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Commu- nicative agents for software development. In Lun- Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand,, August 11-16, 2024, pp. 151 1174615186. Association for Computational Linguistics, 2024. doi: 1211666//1//22AAALLLLLL
URL
810
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean- Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mcllroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thomton, Zhen Yang. Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
URL https: /etpss//arxivv org/ abs/ 200es/2400 2403. 05530
Samuel Schmidgall, Yusheng Su, Ze Wang. Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using 1lm agents as research assistants, 2025.
URL httphhttt///xiiiiee org/
04227
Minju Seo, Jinheon Back, James Thore, and Sung Ju Hwang. detrieval-augmentte data aug- mentation for low-resource domain tasks. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning, 2024. URL httt.http:////t http///pppeeveeeeeeeeeeeeeee /oppe/ee/// net/ ftrfformmmiiio
rV1 TelxeaaTee TKh
Minju Sco, Jinheon Back, Seongyun Lee, and Sung Ju Hwang. Efficient long context language model retrieval with compression. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computationnl Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 15251-15266 Association for Computational Linguistics, 2025. URL
/ettps//aclaane low.aanthhooooyyoooooooooo ogy. wttp::ooor///.....1111oooo 7407
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? large-scale human study with 100+ nlp researchers, 2024. URL httpt:::/./ti...rrrrttre arbbbs///
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai's ability to replicate ai research, 2025. URL org.
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang. Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. MI-bench: Evaluating large language models and agents for machine learning tasks on repository-level code, 2024. URL https: arxiv. org /bss/
2311 09835
15

---

Published as a conference paper at ICLR 2026
Trieu H. Trinh, Yuhuai Wu, Quae V. Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625:476 - 482, 2024. URL /twp::://wwww.... nature
com/ aat.a//eee//1111--0--00177711111
Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Automl-agent: A multi-agent Il framework for full-pipeline automl, 2024. URL https:/ arxiv. www.i..rrg/oooo.o.oooo...o.ooorr abs /twb////11/ 02958
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 2017. URL https: /ttpp::: neur ps CC/ htwp//aappprp//aaa 201 7/hash/ html
Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. Mineru: An open-source solution for precise document content extraction, 2024a. URL /ttppis///axiv..oo org.
wwwpwx.o.o..b////.//.. wwtaaabs/2449 18839
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in lims for enhanced mathematical reasoning. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL
httts:://ppenev// www.breiiew.ee//eeee www.w...e..e...... wwt.:e/eoooomimd20000000000
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 13484- 13508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023 a023.acl--ong.751 754. URL htt.htppps///ccllaaoollllyyy wtt.:////22..1-1ooogo...... 7547
a http//acllaa
acl.
Haoran Wei, Yaofeng Sun, and Yukun Li. Deepseek-oer: Contexts optical compression. arXiv preprint arXiv:2510. 1...:25510...4.... 2025.
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and
Linyi Yang. Cycleresearcher: Improving automated research via automated review, 2025. URL
https://arxx wwwpaxiwww..gga///2111111 00816
Chunqiu Steven Xia, Yinlin Deng. Soren Dunn, and Lingming Zhang. Agentless: Demystifying Ilm- 01489
based software engineering agents, 2024.
Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, and Yulan He. Scireplicate-bench Bench- marking 1lms agent-driven algorithmic reproduction from research papers, 2025. URL
https://aaxiv www.ooggg/s.... 2504. 00255
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search, 2025. URL http:ss://txii..cooot
/2504. 08066
Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. Large language models for automated open-domain scientific hypotheses discovery. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 13545-13565 Association for Computational Linguistics, 2024. doi: 10. 1B855311/2222FFFNNNGS-AAAAA
URL https
7//60..02/11011853//1// http:///22222....iinn.-----e..-. 804
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. htt.hhttss://oeneeeeet
net. foot/foorm?? d=WF_ wluYUI-XX
16

---

Published as a conference paper at ICLR 2026
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian- GuanGu Lou, and Weizhu Chen. Repocoder: Repository-Level code completion through iterative retrieval and generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6- 10, 2023, pp. 2471-2488 Association for Computational Linguistics, 2023. doi: 10tp///1..1663/1//222222 120222..EMLLPMMMII..... URL https::///eete BLLpS:///Ba...rr//..111////////0
emnlp main. 151
Jiarui Zhang, Yuliang Liu, Zijun Wu, Guosheng Pang. Zhili Ye, Yupei Zhong, Junteng Ma, Tao Wei, Haiyang Xu, Weikai Chen, 7een Wang, Qiangjun Ji, Fanxi Zhou, Qi Zhang, Yuanrui Hu, Jiahao Liu, Zhang Li, Ziyang Zhang, Qiang Liu, and Xiang Bai. Monkeyocr v1.5 technical report: Unlocking robust document parsing for complex patterns, 2025. URL /ttpp:://axii.....
abbs/2511.0090
Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, and Yuqing Yang. Micopilot: Unleashing the power of large language models in solving machine learning tasks. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 Volume 1: Long Papers, St. Julian's Malta, March 17-22. 2024, pp. 1931-2955 Association for Computational Linguistics, 2024. URL https: bttps://aclantee hataa/aaannttooy..... wtt.:y/orrr/2222accl111ooo110
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging ilm-as-a-judge with mt-bench and chatbot arena. Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NearlPS 2023, New Orleans, LA, USA, December /ttpttp::/ppperreet
10 16, 2023, 2023. URL http: http:::/pppprssee nips wwwp:ipppppccpppppeeeppepppppppe files/ wiles/paperi pappp/2223////////////
cattrraat-Dattatss.... Datasets and
wtr.chmarks.htm..
17

---

Published as a conference paper at ICLR 2026
APPENDIX
A ADDITIONAL EXPERIMENTAL DESIGNS
A.1 IMPLEMENTATION DETAILS
All experiments are conducted using o with high reasoning effort version (c3-minni-f gh) as the default backbone, released on January 31, 2025. To collect paper metadata and content, we use open revi ew_ www_srraeeee =[ with the OpenReview and Semantic Scholar AP For document processing, we convert papers into structured ISON format using the s = 2 json library (Lo) al. 2020 Notably, with 03-mini hi gh to generate repositories for 90 papers, the total API cost of PaperCoder amounts to $76.65, resulting in an average cost of approximately $0.90 per paper.
A.2 HUMAN EVALUATION PROCESS
Given the complexity of the task (requiring comprehension of scientific papers and their associated implementations), we recruit participants who have at least one peer-reviewed paper and a degree in computer science. We note that they were compensated at a rate of $15 per hour. For annotation, they were provided with a 4-page document, which includes task instructions, annotation examples, and 10 generated repositories grouped into three sets, follows: (Group 1) Model Variants of Our Method that includes repositorics generated by our system using different backbone models (c.g., o3 03-mini vs. three open-source alternatives); (Group 2) Naive Baselines that includes repositories generated using only the Paper or the Abstract as input; and (Group 3) Related Works that includes repositories generated by existing software development frameworks, such as MetaGPT and ChatDev. Each repository was anonymized using repo X naming format to prevent bias regarding the generation method. Following the question guidelines in the document, annotators reviewed and evaluated the repositories generated by different methods and models. Also, on average, evaluating 10 repositories for single paper took approximately 45 minutes. Table shows a detailed annotation example.
A.3 EIPERENCE-BASEE EVALUATION
In the reference-baseeeeeeeeee evaluation setup, the repository may exceed the context length (even frontier) LLMs. Following Starace et al. (2025), when this occurs, prompt the model to select the most relevant files for evaluation. The selected subset is then used as the reference for scoring. We use the 2ot-4o-2024-10 1-20 as the evaluation model.
A.4 PAPERBENCH CODE-DEV EVALUATION
While PaperCoder is designed to generate only the source code, the PaperBench Code-Dev bench- mark used for evaluation requires additional script file called reproduce ch. To meet this requirement, we further prompt the coding agent to generate it and evaluate the code with it.
A.5 ADDITIONAL. DETAILS ON EXECUTION AND REPRODUCIBILIT EXPERIMENTS
To assist the reproduction of repositories from PaperCoder, we perform LLM-assisted automatic debugging. Specifically, primarily o4-mini for debugging, with GPT-5 used fallback when identical errors persist. Furthermore, all executions are performed in a Docker environment with an NVIDIA GeForce RTX 2080 GPU, and for experiments requiring larger memory, an NVIDIA RTX A6000. Lastly, due to hardware constraints, we adjust certain hyperparameters (e.g., batch size or learning rate), and in rare cases, subsampled the training data enable successful execution. We provide the prompts in Figure 22 and statistics on the number of modified lines in Table 19
http:://docs..pperevieeeee http///wwwwi.eeeeecceeeeeeaa///p
18

---

Published as a conference paper at ICLR 2026
Table 9: Code availability across major machine learning conferences. We report the total number of accepted papers, the number papers with publicly available code (identified GitHub URLs in ArXiv abstracts), and the corresponding percentage for each venue. The last row shows the average across all three conferences.
Conference
# of Accepted
w/ Code
Percentage (%)
ICLR 2024
2207
467
21.2
ICML 2024
2610
435
16.7
NeurlPS 2024
4006
825
20,6
Average
2941
576
19.5
Table 10: Average Replication Scores (%) on PaperBench Code-Der For all OpenAI models. the reasoning effort is set to high, and we take results for BasicAgent and IterativAAgent from Sttraaee al. (2025). For PaperCoder, we report the average and standard deviation over three runs, except
and 03 due to costs.
Model
Replication Score (%)
Cost per Paper (5)
BasicAgent (63 mini)
5.1 = 0.5
N/A
BasicAgent
1.2
N/A
BasicAgent clonde-3--soomee)
35.4
AerrttivAAgen (o3-mini)
16.4 1.4
N/A 400.00
IeerativAAgent (ol)
49.3 =: 1.1
BerativeAgent (clande 3-5 5-somnet)
27.5 1.6
NiA
PoperCoder (o3-mini)
45.14
Coder (ol)
38.31
8.81
PaperCoder (o3)
60.86
8,99
PoperCoder (claude-3-3 3laaule-35-scomeel 3-5-sennet)
51.14
3.61
B ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
B. CODE AVAILABILITY
To estimate the proportion of accepted papers that release official code repositories, we collect data from three major machine learning conferences in 2024: ICLR, ICML,, and NeurIPS. Specifically, we first retrieve the list of accepted papers from each conference using the OpenReview AP via openreview wew_ewwwweeeeeeeeeeeeeeee___ee_e - While OeenReview abstracts sometimes include repository links, they are more commonly found in ArXiv 10 abstracts. Therefore, we additionally use the Semantic Scholar to obtain ArXiv abstracts corresponding to the accepted papers. We then check whether the
AP I abstract includes GitHub URL
indicator of released code. Table Summmarizes the number of accepted papers, the number with publicly available repositories, and the corresponding percentages for each conference. On average, only 19.5% of accepted papers in them provide official code.
B.2 ADDITIONAL ANALYSIS ON EXECUTABILITY
Table 11: Developer time comparison.
As discussed in Section 43 quired for execution are overwhelmingly simple, which are minor syntax issues, missing imports, small adjustments to variable names rather than logic- or architecture-level revi- sions, as shown in Figure 12 with statistics in Table [5]
observe that the fixes re-
Method
Developer Time
CoLoR
cognitive-behavion RADA
self-instruct
To quantify this more concretely, we further estimate devel- oper time by multiplying the number of modified lines by a difficulty factor (1 3);
geval
1: simple syntax or typo fixes, variable renaming, comment adjustments
2: fixes requiring local reasoning (c.g., adjusting conditions or API usage)
3: nontrivial issues requiring deeper debugging (asynchronous memory-related errors)
The results of this estimation are then reported in the developer time column of Table [1] which demonstrates that correcting the emrors does not require a significant amount of time.
http://sshtl..coomaaaaaaaaaapper hhttpst//arxxvxorr//
19

---

Published as a conference paper at ICLR 2026
In addition to human debugging, also perform LLM-assisted repair to examine whether the generated code can be fixed automatically without human intervention. We use 04-mini-hiqe and GPT-5 during the repair stage. For each repair round, record (i) the number of iterations required, and (ii) the error categories identified using GPT-5.. 5. 1. The results, included in Table [4] then show that this automatic repair strategy resolves all the issues mostly within small number of iterations and that the errors are primarily syntactic or import-related rather than logic-level.
B.3 PAPERBENCH CODE-DEV RESULTS
We conduct additional experiments using various reasoning models, as shown in Table Overall, our method achieves strong replication scores across models. Notably, when using o3, PaperCoder records the highest score of 60.86%. These results suggest that the latest and larger models, particularly those with stronger reasoning and coding capabilities, tend to yield better performance.
B.4 IMPACT OF PAPER CONTENT ON CODE GENERATION
To examine the extent to which the clarity and speci- ficity of the paper content influence code generation quality, we remove the Methodology section from each paper and use PaperCoder to generate the correspond- ing code repository. Specifically, this experiment is conducted with 30 papers (10 from each conference) in Paper2CodeBencch with o3 33-minm n i gigh as the backbone LLM. As shown in Table 12 the average score drops from 4.26 to 3.75 without the Method- ology section, indicating that when detailed specifications are absent, the generated code quality degrades substantially, which supports the importance of precise and explicit descriptions for faithful paper-to-code generation, as well as for human readers seeking to understand and reproduce the work.
Table 12: Comparison of reference-based aver- age scores between the full paper content and the paper content without the methodology section on the subsampled Paper2CodeBench. Values in parentheses indicate the standard deviation.
Full (Original)
wlo Methodology
Ref-based Average Score
4.26 (0.28)
3.75 (0.55)
B.5 MOST COMMON TYPES OF ERRORS AND FAILURE MODES
To analyze failure cases, we execute the generated repositories on Paper2CodeBench (without debug- ging) and inspect the resulting errors. We note that each error is automatically categorized by prompt- ing 14-mini-high. with the raw error message and mapping its response to a canonical taxonomy. As summarized in Table 13 the most frequent causes are MissingDDependency, ImportError, and ModuleNot- FoundÉrror, that order. This pattern suggests that environment and packaging issues dominate over al- gorithmic or logic errors in practice.
Table 13: Categories of error types observed when running Paper2CodeBench Categories are ana- lyzed using 04 04-mini-high and Count indicates the number of papers belonging to each category.
Category
Count
Category
Count
Dependenyyy
ConfgurraiooEEmr
Importirnor
SystudEnor
ModoolNool
Success
Valoetmror
OSenrr
Pypelirnoe
RuntineFnor
Auri
B.6 ADDRESSING ENVIRONMENT SETUP
Our framework focuses on faithfully reconstructing the methodological pipeline, rather than fully automating environment configuration. In practice, environment setup tends to be far easier and more reliable for users to adjust manually, whereas reconstructing the core algorithmic logic from natural language is substantially more challenging and central to our goal. For this reason, PaperCoder prioritizes method-level faithfulness over full automation of environment specification, which is highly paper-specific and often varies across systems. Nevertheless, to further handle this environmental issue, we have conducted an additional experiment on 30 papers (10 from each conference in Paper2CodeBenchh) augmenting PaperCoder with prompts (shown in Figure 33) explicitly aimed at inferring and repairing environment requirements. Then, in these cases, we do not observe gependency-relattd failures, suggesting that dependency issues in Table [3 are not from the intrinsic weaknesses of the framework. We view incorporating a dedicated environment "DevOps" agent an interesting extension for future work.
or
20

---

Published as a conference paper at ICLR 2026
B.7 ANALYSIS PERFORMANCE ACROSS PAPER CATEGORIES
Examining performance across different paper categories helps reveal where code generation is easier more challenging. To achieve this, categorize 90 papers in Paper2CodeBench using 04-mini-high, and then report the average reference-based scores per category in Figure T3 First, we observe that the scores range from 3.38 to 4.21 maximum gap of about 0.83). Specifically, theory and
achieve the highest scores (4.21 and 3.97), while reinforcement learning/conti and focusse papers yield the lowest (3.38 each). These results suggest that there are measurable variations across different categories of papers when implementing them with PaperCoder, with some types of papers being easier for PaperCoder to implement than others.
B.8 DISCUSSION AGAINST DATA CONTAMINATION IN PAPER2CODEBENNCH
We evaluate target papers from ICLR/CCML/Nuurl/ 2024, because the primary models used in experiments (OpenAl 03-mini and GPT-4o) have a knowledge cutoff of October 2023, meaning these 2024 paper/code repositories fall after the training window of models. While we cannot fully rule all forms of indirect exposure, the temporal gap substantially reduces the likelihood of contamination, and thus, data contamination is unlikely to meaningfully affect the results.
B.9 FOOSS-FAMILY VALIDATION FOR PAPER2CODEBENCH MODEL EVALUATION
While human evaluation might be the most reliable form of assessment, conducting full-scale human evaluation for all baselines and papers would be prohibitively expensive (Zheng et al., 2023). Therefore, adopt the standard LLM-as-a-juude T5 03-mini-high correlates strongly with human judgments (0.78 in the reference-basee and 0.73 in the reference-fie free setting), suggesting that the LLM--bssed evaluation could be reliable proxy for human assessments. Nevertheless, to further address the concern on the circularity and bias, specifically that an evaluator from the same model family might favor code generated by the same model family, we perform a cross-family evaluation with Gemini-Flash 2.5 on 30 randomly selected papers of the Paper2Code benchmark. The resulting Spearman correlation of 0.73 between Gemini- Flash 2.5 and o3 13--ini-hight indicates that the evaluation remains consistent across different model families, providing strong evidence that our results are not an artifact model-family alignment.
strategy for primary assessment. Also, as shown in
setting
C LIMITATIONS AND FUTURE WORK
While PaperCoder demonstrates strong performance reproducing machine learning papers (where code implementations are particularly helpful and usually necessary for validating research ideas), its current scope limited to this domain. Beyond this, we believe accelerating the reproduction of scientific discovery other domains where code is not the primary medium for validation, such as theoretical mathematics, is an exciting direction for future work. In addition, the current version of PaperCoder processes only textual inputs, and extending
process visual inputs (such figures in papers) with recent OCR models capable of extracting figures and tables (in addition to text) (Wang
al. 2024a He al., 2024 Wei al., 2025 Li et al. 2025 Zhang al. 2025 Niu al. 2025
an interesting avenue. Lastly, as with other repository-leve code generation approaches, improving executability remains an important (but still challenging) direction for future work.
21

---

Published as a conference paper at ICLR 2026
Table The number repair iterations (with the LLM) and error categories. We use o4-mini -high and GPT-5 for the automatic repair, and GPT-5. . for identifying the error categories.
RepoName
Tteration
Error Category
CoLoR
Configuration Update
2
Code Refactor Robustness Enhancement
Bug Fix
Performance Adjustment/ Behavior Control Update
Cognitiee-cchavions Behaviors
Configuration Update
Bug Fix / Type Compatibility Correction
Autograd Control Memory Optimization
API Modernization/ Correct Mask Handling Behavior Tuning (Token Budget)
Bug Fix / RRhhssnesss Improvement
Representation Handling Feature Refactor
Configuration Robustness Type Validation
Training Efficicncy Optimization/ Memory Reduction/ Autograd Hygiene
Memory & Performance Optimization/ Behavioral Correction
Configuration Robustness Type Validation
11
Algorithmic Refactor Performance & Memory Optimization
API Modemization/ Robustness Improvement
RADA
Configuration Update
API Modemization Compatibility Update
Self-Instruct
Configuration Update
2
API Migration Compatibility Update
API Modemiization SDK Refactor
API Rollbuck/ Correct Endpoint Restoration
API Consistency Fix
API Field Access Modernization
G-EVAL
Modemnizaiion SDK Migration
Debug Cleanup/ Schema Update
Table Executability analysis results on the repositories: we sample five papers and generate corresponding repositories using PaperCoder. For each repository. we report the number of lines modified during debugging. the total number of lines, and the percentage of modified lines.
Repo Name
CoLok
cognitive-behavios
RADA
Self-Instruct
G-EVAL
Average
Modified lines (*.py)
26
10
8
Modified lines (config.
7
1
4
3.5
Total lines
1132
2060
1609
1334
1374
1251.5
Percentage
0.44
0.29
1.06
2.02
1.02
0.81
Table 16: Qualitative analysis of top-ranked repositories. categorize the reasons why human annotators select the repositories generated by our PaperCoder framework as their top choice into six (described in the first row). We also show example of the response in Table
Completeness
Clean Structure Faithfulness Paper
Ease of Use
Code Quality
Unique Strengths
16
22

---

Published as a conference paper at ICLR 2026
Table 17: Example responses from human annotators for the reasons top-ranked repositories evaluation. We ask the human evaluator (who is the first author of the paper to be reproduced by PaperCoder) the following question: "Among the top-ranked repositories in each group, which one do you think is the best? If the repositories are the same, you may select any of them. Please bricfly explain your reasoning."
human
RepoName
Response for the top-ranked repositories
Janus
The code includes everything needed to implement the paper. The code is clean and easy to understand (not overloaded with comments). It's not overly packaged, and since config files are provided, it's convenient running various experiments.
VideoRAG
Each component- dayaonnnt--daa processing. retrieval, frame selection, generation, and evalua- tion-is clearly separated into own module, making the system easy maintain and extend. Morcover, the most critical modules are fully implemented, covering the essential functionality required by the framework.
T1
This repository successfully implements the following: Tool-based filtering, Calculate score of cach generation after filtering using verifier model, Training code for distillation.
Table 18: Analysis of the results from the reproducibility case study.
Repo Name
Analysis of Reproducibility
CoLoR
Exccution was successful, but the ORPO loss likely mis-specified, causing the compression model to fail training intended. This issuc stems from the overly simplified description the loss function the paper.
cognitive-behaviors
Successfully reproduced SFT RL training processes but encountered minor error parsing model responses during evaluation.
RADA
Implementation closely matched the paper, missing details prevented full reproduction of the reported results, leading to identical samples.
Self-Instruct
Executed smoothly and accurately reflected procedure described in paper.
G-EVAL
Implemented only the Coherence metric, though the original paper included Coherence, Consistency, Fluency, and Relevance. The Coherence implementation was faithful and correct.
Table 19: Comparison of the total lines, modified lines, and percentages when applying automatic dehugging on 10 papers from Paper2CodeBencch used for human evaluation.
Abstract
Puper
MetaGPT
ChatDEV
PaperCoder
Modified lines (*.py, #,sh, (.yaml)
30
705
226
275
780
Total lines
3517
3047
8225
4185
16189
Percentage
0.85
23.14
2.75
6.57
4.82
23

---

Published as a conference paper at ICLR 2026
+ Arch. Design
main.py
Goe onitted ..))
DatasetLoader:
(... onitred ...)
Logic Design
[uple|Dateloaser,
Datakoader, D
10
"*"Returns Dataicaders for train, validation
moin.py
and best splits.
train_datases TimssarisdDeaaeet
G... ...)
12
-staset_namessett..o www.sseeettttssseeeeee..... mode-"
trainf,
class Maint
13
Lookaack htt.:/too_..cctt...........e.ee. pun_ssomeee-eee 1at.ppi.c...11.............
G... ...)
14
15
def ron_oxperiment (cclx) -> None:
16
col_osttset TimeBerierDatasett
nenuruns: seli.confic.
val",
Istursrrrr"""" 51
18
11
seed send_paces int selt.
nun_semp_cc so1...on,
20
for run in range(1, nun_rana 1)
21
dest_dataset FincsoriesDttazes/
22
mode
14
dataset Leader
ceat",
DatasetLoudor (sel1.contio
23
Lookback iet.ikkkkrei....obacee..
caders dade_i
prodiction_tariion wtw.roooooottoooooorsoooee selt, wtt.aateo.......p.ts...oooooooo trotoooo.oo.....
dataset_ Loader.
Low.eet___ddor..eeee load_caca((
Esain_loader
data_
26
batellaize zw: int self wwlffeoaffg "aaiiign"iiiigg"
17
val roadee
gst ""batchhsszze" 32)
latd__ ["Va1"
27
tradn_icader DataLoader derasn_darasee,
bast_icader
batch iast.aaaaeaacccssii shuffle=Trual
data__oadees [atk_loadees("aaatt ("Laath
28
val voa_a bataloadar ivai dataset,
bacch_sstz-bbcch_eeo. shucric Falsee
Lesi_loades Databeadex (sea-_dat batchhaizeebbachl.u.ee shurfla-Falso)
lataset_loader..
30
eeturn lowde_, loa_ee, loader
(... ...)
31
32
onitted ...)
class Datasel Loadwes
34
class Hain:
G... oritted ...)
35
oritled ...)
9
def load_ dat__dat))) -> Tuplet
38
def pun experiininn (self) -> Nora:
Datalonde:]:
39
for detasel in dalagel nnnees
10
40
zange (self. (self.ran_
11
Loads the tine aeriex date, training, velidetion, and tess sels, and returna the correspending
loader_i instance Decaset Loader leelf
into
.config, dat-aall_ramm)) toade_ val_loadar,
42
dete
12
objecta.
dataset__oader.p loader.py
13
14
Returns:
oritled ...)
Tuple (acaaladee,,
Datatosder, Datslesder tra n. leader, val loader, Lesi_]oadee]
class Pntasetfaaders
G... ...)
food CSv using pandes de: pd-DataPrame = pd. read dav leelt dat......attt.......o.. till)
def load_ > Tuple Tuple[Balaloasis Datatosder, Batstoader] batchusizes
18
int (ntl.. fig.get
craining", (ll.get ("basch_size" size", 3211
G... omitted ...)
11
train Datalceder bataleader
wiwebaiiteeeehh... True, =on_oorker--01 val loaders Datalosder Dateloeser(
drain_datased
shufflee
12
val daladataset, vid__xiaataattluuii shoffles Falce, vorkeeee))))))
test. bazch_sizz-ban biw.hiiattcccsteee shutrie" :est_dataset, False, workeree))))
14
zeturn train, loader, val. loader, test loader
Case study ablation experiments. Without dependency-awware generation order, the Architecture Design stage causes main.py to duplicate functions from Joader.p. In contrast, adding Logic Design phase resolves this issue by aligning the generation order.
Figure 7:
24

---

Published as a conference paper at ICLR 2026
CoLoR
condii.yyan
model:
hoce_nodee
sitarnacivo_neddli
"mistraiii /tw.:osa-----nnneeeeee---
*meta- "lana/Llaaa Llana-naa..- 2.2-38--naaeeee "B-Innacuuct
trainer.py
self.optine
mow.e..el.l......mo.oo.oooool mndell paaannetera (1,, Ire1s]
self. self.optimis optimizer AdanM wew...nodde model. peedel..paameeeeee parameters OJ le=float
nudel.py moldl AuroMoode1FoCCaaaall Ccon_pratraione (nase_roddl))
sel L.ndel AutoiNadil ForCause 11M, Fron_pret gretre ra rained
mose_n Lcust__anee 2_odde=22ue
Figure 8: Case study reproducing paper, called Efficient Long- Context Language Model Retrieval with Compression (Seo al. 2025) (repository: CoLoR), displaying the files modified during manual debugging.
bognitive-behaviors
oun.fig .ygl
1e-5
peak looningggnngge 0.00001
actor acter_learnin perraninnnnneetee 16-6
critin Ieetineeeeeeleee Jearning leerni 1e
lsasaing_rater 0.000001
critic teaniingatrn
10
nodel:
base_nodals:
*Ilame 38*
13
2.ven-2.B-20 33"
14
"reea-1 Tamma//a/// /1. ams- 322-3.
(0enn/ 1.58*
17
addit Fanal made 81
18
11 amlam 3.1 708*
Figure 9; Case study reproducing paper, called Cognitive Behaviors that Enable Self' Self- Improving Reasoners, or, Four Habits of Highly Effective STaRs (Gandhi 2025) (repository: bognitive-bhhavior),, displaying files modified during manual dehugging.
25

---

Published as a conference paper at ICLR 2026
RADA
fig.yanl
t bbbee
model nemer "t5-baze*
"goole=tt/tt--zse"
1lans2
model_ name: "Lieno2-7B"
"nete-LLomm//laam-2--0
10 summntattionmmee::: "Slamo2-1B-(taz"
[1
augrantation_nodeli
12
rotrioval:
entedding mose_1 peattilbert-bare-te -i--ee--r----e st-------------oeeeeeee
15
encessinn_nocall "conten:o-troncoommees bor.rr/mmeeettt.tooooooooeeeeeee
16
path: deon
18
aatarnai_dato_peet tttts//xterna_datt.....
genezator: Generater Sonaratoz otw.m/oddi-ommmtttttssooooooo
generator: Generster Denorator
Hnnmmedd-aaugeentttiiuuuu renaratien_perrneee
"nex_icngth"; 2048, "Lerperatura": 0.7, "top_k": 50, "Lup_p"s "pon__eterr_pppraaassr"
11
12
Figure 10: Case study on reproducing a paper, called Retrieval- Retrieval-Augmented Data Augmentation for Low- Resource Domain Tasks (Sco et al. 2024) (repository: RADA), displaying the files modified during manual debugging.
self-instruct
contig.yani
davinal angine: 3pt-4.1-nam
detasee dettaset__ldder... Joader.py import
from openal import CoenAI
client Openkl I lepl__ky-oss www.kkkyyssss..e...... env environ-gee ron-ge: (TOPENAT_AP APT_KE11))
wngineeeell: E.engine,
psompi=proupl
tokenee 50,
zesponie client wtw...t..ppllltt................ iors.cratee
model=self. engiine
12
meszagese fro et: "oswef, Foontent prompt H1,
13
nex_complel Lon__okeens1150
15
rew.
reagon se, aho loes [0], stees.as (p
16
taw_ exL
resounse olio Lroes [0] meessage,, content, steip()
17
18
resgonse
rip loweri
answes zwapomse chs [0] .ww.saage content catr lowes ()
20
21
generated
oho own [0] wettaatriii
gen.erabed swponaseeco [0] mew.aage.....om ipt
Figure 11: Case study reproducing paper, called Self-Instruct: Aligning Language Models with Self- Generated Instructions [Wang et al. 2023) (repository: self-instruct), displaying the files modified during manual debugging.
26

---

Published as a conference paper at ICLR 2026
geval
con giggyanl
name: toxt-Gavinct-003
nome: gpt gpt-ne1-nanc
Lerm__evaa_aarr data/eunneva 11l1___.0 dinlogue data/topical csv mallacinatiinnnam. dato/qa35.cop
contig.py
"nano": "ept-d",
*name": "spt-lo-mina",
lin_evaluatae .9
from openal import OpenAΣ
elleat OpenkI www.k/yaw.....oo........ ("OPEONAT_APT_RREC
seit.campies:
("sampics", 231) if self meli..ndel_panneeo model__ane..everee Lewer)) "apt-(" else
self.samples:
(parans..ell ("sanplea", 201) "spt-4" 44" mel...ndde_.mme.eeee else
1t aal..modei_ncme 1oNCE "gpt-1":
reaponse
"apt-d" sold.mode nane. 1o
11
reaponse ol ient chhll. complet cos.ello.....eeeeee....
12
nax_tokans te.ena,
nex_pomplel iin Lokensseee (itax. Lokeme,
15
16
coN:: stz zesponse ("choicos") [0] ("massage") ["ccatent") ".trrippp
zwapunse, cho messaageee..m cuntent. stx
19
choicel ("ncssage") ["contsat") stsrrpp() for choice in response.gcttshhceeeee. ("chcioss" (3)
20
cliccow content sixigit For choice in
choices
score normall
awlf fondellll Jover "gpt nocmall Exacion and "ypt-4" in sel_.nndel__aarr nowee (3
23
1f
Figure 12: Case study on reproducing a paper, called G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment| ( Liu et al. 2023) (repository: geval), displaying the files modified during manual debugging.
Average Score by Category
Theory
4.21 (6)
Interpretability Explainability
3.97 (8)
Application Time Series
3.95
Evaluation Benchmarking
3.94 (13)
Application Multimodal
3.83
Category
Application NLP
3.80 (1.8)
Training Technique
3.79 (42)
Application Vision
3.74 (23)
Optimization
3.67
Architecture/ Model
3.65 (48)
Application Speech/Audio
(1)
Application RL/ Control
3.38 (4)
Dataset
3.38
3.2
3.4
3.6
3.8 Average Score
Figure 13: Average (measured by reference-based evaluation) per category Paper2CodeBench. The numbers to the right each bar indicate the average score, along with the number papers parentheses. Bar transparency is proportional to the count, highlighting categories with more or fewer papers.
27

---

Published as a conference paper at ICLR 2026
D PROMPTS
Prompt for generating the overall plan in the planning stage
[System]
You are an expert researcher and stralegiic planner with deep understandingg of experimental design and reproducciility in scientific
You will receive research puper JSON formal.
Your lask create detailed and efficient plan to reproduce the experiments and methodologies descrihed the paper. This plan should align precisely with the paper's methodology, experimental setup, and evaluation metrics,
Instructions:
Align Paper: plan must strictly
methods, datasets, model configurations hyperparameters,
experimental sctups described in the Be Clear and Structured: Present the plan in a well vell-orranized and easy-to-fain follow format, breaking it down into actionable steps. 3. Prioritize Efficiency: Optimize the plan for clarity and practical implementation while ensuring fidelity original experiments.
[Uber] Hn Paper [paper_ison
Task
We reproduce method deseribed attached paper.
2. The authors did not releuse any official code, so we have to plan our own implementation.
Before writing any Python code, please outline comprehensive plan covers:
Key details from the paper's "Merhhddooogy"*
Important aspects "Epprriments** including dataset roquirements, experimental settings, hyperperameters, or evaluation
metrics.
4. The plan should be **dttailed and informative** as possible to help us write the final code later.
#* Requirermnnt
yet: focus **thorough, clear strategy**. If something is unclear from the paper, mention explicitly.
need provide
** Instraction
The response should give us a strong roadmap, making it casier to write the codc later.
Figure 14: Prompt for generating the overall plan in the planning stage.
28

---

Published as a conference paper at ICLR 2026
Prompt generating architecture design planning stage
[User]
Your goal to create concise, usable, and complete software system design for reproducing the paper's method. Use appropriate open-source libraries keep the overall architecture simple.
reproducing paper's 's method, please design concise, complete system. Keep the architecture simple and make effective use of open-source libraries.
Example
[CONTENT]
"Ioplementation appronch": "We will
"File list":
"main.py",
"dataset_ "onderpp.
"model.py" py". "trainerpp"" "evaluation.p"
"Data structures and interfaces": "InclassDiarram'n
class Main
+ Liiiit___00 (im
+run experrmmmin
In
class
DatasctLonder In inin (purams: dictim
init (config: dictikn
+load datal)
Any'a 'n
class
Model
In
+forwardix:
Tensor)
-> Tensoria
In class Trainer
(moudel: Model, data:
Anyja
+train)
Evalustion
In
init (model: Model, data: Anylin
+evalunte() dictin
bn
Main >> Datasetl anderin
Main
Trainerin
Main Evaluation'n Trainer -> Modelin..
"Program call flow": "assquccneDiagrramaa
participant M Mainin
participant DL DatasetLonder'a
participant
MD us Modella
participant TR as Trainer'a
participunt as Evaluationia
M--DL: lond dataa))a
DL-oM:
datasetln M-MMD: initialize model(in
TRR train(medel, dataset)in
TR.MM:: forward(xjan
MD wTR:
predictionsán
TR-MM: training completela
M--V: evaluate(model, dataset)ln
EV-MD:: forwardix/ln
MD--EV
predictionsin
metricsla".
"Anything UNCLEAR": "Need clarification the exact dataset format and specialized hyperparameters."
ICONTENTI
Nodes: "enndes:; <type> cinstmuctions"
Implementation approuch: sclass 'stt'> # Summarize the chusen solution straliegy
paths. WAYS Data structures and interfaces: typing Optional[str] Use mermaid classDiagram code syntax, including classes, methodr init etc.) and functions with type annotations, CLEARLY MARK RELATIONSHIPS between classes, and comply with PEPS standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with complete design. Program call flow: typing.Optionii [ptional[st]]] 8 Use sequenceDiugran code syntax, COMPLETE. and VERY DETAILER ED, using CLASSES AND API DEFINED ABOVE accurately, covering the CRUD AND INIT each object, SYNTAX MUST BE CORRECT. Anything UNCLEAR: sclass 'ar'> Mention ambiguities and ask for clarifications.
File list: typing.1 Lis([str] Only
main.py py app-py
## Constraint
Format: output wrapped inside ICONTENTI like the format example, nothing else.
Action
Follow the instructions for nodes, generate the output, it follows the format example.
Figure 15:
Prompt for generating the architecture design in the planning stage. This prompt follows the previous prompt and response shown in Figure 14
29

---

Published as a conference paper at ICLR 2026
Prompt for generating the logic design in the planning
[User]
Your goal break down tasks according to PRDicchnical dosign, generate a task list, and analyze task dependencics..
You beeak tasks, unalyze dependencies.
You outline a clear PRDMtechnccal design for repreduciig the puper's method and experiments.
Now, let's break down tasks according PRDiechnical design, generate task list, and analyze task dependencics.
The Logic Analysis should not only consider the dependencies between files but also provide detailed descriptions to assist writing the code needed reproduce the paper.
Format Example [CONTENT]
"Roquired packages":
"bumpy==1.21U"
"Rqquired Other language third-party packages": "No third-party dependencies required"
"Logic Analysis":
preprocessingg
Prepoccspgg class
"traincr.py".
"Trainer...
"dataset_ "dataset_loader.p"""
"Handles londing and
"model.py".
"Defines the model
"evaluation.py"
"Evaluation class
}.
"main. py",
"Entry point c.m.."
1.
"Task list":
"dataset_ loaderpp." "model.pp". py".
'trainerpy''
"evaluation.py"
"main.py"
"Full API spec": 'openapi: 3.0.0
"Shared Knowladge": "Both preprocccco.ciinggggoo py and trainer.py share ..........
"Anything UNCLFEAR": "Clarification needed recommended configuration large-scale experiments."
[CONTENT]
Nodes: "enodes:; <type> cinstructions"
Required packages: typing. SrtnngOpptonaalypiing List[str]] # Provide roquired third-party packages in requisements.co txt format,(c... 'mmmpywml.. 1.21.0").
Required Other language third-party packages: typing. typing. List(str) List down packages required for non-Python langungos. If none, specify "No third-party dependencies required".
be implemented., including Task list: typing List[str] Break down the tasks list filenames, prioritized based dependency order. The task list include the previously genarated tile list.
Logic Analysis: typing-Listlyppen List[yping. Listjurr Provide list of files with the classevinethncd///neeee
dependency analysis and imports. Include as much detailed description as possihle.
Describe all front-end and back-enn communication not required. leave blank.
APT spec: <olass
using OpenAPI 3.0 spec may used by both frontend and backend.
Shared Knowledge: cclass 'stt' > Detail any shared knowledge. like commen uiility liunctions or configuration variables. Anything UNCLEAR: <class 'ar'>> Mention any unresolved questions clarifications needed from the paper project scope.
Constraint
Format: output wrapped inside (CONTENT like the Format example, nothing else.
## Action
Follow the mode instructions above, generate your oulput accordingly. and ensure it follows the given format example.
Prompt generating the logic design planning stage. This prompt follows previous prompt and response shown Figure 15
Figure 16:
30

---

Published as a conference paper at ICLR 2026
Prompt generating the configuration file in the planning stage
[User]
You write clegant, modular, and maintainable code. Adhere Google-style guidelines.
Based the paper, plan, design specified previously, follow the "Formar Example" and generate the code. Extract the training details from the above paper (c.g., learning rate, batch size, epochs, otc.), follow the "Format example" and generate code.
FABRICATE DETAILS only
paper provides.
must write 'config yaml'.
ATTENTION: Use "He" SPLIT SECTIONS, not 'H', Your output Format must follow the example below exactly.
Firmmt Example
Code: config yaml
"'yaml conlig. yaml
training:
learning rate:
epochs:
Code:
config yaml
Figure 17: Prompt for generating the configuration file in the planning stage. This prompt follows the previous prompt and response shown in Figure l6
31

---

Published as a conference paper at ICLR 2026
Prompt for analysis
[System]
You expert researcher, strategic analyzer software engineer deep understanding experimental design reproducibility in scientific research.
You will receive rescarch paper ISON format, an overview plan. design in ISON format consisting "Implementtatinn appreach"" "File list", "Data structures and interfaces" and "Program call flow", Jollowed by task JSON format that includes "Required packagos", "Roquired other language third-party packages'. "Logic Analysis", and "Task list", along with configuration file named condig.yamil"
comprehensive logic analysis to accurately reproduce experiments methadologies
research
This analysis must aliga precisely with paper's methodology. experimental setup, and evaluation criteria.
1. Align with the Paper: Your analysis must strictly follow the methods, datasets, model configuration,, hyperparameters, and experimental setups described in the paper.
2. Be Clear and Structured: Present your analysis in logical, well bell-oogamized, and actionable format that is easy to follow and implement.
3. Prioritize Efflciency: Optimize the analysis for clarity and practical implementation while ensuring fidelity to the original
experiments.
4. Follow design: YOU MUST FOLLOW "Data structures and interfaces" DONT CHANGE ANY DESIGN. not public memher functions that not exist in your design.
CONFIGURATION Always
vetlings from config- lile. assume any only
use configurations explicitly provied.
[User]
Context
#t Paper
[hhe content the paper json format)
Overview of
(The content the overall plan]
#* Design
The content the architecture design)
## Task
logic design)
Configuration file
"'yaml
[The content of the configuration file)
#A Instraction
Conduct a Logic Annlysis to assist in writing the code, based on the paper, the plan, the design, the task and the previously specified configuration (config, yaml). You DON'T I' need to provide the actual code yet; focus on thorough. clear analysis.
Write the logic analysis in T[The name of the file gencrated")" J", which is intended for [Description of the file generated through the 'Logic Analysis" step of the logic design.)".
#A Logic Analysis: [todo_like lile name)
Figure 18: Prompt for analysis. (} indicate placeholders be filled with the content described the accompa- nying explanation. The prompt presented the LLM for each following the sequence defined the logic design.
32

---

Published as a conference paper at ICLR 2026
Prompt for coding
[System]
You expent researcher research.
engineer a deep understanding experimental design reprducibilit scientific
You will receive a research paper ISON format, an overview the plan, Design JSON format consisting of 'Impplemnttation approuch". "File list". "Data structures and intertaces" "Program call flow". followed Task ISON format that includes 'Required packages', "Required other language third party packages', "Logic Analysis", and "Task list along with configuration lile named 'condig yami"" Your lask is write code to repreduce the experiments and methodologies described in the paper.
The code you write must be elegant, modular, and maintainable, adhering to Google-style guidelines. The code must strictly align with the paper's s methodology. experimental sctup, and evaluation metrics. Write code with triple quali..
Context
Paper
{The content of the paper ison format)
plan
The content of the overall plan]
#i Design
[hh content the architecture design)
Tisk
[The content the logic design]
## Configuration file
yaam
[The content the configuration lile]
Code Files
{The content of the code files genorated the previous stop.)
Format example
#a Code: [todo_tile_net
"python
## todo, _ilee_nmeeeeeeeeeee
Instruction
Based the paper. plan. design, task and configuration lile(condig. specitied previously, follow
"Format example", write the
code.
We have [done_fft Jst].
Next, you
only the "[todo name)*
1. Only One lile: do your besi to implement THIS ONLY ONE FILE.
will part Set default value: there any setting, ALWAYS SET DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE. AVOID circular import.
COMPLETE,
project, please implement complete,
snippets.
4, Follow design: YOU MUST FOLLOW 'Data structures interfaces". DONT CHANGE ANY DESIGN public
member functions that do not exist in your design.
CAREFULLY CHBCK
DONT
NECESSARY CLASS/FUNCTION
6. Before using external variable/module, make sure import it first.
out
DETAIL., DON'T TODO.
REFER CONFIGURATION; you configuration from "conlig-yanl", DO NOT FABRICATE any configuration values.
[detailed_loghh logic_analysis!
#A Code: [todo_ file, name]
Figure Prompt coding. {) indicate placeholders be filled with content described in the accompanying explanation. The prompt is presented to the LLM for each file, following the soquence defined in the logic design. Previously generated code files are accumulated and provided as part of the ## Code Files input.
33

---

Published as a conference paper at ICLR 2026
Prompt for model-based reference-busee evaluation
[System]
You will be given research paper aloog two correponning code repositories: a gold repository target repository.
Your task to compare the target repository against the gold repository, rate the target repository on one metric, and provide a crilique highlighting key differences.
Please make sure you understand instructions carefully. Keep decument open while reviewing. and refer needed.
Evaluation Criteria:
Curreciness (1-5): The quality of the larget reposilory accurately implementing the paper's concepts, methodology. and algurithms without logical erroes, as compared to the gold repository. Additionally, provide a critique focusing on the completeness, accuracy, and implementation choices made in the target repository relative to the gold repository.
I: Very Poor. The larget repository does not correcdly implement the core concepts, methodology. or algorithms from the paper. Major logical errors missing components present, especially when compared to the gold repository.
2: Poor. The target repocitory attempts to implement the paper's is concapes but contains significant mistakes or missing components, making implementation insoreet when compared the gold repusitury.
3: Fair. Some core components and concepes are correctly implemented in the target repository, but there are notable logical errors inaccuracies compared the gold repository.
4: Good. The target repository correctly implements the components and methodology, with only minor inaccuracies deviations from the gold reposilory.
5: Excellent. The target repository fully and accurately implements all relevant key components, methodology, and algorithms from the paper, matching the quality of the gold repository.
Evaluation Steps
Identify Key Aspects the Paper: Carefully read the research paper understand its concepts, methodology, and algorichms, Pay close atteation to the key aspects that are crucial for implementing the paper's results (c.g. speoific algorithuns, data preprocessing steps, evaluation protocols).
2. Analyze the Gold Repository: Examine the gold repository to understand bow these key aspects have been implemented. Use the gold repusitory reference for how the puper's methodology should be translated into code. Note the completeness, accuracy, and design choices in the gold repository that faithfully represent the paper's concepts.
3. Examine the Target Ropository: Analyzo the target repository to assoss how well it implements the key aspects of the paper. Reference the gold repository as a guide understanding these key aspects in the targel repository. Focus on whether the Larget repository's core logic, algorithms, and structure align with the methodology and experiments described in the paper.
4. Identify Logical Emors and Deviations: Check for logical errors, missing steps, deviations from the paper's 's methodology. Note any incorrett representations, inconsistencies, or incomplete implementations that could affect the conretnness of the target repository.
5. Provide a Critique: Consider both the completeness and accuracy of the implementation relative to the paper's ssss goals and the gold repesitory's standard. You not need analyze minor details like logging functions, script organization, or documentation quality. Instead, concentrate on the correctness of the logic and implementation that ensures the core concepts from the paper are fully rellected the largel repusitory. For each mismulch or deviation in implementation. note down specilic critiques comparing relevant functions the target repository the conrsponning functions in the gold repository. Highlight incomect logic, deviations that affect the correct implementation of the paper's methodology.
steps,
5. Assess the Correctness: Determine whether the target repository includes all the critical elements described in the paper and implemented in the gold repository. Identify missing components, significant deviations, or incorrect implementations that could affect the correctness of the larget repository.
6. Assign Score: Rased on your evaluation, provide critique and assign correcteess from for the target repository, reflecting how well it implements the key aspects of the paper refor to the gold repository. Include a detailed critique in the specified JSON format.
Severity Level:
Each idcntified critique will be assigned severity level based on its impact on the correctncss of the mothooology implementation.
High: Missing or incorrect implementation of the paper's 's core concepts, major loss functions, or experiment components that are fundamental to reproducing the paper's methodology.
Example: The main algorithm is missing fundamentally incorrect.
Medium: Issues affecting training legic, data preprocessing, or other care funetionalities that significantly impact performance hut do not completely break the system.
Example: Improper training loop structure, incorrect data augmentation, or missing esscutial components in data processing. expected results but errors in the evaluation process helong to this category unless they impact the core concepts. These include minor issues like logging. enror handling mechanisms, configuraton settings, evaluation steps that do not alter the fundamental implementation and additional implementations not explicitly stated in the paper.
Low: Errors specilic features that cause deviations
worked around with mudifications. Any
Example: Suboptimal hyperparameter initialization, incorrect learning rate schedule, inaccuracies in evaluation metrics, using a diffcrent random seed, variations batch processing. different weight initialization, issucs result logging reporting. variations in evaluation dataset splits, improper error handling in non-critical steps, mismatches in secondary evaluation criteria, or additional implementation details not specified the paper do not interfere with core results.
34

---

Published as a conference paper at ICLR 2026
Prumpt model-based reference-buved
Example ISON format:
""json
critiqqe list":
"gold_fibe name": "preprocessing.py py"
'gold, fine_namss "data_ processs
"target_fila finget_fll_ammm..:
name": "datasct.py".
"target fune name": "train, preprocessss "severity level"; levell" "medium", "critique": critique
target repository's 's to gold repository."
'gold_fill name": "utils "gold_fune_nen name": "caleulate_metric metric".
"Larget name": pmetric.pp". "target func name": "fl "fl_at"""
"severity_ leve!"; "low".
"critique": : "A critique the target repository's file with reference to the gold repository."
"score":
Sample:
Research Paper:
[[The content the paper)]
Gold Repository:
[[The gold repositury, officially released by the authors, serves the reference implementattion])
Target Repository:
[[The generated repository, which the target repository for evalustion..)
provide critique target repository single numerical rating following the Example JSON format, without any additional commentary. formatting, chartinoss.
5) based quality sample,
Figure Prompt for model-based reference- based evaluation. [{)] indicate placeholders to be filled the content described the accompanying explanation.
35

---

Published as a conference paper at ICLR 2026
Prompt for model-based reference-free evaluation
[System]
You will be given research paper along its corresponding code repository.
Your task code repository on one metric and provide critique highlighting key differences.
Please make sure you read and understand these instructions carefully, Keep document open while reviowing, and refer to as needed.
Evaluation Criteria:
Correctness (1-5): The quality of the repository in accurately implementing the paper's is concepts, methodology, and algorithmns without logical erroes, Additionaly, provide crilique locusing the completeness, accuracy, and implemeniation choices made the repository relative to the methodology and algorithms descrihed in the paper.
I: Very Poor. The repository does not correctly implement the core concepts, methodology, or algorithms from the paper. Major logical errors or missing components are present.
2: Poor. The repository sttempts to implement the paper's concepts but contains significant mistakes or missing componsents, making the impleceeniatio incorrect.
Fair. Some components and concepts corectlly implemented, but there notable logical methodology.
inaccuracies in the
Good. repusitory correctly implements key companents and methodologyy with only minor inaccuracies that not significanlly affect correcnees. Excellent. The repository fully and accurately implements all key components, mothodology, and algorithms from the paper without logical errors.
Evaluation Steps
1. Ideatify Key Aspocts of the Paper: Carcfully read the paper to understand its core concepts, methodelogy, and algorithms. Pay close attention to key aspects crucial for impletenting the paper's results (e.g.. specilic algorithus, data preprocessing steps, evaluation prolocols).
2. Examine the Code Repository: Analyze the repository to determine how well it implements the key aspects of the paper. Focus on whether repository's core logic, algorithms, stricture align with the methadology and experiments descrihed
paper:
3. Ideatify Logical Erroes and Deviations: Check for logical errors, missing steps, or deviations from the poper's 's methodology. Note any incorrect representations, inconsistencies, incomplete implementatiinns that could affect the correctness of the reposilory.
4. Provide Critique: Consider the completeness and accuracy of the implementation relative the paper's goals. You do not need to analyze minor details like logging functions, script organization, or documentation quality. Instead, concentrate on the correctnees of the logic implemeniation to ensure core concepts from paper are fully reflected repusitary. each identified issue, write detailed critiquc specifying the affected files and functions in the repository. Highlight missing or incorrectly implemented steps that impact the correctness and alignment with the paper's 's methodology.
5. Assess Completeness and Accumey: Evaluate the repository for its completeness and accuracy relative to the paper's methodology. Ensure that all critical componcuts-succ data preprocessing, core algorithms, and evaluation steps-are implemented and consistent with the paper's 's descriptions.
6. Assign& Scere: Based your ovaluation. provide critique and assign correctness from for the repository, rellecting how well it implements the key aspects of the paper. Include a detailed critique in the specified JSON formal.
Severity Level:
Each identified critique will be assigned severity level based impact on the correctness the methodology implementation.
High: Missing incorrect implementation of the paper's concepts, major loss functions, experiment components that fundamental to reproducing the paper's methodology.
Example: main algorithm missing fundamentally incorrect. Medium: Issues aflecting training logic, data preprocessing.
other cure functionalities that significamlly impact performance but de not completely break the system.
Example: Improper training loop structure, incomect augmentation, or missing essential companents data pmocesing.
Low: Errors specilic features that cause deviations from expected resalts but be worked around with modifications. ervors in the evaluation process belong to this category unless they impact the care concepts. These include minor issues like logging, handling mechanisms, configuration settings, evaluation steps that do alter the fundamental implementation and additional implementations not explicitly stated in the paper.
Example: Suboptimal by perppurmeeeerr initialization, incorrect learning rate schedule, inaccuracies evalustion metrics, using a different random seed, variations in hatch processing, different weight initialization, issues in result logging or reporting, variations evaluation dataset splits, improper error handling non-critical steps, mismatches secondary evaluation criteria, or additional implementation details not specified the paper that do not interfere with results.
36

---

Published as a conference paper at ICLR 2026
Prompt for model-based reference-free evaluation
Example ISON format:
'"json
ccritqque_iis list":
"ille name": "dataset.py".
'func name": "train, preprocess",
"severity level"; "medium",
"critique": : critique target repositoy''s file."
"file "metrics.p"" 'func name":: "TI [al_aa__ k" "severity_ level"; "low". "critique": "A critique of the target repository's file."
"score":
Sample:
Research Paper:
[[The content the paper)]
Code Repository:
[[The generated repusitory, which serves us the target repository for evalustion.]
Please provide critique list for the code repository and single numerical rating 2, 3, based quality of the sample, fallowing the Example ISON format, without any additional commentary. formatting, or chattiness.
Prompt for model-based reference-fee free evaluation. (0) indicate placcholders to be filled with the content described in the accompanying explanation.
Figure 21:
37

---

Published as a conference paper at ICLR 2026
Prompt for LLM-assisted debugging
[System]
You are highly capable assistant specializing dehugging real-world repositories. provided
(1) code repository (in in full), and
(2) one or more execution error messages generated during execution repository.
Your objective is to debug the code so that it execules successfully. may involve identifying ying roat causes emoors, modifying faulty logic synlax, handling missing dependencies, making other appropriate corrcctons.
Guidelines:
Provide the exact lines lile changes needed to resolve the issue.
When nocessar. suggest practices improvements to prevent similar
Show only the moditied lines using unifted dift' format:
wwac SEARCH original
corrected line
REPI ,ACE
multiple needed, provide sequentially sepuration.
If external dependencies or environment setips are roquired te.g,, packages, versions, file pathsj, specify them explicitly.
Constraints:
Do not make speculative edits without justification.
Do
internet connection for installation retrieval unless explicitly stated.
Prioritize minimal and effective lixes that preserve the original intent of the code.
Maintain coding style structure original repository refactoring necessary correctness.
[User]
Hht Code Repository [[oodss]]
MA# Execution Eeror Messages [[execution mogg__
Instruction
Now, need debug the above code that it runs without errors. Identify the cause of the execution error and modify the code appropriately. Your output must follow exact Formal as shown in the example below.
## Format Example
Filename: train.py
SEARCH
resuli model, predict(inpul.daaa)
result mwdeeinput..coooommm data)
REPLACE
Answer
Figure 22: Prompt for LLM-assisted debugging. (0) indicate placeholders to be filled with the content deseribed
the accompanying explanation.
38

---

Published as a conference paper at ICLR 2026
Prompt for verifying overall planning
[System]
You given research paper an accompanying reproduction plan.
Your task to rate plan on one metric provide critique highlighting key differenees between plan and what the puper actually requires.
you understand refer to needed.
carefully. Keep document open reviewing.
Evaluation Criteria
Man--aper Alignment (1-5):
plan aligns puper's methodulogy. experimental setup,
evaluation metrics.
Very Poor. plan is largely misaligned
paper's goals methods, omits critical components (datasets, algorithms,
ovaluation), and shows major misundersanndngss
2: Poor. The plan attempis to follow the paper but significant gaps (key experiments missing, wrong resource assumptions. uncleur success criteria).
3: Fair. The plan covers several core needs but contains notable inaccuracies or omissions (partial experiments, vague milestones, unspecified riskssassumpionns..
4: Good. The plan aligns with most paper requirements, has clear milestones and resources; only minor or ambiguities remain. Excellent. The plan fully aligns with the paper's s methodology and experiments, specifies resources risks procisely, and defines clear, measurable success criteria.
Evaluation Steps
1. Extract Paper Requirements:
Identily objectives, datasets, moxeels/aggoriitmss, and trainingievaluation protocels needed for reprodaction.
2. Map Requirements Plan:
Chesk whether the plan includes corresponding milestones, deliverables, estimates (compute, data, libraries).
3. Assess Success Criteria:
plan defines measurable outcomes
paper's metrics
(e.g.. confidence intervals).
4. Critique:
List concrete misalignments, missing items, and unrealistic assumption;; point to specific plan sections.
5. Score:
Provide single 1-5 rating and detailed critique the specified ISON format.
Severity Level
High: Missing core experiments, datasets, or objectives: success criteria not tied to puper metrics, Medium: Incumplete milessones/essuunees uncleur ablations; weak risk mitigation.
Low: Minor amhiguity timelines, non-critical tooling choices, formatting.
Example JSON format
"json
"critique_lis list":
"plan_sections section": "Milestones".
'severityy level": "high",
"critique": "No milestone for ablation studies described Section of the paper: plan skips required variant training." "
"plan_ soction""oo" "Resourees".
"severity Jovel": lovell" "medium". "critique": : "GPU estimate does not account For seeds experiment as required by the paper's evaluation."
"score":
Sample: Research Paper: ([Paper) Overall Plan: [[Paan]
Please provide critique weaknesses overall plan a single numerical rating (1, 4, or 5), following the Example JSON Format, without additional commentary, formattnng. chattiness.
Figure 23:
Prompt verification in overall planning. (()) indicate placeholders filled with the content described in the accompanying explanation.
39

---

Published as a conference paper at ICLR 2026
Prompt for refining overall planning
[System]
expert researcher strategic planner deep understanding experimental design reproducibilty scientitic
research.
You will receive research paper (ISON format), the original overall plan, and ovaluation critique+score of that plan.
Your task revise and improve the overall based the critique, ensuring fully aligns with the paper.
This plan should align precisely with the paper's methodology. experimental setup, and evaluation metrics.
## Instructions:
1. Fix High/Medium Issues: Correct all critical omissions and misaligramntt from the critique.
Preserve Correct Elements: Keep valid, well-aligned parts original plan.
3. Add Completeness Ensure all methods, datasets, experimental setups, and evaluation metrics from the are included.
Clear and Structured: Present imgroved plan roudmap format actionable steps.
5. Prinriize Ffficiency: Optimize the plan for clarity and practical implementation while ensuring fidelity to the original experiments.
Highlight Changes: Provide summary the key changes you made relative to the critique.
ee Format Example
[CONTENT]
"summary changes":
"Added ablation milestones that missing".
"Specified required GPU hours bused on experiment scale".
"Clarified success criteria tied accuracy Fl metries"
Timproved version": "wcRevised detailed plan heress"
[COOTENT]]
Notes
We reproduce method described attached paper.
2. The authors did not releuse official code, sa we have plan our own implementation.
Before writing any Python code, please outline comprehensive plan covers:
Key details from the paper's " "Methodology**
Important aspects "Epperiments** including dataset requirements, experimental settings, hyperparameters, or evalustion
metrics.
4. The plan should be **detailed and informative** as possible to help us write the final code later.
#A Requirements
't need provide actual code yet: focus on **thorough, clear stratogy**. If something unclear from the paper, mention explicitly.
#i Action
The response should give us a strong roadmap, making it casier to write the code later.
Follow the instructions for and requirements, geuerate the output, and follows format example.
Inputs:
Rescarch Paper: [[Paper]]
Original Overall Plan: [[Pian]]
Critique+Score: [[Critique))
Figure 24: Prompt for refinement overall planning. [()] indicate placeholders be filled with the content described the accompanying explanation.
40

---

Published as a conference paper at ICLR 2026
Prompt for verifying archifecture design
[System]
You will be given research paper and architecture design consisting of Implementation approach, File Data structures and cenefaceatclassdae classDa) Program call (owessqucccelaagaam)) and Anything UNCLEAR intended to complete software system design reproducing the paper's
Your task to rate the architecture on one metric and provide a critique highlighting key differences between the dia- grams and what the paper requires.
document open while reviewing. and Please make you read understand these instructions carcfully. refer to it needed.
Evaluation Criteria
How faithfully architecture design Implementation approach, list, structures and interfaces (classDiagrami, Program call flow (sequenceDDagram) - captures the paper's components, datz/control Blws, responsihiliiies key interfaces.
Architecture-Me Fidelity
Section specific indicators (uxed to inform the 1-5 rating):
Implementation approach
Faithfully reflects paper's algorithmic pipeline, major assumptions, training/evaluution protocols.
Mentions all required opüinizer/salvrr variants, loss termis, constraints, and data preprocessing the puper relies on. Notes reproduciiility critical details (random seeds, determinism settings, hardwaree/recisson) the paper depends them,
File list
Provides clear, minimal, and traceable mapping paper sections code modules.
Encodes stratogy/factory points for ablations (optimiacrs, model variants, datascts) without over-coupling.
Separates concems (IOO vs. training vs. evaluation vs. plotting) anticipates extemsibility.
structures interfaces (classDiagram)
Defines interfaces that match the paper's abstractions (e.g.. loss components, physics constraints, evaluation metries).
Shows inpurs/output and typing consistent with the paper's notation (temsor shapes, units, domains). Exhibits low coupling/high cohesion; substitution components (optimizers, backends) possible without ripple changes.
Program (sequenceDiagran))
Preserves the paper's control flow order (training - validation - testing: optimizer switching: line-search loops). Includes erronledge handling the paper roquires (e.g.. fallback when search fails, carly stopping, tolerance checks). Captures logging, checkpointing, and metric computation the times the paper specifies.
1: Very Poor. Core algorithmic components or llows from the are missing or fundamentally wrong; responsibiltties misplaced.
2: Poor. Attempes the paper's structure hat with major omissions (e.g. missing loss path, preprocessing stage, or evaluation path) incorrect interactions.
Fair, Most components exist, but interactions partially incerrect responsibilities muddled (tight coupling. unclear interfaces).
paper. miner omissions coupling don't block correctness. S: Excellent. Diagrams uccurately reflect all care components and flows, with clear interfaces, appropriate sepuration of concerns. and Imceabiiiity to paper sections.
Components interactions largely
Evaluation Steps
1. Identify Core Components:
From the puper. list modules (data loader, model submodules, functions, trainers, evaluators) key messgges/lowss Implementation approch: Extract all algorithmic steps (data preprocessing. model construction, loss formulation, optimization schedule, evaluation protocols).
File list: Map each paper section/subseccionn to candidate module; mark where ablation knobs (c optimizer choice) must exist.
Data structures and interfaces: Enumeraae the required dlasses/struts//lume and their signatures implied by the paper (input domains, shapes, units).
Program call flow: Outline the exact order of operations (incloding optimizer switching. line-scarch/unee loops, validation checkpoints, and plotting/metris export).
Assess Implementation Approach:
Check whether the description faithfully covers all algorithmic componeuts from the paper (optimizers, loss terms, constraints, PDE Jormulations, evaluntion metrics). Verily clarity on critical reproducibility details (byperparameters tolerance values, data handling).
3. Assess File List:
Judge whether files sufficient, appropriately separated, aligned puper's modules (e.g. configs, logging, checkpointin) over-coupling between responsibiliries.
structure. missing utility
4. Assess Data Structures and Interfaces (Class Diagrams):
Check class respansibilitie, interfaces, cohesion/coupling, extensibility, and fidelity to the paper's abstractions. Confirm that class APIs expose exactly what puper specifies (inputs, outputs, and typing).
Assess Program Call (Sequence Diagrams):
Verify message order, sync/ssync boundaries, optimizer switching, ermoredge handling, and inclusion of triiningdevllaaioonvvliiiiii paths. Confirm that ovaluation and logging happen the correct cadence.
41

---

Published as a conference paper at ICLR 2026
Prompl for verifying architecture design
6. Critique:
missing componenttsrelaaioons, faithful implementation.
message ordering, pour modularity. violation design principles
For cach identified weakness, provide JSON entry that includes:
section: One Implemeniation approach, File list. Data structures and interfaces, Program call flow element: The concrete element under critigue
severity Jevel: high, medium, or low
crilique: cuncise explanation
Score:
Provide single 1-5 rating that reflects overall Archilsecture-Metthoe Fidelity and detailed critique in the specified JSON format.
Severity Level
High: Missing/incorrett modeling core algorithm modules or loss/ovaluation flows; sequence order contradicts pa-
method.
Over Over-coupling, unclear interfaces hindering ablations or reproducibility: partial llww omissions (e-g.. missing validation loop).
Medium:
Low:
Naming incomsistencies, minor UML notation issues, optional utilities misplaced.
Example JSON formal
"json
critique_ list":
list:
"section": 'Implemeatattion appreach'.
"element": : "NysNewton-CUS details".
Leverilt_leveel" "high".
"critique": "Implemenration approch lacks specifies on Nyströmm preconditioner update frequency and PCG tolerance, which essential faithful reproduction."
"section": 'File list".
"element": :: "conlig-py"
'severity level"; "mnedium.. "critique:: "No configuration file listed; paper requires reproducibility across experiments with tunable
hyperparameters.
"section": "Duta structures and interfaces""
"element" :: "LossFunction""
"severity level": "high",
"critique": "Loss components for PDE residuals and boundaryfinitial conditions are nor represented as separate classes; paper emplasizess modularity for ablation studics."
"section": 'Program call flow".
"element": : "Evaluation ordering".
"severity level"; "medium". "critique": "Evaluation occurs only at the end; the paper requires intermediate validation steps for monitoring
convergence."
"scure":
Sample:
Research Paper:
[[Paper))
Architecture Design:
[Atchittccuueeessinl))
provide critique of weaknesses architecture design and single numerical rating lowing the Example ISON formal, without any additional commenlary, lormatting, or chuttiness.
5), fol-
Figure 25: Prompt for verification in architecture design. [() indicate placeholders to be filled with the content described the accompanying explanation.
42

---

Published as a conference paper at ICLR 2026
Prompt for refining architecture design
[System]
You an expert researcher strategic planner deep understanding experimental design and reproducihilit scientific research.
You will receive reseurch puper (ISON format), the overall plan. the original architecture design and evaluation cri- tiquetszore that architecture design.
Your task is to revise and improve the software architecture design for reproducing the paper's method based on the cri- tique, keeping it aligned both the paper and ovenall plan.
This software architecture design design should align precisely with the paper's methodology, experimental setup, and evaluation metrics. Keep the architecture simple and make effective use of open-source lihraries.
#* Instructions
1. Fix High/Mecdim Issues: Correct missing or mis-specified modules, incorrect sequence llows, or over-coupled class designs.
Trace Plan/Paper: Ensure diagrams and modules reflect the methous and milestones in the paper overall plan.
3. Keep Correct Parts: Retain any well-designdd files, class structures, or flows.
Improve Clarity: Rewrite diagrams (Mermaid syntax), sequence diagrams, and with complete detail.
5. Highlight Changes: Provide summary of what was fixed or added.
ta Format Example [CONTENT]
"summary_ of changes":
'Separated DataLoader and TokenizerAdaptrr into distinct modules",
"Added validation to sequence disgram".
"Improved interface design for Evaluation class"
"Improved__i version": [
'Implemmntattio approuch:: "We will....
File list":
"main.py".
"datasel_ londerpy"
"model.py". "trainer.py".
"evaluation.py"
"Data stractures and interfaces": "iaclass class Main In M i___i_ni___0___ (Yn caat--uun_cpppp experiment(oo class DatasetLonder
In +_ init (conlig: dietiln +load dataf) >> Any'n class Model la +a__ini iini +=__ini___pprrmmss_. diet/ln +orwaad(x: Tensor) Tensoria class Trainer la (mode_: Model, data: AnyJin +train() Noneln class Evaluntion
jiiiin (model: Model, data:
Anylin sevaluate( >> dictin in Main -> Dataseif caderin Main - Trainernn Main -s Exaluation'n Trainer Modellnn.
"Program call flow': "usequenceiiegraani participant as Main'n participant DL DatasetLoader'n participant MD as Modelin participant TR as Traine'n participant EV as Evaluationln M--DL: load datal An DL- return datasetia M-MD: forwardix wTR: predictionsla
initialize model()in M.TTR: train(medel,, datasel)a
~M: training completeln M-oEV: evaluatetmodel. dataset/un EV--MD: forward(gjaa MD--EV predictionska EV-MM: metricsin".
thiything UNCLEAR": "Need clarification on the exact dataset formal and any specialized hyperparameters"
ICONTENT]
"enodese <type>>
Implementation approuch: <elass 'stt's Summarize the chosen solution strategy.
File list: typing.Lisi(str iss[sar]* # Only need relative puths. ALWAYS write main.py or app-py here.
Data structures and interfaces: typing Optional[str] Use mermaid claseDingram onde syntax, including classes, methodr init etc.) and functions with type annotations, CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEPS API
standards. The structures SHOULD BE VERY DETAILED
complete design. Program call Dew: typing. Optional[ # Use sequenceDiagram code syntas, COMPLETE and VERY DETAILED, using CLASSES ABOVE accurately, covering CRUD INIT object, SYNTAX MUST CORRECT. Anything UNCLEAR: <cluss 'str'> # Mention umbiguities and ask for clarifications.
be compechensive
AND
#a Constraint
Format: output wrapped insids [CONTENTIJCONTENT] like the format example, nothing else.
#A Action
instructions for nodes, generate output, ensure it follows format example.
## Inputs:
Research Paper: [[Paper)]
Overall Plan: [ [Plan]]
Original Architecture Design: [[Architectureeesseee
Critique+Score: ([Critique))
Figure Prompt for refinement architecture design. [()) indicate placeholders to be filled content
described in the accompanying explanation.
43

---

Published as a conference paper at ICLR 2026
Prompt for verifying lugic design
[System]
You will be given research paper and logic design describing the ordered sequence of files/modules to be generated (e.g.. scaffolding, filenames,, module boundaries, dependency urder, build/run scripis).
Your task to rate the logic design on one metric and provide a crilique highlighting key differrnnes between the pro- posed generation sequence and what the paper requires.
make you read understand these instructions carefully. Keep document open while reviowing. refer to it needed.
Evaluation Criteria
Executable Dependency Correctness (1-5): Whether generation
module boundaries produce
buildable sysiem that correcilyy rellects the puper's pipeline (data -3 train - eval) and enables required experiments.
I: Very Poor. Order/boundarie prevent successful baild or omit essential artifacts; critical dependencies unresolved.
2: Poor. Major steps are out order or missing (egg. metrics defined after their use); build/run impossible without substantial rework.
3: Fair. Core path is present but with notable dependency loaks or circularity: buildable with nce-trivial fixes.
Mostly correct ordering boundaries; minor script 5: Excellent. Fully coherent generation sequence with clear depentencies, reproducible builds, and explicit hooks for experiments/abb
don't block
Evaluation Steps
1. Identify Required Pipeline: Identify main stages from the paper (e.g.. proprocessing, model, training, evaluation) must reflected logic design.
2. Check Ordering & Boundaries:
Confirm that medule ordering respects dependencies (e.g.. data before training. training ovaluation) avoids circular imports.
Reproducibility Hooks:
Verify configuration, seed contro,, CI Clntentry points, and script orchestration match the paper's eval protncol.
4. Assess Logic Analysis:
Evaluate whether the logic analysis correctly captures the roles, dependencies, and data flow of each file/module. Look for missing modules, unclear roles, mismatched dependencies. Check whether shared knwwledgeiomiiiuaaaionn is pruperly integrated.
5. Assess Task List:
Ensure the listed files/modules fully cover roquired pipeline and appear an exccutable order. Flug scripts missing, duplicated, misaligned with the analysis.
Critique:
Identify misplaced steps, missing files, circular dependencies, non reproducible sequencing: reference specific stepw/filenames. Summarize weaknesses mismatches. Categorize severity (High/eddimml oww and reference specific sections (1 (Logic Analysis Task list).
Score:
Provide single 1-5 rating and detailed critique in the specifled ISON format
Severity Level
High: Missing generation of motiles ondering that makes the pipeline executable trainer created he- fore modeV/oss interfaces exist).
Medium: secondary components (conligs, metrics, dataset splits) significantly hinder correct
Low: Naming incomiistencies, minor script flags, optional packaging artifacts.
44

---

Published as a conference paper at ICLR 2026
Prompt for verifying logic design
Example JSON format
""joo
Example ISON format
"critique__is list":
"section": 'Logic Analysis".
"slep_ref"; ref": "evaluation.pp" 'severity level": "high",
"critique": "Evaluator script depends metrics that are not defined before its use; imports would fail."
"section": 'Logic Analysis".
'slep "trainer
"severityy level"; levell!: "meddium", "critique": "Trainer references optimizer variants, configuration hooks not clearly
"section": : "Task list".
"step_ref" "main.py".
"severily_ level": "low".
"critique": : "Entrypoint is listed hat lacks mention of configuration flags or seed injection for reproducibility"
"scure":
Sample:
Research Paper: [[uper]]]
Logic Design: ((LogicDesiga))
Please provide critique of the weaknesses in the logic design and a single numerical rating (1, 2, 4, or 5), fallowing Example ISON formar, without any additional commentary: formatting, chattiness.
Figure 27: Prompt for verification in logic design. {O) indicate placeholders to be filled with the content described in the accompanying explanation.
45

---

Published as a conference paper at ICLR 2026
Prompt for refining logic design
[System]
You an expert researcher strategic planner deep understanding experimental design and reproducihilit scientific research.
You will receive research paper (SONN format), overall the architecture design, the original logic design and an evalnation critique-soore that logic design.
Your tusk is to revise and impeove the logic design based on the critiqee, ensuring it is executable, complete, and aligned hoth the paper, overall plan and architecture design.
The logic design breaks down tasks according to the PRDMechnicaa design, generates a task list, and analyzes task depen- dencies.
logic design
PRDAechhical plun reproducing puper's methods experiments.
'Logic Analysis" should only consider dependencies betweon files also provide detailed descriptions assist in writing the code needed to reproduce the paper.
1 Fix High/Medium Issuec Correct misordered dependencies, missing files, or incomplete AP1 specs.
Ensure Exccutability: Verify dependency order supports a buildable and runnable system.
3. Align with Architecture: Ensure file breakdown matches the architecture's is file list and APls.
4. Highlight Changes: Provide a clear summury of modifications.
#* Formar Example [CONTENT]
'summary_ of changes""
"Moved metric definition before evaluator script task list",
"Expanded API spec to include ablation toggle cadpoints".
"Clarified shared conlig variables Trainer and DataLoader"
'improved_ version": {
"Required packages":
'numpy==1.211. 1,210'"" 'torch==1.9.0"
"Required Other language third party packages": "No third-party dependencies required"
"Logic Analysis":
'data_ preprocessing py"... "DataPreroccessin class
"trainer.py".
"Trainer
"dataset loaderpp".
"Handles loading
"model.pp"
'Defines the model
"evaluation, py".
"Evalsation class
"main.py"
"Entry point.....
1.
"Task list":
"datuset leader.py", 'model.py" "trainer.py". "evaluntion.pp"
"main. py"
1.
"Full API spes": 'openapi: 3.0.0
"Shared Knowlodge": "Both data preprocessing, py and trainer py share .......
"Anything UNCLEAR". "Clarification needed recommended bardware configuration for largo-scale experiments."
[CONTENT]
46

---

Published as a conference paper at ICLR 2026
Prompt for refining logic design
Nodes: "cnode>: <lype> cinstruction>"
Required packages:
Optinmalyyyinng..iilsttt # Provide required third party packages in requirements..tt format.
'numpy=11221. -1.2100).
Required Other language third party packages: Lypiss.Lisl(st]] down packages required non Python languages. none, specify "No third-party dependencies required".
Logic Analysis: Lyping.Listttying Lis(sr] Provide list files the clasos/mcchods/funciios be implemented. including dependency analysis and imports. Include much detailed description possible.
Task list: typing List[str] a Break down the lasks into list filenumes, prioritized based on dependency order. The task list must include the previoudy generated file list.
Full API spec: <class 'str's # Describe all APIs using OpenAPI 3.0 spec that may be used by both frontend and buckend. If' and back communication
required, Shared Knowledge: celass 'stt's > # Detail any shared knowledge, like common utility functions configuration variables.
Anything UNCLEAR: <elass 'str'> Meution any unresolved questions or clarifications needed from the paper or project scope.
scope.
Constraint
Format: output wrapped inside [CONTENTIJCONTENT] like the format example, nothing else.
Follow the node instructions above, generate outpur accordingly, and easure it follows the given formar example." ""]]]
Inputs:
Rescarch Paper: [[aapr]]
Overal Plan: [[Plan]]
Architecture Design: [[ [Archittcttre
Original Logic Design: ((sigicDesiign)
Critique+Scone: [[Criique])
Figure 28: Prompt for refinement in logic design. (0) indicate placeholders to filled with the content described the accompanying explanation.
47

---

Published as a conference paper at ICLR 2026
Prompt for verifying the configuration
[System]
You will given research paper, accompanying overall reproduction plan, architocture design consisting Implementationn appreach, File list, Data structures and sesraaooollasDiaaaaaan Program call low.sequeneeDigraa and Anything UNCLEAR intended to complete
system design reproducing poper's method. logic design describing urdered sequence ftles/modules to be generated (e.g.. scaffolding. filenames, module boundaries. dependency order. buildrrun scripts) and "config- yami" file generaled from those artifacts.
Your task is to evaluate the quality of the 'confi.yaml' file in supporting reproduction the paper's experiments.
Pleuse make sure you read and understand these insiructions carefully. Keep this document open while reviewing, and
Evaluation Criteria
Conliguration Fidelity (1-5): The extent to which the 'config.yaml" accurately rellects the puper's methodology. datasets, hyperparameters, and evaluation settings, while aligning planning artifacts.
I: Very Poor. The config amits or misrepresents critical settings (datasets, hyperparameters evaluation parameters). Can- reproduce the experiment.
2: Poor. Includes some relevant parameters but misses major components or sets them incorrectly; partial repreducibility at best. gaps, inconsistencies,
defaults, Requires manual correction. 4: Good. Mostly faithful and complete, with only minor ambiguities (e.g. default values, logging frequency). Reproducible with adjustment.
essential parumeters,
S: Excellent. Fully specifics all required datasets, preprocessing, model parameters, training/evaloation settings, and reproducibility details (seeds, logging). Ready to run directly.
Evaluation Steps
1. Check Paper Alignment:
Extract required datasets, hyperparameters, evaluation protocols, and reproducibility factors from the paper.
2. Compure to Planning Artifacts:
'config, yaml' contains entries
improved plan, design, logic design.
Evaluate Completeness:
Confirm inclusion of sections:
Dataset paths and preprocessing details
Model hy typerparaeeters (hidden sizc. leamning rate, optimizer, etc.)
Training/evaluae evaluation settings (batch size, epuchs, metrics)
Ahlation/variant toggles experiments require
Random seed and reproducibility parameters
4. Check Consistency:
Verify keys, structure, and naming match the architccture and logic dosign (file names, module references, eto.).
S. Critique:
Identify missing inconsistent config
values, misaligned
Score:
Assign from 1-5 and oulput your critique in JSON formal.
Severity
High: Missing/fncoreet core parameters (datasets, learning rale, epochs, evaluation metries).
Medium: Incumplete experiment coverage (ablations missing, evaluation variants absent, inconsistent naming).
Low: Fommating/naming issues, minor logging/d:bugging configs, optional parameters not critical to reproducibility.
48

---

Published as a conference paper at ICLR 2026
Prompt for verifying the configuration file
Example JSON Output
"json
"rittquee_ist"" list":
"config "dataset.pahh"
"severity level": "high",
"crilique": "Dataset path missing; cannol locate dataset specified puper."
"config_ key":
"training,o sood""
"severity_ level": "medium"
"critique": : "Random seed not set, reducing reproducibility across runs."
"config_ "logging.sne save, dir",
"severity_lie level"; "low".
"critique": "Output directory not cleatly defined; may default an unintended location."
"soore::
Sample:
Research Paper: [[Papor]]
Overall Plan: [[Plan]]
Architecture Design: [[Architecureeessgnn]]
Logic Design: ((LogieDesign))
Config File: [[Condig YAML)]
Please provide critique of the weaknesses in the 'config-yall file and single numerical rating (1, 2, 3, 4, or 5), fol- lowing Example JSON formar, without any additional commentary. formatting, chattiness.
Figure 29: Prompt for verification in the configuration file. (1H) indicate placeholders to be filled with the content described in the accompanying explanation.
49

---

Published as a conference paper at ICLR 2026
Prompt refining configuration
[System]
You are expert ML engineer and experiment reproducibiiity specialist.
You will receive research paper (JSON format), overall the architecture design, the logic design, the original "config.yami" file and an evaluation critique+score of that 'config.ymm! file.
Your task is to revise and ingrove the "config.yaml so that fully supports reproducing the paper's method based on the critique, ensuring exccutable, complete, and aligned with the paper, the overall plan, architecture design and logic design.
Instructions
1. Fix High/Meeiim Issues:
Correct missing datasel puths, hyperparameters, evaluation metrics, or other essential fields noted in the
critique.
Preserve Correct Fields: Keep all valid and wolltconst conlig entries inlact.
3. Ensure Completeness: Add all missing sections required by paper:
Dataset specifications
Model hyperparameters
Training settings
Evallnation metrics und protoculs
Ablation/variaa toggles required
Reproducibility controls (random seeds, checkpoints, logging)
4. Consistency: Fasure keys and structure match the architecture and logic design (file references, module naming).
Clarity: Use standard YAML conventions with clear bicrarchical structure.
6. Highlight Changes: Provide summary of what was changed relative to the critique.
en Farmat Example [CONTENT]
summury. of ccaaggss:
'Addddd dataset path and preprocessing parameters',
"Specified random seed for reproducibiiity
Aligned optimizer settings with paper (AdamW, Ir=3e-5)",
"Included ablation toggles for haseline variant experiments"
"improved__ verssoon "ecFull corrected yamll berees"
[cCONTENT]
## Inputs:
Research Paper: [[aape)]]
Overall Plan: [[Plan]]
Architecture Design: [[ [eschitectureeessign))
Logic Design: ((LogicDesign))
Original Contig Hile: ([MoL)igYAML.))
Ccitique-Soore: {[Criique))
Figure Prompt refinement configuration file. [()) indicate placeholders filled with content described in the accompanying explanation.
50

---

Published as a conference paper at ICLR 2026
Prompt for verifying the analysis file
[System]
You will be given a research paper ISON format, an overview of the plan, design in JSON format consisting of "Implemeutatiinn approwch"" "File list", 'Required packagos". "Required other language third-party packages". "Logic Analysis", and "Task list", configuration file named yanl along with an analysis lile containing comprehennive logic analysis to accurately reproduce the experiments methodologies described in the research paper. This unalysis must align precisely with puper's methodology, experimental setup, and evaluation criteria.
structures interfaces" "Program call fellowed by
JSON format that includes
Your task is to evaluate the quality of the analysis file in preparing to implement the code, and how well it aligus with puper's methodology planning artifacts.
Evaluation Criteria
Analysis Fidelity (1-5): The extent to which the analysis Gile clearly and correctly specilies the responsibilities, methods. and workflows required reproduce puper's experiments methondalgiies.
I: Very Poor. The analysis is vague, missing care methods, or contradicts the puper/planning unifacts. Cannot guide im- plementation.
partial method outlines critical functionality (e.g.. evaluation loop, conlig integration). Would mislead implementation.
3: Fair. Covers most key components, but lacks detail in methed responsibilities misorders dependencies. Usable with significant manual fixing.
4: Good. Clear and structured, with most responsibilities conrectly assigned and aligued with the paper. Only minor or ambiguities.
Excellent. Complete, precise, executable outline. methods workflows are included, responsibilities are clear, and directly enables faithful code implementation.
Steps
Check Paper Aligoment:
Verify that classes and methods in the analysis match the paper's mothodology (datasets, training, evaluation, metrics).
Chock Plan Consistoncy:
Ensure responsiiilttes mutch the overall plan, architecture design. logic design (naming, APIs, flows), and configuration tile. The analysis file must follow structures interfaces" not public member functions that do exist in your design. Also, always reference settings from the conf file. Do not invent or assume values values-only use configuration explicitly provided.
Check Completeness:
Confirm that the analysis covers the lile's role in the overall experiment pipeline, including relevant aspects such as:
entry-pun entry-point logic (if Dataset handling (loading, preprocessing, augmentation, batching)
workflows, execution flow, script-level commands)
Model initialization furchitectures, weights. optimizers, schedulers)
Training loop and checkpoints (iteration structure. less computation, savingrresorring models)
Evaluation loop and metries (validation/tssting, performance measurement)
Configuration logging integration perparameters, experiment tracking. reproducibilitt)
Utility methods and shared functionality (helper functions, abstractions, or cross module dependencies that support multiple parts of codehase)
4. Check Clarity:
Evaluate whether the method steps sufficienlly detailed and logically ondered be implemented directly. The analysis should present a logical, well-organized, and actionable format that is casy to follow and apply.
5. Critique:
List missing steps, unclear method responsibilities, or inconsistencies with prior planning artifacts.
Score:
Assign single 5 score and provide critiques JSON format.
Soverity Levels
High: Missing orchestration, thilse//mmsee//tannnnneeall flows, or analysis contradicts paper's methods.
Medium: Incomplete detail dependencies, unclear method responsibilties, inconsistent naming compared to planning artifacts.
Low: Minor formatting, naming clarity, or logging/dchugging omissions.
51

---

Published as a conference paper at ICLR 2026
Prompt for verifying the analysis file
Example JSON Output
"Juum
"criique list':
"section": : "conduct, training",
"evverityleveee"" "high". "critique": "Training method does not mention checkpoint saving/loulingg which is required for reproducibilit in the
puper.
"scctinn: " "initialize model",
"soverity_ Jevel": "modium",
"critique" "Model initialization does specify tokenizer embedding setup described in the architecture
desigm.""
"vection": "setup. logging".
seeverity_eeeet level": "low", "critique": "Logging configuration not aligned shared logging utilitics outtined the logic design."
"score":
}
..
Sample:
Rescarch Paper: [[Pape)]]
Overall Plan: [[Plan]]
Architecture Design: (AArchitcctureeesggaa)
Logic Design: ((LigicDesign))
Config File: (AML))
Analysis File: {[Analysistile]]
provide critique weaknesses analysis single numerical rating the Example ISON Format, without additional commeniary, Jommatting, or chattiness.
5), following
Figure 31: Prompt for verification in the analysis file. (0) indicate placeholders be filled with the content. described the accompanying explanation.
52

---

Published as a conference paper at ICLR 2026
Prompt for refining the analysis file
[System]
You expert researcher, strategic analyzer software engineer deep understanding experimental design
reproducibility in scientific research.
You will receive research puper (JSON format), the overall plan. the architecture design, logic design, conligura- tion file named "config, yamil the original milysis file and an evaluation criliquetsoore of the analysis file.
Your task is to revise and impeove the unalysis file bused on the critique and ensure that it aligns with the reseurch paper. overall plan, the architecture design, logic design, configuration file.
This analysis must aliga precisely with the paper's methodology. experimental setup, and evaluation criteria.
The analysis must he conducted with ahsolute fidelity to the paper's methodology, ensuring that every element- from datasets and model configurations perpurameeers experimental selups
original specification without
assumption.
presentation with case.
be clear, logically structured, actionable, allowing others to replicate or extend the work
The established architecture design of "Dtaa structures and interfaces" remain intact; under circumstances should this design he altered, nor should functions outside those explicitly defined be introduced.
Every reference experimental settings must he drawn directly from the config.ymml file, with no invented or inferred values permitted.
#* Instructions
High/Medium Resolve critical omissions contradictions critique (e-g., missing training/eval loops, incorrect method responsibilities, ignoring config yaml values).
Preserve Correct Elements: Keep valid, accurate, and consistent sections of the original analysis lile.
Ensure Completeness: The improved analysis the file's moee the experiment pipeline, including relevant
aspeets such as:
OOchesttationkentt-piiee logic
Dataset handling
Model initialization
Training koop & checkpoints
Evaluation loop & metrics
Config and logging integration
Utility methods and shared knowledge
4. Consistency:
Match class/method names and APIs to those in anchitecture design.
Respect dependencies order defined in the logic design. Always reference hyperparameter//ettinn from config yaml never invent values.
5. Clarity: Write method responsibilttis and steps in clear, logically ordered, and directly implementable way.
6. Highlight Changes: Provide summary of the changes relative the critique.
Format Example [CONTENT]
"summary. of changes": "Added checkpoint saving/loading steps in training. aligned model initialization with architecture design and conlig. clarified logging setup to use shared utilities, expanded evaluation flow with metrics defined config yaml." "impoved_ version": : "ooFFull improved analysis file heresy"
iCONNTNTI
Inputs:
Research Paper: ([Paper)
Overall Plan: [[Plan]]
Architecture Design: (
Logic Design: ([LggiDDsiign)
Conlig ((ConligYAMLL))
Original Analysis File: [[AnalsiiFFlle]
Critique+Score: [[Critique]]
Figure 32: Prompt for refinement in the analysis file. (() indicate placeholders to be filled with the content described in the accompanying explanation.
53

---

Published as a conference paper at ICLR 2026
Prompt for addressing the environment in the coding stage
[System]
You are expert researcher and sollware engineer with a deep understanding of experimenial design and reproduciiilit in scientific rescarch. You receive configuration file named "condig-yame yaml". and impimented code repository. Your task is write Bash given repository from scraich. The script should create and activate the required envireament, install all
script that can dependencies, and include the commands needed to execute main file or entry point. Make sure the script self contained and can be executed without any manual setup.
Write code triple quota.
[User]
# Context
Configuration file
"yaml
configuration lile)
Code Files
the generated repository)
Format example
#i Code: reproduce.sh.
"bash seproduce.sh
# Instruction
Based code follow example", code.
have [done Ist].
Next, must write only the "reproducc.sh"
Code: reproducc.sh
Prompt addressing the environment the coding stage. (() indicate placeholders to filled with the content described in the accompanying explanation.
Figure 33:
54

---

Published as a conference paper at ICLR 2026
E EXAMPLES OUTPUT OF THE PLANNING PHASE
Relow compowenske dabal
kello fannkely mnnooce experientt mi desutbed "Attrdion You componetts foo dut propecoasinng Tho nooe exth bocaam, sariing riginen, and evalutionn redictengh prowding e chotagy teloco wiiting any Pytiic code, You can bater Frnskas eooh sedion (eq.
OVERVIEW DETALSS
mirducaee Tranetorar arctfachum--- pumnly atwnttnntssed uncoden-decxder model consiinety
convelaiion
tuo spperrante tricke
cosddationn WNIT Fng German
sackenco
Einliih- Freezh (ctil enntenne darowats
gight cand hanns parr the WDJ porker of the Pain Trosdera a (win both anull duta and sereospenneel actirpal
Hedd retende pooibooo ancodd nor (us cin sote mediuul connactions sctioon andityee zomokmiine
deopour (ypicoll though
Eagiet Frencc thy monel)
DATA PREPARATION EXPEREMENTAL
Modiose Foaiiitionn Dutasets
Eate WMT ENOC pe jer SMd sartence pars)
Tukarics santlnaos
Apply syte-pai urecding (BPE) sind scunoe-larg largee rocdaalayy 000 pikura golighhhhh
approdrnane
balch wach for
Frgllch Freech:
3014 ENIRR dansee pooad Prirrocussiig
Use worl poci-lecs segmenttten becat siory alse of 50,000 oumm. g shoong nny approwl tivon
Coneltanno Parning:
WEU Parm Treasans
axpermants.
sontenoee (une vosebuarry of takons)
Phupnaans senkenso ts gansonts oic sput regrosndtiinn sationo and la perisponing legul buo dlody vounted ca innestod brashated bood.
Somne sceaiies toke noston BrEiwardd place looki detatt
choose catibiinned bots (eg.. Berienadiaccs, Subword-MMT docament
chokes
MODE ARCHITETTRRE MODULES
Erboddi ng Layers
deddiing
enoode (ooures decader Bergefs
matin Lananal
untookking endll prosoitmax laynn) with soiling locka -i. miod.
Enouding (ansue
Iepterent coottorol encoding vitt
Firyoe, 20 engooe inttorppi model))
10000h13k
The each unbodding the sume cimmmaaion up duraddal (9112 for tho base mode))
Proveen
Eachh bayer conta
Mull-wead Futtention sub-lion
attention
For euch food:
model & 66
Powikan-eae buly con voded loes llevad retwar.
Dereracts by inpustuatur 512 innendayr ii 201n
Each sob-lape ensppod with rea connec knlloed layer non ronnullation.
Decoder
Mentccll
ardiionn
Mesied
Hullhnadd ecca-ate attmmtt fon sut over the encoder culput oinccee-dacoder sttendont
Rosicual to vodtine and comoal cation are ippledd moch sut-tyee
anbedding
posiion ensure proper sutoogeessive modiiing.
Atemmoo Cebale
Del Pel-rrrcue Ateenior
Gaee query laye
compule strenton weights out suthaa//or kytmme
cutu project K. sito sulopacco soving lsunned mur cer (ML O. W.K. NLVI
2o conanaaaa the forn prgper with W.
Note: dtscrioos modifiedoon (mealing decodde vanaaiona head conraacimannleen: panams
Addioonall Componens:
SSftraa over shaned eocebolory wrights for preticiion
Apply propest gP, drop - bu fo tess models and or varbent notes for ENunt tail model Sun enttrddinng and poetional encecing
eg.
- daiing
sollan languls
TRABING PROCFTURE AYPRORARAMETEERS
Optimizr Learning Rale Scheolle
31 Dri 32 0.08. 1e-6. Leam rate modele--.... mipptta (-05 stop "warmup, jmape wammupp alaps
koaning
incaanee laaming
Inverse rool number.
Farm barchee by grouping sentence pain with ampro/mssely by 25 000 manne both esurce and tanget sequencee. memory wiicleetty
Figure Artifact from Step 1.1 (Overall Plan) the planning stage of PaperCoder, generated during repository construction for the Transformer /Vaswani 2017) (1/2).
55

---

Published as a conference paper at ICLR 2026
Traniin Scheskles jor eguetrents
The for 100,C00 stix (- hours on P120 dPUU))
Avorago
cheipporin (1ocodded
Traaaorennn by Mudei.
aguationo
sererwant une liia rooki
40d6, porsblly uijost the number of and dispoot speclficiii "tig confige
500 1-3.5.
GPMM)
Acoegee medipoate
Use 4doye Transfores with model- 1004.
For sento pencead petting vecituuary 59% tnowe.
Bean search menta Use bean 21, length
and asuman ougul lingth (routieengh lengh
Regelortation Other Training Datals:
Apple drpout presmbed poins
syidaaa Rive The paper does not poals gradlnt oo sanig wau used leis onimplonen Butdon dodision Oooune ii your dhokee you ook) poodent disping.
DEGODING EVILUAT IIN
Teowling Trandation Todl::
beom ooarm with
Lergih semaity o 04
varty
Exdtuste to Bi accoes
WMMT 2014 wowess fins bos perform
Conuiderr using srandart evaluation reg, Sacrd_UU) to enaure companst By reporied numbers.
Evolutionn for Cuesiitunce Rarsnng
Eeuilade on sagi sen 1951 pectoeo us oulrnod thee paper.
IMPLMMENTATOON INN OROAMEATION RODDSMM
Fndioomerr Swisr
Choose your doep woining forrwwook (u.g. Tenee you or Pyftrshu Docement farnewert vansoon and GPU logia rements NVVEA P100 SPUk assured) Pregans preprcomsing scripe for BPE wordislece segtwenttdon
Stpp2: Modulo who instomentaionn
"Wee
beciligg and pouilomul encscingss
cotepndich sttenton
1au55- hlads-hedd
masking fundiosiity decoder
orn thed docuadd byees.
Depoddr liper build pub-leee block insoked sell saltartioon uncocan-deoon docoda adaritons and lood Fomnard)) coacibod.
Finem weight srriiig fetwner inpur enteerinng and the cortnaa projection
Trannin Leaaning Schecuic
integrate
leaming actedlle incude warmug
baiding toxan
ieduds dlopout and latel umooteiig ig looe out outaiinn
CCrde the beam ouach decoding with parsowerr thei own nelp usted theams eone teegth penalty, max oirut lenght)
- nond cury skops up shanng doook tgg sugpessed
Step Checipoore Averging
(Wen
possration from last
mndelic chackppints frol wunkutiee.
LLgg traniig ng bos vall celinn BLCUpsetteds bledit and cecoding ourputs.
Ourng ao veaimodiure resuito EU scores on dow soll to trose reporedd the poper
any vergences wasileloo polo
foooooss, iitonization offerences ota
Step
Conellanno Parning Eapetments
"Arept Tranetomee
veriic conett
lalining the namber layos und solen dinse sane lays' duneed 1004)
Turnig boeemaannt soarch penumokrr Boaer slco 21, parulty O 3, max cuput sppar
AMENCU TIES UINERTUNN DE DETALS
Tolenibriion Nood The sead RPC
processing
desovibed Choose elandord report
Dara Stufling Batciing oi "aadrood 28,000 kahone por directon' rule aocesalie open endted You may nend repertont masuh GPU memorr ssage desccittdd G tad pipppiiil is mantioned expi by the paper decidl you veett adill stac abai be and coc ament your choice.
Bagin
hyperpasmetenn
Insirrry
Allicugh paper wacc deockpoint uwarag ng. dulabl aoah Euqqunccy checcipoin seavec (owary 12 (iminatoo)
repped your compute cnvinement
FINAL REWARNSS
Choe all modulca and trainng loopi are place, vun pilo copttrenta q. on suboet of catar to warty that your enpirennation teapociall the atamntoon readhararra and toaniig mate schodu Commpan innemodkone logs propleely on toaniig teathes new net BU) with paper's repored numbers. and retine your plonssttaiin rageiicrr cafiveries persia.
should you with foundaation trandldiigg poper's mechedsloyy into robust code. Each black stop diaccly traced back poper's sections medel ardhtacture traiiin affenn annuing that preserve Bitel by to oiiginal expertmant design wiile allowtng room for iitorr acgustmant niceeattryy
Figure 35: Artifact from Step 1.1 (Overall Plan) in the planning stage of PaperCoder, generated during repository construction for the Transformer Vassaani al. 2017 (2/2).
56

---

Published as a conference paper at ICLR 2026
Deuuleeen calo uppeovar)
Gartenaflecatt inwondkad KSTT booestritnn Our ayetser regsore concrete ineo distec modk musoe loading pemegeeeeeeee coppocooing and boktiing mode sefieton implameeding Tronstomss
inckuiing shrmupy, and woo jaiing FU Furforr traniotton and parding) design veraine dtrpin lewerging senblatedd Brreie code concie and rommiiatiin
daraset Juader
abuudancc intankooo))
caretlagoam
Jen Jocily diet
Vir morely dist
"load, dimal)
Chex Thaneterrenicdee
Hinnardoo Tersos lgi. Terser) Toroor
chee Encwder:
Honnardo: lamax) Tanax
clea Decadar!
(porane
Tanner
jrt. HE, mode: 1t nam, Jeacs: 10
<ipwardkguury way.
GytonsiTerser
F
mode:
-boaando Temory
ckou PradonnEreedrig
<bowario: Tentoyy "hesore
Zaex Tininr
rnformathodde. cortig dit)
+bain)) Home
pr modee: roformeiidddee cabe dil contog did)
Man Ttyuftmmrtioee
Mis Funkintin
Tharo bemaettecdallloo o- Breccirr
Ehoader un Must Aeathiattion
Enooder .o- FoodFi ward
Decader Rostknalthcoddgg
Decoder Poedonal
Endlarian Turokemneoott
secoorochhe
parkdpart
Main
Dutzadizzddr
cort-quen Triettmeiittel
Transer
jerlidpart 27 Evalubon
Mixdl: +1_ [coefo)
DL-0-M natan (reia wwuoonipeeiinnggg (ktrwe))
Ma-TP: (Todee-Tu des, contel
(rocaeTMM, mas contg)
Tiso0/tt fomardion:: Uprriuiing buid!
T2ooTRR savn checipilet) paotdically
MoneEi anduste)) rnfernne decodinnl
Ti predicionn
Ediovi evalatiion nekios (BLEU) soorss
Covilcation in needed woet entual tormass for machine trandrtin and conetiuanty panding (hgg strucion Indaniztionn ouburj Additonaly, cecale whethrts grodent dissing. spocialion youporanslls boppocel the comligu wtoon//oolll
peccio theoppar seving
Figure 36: Artifact from Step 1.2 (Architecture Design) in planning stage of PaperCoder, generated during repository construction for the Transformer /Vaswani el al. 2017).
57

---

Published as a conference paper at ICLR 2026
HARXorig dCC)
dibyyade: Trartammniodd, daix dir,, cartee dit roaing Hore
(nbjaarii Awad)
inntimodei ThraformnMMess dotec det, casti:
clecppekopat 101
iittipaani
Encader
fandFonvard
dernardgpey::
Figure 37: Class diagram visualization generated from Step 1.2 (Architecture Design) in the planning stage of PaperCoder, generated during repository construction for the Transformer Vaswani et al. 2017)
Aatm
Datesetloeder
TransformerHddd
Trainer
Evoluacion
lod. deta)
return (data, transationepari datzse]]
JoP inocel procel-Te,, data, conffgi
moltd-T dats, configl
forwartior,, tat) [dring each botch]
mode: prodictions
cne__ctecperett pertoctcally
complete
evaluate))
forwandters, inforence deceding)
csalaation netric (BLEU F1 scores)
ceder
Transformentodel
Eveluation
Figure 38: Program call flow visualization generated from Step 1.2 (Architecture Design) the planning stage of PaperCoder, generated during repository construction for the Transformer (Vaswani et al. 2017).
58

---

Published as a conference paper at ICLR 2026
Pegg rnd poccagee)
-orddee 1003
-ardftaniod 100
No pind dependardis requines
Logg Anuteaat
bacenps Shrpwed coder sivas handies
datasses bot naciis
fomnc by Genteraa CPC)
cantton synamic bunch ne saued
tours, timport
"coriondEreedde u. Thex rodule loatagea Py foeett en Moduls refccc doagg cstataa fun pape with dlar dlanlion modia nerra Hosdual con hedorr. laps' rormal odton, and ahured unbooddnng
("oanerr oy teplomenti the Thaner cous whit mede the niring ueing the hantrmrlleee The Traiee bnacle opiiniw seve (axam with cuetes by rerprmmnttin and leaming man schwolds
(eea unkon py:. Donsais the Evok salion clsss repprrs bie naninn rlevenoe the lolned model inp novking hoam owarch decocling and tomputt fig evak oeltc meticc (RI EU tranditionn lock
sponallly apc and non artoniitee the Trarne and Evaluel closass con manages oull eg raining follwed salion and motic rejo ring too mooo togetior oor and handhrs owardl
tmaa
haenc parangi. may external lemettes secrebi
mediu depanda modelay dateect
argumennts
Entamticoder
po
model senffmmmeeedee
Thaa toj crooslay
Shaned Include contiuustss
fundtons, chedkpote (eg. model sum, hwads, If commap ships, emopiit tree shoutt contrilieed common corti guatton modkle didionary to conssteery scroe
damaet Joaderay hypamarartten model
Covilcation in nedder mecommanden handowe ovn gurrio bege-azale equerments, the wand fomor denset (fe drudtre skerbston ratare deth qpli)) the trog checipoire Khiing, grndied ripiing
Figure 39: Artifact from Step 1.3 (Logic Design) in the planning stage of PaperCoder, generated during repository construction for Transformer Vaswani 2017).
#* config.yaml
training:
opt sptimier:
betal: 0.9
beta2: 0.98
epailion: 1e-9
wazmup_ateps: 4000
total_steps: learning_rate_cctte
training steps (-12 houre 2100 *d_modeI1-0.5 min (step""O...5 step warmup_atepa"-1.5)
batchhtokkns::
source: 25000
target: 25000
drosout:
label_ sabel__moothing: 0.1
madel:
num num_laye num_layera: ayers: 6
d_model: 512
d_ff: 2018 nun_heads:
d_k: E4
d.v sharod__mmeddiing true
data:
dstaset: "wor 2014 English-to--Grman" encoding: "byte pair encoding* wocsbulary_eāze: 37020
evaluatioon
beam_eise:
length_penalty: 0.6
max mox_outpat_ "ingut_length 50"
Figure 40: Artifact from Step 1.4 (Configuration File) in the planning stage of PaperCoder, generated during
repository construction for the Transformer VVaswani et al. 2017)
59

---

Published as a conference paper at ICLR 2026
Table 20: ICLR 2024 papers used in our Paper2CodeBench benchmark. We evaluate each paper using the model-based, reference-free setting, with gpt-4o-2024- 1114--11--11 -20 as the evaluation model.
Paper
Source
Score
Generative Judge for Evaluating Alignment
Poster
Distributional Preference Learning: Understanding and Accounting for Hidden Context RLHF
Poster
4
Inherently Interpretable Time Series Classification via Multiple Instance Learning
Oral
3.9
iTransformer: Inverted Transformers Are Effective for Time Series Forecasting
Oral
3.9
Tell Your Model Where Attend: Post-hoc Atlention Steering LLMs
Poster
Knowledge Distillation Based Transformed Teacher Matching
Poster
3.9
Meaning Representations from Trajectories in Autoregressive Models
Poster
3.8
Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis
Poster
3.8
VDC: Versatile Data Cleanser based Visual-Linguustic Inconsistency Mul- timodal Large Language Models
Poster
3.8
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis
Poster
3.8
SliceGPT: Compress Large Language Models by Deleting Rows and Columns
Poster
3.8
Beyond Accuracy: Evaluating Self- Self-Consistency Code Large Language Models with Identity Chain
Poster
3.8
Guiding Masked Representation Learning to Capture Spatio- Temporal Relation- ship of Electrocardiogram
Poster
3.8
Social Reward: Evaluating and Enhancing Generative Al through Million-User Feedback from an Online Creative Community
Oral
3.7
Language Model Detectors Are Easily Optimized Against
Poster
3.7
Improving protein optimization with smoothed fitness landscapes
Poster
3.7
SparseFormer: Sparse Visual Recognition via Limited Latent Tokens
Poster
3.7
AutoVP: An Automated Visual Prompting Framework and Benchmark
Poster
3.7
Hierarchical Context Merging: Better Long Context Understanding for Pre- trained LLMs
Poster
3.7
SEABO: A Simple Search-Based Method for Offiine Imitation Learning
Poster
3.7
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data
Poster
3.7
Rethinking The Uniformity Metric Self-Supervised Learning
Poster
3.7
VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE
Poster
3.6
Efficient Backpropagation with Variance Controlled Variance-Controlled Adaptive Sampling
Poster
3.6
Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning
Poster
3.6
Contro/Video: Training-free Controllable Text-to-Videe Generation
Poster
Context-Aware Meta-l Meta-L_caming
Poster
3.6
RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit Neural Representations
Poster
3.6
Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
Poster
3.6
Modulate Your Spectrum in Self- Saf--uupeeeeeed Learning
Poster
3.6
60

---

Published as a conference paper at ICLR 2026
Table 21: List of ICMI. 2024 papers used in our Paper2CodeeBench benchmark. We evaluate each paper using the model-based, reference-free setting, with gpt-40-20246 1-20 the evaluation model.
Paper
Source
Score
Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wiss Attention
SAMformer:
Oral
Autoformalizing Euclidean Geometry
Poster
4
Recurrent Distance Filtering for Graph Representation l.carning
Poster
CosPGD: efficient white-box adversarial attack for pixel-wise prediction tasks
Poster
3.9
Token-level Direct Preference Optimization
Poster
3.9
BayOTIDE: Bayesian Online Multivariate Time Series Imputation with Func- tional Decomposition
Oral
3.8
CurBench: Curriculum Learning Benchmark
Poster
3.8
Exploring the Low-Pass Fiftering Behavior in Image Super-Resolution
Poster
3.8
Towards Efficient Exact Oplimization of Language Model Aligument
Poster
3.7
Effectiveness of Supervision Asymmetric Non-Contrastive Learning
Poster
Drug Discovery with Dynamic aware Fragments
Poster
3.7
Fool Your (Vision and) Language Model With Embarrassingly Simple Permuta-
Poster
tions
Image Restoration Through Generalized
Bridge
Poster
3.7
Timer: Generative Pre-trained Transformers Are Large Time Series Models
Poster
3.7
Mitigating Oversmoothing Through Reverse Process of GNNs for Heterophilic Graphs
Poster
3.7
Suppevi-S Semantic Segmentation with Prototype-base Feature Aug- mentation
Poster
3.7
ConvNet Transformer, Supervised CLIP: Beyond ImageNet Accuracy
Poster
CLIF: Complementary Leaky Integrate-and anditte--ndFFiiin Fire Neuron for Spiking Neural Networks
Oral
3.6
FiT: Flexible Vision Transformer for Diffusion Model
Oral
3.6
Decomposing Uncertainty for Large Language Models through Input Clarifica- tion Ensembling
Oral
3.6
SparseTSF: Modeling Long-term Time Series Forecasting with # *kk Parameters
Oral
3.6
Sample-specific Masks for Visual Reprogramming based Prompting
Oral
3.6
Boundary Exploration for Bayesian Optimization With Unknown Physical Con-
Poster
3.6
straints
Listwise Reward Estimation for Offline Preference-based Reinforcement Lear- ing
Poster
3.6
Graph Distillation with Eigenbasis Matching
Poster
3.6
Temporal Spiking Neural Networks with Synaptic Delay for Graph Reasoning
Poster
3.6
Position: Quo Vadis, Unsupervised Time Series Anomaly Detection?
Poster
Neural SPH: Improved Neural Modeling of Lagrangian Fluid Dynamics
Poster
3.6
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
Poster
Models
Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calihration
Poster
3.6
61

---

Published as a conference paper at ICLR 2026
Table 22: List of NeurIPS 2024 papers used in our Paper2CodeBench benchmark. We evaluate each paper using the model-based, reference-free setting, with gpt-40-2002-0 the evaluation model.
Paper
Source
Score
PACE: marrying generalization in PArameter-efficient fine-tuning with Consis- lency rlEgularization
Oral
The Road Less Scheduled
Oral
4
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
Poster
Binarized Diffusion Model for Image Super- Resoluton
Poster
Learning lo Predict Structural Vibrations
Poster
4
Attack- Aware Noise Calibration for Differential Privacy
Poster
Make Your LLM Fully Utilize the Context
Poster
Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention
Poster
3.9
Sm: enhanced localization in Multiple Instance Learning for medical imaging classification
Poster
3.9
AutoTimes: Autoregressive Time Series Forecasters via Large Language Models
Poster
End-to-End Ontology Learning with Large Language Models
Poster
3.8
Scaling transformer neural networks for skillful and reliable medium-range weather forecasting
Poster
3.8
Autoregressive Image Generation without Vector Quantization
Oral
3.7
Adaptive Randomized Smoothing: Certified Adversarial Robustness for Multi- Step Defences
Oral
3.7
Generalizable Person Re-identification via Balancing Alignment and Uniformity
Poster
3.7
Universal Neural Functionals
Poster
3.7
Are Self-Attentions Effective for Time Series Forecasting?
Poster
XMIL: Insightful Explanations for Multiple Instance Learning in Histopathology
Poster
3.7
Leveraging Environment Interaction for Automated PDDL Translation and Plan-
Poster
ning with Large Language Models
Task Aagostt Machine Learning-Assisted Inference
Poster
3.7
Make Continual Learning Stronger via C-Flat
Poster
3.7
DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning
Poster
3.7
Graph
AsyneDiff: Parallelizing Diffusion Models by Asynchronous Denoising
Poster
3.7
You Only Look Around: Learning Illumination Invariant Feature for Low-light Object Detection
Poster
3.6
MutaPLM: Protein Language Modeling for Mutation Explanation and Engineer- ing
Poster
3.6
Advancing Training Efficiency of Deep Spiking Neural Networks through Rate- based Backpropagation
Poster
Improved off- toliillc training of diffusion samplers
Poster
3.6
Navigating the Effect of Parametrization for Dimensionality Reduction
Poster
3.6
Long- Range Feedback Spiking Network Captures Dynamic and Static Represen-
Poster
tations of the Visual Cortex under Movie Stimuli
InfLLM: Training-Fres Free Long- Long-Context Extrapolation for LLMs with an Efficient Context Memory
Poster
3.6
62

---

Published as a conference paper at ICLR 2026
Table 23: List of papers used in human evaluation. We evaluate the official repository of each paper, released by the authors, using the model-basee reference-free setting with 100-44-2224-1---111 as the evaluation model.
RepoName
Paper
Score
VideoICL: Confidence-base Iterative In-context Learning for Out-of-
VideoICI.
2.6
Distribution Video Understanding
Identity Decoupling for Multi-Subject Personalization of of Text-to-Imagee Models
MuDI
KALMV
Knowledge-Augmented Language Model Verification
3.3
sea-attention
SEA: Sparse Linear Attention with Estimated Attention Mask
HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models
HarmAug
GruM
Graph Generation with Diffusion Mixture
3.7
Adaptive-RAGe Learning Adapt Retrieval Auggeeete Large Lan-
Adaptive-RAG
2.7
guage Models through Question Complexity
Sketch-of-Thhugge Efficient L.I. Reasoning with Adaptive Cognitive- Inspired Sketching
SoT
4.0
Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model
Mol-LLaMA
judge_code_ lat___odd_efffciineeee
Rethinking Code Refinement: Learning Judge Code Efficiency
3.1
mhowledgeAAugmenter Reasoning Distillation for Small Language
KARD
3.2
Models in Knowledge-Intenne Intensivv Tasks
Concept-skill Transferability-basee Data Selection for Large Vision- Language Models
COINCIDE_cod
3.0
Janus
Aligning to thousands of preferences via system message generalization
3.5
Silent Branding Attack: Trigger-fres Data Poisoning Attack on Text- to-Image Diffusion Models
N/A
N/A
VideoRAG
VideoRAG: Cettieval-Augmennee Generation Video Corpus
3.0
RADA
Retrieval-augmence data augmentation for low-resource domain tasks
3.0
Continual Audio-Video Pre-training with Localized Alignment
STELLA:
STELLA_code
3.3
Prometheus-vision. Vision-language model a judge for fine-grained evaluation
prometheus-vision
3.1
CoLoR
Efficient Long Context Language Model Retrieval with Compression
3.0
Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision
Volcano
cool-integrated Self-verification for Test-time Compute Scaling Small Language Models
Tool
N/A
N/A
Table 24: List of papers used in executability analysis.
Repo Name
Paper
CoLoR
Efficient Long Context Language Model Retrieval with Compression
cognitive-behaviors
Cognitive Behaviors that Enable Self- Self- hnll--ippoiing Reasoners, or, Four Habits of Hlighly Effective STaRs
RADA
Retrieval-Augmented Data Augmentation for Low- Row-Resouree Domain Tasks
Self-Instruct
Self Instruutt Aligning Language Models with Self- Instructions
G-EVAL
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
63

---

Published as a conference paper at ICLR 2026
Paper:
Github:
[General]
1. If someone wants to reproduce the methods and experiments in your paper, which components would they need to implement? Please break down into the following sections: (1) data processing, (2) method (e.g., model training or main pipeline), and (3) evaluation.
For example, in Self-Instruct (TLDR; Self-Instruct, a framework for improving the instruction-flllowinn capabilities of language models bootstrapping off their own generations.):
Data Processing
N/A
Method (e.g., Model training Main pipeline)
1. Instruction Generation
Classification Task Identification
Instance Generation Filtering
Evaluation
Training the model using the generated synthetic data via our methods Evaluating the trained model
Your Answer
Data Processing
Method (e.g.. Model training Main pipeline)
Evaluation
Figure 41: Human Evaluation Guideline (1/3)
64

---

Published as a conference paper at ICLR 2026
[Comparison]
Given set of repositories, which which best re-implements the methods and experiments intended by the paper?
helpful reproducibility
Please review the provided repositories (Group 1: repo1- Group 2: repoS-repo7, Group 3: repo8-repo 10) and rank them based how well they are implemented.
It is worth noting that the same repository may appear more than once between repo1 and repo10; this is not error.
(Optional things: Feel free to leave comment explaining why you ranked them that way)
[Group1: repo1-)epo4]
2nd
3rd
4th
[Group2: repo5-repo7]
1st
2nd
3rd
[Group3: repo8-repo10)
1st
2nd
3rd
Among the top-ranked repositories in each group, which you think the best? If the repositories the same, you select any them. Please briefly explain your reason.
(All: repo1 -rpoo10
1st
Reason
Figure 42: Human Evaluation Guideline (2/3)
65

---

Published as a conference paper at ICLR 2026
[Detailed Analysis about the 1st Repository]
3. Do you think the first-ranked repository you chose would make easier to reproduce the paper's methods and experiments than starting from scratch?
Yes
No
you selected 'No', please briefly explain why. Otherwise, you may leave this blank.
Reason No
Based on the key components you mentioned question how well does "repo10" repository support them?
Please check one of the following for each component: (o fully implemented, A partially implemented, not implemented) If select A please briefly explain your reason.
Example: Seif-Instruct (TLDR; Self-Instruct, a framework for improving the instructionffolloiing capabllities of language models by bootstrapping off their own generations.)
Data Processing
N/A
Method (e.g., Model training Main pipeliine)
1. Instruction Generation (o)
2. Classification Task Identification (o)
3. Instance Generation (4)) They don't implement output-first and input-first seperately.
4. Filtering (A.: They only implemented it using the ROUGE-L-based filter, not with the exact input-output pairs.
Evaluation
Training the model using the generated synthetic data via our methods (o) Evaluating the trained model (x): They only provided the training code.
Your Answer
Data Processing
Method (e.g., Model training or Main pipeline)
Evaluation
Figure 43: Human Evaluation Guideline (3/3)
66


Reviews:
[{"review_id": "EzcApkZYoy", "reviewer": "Reviewer_FaxH", "summary": "The paper presents paper2code, an ambitious system for translating scientific ML papers into executable code. The pipeline comprises three stages—planning, analysis, and coding—aimed at extracting algorithmic intent and producing runnable implementations. Evaluations on the authors’ selected metrics indicate promising effectiveness of the approach.", "questions": "1, How is correctness evaluated when multiple valid implementations exist, and what does the reported 0.81% edit rate imply for developer time and repair iterations?\n2. How robust is the system across domains/frameworks/formats, how are licensing/leakage and security risks handled, and where does human review add the most value?", "limitations": "", "rating": null, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. Framing end-to-end paper to code generation is both challenging and potentially transformative for reproducibility and rapid prototyping.\n2. The decomposition into planning, analysis, and coding is reasonable and provides a clear, modular workflow for bridging natural language descriptions and implementation.", "weaknesses": "1. It is not evident how the paper defines “correct” generated code (e.g., functional equivalence, numerical agreement within tolerance, adherence to algorithmic specifications). Please clarify ground truth, acceptance tests, and evaluation thresholds.\n2. The claim that fixes average 0.81% of total lines may understate practical burden: small line edits can involve significant debugging time. Consider reporting developer time, number of repair iterations, error categories (syntax/import/runtime/logic), and success rates after each repair round.\n3. Correct coding implementations can vary (e.g., refactoring choices, hyperparameters, library calls). Explain how the evaluation accommodates non-uniqueness—e.g., via canonical reference implementations, specification-driven unit tests, layout-/refactor-invariant metrics, or behavior-based measures on hidden test sets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XtXhjZF25O", "reviewer": "Reviewer_aZeb", "summary": "*Summary*\n\nThis paper addresses the critical reproducibility challenge in machine learning, where a significant fraction of published papers (e.g., ~80.5% in 2024) do not release their code. To combat this, the authors introduce **PaperCoder**, a multi-agent Large Language Model (LLM) framework designed to automatically generate functional code repositories directly from scientific papers\n\nThe core of PaperCoder is a structured, three-stage pipeline that mimics a human developer's workflow:\n1.  **Planning:** A set of specialized agents first constructs a high-level roadmap, designs the system architecture (including class and sequence diagrams), identifies file dependencies and execution order, and generates configuration files (e.g., `config.yaml`)\n2. **Analysis:** The framework then performs a fine-grained interpretation of the plan, defining implementation-specific details, inputs/outputs, and algorithmic constraints for each module\n3.  **Generation:** Finally, the framework synthesizes the modular, dependency-aware code file-by-file, using all previously generated artifacts as context.\n\nTo evaluate their framework, the authors introduce a new benchmark, **Paper2CodeBench**, consisting of 90 recent papers from ICLR, ICML, and NeurIPS 2024. They conduct extensive evaluations using both model-based (reference-free and reference-based) and expert human evaluations, notably involving the authors of the original papers. The results demonstrate that PaperCoder substantially outperforms strong baselines, with 92% of human judges reporting the generated repositories as helpful.", "questions": "1.  Given that only 4/90 automated runs succeeded and the primary failure was dependency-related, why isn't a \"DevOps Agent\" (to generate `requirements.txt`, `Dockerfile`, etc.) a core part of the \"Planning\" and \"Generation\" stages? This seems like a crucial missing piece for generating \"functional repositories.\"\n2.  The paper uses \"LLM-assisted debugging\" as an *evaluation* step in Section 4.3. This proved effective. Have the authors considered integrating this as a *closed-loop refinement step* within the PaperCoder framework itself? For example, the framework could generate the code, attempt to execute it in a sandbox, catch the `stderr`, and feed the error message back to the \"Coding\" agent to self-correct. Or, as an alternative, leverage unit tests for all the components.\n3.  The \"Data Processing\" stage has the lowest coverage (56%), and a case study failed on an ambiguous loss function. How does the framework currently handle ambiguity? Would it be feasible for the \"Analysis\" agent to *stop and query the user* when it detects high ambiguity (e.g., an \"overly simplified\" description), rather than proceeding to generate a potentially incorrect implementation? Or, at least marking potential issues coming from unders-specification of the paper's description in the code?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper makes a highly significant contribution to the ICLR community.\n\n1 **Significance of Problem:** The work directly addresses the critical and widely-felt reproducibility crisis in machine learning, where the lack of code is a major barrier to scientific progress.\n\n2 **Novel and Sound Methodology:** The proposed **PaperCoder** framework is a strong contribution. Its multi-stage (Planning, Analysis, Coding) pipeline is a logical and novel approach that mimics how a human expert would translate a paper to code. The \"Planning\" stage, with its decomposition into an overall plan, architectural design, logic design, and configuration generation, is particularly strong and provides the necessary scaffolding that simpler one-shot methods lack.\n\n3 **Exceptional Evaluation Protocol:** The evaluation is a major strength.\n    1.  The authors introduce **Paper2CodeBench**, a new, relevant, and large-scale benchmark of 90 papers from top 2024 conferences.\n    2.  They employ a robust LLM-as-judge evaluation with both reference-based (vs. author code) and reference-free (vs. paper text) settings.\n    3.  Most impressively, they validate their findings using **human evaluations from the original paper authors**, which is the highest possible standard for this task. The strong results from this (e.g., 92% of authors found the generated code helpful) are highly persuasive.\n\n4 **Clarity and Presentation:** The paper is extremely clear, well-written, and easy to follow. The figures and tables are informative and effectively communicate the framework's design and results.", "weaknesses": "1 **Overstated Executability Claims:** The abstract's claim of \"functional code\" and the conclusion's \"strong executability\" are misleading. These claims appear to rest on a *manual* debugging analysis of only **five** papers, which found an 0.81% fix rate. This is a very small and likely biased sample.\n\n2 **Contradictory Automated Execution Results:** A more comprehensive analysis in Appendix B.4 (Table 12) shows that when the generated code was run automatically on the *full 90-paper benchmark*, only **4 out of 90** repositories executed successfully. This suggests the system is not \"near-executable\" out-of-the-box.\n\n3 **Failure to Address Environment Setup:** The primary failure mode identified in Table 12 is environmental: \"Missing Dependency\" (23 papers), \"ImportError\" (14 papers), and \"ModuleNotFoundError\" (14 papers). Although the \"Logic Design\" phase plans for \"Required packages\", the generation stage clearly fails to implement this reliably. The framework is good at generating Python *syntax* but poor at generating the *runnable environment* (e.g., a `requirements.txt` file).\n\n4 **Unresolved Ambiguity Bottleneck:** The paper's own fine-grained analysis (Figure 5) reveals that the framework's implementation coverage is lowest for **\"Data Processing\" (56%)**. The authors attribute this to papers being \"under-specified\", but handling this ambiguity *is* the core difficulty of the task. A case study (Table 15) confirms this, showing one failure was due to an \"overly simplified\" loss function description. The framework currently guesses (and fails) in the face of ambiguity.\n\n5 **Potential Evaluation Bias:** In the human evaluation (Table 14), the top reasons for preferring PaperCoder were \"Completeness\" and \"Clean Structure\". Since the PaperCoder framework is *explicitly designed* to produce a complete, modular structure (via its Planning stage), the evaluation may be rewarding the framework's *scaffolding* more than its *algorithmic correctness* when compared to one-shot baselines that lack this scaffolding by default.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "27U97mMQkn", "reviewer": "Reviewer_V6cT", "summary": "This paper introduces PaperCoder, a multi-agent large language model (LLM) framework designed to transform machine learning papers into functional code repositories. The system emulates a human development pipeline, consisting of three key stages: planning, analysis, and generation. The authors evaluate PaperCoder using two benchmarks: Paper2CodeBench and the newly released PaperBench, employing both automated reference-free and reference-based metrics, along with expert human assessments. The results demonstrate significant improvements over baselines: 88% of the generated repositories receive the highest ratings, 92% of human judges find them useful, and the generated code typically requires only minor adjustments (with an average of 0.81% of lines modified). Ablation studies confirm the contribution of each stage to the overall performance. In summary, this paper addresses a valuable problem, and the proposed methodology and evaluation approach will provide meaningful support for future research.", "questions": "See the weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper introduces PaperCoder, a multi-agent large language model (LLM) framework designed to transform machine learning papers into functional code repositories through three key stages: planning, analysis, and generation. The results demonstrate significant improvements over baselines and ablation studies in Table 6 demonstrate the effectiveness of each proposed component.\n2.\tThis paper constructs Paper2CodeBench by collecting recent machine learning papers with available code, filtering repositories for manageable size, and selecting high-quality papers through model-based evaluation. The benchmark provides reliable evaluation support for future research on paper-to-code generation.", "weaknesses": "1.\tReproducibility is a crucial metric for assessing code quality. However, this study conducts reproducibility analysis on only a limited subset, while the primary evaluation relies on the LLM’s judgment of code quality. It remains unclear whether the LLM’s scores reliably correlate with actual reproducibility, i.e., whether code receiving higher scores is truly more reproducible.\n2.\tThe selected papers are drawn from ICLR, ICML, and NeurIPS 2024, along with their associated code repositories. It is important to consider potential data leakage risks, as these codebases may have been incorporated into the training of open-source models or closed-source APIs.\n3.\tIn the LIMITATIONS section, the authors note that the current model only processes textual inputs. However, many methods and model architectures depend on the joint interpretation of diagrams and text. Consequently, relying solely on textual information may not faithfully reproduce the intended model structures. Exploring the use of PDFs as images input to a multimodal LLM for code generation represents an interesting direction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XuK7CPeVkF", "reviewer": "Reviewer_QcnK", "summary": "The paper introduces PaperCoder, a multi-agent large language model (LLM) framework that automatically converts machine learning research papers into functional, repository-level code implementations. The system operates in three structured phases: planning, analysis, and coding. PaperCoder is evaluated on a benchmark of recent ML papers (ICLR, ICML, NeurIPS) and compared against baselines like ChatDev and MetaGPT, using both model-based and human evaluations. Results show that PaperCoder consistently generates more accurate and executable implementations, outperforming all baselines on both fidelity and usability metrics.", "questions": "none", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper tackles a highly impactful and underexplored task—automatic code generation from scientific papers—which directly addresses the reproducibility crisis in ML research. The framing of “Paper2Code” as a structured, multi-agent reasoning pipeline is both original and well-motivated.\n\n2. The decomposition into Planning–Analysis–Coding is conceptually sound and empirically justified. Each stage mirrors human reasoning during software development, which gives the approach interpretability and modularity. The inclusion of architecture diagrams, execution order reasoning, and configuration synthesis adds practical depth.\n\n3. The paper conducts extensive quantitative and qualitative evaluations, showing the effectiveness.", "weaknesses": "1. While the use of LLMs as judges is increasingly common, it introduces potential circularity and bias, especially since the same family of models may be used for both generation and evaluation. More independent human-based verification (beyond author feedback) would strengthen the claims.\n\n2. Although executability is discussed, actual reproduction of original experimental results remains limited (only partial success in 4/5 cases). The paper does not sufficiently quantify semantic correctness or result fidelity, which are crucial for scientific reproducibility.\n\n3. The ablation study (Table 6) shows some counterintuitive drops (e.g., architecture design hurting performance). While the authors provide a plausible rationale, a deeper qualitative analysis of failure cases would improve interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)