SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Authors: Yanzheng Xiang
Hanqi Yan
Shuyin Ouyang
Lin Gui
Yulan He
Venue: COLM 2025 (2025)
Keywords: Code Generation
Agent
Scientific Paper Understanding
TL;DR: We investigate the use of LLMs to aid in reproducing the core algorithms proposed in real-world NLP publications. To support this task, we introduce a benchmark and a multi-agent framework.

Abstract: This study evaluates large language models (LLMs) in generating code from algorithm descriptions in recent NLP papers. The task requires two key competencies: (1) algorithm comprehension: synthesizing information from papers and academic literature to understand implementation logic, and (2) coding expertise: identifying dependencies and correctly implementing necessary APIs. To facilitate rigorous evaluation, we introduce SciReplicate-Bench, a benchmark of 100 tasks from 36 NLP papers published in 2024, featuring detailed annotations and comprehensive test cases. Building on SciReplicate-Bench, we propose Sci-Reproducer, a dual-agent framework consisting of a Paper Agent that interprets algorithmic concepts from literature and a Code Agent that retrieves dependencies from repositories and implements solutions. To assess algorithm understanding, we introduce reasoning graph accuracy, which quantifies similarity between generated and reference reasoning graphs derived from code comments and structure. For evaluating implementation quality, we employ execution accuracy, CodeBLEU, and repository dependency/API recall metrics. In our experiments, we evaluate various powerful non-reasoning and reasoning LLMs as foundational models. The best-performing LLM using \ModelName~achieves only 39\% execution accuracy, highlighting the benchmark's difficulty. Our analysis identifies missing or inconsistent algorithm descriptions as key barriers to successful reproduction. We make available our benchmark and code at https://github.com/xyzCS/SciReplicate-Bench and project homepage at  https://xyzcs.github.io/scireplicate.github.io/.

Published as a conference paper at COLM 2025
SciReplicat-Bennhh Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Yanzheng Xiang', Hanqi Yan', Shuyin Ouyang', Lin Gui', Yulan Hel2 *King's College London, 2The Alan Turing Institute
(yanzheng.. xiang, hanqi yan, shuyin. ouyang,
Shuyan,.uyaan, 1in. 1. 1.gui, yulan. he)ann.eeeer heeekel...... ac uk
Abstract
This study evaluates large language models (LLMs) in generating code from algorithm descriptions in recent NLP papers. The task requires two key competencies: (1) algorithm comprehension: synthesizing informa- tion from papers and academic literature to undersland implementation logic, and (2) coding expertise: identifying dependencies and correctly implementing necessary APIs. To facilitate rigorous evaluation, we intro- duce SciReplicate-Bende
a benchmark 100 tasks from 36 NLP papers published in 2024, featuring detailed annotations and comprehensive test cases. Building on SciReplicate Bench, we propose Sci-Reproducer, a dual- agent framework consisting of Paper Agent that interprets algorithmic concepts from literature and a Code Agent that retrieves dependencies from repositories and implements solutions. To assess algorithm under- standing, we introduce reasoning graph accuracy, which quantifies similarity between generated and reference reasoning graphs derived from code com- ments and structure. For evaluating implementation quality, we employ execution accuracy, CodeBLEU, and repository dependency / API recall met- rics. In our experiments, we evaluate various powerful non-reasoning and reasoning LLMs as foundational models. The pest-performing LLM using Sci-Reprod uce achieves only 39% execution accuracy, highlighting the benchmark's s difficulty. Our analysis identifies missing or inconsistent algorithm descriptions as key barriers to successful reproduction. We make available our benchmark and code at O GitHub and project homepage at & Homepage.
Introduction
The evolution of Large Language Models (LLMs) has ushered in a transformative era in scientific discovery, positioning them powerful tools for streamlining research (Gridach et al., 2025; Buehler, 2024; Lu et al., 2024), from idea generation to verification and publi- cation writing. For instance, Si et al. (2025); Gu & Krenn (2025) demonstrated how LLMs can be prompted to generate novel research ideas, while Yuan et (2022); Du et al. (2024) explored their use in producing literature reviews for idea evaluation. Additionally, LLMs are increasingly integrated into tools like Semantic Scholar Research Rabbit?, and Under- mind Research Assistant enhancing literature discovery, citation analysis, and knowledge synthesis. These advancements, both in research methodologies and practical applications, suggest that LLMs have the potential to assist across multiple stages of scientific discovery.
Among the aforementioned advancements in research acceleration, the ability of LLMs to correctly generate code for validating real-world scientific ideas particularly noteworthy. Computational validation is crucial across many fields, yet researchers often face barriers due to limited coding expertise or inaccessible implementations. By converting scientific
http:/ww..emmnticcchhlllooo org
chttps::/www/ www.wereeeaaaahhabboo Phttps://wwwe wwtp://wwwdermiiddcco ai./

---

Published as a conference paper at COLM 2025
algorithm descriptions into executable code, LLMs could enhance reproducibility and accelerate scientific discovery. However, despite Progress in LLM-based code generation, a sigmificant gap remains in generating code directly from scholarly papers. First, algorithm comprehension from scientific papers is challenging. Research papers are characterized by their brevity, methodologica rigor, and extensive citations, with critical details about algorithms often dispersed across multiple sections of the paper. Understanding these algorithms requires synthesizing information from internal references and external scholarly works. Second, code repositories typically consist of multiple interdependent files and directories. To implement an algorithm, LLMs must comprehensively examine file dependencies, identify reusable components, and correctly handle both internal dependencies and external APIs.
Despite the importance of automated scientific idea verification, there exists no dataset specifically designed to evaluate the ability of LLMs to reproduce real-world algorithms proposed in peer-reviewed publications. As shown in Table 1, there are several machine learning software engineering benchmarks primarily focused on evaluating algorithmic design or straightforward implementations, which are significantly less complex than the methods typically described in academic research papers. For example, MLE-BENCH (Liu et al., 2023) and MLAgentBench (Huang et al., 2023) utilize Kaggle competitions, where LLMs must develop and implement solutions based on provided task specifications. ML- BENCH (Chan et al., 2024) uses Machine Learning (ML) GitHub repositories to assess LLMs' text-to-code capabilities and test autonomous agents in task execution.
Benchmark
Paper Understanding
Repo-Search
Test Case
Source
Task Types
E-BENHH
x
Kaggle
Algorithm design and code gen. Algorithm design and code gen. Code gen.
MLAgentBench
Kaggle
ML-BENCH
X
/
Github
SciReplicat--enchh
/
Publications
Replicate code for algorithuss woal-world NL.P publications.
Table 1: Comparisons of different machine learning software engineering benchmarks.
To address this gap, we developed CedReplicate-Benchh the first benchmark specifically designed to evaluate LLMs' capabilities in code generation for reproducing research findings from academic papers. It consists of 100 code reproduction tasks derived from 36 papers published in leading conferences in 2024. This recent publication window was deliberately chosen to minimize the risk of data leakage. An overview of the task is illustrated in Figure 1, with concrete example provided in Figure A2 in Appendix E. The task consists of two main steps: 1. Algorithm understanding. LLMs must extract essential information from the paper, such as workflow details, algorithm descriptions, and hyperparameter values. 2. Code implementation. LLMs then implement function or method within provided repository, using both the extracted information and the LaTeX representation of the algorithm from the paper. We introduce Sei-Repr,,,,,r, dual-agent system that combines Paper Agent and Code Agent to handle these two steps collaboratively and implement code for the target algorithm.
To rigorously assess LLM performance on this benchmark, we evaluate two dimensions corresponding to the aforementioned two steps: algorithm comprehension correctness and code correctness. To evaluate algorithm comprehension, we introduce a reasoning graph to repre- sent the reasoning logic behind algorithm reproduction. Each node in the graph represents code comment, which reflects a single reasoning step and is aligned with a specific segment of code. Edges between nodes are defined based on data flow relationships across different code segments. We compute the similarity between the generated reasoning graph and reference graph to derive the reasoning graph accuracy. To evaluate code correctness, we employ established metrics including execution accuracy (Rajkumar et al., 2022; Xiang et al., 2023), CodeBLEU (Ren et al., 2020), and recall of intra/ cross/crssefile dependencies and APIs.
Our work makes the following contributions:
Benchmarks: SciReplicate-Bende Bench, a benchmark of 100 algorithm reproduction tasks from recent NL.P publications.
2

---

Published as a conference paper at COLM 2025
Metric:
We propose novel reasoning graph accuracy metric for evaluating algorithmic comprehension.
Approach: Sci-Reprouueer, dual-agent framework combining paper understanding and code implementation.
Insights: Comprehensive evaluation across state-of-the-art LLMs reveals four key find- ings: (i) the task remains highly challenging, with execution accuracy below 40% for all models; (ii) reasoning models exhibit "overthinking" behavior (Cuadron et al., 2025; Sui al., 2025), over-relying on internal reasoning rather than utilizing available tools for information extraction; (iii) while LLMs demonstrate strong algorithmic comprehension, they struggle with practical implementation; and (iv) algorithm reproduction failures often stem from incomplete paper descriptions, which our Sci-Rpproducer effectively addresses.
2
Related Work
Our work lies the intersection of AI for automating scientific discovery and LLM-based code generation.
2.1 AI for Automating Scientific Discovery
The application of LLMs to accelerate scientific research has emerged rapidly growing field with diverse approaches. Several studies have demonstrated the potential for compre- hensive research automation through end-to-end AI systems. Schmidgall et al. (2025); Lu et al. (2024) developed frameworks that integrate idea generation, experimental validation, and manuscript composition, with some AT-authored papers successfully passing workshop review processes (Yamada et al., 2025). Complementary research has focused on the creative aspects of scientific inquiry, with Wang et al. (2023); Ghafarollahi & Buehler (2024); O'Neill et al. (2025) investigating LLMs' capacity for generating novel research hypotheses. Notably, recent evaluations by Gu et al. (2024); Kumar et al. (2024); Liu et al. (2025); Si et al. (2024) suggest that Al-generated research concepts may occasionally exceed human-generated ideas in terms of novelty and originality.
Within computational disciplines where implementation validation is essential, LLMs have shown promise in algorithm design and code development tasks. Our proposed SciReplicate-Bencl addresses previously underexplored area: the automated reproduction of algorithms directly from academic publications. This represents a unique challenge at the intersection of scientific literature comprehension and executable code synthesis. While recent parallel efforts such as PaperBench (Starace et al., 2025) and Paper2CodeBenchh (Seo et al., 2025) have tackled related problems by exploring full codebase reconstruction, these evaluation approaches rely substantially on manual assessment criteria and LLM-based correctness judgments, introducing potential inconsistencies and reliability concerns. Our approach prioritizes objective evaluation through execution accuracy, providing more rig- orous validation than non-executable assessment methodologies (Wang et al., 2022; Chen et al., 2021).
The broader ecosystem of computational reproducibility research includes specialized frameworks such MLGym (Nathani et al., 2025) for baseline improvement, and evaluation benchmarks developed by Siegel et al. (2024); Ren et al. (2023) that assess LLMs' ability to reproduce published experimental results using existing codebases.
2.2 LLMs for Code Generation
Code generation has emerged as prominent application of LLMs, with benchmarks ranging from basic programming tasks (Chen et al., 2021; Jain et al., 2024; Austin et al., 2021; Hendrycks et al., 2021; Liu et al., 2022) to realistic software engineering challenges like SWE-bench (Jimenez et al., 2023), which uses actual repository pull requests. However, these benchmarks primarily target general software engineering rather than scientific algorithm reproduction.
3

---

Published as a conference paper at COLM 2025
Recent efforts have developed machine leamiig-speecffic benchimrbbssssssss (Liu et al., 2023; Huang et al., 2023; Chan et al., 2024), but these typically involve implementing algorithms proposed by the models themselves solving relatively straightforward tasks. They lack the depth of algorithmic understanding and rigorous paper analysis required for reproducing algorithms from peer-reviewed publications.
Despite advances in tool-augmented code generation (Schick et al., 2023; Zhang et al., 2024a;a; 2023b), no existing system specifically addresses the unique challenge of translating academic papers into executable code. Our Sci-Reproducer framework demonstrates the ability to comprehend academic publications and convert abstract algorithm descriptions into functional implementations.
3 SciReplicate-Bencl
Overview SciReplicate- Bennchhhtteeechhhhee is designed to evaluate LLMs' ability to reproduce algo- rithms from academic papers, consisting of 100 tasks curated from 36 recent NLP publi- cations with their corresponding open-source implementations. The task categories are detailed in Figure A1 in Appendix B. The benchmark focuses on repository-level code generation, where each task is centered around implementing specific function or class method. As illustrated in Figure A3 in Appendix E, each task comprises nine components, which can be categorized into three groups corresponding to code generation, evaluation, and analysis, respectively.
For code generation, the following components are provided as inputs to LLMs:
Function signature: the definition of the target function, including detailed descriptions of its input and output variables.
Algorithm Description: The LaTeX code description of the target algorithm, typically located within subsection paragraph of the target paper.
Literature context: the original paper along with its cited references, providing broader conceptual context.
Repository context: all source files and code in the repository that inform or support the target implementation.
For evaluation, the following components are provided for code execution and metrics calculation:
Reference implementation: ground-truth code serving as the reference for CodeBLEU evalu- ation.
Reasoning graph annotations: structured representations of the algorithmic logic and implementation flow, enabling assessment of reasoning graph accuracy.
Dependency annotations: comprehensive documentation of internal dependencies, cross- file relationships, and external API usage for computing recall metrics across all depen- dency categories.
Test environment: isolated Python execution environment containing validation cases and automated verification scripts for assessing implementation correctness.
To enable further analysis of the underlying causes of LLM failures, the benchmark includes:
Missing/Mismattch Information: the LaTeX description of the algorithm may omit certain implementation details, which could either appear elsewhere in the paper or be entirely absent. We also annotate mismatches between the paper description and the reference implementation.
Task Definition Based on SciReplicate an LLM given the algorithm description, function signature, literature context, and repository context as input. The LLM is asked to output function that implements the target algorithm.
Benchmark Construction The benchmark construction process comprises four key steps: paper selection, Python environment setup, documentation, and verification suite prepa- ration. To mitigate the risk of data leakage, selected papers published in 2024 that
4

---

Published as a conference paper at COLM 2025
provide publicly available code repositories. During the annotation process, each repository was refactored to isolate the core algorithm into a standalone function, and all sources of randomness were removed to ensure reproducibility and prevent leakage. On average, annotating each paper requires approximately 12 hours. Details of the annotation process are provided in Appendix A.
3.1 Evaluation Metrics
3.1.1 Evaluating Algorithm Comprehension
We propose the reasoning graph accuracy metric evaluate how well LLMs understand the logic and implementation of algorithms. During code generation, LLMs are prompted to insert specially formatted, non-overlapping, non-nested comments that mark reasoning steps derived from the algorithm's 's LaTeX code (The prompt can be found in Figure A5). We then construct reasoning graph = {N, (illustrated in Figure A3), modeled as Directed Acyclic Graph (DAG). Each node n; = (W; R2 E N represents reasoning step with a comment wi and corresponding code snippet cj. An edge ej = (n, nj), ej E E is added a variable used in ej is defined or last modified in c₁. To compute the reasoning graph accuracy, we compare the generated graph Gg with the reference graph G, via node and edge matching:
Node matching: comments from Gy and Gg are passed to GPT-4o, which maps each reference node one or more nodes in the generated graph. A node in G, is considered matched if it has at least one corresponding node in Gg. The prompt template used for this process is available in Figure A4.
Edge matching: for each reference edge e, (n,, if both endpoint nodes have corresponding matches in Gg, we apply Breadth-First Search(BFS) to verify whether corresponding edge exists in Gg-
The reasoning graph accuracy S, is computed
N
E
S
Σ
(1)
where Nm and Em denote the sets of matched nodes and edges, respectively, and s/ and sy represent their corresponding significance scores. Node significance is determined by the complexity of its corresponding code segment, measured by the number of variable defini- tions and usages, function calls, arithmetic operations, and lines of code, then normalized across the reference graph. Edge significance is calculated as the product of the significance scores of its connected nodes, followed by normalization.
3.1.2 Evaluating Code Generation
For assessing coding ability, we use the following evaluation metrics:
Execution accuracy (Xiang et al., 2023; Zhang et al., 2024b; Long et al., 2022): we integrate the generated code into the repository and execute it to obtain results. all test cases match the reference results, we consider the code correct.
CodeBLEU (Ren et al., 2020): this metric evaluates how similar the generated code is to reference code by using the traditional BLEU metric (Papineni et al., 2002) while incorporating syntactic information through abstract syntax trees (AST) and semantic understanding via data-flow graphs (DFG).
Recall (Li et al., 2024): we calculate recall scores specifically for intra-file dependencies, cross-file dependencies, and external APIs.

---

Published as a conference paper at COLM 2025
Literature Context
Code Context
t
Python Envronment
Target Relevont Litrrture
Reposttory
Webate
Interact
Interact
Paper Agant
Code Agent
Literature Report
Aggent Strategy
Agpent
Actions
Actions
Search web
Search Poper/Extroct Section
Seurch Code
Search Liferature
Gode Interpinter
Figure 1: Overview of the task and the proposed Sci-Reproducer framework. The task involves algorithm understanding and code implementation, handled by Paper Agent and Code Agent operating in separate contexts with specialized actions.
Action Name
Input
Observation
Paper Agent
SearchPaper
Query
The retrieved response from the target paper relation
the query.
SearchSection
Section
entire content of a section based on section label.
SearchLiterature
Paper ID, query
The answer to the query searched from the literature (identified by Paper ID).
Code Agent
SearchCode
Name
definition a specific code element repository.
SearchFile
Name
The content of certain file in repository.
SearchWeb
Query
information obtained from the website.
Compiler
code
The feedback from the compiler after executing the code.
Table 2: The pre-defined actions for the Paper Agent and the Code Agent.
Sci-Reproducer
To address this task, we introduce Sci-Reproducer 4, dual-agent framework designed for scientific paper algorithm replication. As illustrated in Figure 1, Sci-Reproducer comprises P'aper Agent and a Code Agent that collaboratively work to replicate algorithms described in given paper. The predefined actions employed by the agents are summarized in Table 2, with implementation details provided in Appendix C.
4.1 Paper Agent
Based on the provided algorithm description, the Paper Agent systematically retrieves contextual information from the literature context to support algorithmic understanding, Due to the input length limitations of LLMs, is infeasible to input entire paper along with their associated literature. Consequently, the Paper Agent must selectively extract pertinent information, following a strategy akin to Retrieval Augmented Generation (RAG) (Wang et al., 2024b; Sarthi et al., 2024). The Paper Agent incrementally builds an understanding of the target algorithm by executing predefined actions to query the literature context. To facilitate this process, we adopt ReAct (Yao et al., 2022) as the agent strategy, which enables seamless integration of action execution with intermediate reasoning steps.
After the Paper Agent concludes that necessary information has been collected, gen- erates comprehensive literature report comprising key findings that fill in the missing
'A video demonstration showcasing Sei-Reerodseer's capabilities is provided at be/ GcSMMgyehj

---

Published as a conference paper at COLM 2025
Approach
Exe Acc(t)
CodeBLEU(1)
RG Acc(t)
Intra-File(t)
Recall Cross-Filed File(1) API())
GPT-to-mini
No Agent
0.030
0.238
0.694
0.217
Paper Agent
0.040
0.246
0.717
0.024
0.000
0.251
Code Agent
0.140
0.279
0.738
0.563
0.364
0.328
Sci- Reproducer
0.170
0,303
0.741
0.576
0.364
0.362
S GPT-4o
No Agent 0.040
01.259
0.,205
0.059
Paper Agent
0.020
0.263
0.724
0.023
0.000
0.298
Code Agent
0.260
0.325
01.748
0.682
0.576
0.421
Rel-Reproducer
0.270
0.329
0.751
0.688
0.636
0.417
A
Claude-Sonnet-3.7
No Agent 0.070
0.2882
0.725
0.094
0.091
0.362
Paper Agent
0.050
0.291
0.727
0.082
0.091
0.379
Code Agent
0.320
0,394
0.764
0.763
0.545
0.545
Sci- Reproduuccrr
0.390
0.401
0.773
0.776
0.636
0.626
Gemini-2.0-Flash
No Agent 0.070
0.275
0.688
0.071
0.000
0.294
Paper Agent
0.040
0.278
0.082
0.332
Code Agent
0.220
0.323
0.708
0.553
0.212
0.426
Sd-Reproducee
0.250
0.346
0.727
0.588
0.333
0,455
Deepseek-V3
No Agent 0.030
0.260
0.712
0.012
0.061
0.272
Paper Agent
0.050
0.275
0.732
0.012
0.306
Code Agent
0.210
0.312
0.736
0.482
0.182
0.383
SciReerrduucer
0.220
0.334
0.738
0.565
0.333
0,443
03-mini-low
No Agent 0.080
0.259
0.758
0.035
0.000
0.323
Paper gent
0.050
0.262
0..29
0.035
0.000
0.315
Code Agent
0.150
0.278
0.803
0.306
0.000
0.348
Sci- Re-Reeproduer
0.180
0,280
0.771
0.376
0.121
0.328
03-miniimeddim Q
No Agent 0.040
0.263
0.729
0.035
0.000
0.336
Paper Agent
0.060
0.263
0.726
0.047
0.000
0.319
Code Agent
0.289
0.749
0.376
0.404
Reprouerr
0.240
0.283
0.758
0.341
0.061
0.362
S 03-mini-high QQ
No Agent
0.070
0.269
0.718
0.047
0.000
0.345
Paper Agent
0.070
0.267
0.751
0.035
0,000
0.366
Code Agent
0.160
0.277
0.773
0.165
0.152
0.374
Sci-Reproducer 0.160
0.283
01763
0.294
0.091
0.357
Table 3: Performance evaluation on the SciReplicate- benchmark. Models with CQ
notation indicate reasoning LLMs. "Exe Acc" represents execution accuracy while "RG Acc" indicates reasoning graph accuracy.
components of the target algorithm's LaTeX source. An example of the literature report is shown in Figure A8. This report subsequently serves crucial input for the Code Agent. The prompt used to guide the Paper Agent is provided in Figure A6.
4.2 Code Agent
Informed by the algorithm description, literature report, and code context, the Code Agent searches the repository locate essential dependencies required for implementation. It can also browse websites for additional information and use a compiler to test and iteratively debug the code, ensuring proper execution by identifying and fixing syntax errors. The prompt for the Code Agent is provided in Figure A7.
Experiments
We evaluate Sci-Reproducer the SciReplicate-Bench benchmark using 7 advanced LLMs, including five non-reasoning LLMs: GPT-4o-mini (4o mini, 2024), GPT-4o (GPT-40, 2024), Claude-Sonnet-3.7 (Claude-Sonnet-3.7, 2025), 0e (Gemini--200FFlaas 2024),
7

---

Published as a conference paper at COLM 2025
and Deepseek-V3 (DeepSeek-Ale et al., 2024), and different versions of the reasoning models O3-mini (o3 mini, 2024), i.e, three different levels reasoning intensity. For the reasoning graph accuracy metric, node matching is performed using GPT-4o, which may introduce some randomness. To reduce this variability, we set the temperature to and top-p to 1, ensuring more deterministic generation. The calculation is repeated three times, and we report the average score as the final result.
5.1 Results on SciReplicate-Bnnch
Table 3 displays Sci-Reproducer's 's evaluation results and contributions of Code/ Agent. The "No Agent"directly prompts the LLM to generate code based solely on the algorithm description and function signature. "No Paper Agent" allows the LLM to use Code Agent actions, but restricts access to Paper Agent actions. "No Code Agent" grants access to Paper Agent actions but blocks Code Agent capabilities. The results offer key insights, discussed in the following.
LLMs struggles SciReplicate-ennch Most LLMs perform poorly, achieving less than 0.1 execution accuracy without using the agent to examine literature and repository contexts. With enhancement of Sci-Reproducer, these LLMs show notable improvements, with an average increase of 0.181 in execution ACC and 0.057 in CodeBLEU, although even the best-performing model, Claude-Sonnet-3.. only achieved 0.390 execution accuracy. This highlights the exceptional challenge presented by our SciReplicate-Benee Bench.
LLMs comprehend algorithm logic LLMs demonstrate strong algorithmic compre- hension capabilities, as evidenced by reasoning graph accuracy scores averaging 0.716 even without agent assistance. The addition of individual agents provides modest but consistent improvements: the Paper Agent increases understanding by 0.009 on average, while the Code Agent contributes larger gain of 0.036. Combined agent deployment yields a cumu- lative improvement of 0.037. These enhancements stem from complementary mechanisms: the Paper Agent strengthens theoretical comprehension by gathering relevant contextual information from academic literature, while the Code Agent facilitates practical under- standing through extraction of pertinent code patterns and dependency structures from repositories.
LLMs face challenges with actual implementation Although LLMs are capable of under- standing algorithms, their performance in code generation remains suboptimal. Despite using Rerooeerd,,,,r, the average execution accuracy remains low 0.235, with a CodeBLEU score of 0.320.
Accurate dependency and API identification is crucial for code implementation Effectively recognizing and leveraging dependencies from the source repository and external APIs is essential for accurate code implementation. The integration of Code Agent led to substantial gains in recall with average increases of 0.441, 0.239, and 0.100, respectively, compared to cases without the agent. With Sci-Reproducer, Claude-Sonnet--.. attains the highest execution accuracy of 0.390, with the highest recall for intra/ cross file dependency and API usage, at 0.776, 0.636, and 0.626 respectively.
Overthinking leads limited improvement in reasoning LLMs Reasoning LLMs exhibit more modest performance gains when using Sci-Reproducer. While they achieve an average execution accuracy improvement of 0.13, non-reasoning models demonstrate substantially larger gains of 0.212. This pattern extends to recall metrics, where reasoning LLMs show improvements of 0.243, 0.061, and 0.041 respectively, compared more pronounced gains of 0.560, 0.345, and 0.135. We attribute this performance gap to the " "overthinking" phenomenon (Cuadron et al., 2025; Sui al., 2025), where excessive internal reasoning impedes effective action execution, a limitation that we examine in detail in the following subsection.
non-reasoning LLMs'
5.2 Tool Usage Analysis
Figure presents the number of times each LLM invokes actions on the full dataset with Sci-Reproducer. We observe the following:
8

---

Published as a conference paper at COLM 2025
Grouped Dar Chart Showing Tool Usage lrequencyy by Model
Model
c3-mi-iip min-
300
os-mir L-med um
ob mim inigh
opt-do-mih
9p5-40
250
dinepsowk
150
100
SearchFie
SearchCode
SearchWeb
Compller
ScarchRoper
SearchSection
SearchLiterooure
Figure 2: A grouped bar chart illustrating the frequency of tool usage by different models. The x-axis represents various actions, while the y-axis indicates the total number of times each tool was used on this dataset.
For code-related actions, reasoning LLMs demonstrate limited tool usage, employing "SearchFile", "SearchCodeltem' and "SearchWeb" only 25.0, 3.3, and 0.0 times on av- erage, respectively. Non-reasoning LLMs use these same actions far more extensively, with averages of 210.4, 68.2, and 16.8 times respectively. This disparity reveals funda- mental behavioral difference: reasoning models favor internal deliberation over external information gathering. Conversely, reasoning LLMs invoke "Compiler" " more frequently, indicating they require more debugging iterations due to inadequate contextual infor- mation gathering. This over-reliance on internal reasoning undermines performance: advanced models like a3-miní-high and o3-mini-low achieve execution accuracy compara- ble to GPT-4o-mini, negating their theoretical computational ad advantages.
Paper-related actions exhibit similar pattern. Reasoning LLMs use "Searchl'aper" an average of 56.3, 70.0, and 20.0 times respec-
"SearchSection", and "Search tively, while non-reasoning LLMs demonstrate substantially higher usage at 244.8, 188.4, and 58.0 times respectively. Additionally, we observe a clear preference for target paper extraction over external literature consultation. Actions targeting the primary paper ("SearchPaper" and "SearchSection") are invoked 174.1 and 144 times on average, signifi- cantly more than "SearchL which accesses related works only 43.8 times.
5.3 Error Analysis
5.3.1
Syntax Errors
Table A4 shows the syntax error rates for each model across different configurations. With- out the Code Agent, syntax errors occurred at rates of 80.3% ("NoAgent") and 83.3% ("Paper Agent"). After implementing the Code Agent, these error rates dropped significantly to 29.4% ("Code Agent") and 24.9% ""Sci-Reproducer". The remaining syntax errors mainly result from incorrectly using repository dependencies. This occurs because our approach, un- like human developers, cannot dynamically access runtime information through compiler during the code generation process.
5.3.2 Logic Errors
Another issue stems from differences in implementation logic, which can be broadly catego- rized into: (1) discrepancy in algorithm implementation that result in differing outputs, and (2) missing or mismatch information the algorithm descriptions the paper compared to the actual code.
Implementation discrepancy An algorithm may have multiple valid implementation ap- proaches. For example, the cross-entropy loss function be implemented by directly

---

Published as a conference paper at COLM 2025
Model (Sci-Reproducer)
Exe Acc(1)
RG Acc(t)
Intra-File(1)
Cross-File(t)
Recall
API())
GPT-io-mini
0.220
0.316
0.809
0.588
0.485
0.409
Deepseck-V3 03-mini-low Q
0.470
0.378
0.834
0.682
0.424
0.609
0.220
0.292
0.850
0.259
0.091
0.460
Table 4:
Experimental Results when input in the prompt.
information is regard as external
invoking the PyTorch API "torch.nn. "torch.nn.CrossEnnnn CrossEntropy" or by manually coding it from scratch. Such implementation choices may introduce subtle differences that lead to variations in the final output of the function.
information in algorithm description Algorithmic descriptions in research papers often lack concrete implementation details, and in certain cases, the provided code may exhibit minor discrepancies compared the descriptions in the paper. We manually compared the implementation code of all tasks in the dataset with their de- scriptions in the papers identify missing or mismatch information. We then provided this information as additional input and apply Sci-Reproducer framework on three LLMs. The Results is shown Table 4, regarding to Execution Acc, the performance for GPT-4o-mini, Deepseek- V3 and O3-mini-low improved 0.050, 0.250 and 0.040 respectively. The missing information can be divided into four categories:
Missing/Missatche
Hyperparameters and configurations:
descriptions of target algorithms in papers often omit specific hyperparameter settings, such as the batch size.
Numerical stability techniques: standard techniques for ensuring numerical stability, such as handling division by zero.
Implementation logic: common implementation practices and model design choices, such data splitting protocols.
Coding strategy: practical programming techniques that enhance implementation effi- ciency and reliability, such as early stopping criteria.
More examples for each category can be found in Table A5 Appendix E. As for mismatched information, it occurs far less frequently compared missing information, and its categories largely overlap with those mentioned above.
To mitigate the widespread issues of missing and mismatched information, the first category can generally be addressed by referencing the original research paper and related literature, or by inspecting the code repository for explicit configurations. However, addressing the other three categories requires familiarity with general machine learning coding conven- tions, thus necessitating that the LLMs identify and utilize implementation patterns from comparable algorithms to enhance code quality. Future research may improve performance by incorporating implementation insights from similar algorithms through techniques such as in-context learning (Zhou et al., 2023; Xiang et al., 2024), and by leveraging real-time compiler feedback to infer precise variable values.
Conclusion
We evaluate LLMs' ability to replicate algorithms described in recent NLP papers. To support this, we introduce BikepllicateBennhh benchmark with rich annotations, and Sci-Reproducer, a dual-agent framework for bridging algorithm understanding and code generation. We assess performance using reasoning graph accuracy and standard implemen- tation metrics. Results show the task is highly challenging, with failures largely caused by missing or inconsistent algorithm descriptions.
Acknowledgements
This work was supported in part by the UK Engineering and Physical Sciences Research Council (EPSRC) through a Turing AI Fellowship (grant no. EP/V020572 /1, EP /V20579 /2). We thank the authors of the selected papers for making their code openly available.
10

---

Published as a conference paper at COLM 2025
References
GPT 4o mini. https::/platffn http://pplatffrmm.peee... /tt/////////////ee////////oooooo 2024.
Esteban Garces Arias, Julian Rodemann, Meimingwei Li, Christian Heumann, and M. Aßen- macher. Adaptive contrastive search: Uncertainty-guider decoding for open-ended text generation. In Conference on Empirical Methods in Natural Language Processing, 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. ArXiv, abs/2108.07732, 2021.
Markus J. Buehler. Accelerating scientific discovery with generative knowledge extraction, graph-based representation, and multimodal intelligent graph reasoning. Machine Learn- ing: Science and Technology, 2024. URL httt:://oopsceeett iop. org http///1.cceee...../ 1088/ 2632-2153/ad7222
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal A. Patwardhan, Lilian Weng, and Alek- sander Mkadry. Mle-bench: Evaluating machine learning agents on machine learning engineering. ArXiv, abs 00000/400,,7 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Paamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo Bavarian, Clemens Winter, Phil Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. 2021.
Arv,
Qizhou Chen, Taolin Zhang, Dongyang Li, Longtao Huang, Hui Xue, Chengyu Wang, and Xiaofeng He. Lifelong knowledge editing for Ilms with aetrivaa-augmennee continuous prompt learning. In Conference on Empirical Methods in Natural Language Processing, 2024a.
Shuhao Chen, Weisen Jiang, Baijiong Lin, James T. Kwok, and Yu Zhang. Routerdc: Query- based router by dual contrastive learning for assembling large language models. ArXiv, abs/2409 100896409.9888, 2024b.
Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, and Ke Ding. Learning to maximize mutual information for chain-of-then thought distillation. In Annual Meeting of the Association for Computational Linguistics, 2024c.
Ziru Chen, Michael White, Raymond Mooney, Ali Payani, Yu Su, and Huan Sun. When is tree search useful for Ilm planning? it depends on the discriminator. In Annual Meeting of the Association for Computational Linguistics, 2024d.
Yew Ken Chia, Guizhen Chen, Weiwen Xu, Anh Tuan Luu, Soujanya Poria, and Li Bing. Reasoning paths optimization: Learning to reason and explore from diverse paths. ArXiv, 201s//410008888, 2024.
Neil Chowdhury, Franklin Wang, Sumedh Shenoy, Douwe Kiela, Sarah Schwettmann, and Tristan Thrush. Nearest neighbor normalization improves multimodal retrieval. In Conference on Empirical Methods in Natural Language Processing, 2024.
Claude-Sonnet-3.7. http:://wwww..thopp..ccoo
http:ppppcccco/caaa//////senoooo 2025.
Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, Nicholas Thumiger, Aditya Desai, lon Stoica, Ana Klimovic, Graham Neubig, and Joseph E. Gonzalez. The dan- ger of overthinking: Examining the reasoning-action dilemma in agentic tasks. ArXiv, abs/2502.08223, 2025.
11

---

Published as a conference paper at COLM 2025
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bing-Li Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dong-Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, L.. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Jun-Mei Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shao-PPigg Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang. Wenjun Gao, Wen-Xuan Yu, Wentao Zhang, X. Q. Li, Xiangyu Jin, Xianzu Wang, Xiaoling Bi, Xiaodong Liu, Xiaohan Wang, Xi-Cheng Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, X. Zhu, Yang Zhang, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yao Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yi Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yi-Bing Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxiang Ma, Yuting Yan, Yu-Wei Luo, Yu mei You, Yuxuan Liu, Yuyang Zhou, Z. E. Wu, Zehui Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhen guo Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zi-An Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report. ArXiv, abs/ a 19437, 2024.
Zhenyun Deng, M. Schlihtkrull, and Andreas Vlachos. Document-level claim extraction and decontextualisation for fact-checking. In Anmual Meeting of the Association for Computational Linguistics, 2024.
Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ran- ran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj San- jay Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip S. Yu, and Wenpeng Yin. LLMs assist NLP researchers: Critique paper (meta))reviewing
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 5081-5099, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: /et.g///eeeccooooooe///e/een//// //aclanthology.org///
(11663//1122244emlpmmma comlp-mmin.299) 292/
URL https:
Gemini-2.0-Flash https:///een:::
2024.
Alireza Ghafarollahi and Markus J. Buehler. Sciagents: Automating scientific discovery
through multi-agent intelligent graph reasoning. ArXiv, hats/2409.0555 2024.
GPT-4o. https://platfomm www..oppnaai......m www.aa..oo/doocsssseo.e wwt.c//ooo//mooee///gttttoe
2024.
Mourad Gridach, Jay Nanavati, Khaldoun Zine El Abidine, Lenon Mendes, and Christina Mack. Agentic ai for scientific discovery: A survey of progress, challenges, and future directions. 2025.
Tianyang Gu, Jingjin Wang, Zhihao Zhang, and HaoHong Li. Lims can realize combinatorial creativity: generating creative ideas via Ilms for scientific research. ArXiv, abs/ /2//2212111441 2024.
12

---

Published as a conference paper at COLM 2025
Xuemei Gu and Mario Krenn. Interesting scientific idea generation using knowledge graphs and IIms: Evaluations with 100 research group leaders, 2025. URL htttps://axxvv.ort abs/2405.17800 17844.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. ArXiv, abs/2105.00 2105.09938, 2021.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. In International Conference on Machine Learning, 2023.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. ArXiv, abs 2440.007774 2024.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? ArXiv, abs/ abs/2310.06770, 2023.
Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. Exploring concept depth: How large language models acquire knowledge at different layers? ArXiv, abs/ 2ets//40000006, 2024.
Amir Hossein Kargaran, Franccois Yvon, and Hinrich Schutze. Masklid: Code-switching language identification through iterative masking. ArXit, abs/2406.06265 066263,, 2024.
Yeachan Kim, Junho Kim, and SangKeun Lee. Towards robust and generalized parameter- efficient fine-tuning for noisy label learning. ArXiv, abs/2411.008777 2024.
Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal, and Asif Ekbal. Can large language models unlock novel scientific research ideas? ArXiv, abs/2409.06185 2024.
Wen Lai, Viktor Hangya, and Alexander Fraser. Style-specific neurons for steering Ilms in text style transfer. ArXia, abs/2410.00593, 2024.
Jia Li, Ge Li, Yunfei Zhao, Yongming Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. Deveval: A manually-annotatee code generation benchmark aligned with real-world code repositories. ArXiv, abs/ 1905..966,, 2024.
Hongfu Liu, Yuxi Xie, Ye Wang, and Michael Shieh. Advancing adversarial suffix transfer learning on aligned large language models. In Conference Empirical Methods in Natural Language Processing, 2024a.
Wanlong Liu, Li Zhou, Ding Zeng, Yichen Xiao, Shaohuan Cheng, Chen Zhang, Grandee Lee, Malu Zhang, and Wenyu Chen. Beyond single-event extraction: Towards efficient document-level multi-event argument extraction. ArXiv, abs/ flos/2405.00 2405501884 2024b.
Yihong Liu, Chunlan Ma, Haotian Ye, and Hinrich Schütze. Transmi: A framework to create strong baselines from multilingual pretrained language models for transliterated data. Arv, abs//405.09911 2024c.
Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, and Dongzhan Zhou. Researchbench: Benchmarking lims in scientific discovery via inspiration-based task decomposition. ArXiv, abs/ abs/2503.21248, 2025.
13

---

Published as a conference paper at COLM 2025
Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Zheng Li, Liang Chen, Yiming Zong, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark B. Gerstein. MI-bench: Evaluating large language models and agents for machine learning tasks repository-level code. 2023.
Zejie Liu, Xiaoyu Hu, Deyu Zhou, Lin Li, Xu Zhang, and Yanzheng Xiang. Code generation from flowcharts with texts: A benchmark dataset and an approach. In Conference on Empirical Methods Natural Language Processing, 2022.
Lingli Long, Yongjin Zhu, Jun Shao, Zheng Kong, Jian Li, Yanzheng Xiang, and Xu Zhang. N12sql generation with noise labels based on multi-task learning. Journal of Physics: Conference Series, 2294, 2022.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. ArXio, 2024.
Michele Mastromattei and Fabio Massimo Zanzotto. Less is ken: a universal and simple non-parametric pruning algorithm for large language models. ArXiv, abs/240s 2402.03142 2024.
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Niko lay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaura- sia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. Mlgym: A new framework and benchmark for advancing research agents. ArXiv, abs/2502. 2025.
03 mini. https://platfon htt.://m....eaai..cooo/////oeeea
2024.
Charles O'Neill, Tirthankar Ghosal, Roberta Raileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciuca. Sparks of science: Hypothesis generation using structured paper data. ArXiv, abs/ 12976, 2025.
Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson, Michael E. Moseley, Curtis P. Langlotz, Akshay S. Chaudhari, and Jean- Benoit Delbrouck. Green: Generative radiology report evaluation and error notation. ArXiv, abs/ abss/2055.mm 405.03395, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, 2002.
Tianyu Peng and Jiajun Zhang. Enhancing knowledge distillation of large language mod- els through efficient multi-modal distribution alignment. In International Conference on Computational Linguistics, 2024.
Duy-Tung Pham, Thien Trang Nguyen Vu, Tung Nguyen, Linh Ngo Van, Due Anh Nguyen, and Thien Huu Nguyen. Neuromax: Enhancing neural topic modeling via maximizing mutual information and group topic regularization. ArXio, Ab09/9499999499 2024.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the text-to-sql capabilities of large language models. ArXiv, abs/2204.00498, 2022.
Tal Reiss, George Kour, Naama Zwerdling, Ateret Anaby- and Yedid Hoshen. From zero to hero: Cold-start anomaly detection. ArXiv, 240322403, 2024.
Pu Ren, N. Benjamin Erichson, Shashank Subramanian, Omer San, Zarija Lukic, and Michael W. Mahoney. Superbench: A super-resolution benchmark dataset for scien- tific machine learning. ArXiv, 2000./230..40000 14070, 2023.
14

---

Published as a conference paper at COLM 2025
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, M. Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. Xrxiv, abs/2009. 102977, 2020.
Parth Sarthi, Salman Abdullah, Aditi Tuli,, Shubh Khanna, Anna Goldie, and Christopher D. Manning. Raptor: Recursive abstractive processing for tree-organized retrieval. ArXiv, abs/2401.18059, 2024.
Timo Schick, Jane Dwivedi-Yis Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. ArXio, 2402,021,, 2023.
Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using Ilm agents as research assistants. ArXiv, abs/2501.042277 2025.
Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. Paper2code: Automating code generation from scientific papers in machine learning. ArXi,, abs/2504.17792, 2025. URL https://api. semanticscholar org/CorpusID: 278033490.
Dan Shi, Renren Jin, Tianhao Shen, Weilong Dong, Xinwei Wu, and Deyi Xiong. Ircan: Mitigating knowledge conflicts in Ilm generation via identifying and reweighting context- aware neurons. ArXiv, abs/2406. /2/.224401111,, 2024.
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can ilms generate novel research ideas? large-scale human study with 100+ nlp researchers. ArXiv, abs/ (lbs/2409.04 2440900419, 2024.
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs generate novel research ideas? large-scale human study with 100+ NLP researchers. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevien, wtt.eeiwwwwieoooummiiiiii. M23dTGWCZy.
Zachary S. Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. Core-bench: Fostering the credibility of published research through computational reproducibility agent benchmark. Trans. Mach. Learn. Res., 2024, 2024.
Sanghyeob Song, Jaihyun Lew, Hyemi Jang, and Sungroh Yoon. Unsupervised homography estimation on multimodal image pair via alternating optimization. ArXiv, Abs/2411.13306 2024.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai's ability to replicate research. ArXiv, abs/ 25.4.0088,,, 2025.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoingg for large language models. 2025.
Liyan Tang, Philippe Laban, and Greg Durrett. Minicheck: Efficient fact-checking of lims on grounding documents. In Conference on Empirical Methods in Natural Language Processing,
2024a.
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. Language-specific neurons: The key to multilingual capabili- ties in large language models. ArXiv, abs/ 1402..64488 2024b..
Hung Quoe To, Minh Huynh Nguyen, and Nghi D. Q Bui. Functional overlap reranking for neural code generation. In Anmual Meeting of the Association for Computational Linguistics, 2023.
Cunda Wang, Weihua Wang, Qiuyu Liang, Feilong Bao, and Guanglai Gao. Unifying dual- space embedding for entity alignment via contrastive learning. ArXiv, abs/2411.00022 2024a.
15

---

Published as a conference paper at COLM 2025
Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. Scimon: Scientific inspiration machines optimized for novelty. In Annual Meeting of the Association for Computational Linguisfics, 2023.
Xinyu Wang, Yanzheng Xiang, Lin Gui, and Yulan He. Garlic: Llm-guided dynamic progress control with hierarchical weighted graph for long document qa. Ar abs/2410.04790, 2024b.
Yaoke Wang, Yun Zhu, Wenqiao Zhang, Yueting Zhuang, Yunfei Li, and Siliang Tang. Bridging local details and global context in text-attributed graphs. In Conference on Empirical Methods in Natural Language Processing, 2024c.
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig, Execution-based evaluation for open-domain code generation. In Conference on Empirical Methods in Natural Language Processing, 2022.
Yanzheng Xiang, Qian-Wen Zhang, Xu Zhang, Zejie Liu, Yunbo Cao, and Deyu Zhou. G²r: A graph-guided generate-and-reramt framework for complex and cross-domain text-to-sql generation. In Annual Meeting of the Association for Computational Linguistics, 2023.
Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. Addressing order sensitivity of in- context demonstration examples in causal language models. ArXiv, abs/2402.1 2402.11677 2024.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. ArXiv, abs/2402.08983, 2024.
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Nicolaus Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. ArXiv, abs/ abss/20 2504008066, 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv, abs/ /b120.220,,, 2022.
Rui Ying, Mengting Hu, Jianfeng Wu, Yalan Xie, Xiaoyi Liu, Zhunheng Wang, Ming Jiang, Hang Gao, Linlin Zhang, and Renhong Cheng. Simple but effective compound geometric operations for temporal knowleds ledge graph completion. In Annual Meeting of the Association for Computational Linguistics, 2024.
Yao-Ching Yu, Chun-Chih Kuo, Ziqi Ye, Yu-Cheng Chang, and Yueh-Se Li. Breaking the ceiling of the I1m community by treating token generation as classification for ensembling. In Conference on Empirical Methods in Natural Language Processing, 2024.
Zeping Yu and Sophia Ananiadou. Neuron-level knowledge attribution in large language models. In Conference on Empirical Methods in Natural Language Processing, 2023.
Weizhe Yuan, Pengfei Liu, and Graham Neubig. Can automate scientific reviewing? J. Artif. Int. Res., 75, December 2022. ISSN 1076-9757. doi: (0.1613/jairt.11262. URL https://doi. org/10. j.../jair...1220
Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnos.. Gliner: Gen- eralist model for named entity recognition using bidirectional transformer. ArXiv, abs/2311.08526, 2023.
Jiahao Zhang, H. Zhang, Dongmei Zhang, Yong Liu, and Sheng Huang. End-to-end beam retrieval for multi-hop question answering. In North American Chapter of the Association for Computational Linguistics, 2023a.
Kechi Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. Toolcoder: Teach code generation models to use api search tools. ArXiv, abs/2305.00 2023b.
16

---

Published as a conference paper at COLM 2025
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Anmull Meeting of the Association for Computational Linguistics, 2024a.
Xu Zhang, Yanzheng Xiang, Zejie Liu, Xiaoyu Hu, and Deyu Zhou. 12r: Intra and inter- modal representation learning for code search. Intell. Data Anal., 28:807-823, 2023c.
Xu Zhang, Xiaoyu Hu, Zejie Liu, Yanzheng Xiang, and Deyu Zhou. Hfd: Hierarchical feature decoupling for sql generation from text. Intell. Data Anal., 28:991-1005, 2024b.
Xu Zhang, Zexu Lin, Xiaoyu Hu, Jianlei Wang, Wenpeng Lu, and Deyu Zhou. Secon: Maintaining semantic consistency in data augmentation for code search. ACM Transactions on Information Systems, 2024c.
W. Zhao, C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie. Ratescore: A metric for radiology report generation. In Conference on Empirical Methods in Natural Language Processing, 2024.
Yuxiang Zhou, Jiazheng Li, Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. The mystery of in-context learning: A comprehensive survey on interpretation and analysis. In Conference on Empirical Methods in Natural Language Processing, 2023, URL https: //api. semanticchat wrw.aaarorrr//oooto:o/.. 264832783.
Appendix
A Details of the Annotation Process
Step 1: paper selection We curated NLP papers from leading conferences in 2024, includ- ing ACL, EMNLP, ICLR, Neurips, and COLING. Using a web crawler, we collected accepted paper titles and employed the PapersWithCode API$ to identify those with open-source implementations. For each identified paper, we retrieved corresponding GitHub repository links and metadata (e.g., stars, issues, release dates) via the GitHub REST API
To filter candidates, we applied the following criteria:
Removed survey/exploratory papers while retaining method-focused research.
Applied a cutoff date of January 1, 2024 to avoid data leakage.
Excluded repositories with fewer than 5 stars ensure basic quality assurance.
Subsequently, researchers manually reviewed each candidate paper and its repository. We discarded papers with excessive computational demands, poorly structured code, ambigu- ous documentation, missing preprocessing steps, or reported reproduction issues.
Step2: python environment setup For papers passing the initial screening, annotators followed the README to set up the environment and replicate experiments. Common issues included dependency conflicts, data loading failures, and incomplete buggy code. Annotators attempted to resolve these problems; repositories with irrecoverable errors were excluded.
Step3: annotation
Annotation consists of two steps:
1. Algorithm-Function alignment: most papers contain multiple algorithmic compo- nents, often organized as subsections. Annotators segmented these into distinct units and mapped each to its corresponding implementation. Code was refactored to encapsulate each algorithm in standalone function or method. Papers with implementations too fragmented for restructuring were excluded.
/nttps://paperswi www.whhhhowccccoooooooooccoeeeco wttpo/////pp.//co//ooooooooooooo https///sss.ioooececcoooo//o////
hhttps://doc gi httpl//ssiiiiuu...oceeeeeeeeeeee hwtp../ce//eerreeraaaaeeeiiemn2o
17

---

Published as a conference paper at COLM 2025
2. Detailed annotation: for each aligned function, annotators documented in- put/ output variables, intra- and cross-file dependencies, and external API usage. Additionally, they inserted explanatory comments mapping code segments to al- gorithm components. Based on these annotations and variable dependencies, we construct reasoning graph representing the implementation logic. During the annotation process, LLMs were employed to assist with algorithm-function alignment and the generation of variable descriptions and code comments. All outputs were subsequently reviewed and corrected by human annotators to ensure accuracy.
The final selected papers are listed Table A1.
Step 4: verification suite preparation Finally, annotators created verification suites with 10 test cases per task, drawn from the original datasets used in each repository for the majority of papers. For small number of repositories, fewer than 10 test cases could be constructed. For instance, algorithms that analyze LLM parameters may have only single test case. Given the inherent randomness in many NLP implementations and potential machine-related variability, we addressed reproducibility from two angles:
Eliminating code randomness: annotators fixed random seeds and replaced non- deterministic operations (e.g,, unordered sets) with deterministic equivalents to ensure consistent outputs across runs.
Controlling hardware variability: users were instructed to run both reference and generated code locally to eliminate discrepancies caused by hardware differences.
Lastly, annotators implemented task-specific comparison scripts to evaluate output correct- ness, accounting for variations in return types across tasks.
B Details of the Task Categories
The Categories of the Tasks
15%
2.2
24%
20%
19%
Representat and Embedking Methods Loss Functions and Optimization Objectives
Information Extraction and Agg egation
Components
Inference and Search Algarithms
Figure A1: The categories of the tasks within SciReplicat--ennch.
The benchmark encompasses five main task categories in the NLP domain: representation and embedding methods, loss functions and optimization objectives, information extraction and aggregation, model architecture components, and inference and search algorithms. The distribution of each task category is illustrated in Figure A1.
18

---

Published as a conference paper at COLM 2025
C Details of the Actions
In this section, we provide implement details for all actions defined in the Sci-Reproducer.
SearchPaper We obtain the LaTeX source code of the target academic paper from arXiv? and apply regular expression-based parsing extract the content corresponding each section. Subsequently, we iteratively feed the content of each subsection, along with the query generated by the large language model, into GPT-4o-mini. The model extracts relevant information and returns as an observation to the paper agent.
Following the same approach as SearchPaper, the tool begins by parsing the LaleX source code of the target algorithm. Upon receiving section ID from the Paper Agent, it retrieves and returns the content of the corresponding section.
SearchSection
Search Given a paper ID and query, the tool attempts to download the cor- responding LaTeX source code from arXiv. If the LaTeX source code is unavailable, it returns no information. Otherwise, it extracts content relevant the query from the paper, following the same procedure as the SearchPaper action.
SearchCode For each Python file in the code repository, we utilize the Python AST package to parse the file and extract all defined classes, functions, and global variables. Unlike embedding-based code search methods (Zhang et al., 2024c; 2023c), the Code Agent in our framework directly provides the name of a code item. The tool then returns the corresponding definition if it exists; otherwise, it returns an empty response.
SearchFile When the Code Agent provides a file name, the tool returns the full content of the corresponding file.
SearchWeb When the Code Agent issues a query, we use the Google Search APT to retrieve relevant information from websites. These results are then processed by GPT-4o- mini, which filters the content and extracts the information most relevant the query for return.
Compiler Once the Code Agent completes code generation, it invokes the compiler to execute the code. The generated function method is inserted into the original Python file, and the corresponding Python environment is used to run the code. The output from the compiler is then returned as the feedback.
D Human Evaluation of Reasoning Graph Accuracy
To validate the reliability of our LLM-based evaluation metrics, conducted human evaluation study on a subset of our benchmark. Three PhD students in computer science independently assessed the reasoning graph accuracy for 20 tasks generated by GPT-4o-mini and O3-mini-medium using Sci-Reproducer.
As shown in Table A2, human evaluations demonstrate strong alignment with our au- tomated LLM-based assessments. The mean scores show consistent agreement between human annotators and GPT--40 evaluations, with standard deviations indicating reasonable inter-annotator consistency.
Furthermore, we computed the Pearson correlation coefficient between human and LLM- based evaluations across all assessed instances. As presented in Table A3, the correlation (r 0.7518, P 0.005) indicates strong positive relationship between human judgments and automated assessments, supporting the validity of our L.L.- evaluation approach.
7https://arxiv.or
$nttps://docs phttp::://docss.tthnn... hwtp:/rr//////br/////ae/e /httpss//dewellpprr..ooo google. com/ com:/custom wtt.stmcmss////////
19

---

Published as a conference paper at COLM 2025
While this preliminary validation demonstrates the effectiveness of our automated metrics, comprehensive human evaluation across the full benchmark remains an important direction for future work.
E Figures and Tables
a Seatence fumation
def sereace, por Jentiespprriss wellgg
propose model suntancs cotecteon au edtaact we surmertia tart on, the purpase, conaterei the for sentares The setub
sequas found Appendle
Samplas Dutddctt) Contars the homm oeteenses The sayc dittnsory
culading entiltmett butwwan
rxplerant
Dupuu Venable::
nouts Burtian indudingg
Edred information from birget paper ndevent papers
Letermaaee
resduce docaners encoder boocdio 2641 (bortSan)
soole situet for doo montleee DOOVl
DOCAL swwly-toreruased
valur sumplle
Extrait infernadion fore ressitory
keposhory
oheee document car sting sentences tre It Aacooial fermudes input as
for simplitts se select the sentence mumaddl by each marted compartien.
tenp
[[55] and (rar) denats
Detala oftuustion > ptA000
tano
senianee
def www.gagggggghpiaa......
Figure A2: The task consists of two steps:
Algorithm Understanding and Code Implementation. (Left) The model must extract an algorithm's workflow and details from the research paper, including descriptions and variable values from cited papers and other paper sections. (Right) Using this extracted information, the model implements the corresponding function in the code repository, correctly handling dependencies and API calls.
SuiRepliatte
Evaluation Maniice: Sanarion ACE E2: B0, Oodell BUTDA Besec tng Graph Aod a
Algerttimn Code
Randion Sggatare Joiigfm
Recal that our octecio to allgn the rupourttiiin har dtfuratt position winh these at the and got
permusaion gdi indo, Juich indi gel, zanon Los "ampprattreebbi
updans during the Finedart precess. by seednn sis and SLS lnto SMS and SS./S, we obean the
.Aen State parch departarct-faaeee flando avape-tbach you Num Layers, Num 1el
is and Slamolu (oa mirmes respactively
ioec steuuatrr
lunejecuntio)
Cutpus Vedablen:
1999 cww.ee (hoot) Ave www.srggiiittaaaa.oo..ooooo www.eeetoonioooooooooooooo sonoss sempples.
1. Terget Paper: Aadroo rg unher seradiitt ty necnrar
Liseratars Donner
Repicebarry Connert
loss comner--
Implamentation Reterance
Reloware Uteratanc
curtasicelly ucaved prompta und when to Ind thar:
buich numPaSSem ceedeet 1400
Overcoriig tamahut prompt onder aumalliety.
clase LaaraNodeel sndeect
Liamezy
Saphi betrat teheores resemantal tmst bur medence (Suntoornet organd rppesornationn fome the orginal
Stppi: Caiculats poaltive ammaarty acors so coatuars hoy mwaas how cloa
Sleph Caloratenspiinn miketty sourt czatgars camsofonslutrrattti mas how dlsse lual (desscee di which savce regalia
Swest Finel cosmaetiv losefe theae rakene dngiporti gpotiius regar un) This alggnwot
StapS Aww.mutoossaao........ an the loss and upo court and awergg the sora conmende loss
walan companions.
Massing
Verteaatio
Dependancikg danclia Liamu LiamaHodei Cross dineennniiie Noow iorct tarhorrs
Inte-- Dupen
Theinss hovitns scasss
Hizsing Detall:
crighol code repostory
Output Compars Scrige Pybon p prgaam designed varty
Cede Eesmomment
Figure A3: The overview of the SciReplicate-Bench
20

---

Published as a conference paper at COLM 2025
Task: Match each generated step with its functionally equivalent reference step(s).
Input:
Target Method: Scientific research method described LaTeX.
Reference Steps: Ordered steps from the original implementa tion, labeled [Ref 1), [Ref 2), etc.
Generated Steps: Ordered steps from -LM-generated im plementation, labeled [Gen 1), [Gen etc.
Output Requirements:
For each generated step, identify the reference step(s) that implement the specific functionality.
Format your
follows:
Ref
Ref Y, RefZ
Matching criteria:
Match based functional equivalence, textual similarity
Steps perform
specific operation
Steps serve same role overall algorithm
Steps must produce equivalent results given same inputs
Consider sequential position:
Earlier generated steps likely match earlier reference steps
generated steps likely match reference steps
generated step no equivalent or ambiguous, output
Important:
Ensure reference indices actually exist reference steps Do not include explanations in your output
Provide answers for all generated steps
Input Format:
[Target Methad] (target_method]
[Reference Steps) (ref_ comments)
[Generated Steps] (gen_ co____tmment
Your answer:
Figure A4: The prompt for node matching.
21

---

Published as a conference paper at COLM 2025
Title
Conference
1. From Zero to Hero; Cold-Start Anomaly Detection (Reiss et al, 2024)
Findings of ACL 2024
2. Addressing Order Sensitivity of In-Context Demonstration Examples Causal Language
Findings of ACL 2024
Models (Xiang et al., 2024)
3. Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling (Yu et al,, 2024)
Findings of EMNLP 2024
4. Simple but Effective Compound Geometric Operations for Temporal Knowledge Graph Completion (Ying al., 2024)
ACL 2024
5. When is Tree Search Useful for LLM Planning? It Depends on the Discriminator (Chen et al., 2024d)
ACL 2024
6. Functional Overlap Reranking for Neural Code Generation (To et al,, 2023)
Findings of ACL, 2024
7. Beyond Single-Event Extraction: Towards Ffficient I Document-I evel Multi-Event Argument Extraction (Liu et al,, 2024b)
Findings of ACT, 2024
8. Unifying Dual-Space Embedding for Entity Alignment via Contrastive Learning (Wang et al,, 2024a)
COLING 2025
9. Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concepts at Different Layers? (lin et al., 2024)
COLING 2025 2025
10. TRANSMI: A Pramework to Create Strong Baselines from Multilingua Pretrained Language Models for Transliterated Data (Liu et al,, 2024c)
COLING
11. Enhancing Knowledge Distillation of Large Language Models through Efficient Multi- Modal Distribution Alignment (Peng & Zhang, 2024)
COLING
12. Document-level Claim Extraction and Decontextualisation for Fact- Pact-Checking (Deng et al., 2024)
ACL 2024
13. IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweight- ing Context-Awan Aware Neurons (Shi et al., 2024)
Neurips 2024
14. RouterIC:: Query-Bace Based Router by Dual Contrastive Learning for Assembling Large Language Models (Chen et al., 2024b)
Neurips 2024
15. Unsupervised Homography Estimation on Multimodal Image Pair via Alternating Opti- mization (Song et al, 2024)
Neurips 2024
16. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (Sarthi et al., 2024)
ICLR 2024
17. Less is a Universal and Simple Pon-Parametricc Pruning Algorithm for Large Language Models (Mastromatteri Zanzotto, 2024)
Findings of ACL 2024
18. Adaptive Contrastive Search: Uncertainty- Guided Decoding for Open-Ended Text Genera- (Arias et al., 2024)
Findings of EMNLP 2024
19. MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (Tang et al., 2024a)
EMNLP 2024
20. Nearest Neighbor Normalization Improves Multimodal Retrieval (Chowdhury et al,, 2024)
EMNLP 2024
21. Neuron-Level Knowledge Attribution in Large Language Models (Yu & Ananiadou, 2023)
EMNLP 2024
22. RaTEScore: Metric Radiology Report Generation (Zhao et al, 2024)
EMNLP 2024
23. GREEN: Generative Radiology Report Evaluation and Error Notation (Ostmeier et aL, 2024)
Findings of EMNLP 2024
24. Style-Specific Neurons for Steering LLMs in Text Style Transfer et al. 2024)
EMNLP 2024
25. Singuage-Specific Neurons: The Key to Multilingual Capabilities in Large Language
ACL 2024
Models (Tang et al,, 2024b)
26. Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths (Chia et al, 2024)
Findings of EMNLP 2024
27. Lifelong Knowledge Editing for LLMs with Netrieval-Augment Continuous Prompt Learning (Chen et al., 2024a)
EMNLP 2024
28. NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization (Pham et al,, 2024)
Findings of EMNLP 2024
29. Bridging Local Details and Global Context in Text-Attributed Graphs (Wang et al., 2024c)
EMNLP 2024
30. Advancing Adversarial Suffix Transfer Learning on Aligned Large Language Models (Liu
EMNLP 2024
et al. 2024a)
31. SafeDecoding:: Defending against Jailbreak Attacks via Safety-Aware Decoding (Xu et al., 2024)
ACL 2024
32. MaskLII: Code-Swiithhng Language Identification through Iterative Masking (Kargaran et al, 2024)
ACL 2024
33. Towards Robust and Generalized Barameter-Effiieent Fine- Tunnng for Noisy Label Learn- ing (Kim et al., 2024)
ACL 2024
34. Learning to Maximize Mutual Information Chain-af-Thought Distillation (Chen et at., 2024c)
Findings ACL. 2024
35. GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Trans-
NAACL 2024
former (Zaratiana al,, 2023)
36. End-to-End Beam Retrieval for Multi-Hop Question Answering (Zhang et al., 2023a)
NAACL 2024
Table A1: List of papers used in our Sci-Replicate benchmark.
22

---

Published as a conference paper at COLM 2025
Table A2: Reasoning Graph Accuracy of Sci-Reproducer Models Evaluated by LLMs and
Human Annotators
Evaluation Method
Sci-Reproducer Model
GPT-4o
Human (Mean + Std)
GPT-4o-mini
0.782
0.768 (0.066)
O3-mini-medium
0.790
0.789 (0.033)
Table A3: Pearson Correlation between LLM and Human Judgments of Reasoning Graph
Accuracy
Number of Data Points
r
P-value
20
0.7518
<0.005
Approach
Error Ratio (4)
No Agent
80.3
Paper Agent
83.3
Code Agent
29.4
Sci-Reproducer 24.9
Table A4: Syntax error across different settings.
Categories
Examples
Threhools,, batch sizes, maximum ileration counts, exact numbers of clusters, initialization methods for variables or vectors, types of regularization (such as 1.1 or 1.2) and specific distance metrics (egg, using 12 norm for Euclidean distances)
Hyporparameters and configurations
clamping values avoid numerical instability adding small constants during logarithmic calculations, managing division by zero scenarios, addressing rounding and precision issues.
Numerical stability techniques
Implementation lugic
Data splitting, application of dropout, formatting of input sequences, handling special edge cases the input data.
Coding strategy
Caching for performance enhancement, retry mechanisms handle failures, early stopping criteria, and strategies for memory optimization.
Table A5: Some examples for different missing information categories.
23

---

Published as a conference paper at COLM 2025
Please complete the target function (or method) and provide the pure code output without any additional toxt. Include following these specftc guidelines:
Comments:
1. Focus Comments should explain the reasoning behind the code generation
derived from the
LaTeX description.
Implementation Details: including any code-spess aade-specific implementa detailss in
3. Mapping to LaTeX: Each comment must indicate which functionality described in the LaTeX code implemented the
subsequent Python snippet.
4. No overla The code snippets corresponding each comment not overlap, nesting not allowed.
5. Single Function: Implement the code thin single function (or method), without breaking into multiple functions.
6. Import Package: Import packages within the function (or method) ensure the code self contained.
Format,, the comment for each snippet should be the following format:
Snippet Comment here
[Bagin Snippet
Code snippet here
Snippet X]
[Example]:
"' "python
def aapplppppppeeee token_level_ {raarrnsnooooiionn
repressnttaiinn
auxilia representation,
transformatio_ _indiiess batch mask, scaling faciin_factoo-0.001
import tarch Import numpy as transformed tokens token represeetationnclonee)
(start_idx, end enumeradtttransoommttinnnn
Snippet first verify current batch item by checking the
bltch__matt mask, analogous referencing in-context example M/mathbitSS)) that has corre corresponding reference M)mathbr((H rJV in the LaTex snippet.
[Begin Snippet 1]
batch_mask[i]=
[End Snippet 1]
Snipget apply basic token_ regresentttennonnnnnnn incorporating slice from the auxiliary representation, akin to combining M\mathaf[H portion (/mathbf((H_ Brt[Hb(HHHHW_))___ for enhanced alignment.
[Begin Snippet
sogment main transformed tokensji, start iidkeend idk, sogment aux auxillary_ retillayyrrprrsttattonnt start ttart_ikkendd :] [End Snippet 2]
Snippet 3: The tr transformation is simple element-wise operation combined with scaling, factor, illustrating simplifted
alignment concept from the LaTeX, which mightinvolve more complex contrastive sttentional ca lcuulations.
[Begin Snippet 3]
updated segment torch.addlseggeent main, tonch.mullsegment aux, scaling, factor)) _calor)_facttr) tra informed takens(), start iixcand idx, updated_ segment [End 5 nippet 3]
Snippet
The final transformed tokens are now partially aligned with the auxillar reference, refiecting notion zug mugntinig
outputs
|Equation references in the LaTek snippet would correspond to (2-3)
similar definitions reference alignment).
[Begin Snippet 4]
return transooeeed tokens
[Ed Snippet 4]
...
Your answer:
Figure A5:
The prompt for code generation.
24

---

Published as a conference paper at COLM 2025
TTask Overview]
Reprodute Python code corresponding Ta bas methodology from scientific However, due the paper's length, steps:
fully ingested by large nggaage model once. Therefore, the solution requies
details, insights, and supporting information from 2. Code Reproduction (Subsequeet Taskg: implement the Python code based the informa tion gathered and the provided
Information Retrieval
Task]:
scademic paper's 's LaTeX description related literature.
|Your Specific Focus]
You tesked exclusively with Step 1: Informa tion Retrieval, You must gather organize necessary details that later be used im plement the Python code.
[input]]
List sections: The pa includes the following sections (triles provided for reference):
[Section_Stuee String)
LaTex escrrptiio: The LaTex code for the corresponding subsection in the paper, describing the algorithm implemented by the target function.
code)
Tools: Tools that can adopted to gather external information during the information retrieval process. Search Paper(query)
Descript on:
concept or any other element appears tanget section without sufficlent details, use this action to search for the complete information the full paper.
Parameter eetr::
'query' (string): A query describing the information needs be
full paper.
Examples:
contains: "We concept $X_'5
action SearchRaperr"
definition of Sx. $")
If the LaTeX contains: "The function StikIS defined based properties of Ummatha KG)S.", then the action
SearchPaper(" properties 5lymttica/(Ss551
SearchSection(k)
Description: the target section references another section in the paper with the title x extract the information from the referenced section and return SearchSection(s)
Param eters:
wix' [string:: J: The title of the referenced section.
Example:
Latew:
derivation our function can be
Section. Action: SearchSection(" "method")
Searchliteratoo aetrrchiiiteauurekeee query]
Description: the target section cites another paper (label] and you determine that some information needs be the specific information look
retrieved from
Searchüteraturellabe queryl, referenced paper.
Param eters:
(string]:
key of referenced paper.
citing paper, (ycie(j)),
represents the citation key.
"query' (stringl: The spedfiic information to search for the referenced paper.
Example:
- -Latex: "We adopt the metric proposed in teite(wang2025)" Action: Searchüterature"" werchhtterauuee(wwm220511 2wong2025" "The proposed metric paper")
-Latex: 'The algorithm is based on the work of \cite[smith20181" Action: Searchlkerature"" harriieerrrtreemmmt2211111 2mith20088 "The algorithm details the paper")
-Latex:
study by \cttefones20200 Searchüterature j00e
deta in the paper"]
[Instruction]
order complete code reproduction, is necessary to understand the algorithm described the LaTeX description. The tools "Searchfiper" "SearchSection" and "Searchüttrature" should be used to metrivv relevant information
paper you 1. the laTeX Description lacks the defisition of variable, use "SearchPape"" tool to find its definition. la Description references other sections of the paper, "SearchSection" tool retrieve those sections and supplement the missing details, 3. the la TeX Description dies methods from other papers, use "Searchükerature" tool to extract relevant information referenced papers.
methodology proposed
description.
example:
[Action]
Apply tool defined above gather external information.
gathered necessary information, the Code Reproduction stage, the appropriste action is "Finish"
code, prepared proceed
[Observation]
If the action apply prederined tool, then the observation should be the retum response of the tool.
[Response Template]
Thought: Ithink...
Action: Beorch-aperfgueren or SearchSection(abell or Searchüterstureekey, query) or Finish Dbsenvation: Outcome of the action.
|Your Answer)
Please information extraction by strictly adhering the provided tem plate for the format.
Figure A6: The prompt for Paper Agent.
25

---

Published as a conference paper at COLM 2025
You area mide anchttet racced with reproductigg Python function correponding in sigorthi in the nethocs par a srientile poper. The lnce coding enstronment includes GRU and supports cunA, (wil prouud the fillowng information:
1. Repostory strumure: The organication of files within the code repestory, This is repository code generation teskuse you should cagi are the theroug extract use code.
2. Target function: The defirition of the python function you need implement
3. LaTek descrittion piton. The LaTek coce for the corresponding thin the paper, desoribing the sggoithin implemented the
fanction
4. the acted information: the information estracted from the target and relevant lttrature that provide more details when imploment the function..
Tooks Tooks that
vaternal information curing
Inepouatery Strusurre
(nnganbration)
(Target Function]
function located "[Python, _ateoo______ereeeeee Fatth" definition condits of following compotents:
Variables
[Targed,
Extracted Information)
paper agent points imploment you should to Uterature with sorget paper, always
uncerstand sarges aigorithin, information target paper.
(extracted, jnfo]
[T
Descrption: query using Google search engine ne retrieue relecin infoemation, You weam usage, defintions, fixes, implementaton umbier algorthmms Parameters: Quers (ctiigg) search query to renrieve relevant information.
Example: Searchiteeh to instement neural network PyTorch")
2. SearcchFleeM)
Description: Retfieve the conte content of Python from the current tory
Parameters: por legh: nameett the python file search fortn ourrent repository.
Example: Searchfil""m
3. SeirchCode)ee(44]
Descrption: with information about speciiis code rrem the rapoaitory indluding global variables, functions, methads, or clases.
Farametor:: (iringg: The name the code itam to sear forin the ourrent repository.
Foumpler Swarchtadeeenn (Mode")
instuct on:
in order complete this taok it necescory
be to search the code repoaitory for cornest help
impeement the targetfuncttn function for esample:
1. Uoe Shaachillee to retreve the content Python fie from the reposhorr.
3. Une "SwarchC chCoeeleem
find deaall about speitfic rnde infelltem within the repostorry.
3. Use "Sear chiNco" to retrieve information from the website.
To effectively callee the code cedeereeroucctto tack, follow structured process thataltemttes benween Thought, Action, ndd Observation steps
[Thought]
Analize the curient sit.tttttiiinn
ident maxing information from cods. As level code
the relaant
lunctions, daaass, code repasitory.
3. Plan steps to gatner the required mnformation.
(actio))
Apply tool defined above uetemal information.
action should "GerentteGdd""
[Cheenation]
action prediffinss
should
tool.
action is "GenerateCode: observation result generaite codee
Response Templicee
Ithink
Sewrewee/lSurrllle searcheilejs]] Swwsechhodeeteenniccc
Mtep-by-stpp reposioory
wlongssie slgwtthmm.
Thought, siggtt,,,, repprnee
[rour Answer)
Figure A7: The prompt for Code Agent.
26

---

Published as a conference paper at COLM 2025
Information Point 1: Hidden Representation of Last Token
Overview: Explanation of the variable \[bm(H)]ii-]]) used in the consistency enhancement algorithh..
Source: Target Paper
Extracted Information:
((bm((H]V represents the token-level outputs of the last self-attention layer of the model.
((bm((H) \in (mathbb(R))(m (times \times dJAJ, where \mm) the batch size, V(V) the sequence length, and
MdV the dimensionality the hidden layer.
The notation M(I.-1]V refers to accessing the hidden representation of the last token for the \/1)-th input in the
batch.
Information Point 2: LoRA Adaptation
Overview: Description of the LoRA ada technique used the algorithm.
Source: Target Paper
Extracted Information:
LoRA adaptation keeps all model parametes fixed except the trainable query (W_QV value (W_V) matrices within self-attention layers.
This technique integrates information from reference representations into the LoRA parameters during fine-tuning. LLMs.
The rank of LoRA set
Information Point 3: Consistency Enhancement Methodology
Overview: Methodology for enhancing consistency model outputs batch.
Source: Target Paper
Extracte Information:
The approach enforces similarity among hidden representations last token the input batch VIV.
classification head parameters remain fixed during training.
consistency [[mattcaa(L]_((ree)) ((en)t) calculated as [[-m_[[==]mm]] (tt-teexttccos//mm(H[illl)) \bm[H_a]))
110bm(H_a) a/\) average
representations across the batch: V/bm(H_ajp
\frac(ssum_iiss (actsssm_mii11/Mmm [mm]
Figure A8: An example of output report of the Paper Agent.
27


Reviews:
[{"review_id": "Zr3vMffst3", "reviewer": "Reviewer_fdpN", "summary": "The paper presents a new benchmark, SciReplicate-Bench, for reproducing scientific papers, which contains 100 tasks from 36 papers.\nThe authors present a new evaluation metric: reasoning graph accuracy, which checks the correctness of the implicit reasoning process behind code generation.\nThe tasks are also evaluated based on execution accuracy.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task of reproducing numbers in research papers is natural.\n2. The paper experiments with a paper agent with 3 actions and a coding agent with 4 actions, where the actions are all widely used in coding practice.", "reasons_to_reject": "1. The setting of the tasks is not natural enough. The authors still focus on the traditional function/class generation setting (such as in RepoEval (https://arxiv.org/abs/2303.12570) or ClassEval (https://arxiv.org/abs/2308.01861)), where the coding agents aim to implement a specific function or class method in a given repo. This setting is not natural in two ways: (1) the repo context is extracted from the whole repo, not the commit where the target class/function has not been created yet, and (2) in practice, to implement some feature, we typically need to modify multiple classes and functions, instead of a single one.\n\n2. The test case generation part is very important but is not clearly introduced in the paper (even in the Appendix). Here are some potential concerns about test generation: (1) how do you deal with function/classes with `print` or `logging` information? (2) how many tasks require GPU for computation and what kind of GPU? (3) The authors try to eliminate randomness as follows: \"Annotators fixed random seeds and replaced non-deterministic operations (e.g., unordered sets) with deterministic equivalents to ensure consistent outputs across runs\". What if the model-generated code contains some randomness??", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eTRd5kDSaF", "reviewer": "Reviewer_RNJE", "summary": "In this paper, the authors present SciReplicate-Bench, a benchmark that aims to model automated algorithm replication from scientific papers. Specifically, their benchmark is built 36 NLP papers from 2024 (the selection of date was intended to avoid leakage issues) and involves implementing specific functions from these papers (as such, it is a rather simplified version of the general replication task). As outlined at the beginning of Section 2, the benchmark provides the following input: 1) latex code for the function to implement; 2) a function signature for the target function; 3) the original paper for this function and; 4) details of the corresponding repository (this is some mismatch between the description of the input in lines 109-114 and the task description starting in line 131; I would encourage the authors to address this). All such information is extracted manually by annotators for the selected papers (like all such annotation tasks, this appears to be a timely process, taking 12 hours per paper). \n\nTo evaluate models on this benchmark, the authors propose 3 metrics (see lines 168-176): 1) execution-based accuracy (i.e., does the implemented function match the target output); 2) CodeBleu that measures code overlap with the reference code; 3) recall based on whether the code captures certain dependencies (the details here are less clear, I would encourage the authors to make the description more clear in 2.1.2). In addition, they also attempt to measure (what they call) “algorithm comprehension”, or the ability of the model to understand the target code, and do this by prompting models to document their code that facilitates the construction of a “reasoning graph” that they compare against a reference reasoning graph. This part of their evaluation is very interesting and  unique, though I have some questions about these graphs and whether the LLM-based evaluator the employ to compare graphs makes sense (see below). \n\nThey perform experiments using ReACT style agents. Their full approach (sci-reproducer) factorizes the problem into two agents, a paper agent and code agent, and they ablate each agent to measure it’s contribution (all results are in Table 3). They experiments with a comprehensive mixture of reasoning and non-reasoning models and find the tasks to be difficult according to all metrics (e.g., GPT-4o achieves only 0.170 execution accuracy). Additional analysis is performed to better understand certain error cases (e.g., logic errors and syntax errors) and differences in tool usage among different models (I didn’t find these results, summarized in Figure 2, to be particular well motivated or insightful; I would encourage the authors the consider using this space instead for some of the important details in the appendix, such as Figure A2).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-- A new (expert annotated) benchmark for paper replication that I think others working in this area will benefit from (as I note below, however, I think this benchmark has some limitations compared with some missing related work that needs to be directly addressed). \n \n-- A comprehensive set of baseline results that show the task to be difficult, along with a very strong multi-agent baseline (i.e., their \"sci-reproducer\"). \n\n-- Some novel ideas about code evaluation based on their \"Code Reasoning Graph\" that others working in this area might adopt to evaluating similar code generation problems.", "reasons_to_reject": "-- Some key related work is missing. Specifically, a comparison with the SUPER benchmark (Bogin, et al 2024, EMNLP) (I think such a comparison should be included in Table 1). In some ways, the SUPER reproduction setup is more realistic than what's proposed in this paper (e.g., it focuses on building a full experiment pipeline rather than a single function). I would like to see this directly addressed by the authors.\n\n-- (related to above) While SciReplicate-Bench aims to evaluate experiment replication, the specific way this operationalize this (i.e., as a single function implementation) addresses only a small and narrows aspect of this general problem. \n\n-- Lack of human evaluation of automatic metrics. In particular, while the reasoning graph-based evaluation is interesting and novel, they do not appear to perform a manual evaluation of the automatic node matching and edge evaluation (from what I understand, such matching is performed by a LLM). Absent of this, it is very unclear how reliable these metrics are. \n\n-- Similarly, it's unclear whether the CodeBleu, while a known metric is reliable. E.g., is there a strong correlation between code that executes correctly and CodeBleu?", "questions_to_authors": "-- Was any manual evaluation performed on the node matching? \n-- It's still unclear how precisely the recall was computed and how reliable this is.. Was this done via  some static analysis of the code? More details would be helpful here.", "extra_scores": {}}, {"review_id": "7sjkgJIr9C", "reviewer": "Reviewer_saNg", "summary": "The paper introduces SciReplicate-Bench, a benchmark designed to evaluate large language models (LLMs) on the challenging task of reproducing algorithms from recent NLP research papers. The benchmark comprises 100 tasks curated from 36 papers published in 2024, explicitly focusing on translating algorithm descriptions written in LaTeX and scattered scientific text into executable code within corresponding open-source repositories.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. SciReplicate-Bench is a benchmark focused on reproducing realistic, algorithmic code implementations from academic publications rather than simple or toy tasks.\n2. The introduction of reasoning graph accuracy evaluates not only final code correctness but also the intermediate reasoning steps that reflect the model’s comprehension.", "reasons_to_reject": "1.  Considering that there are already many works utilizing multi-agent systems for scientific paper code generation, the novelty of the method in this paper is limited. This paper should elaborate more on the differences from these works. Although the appendix provides some explanations, the unique contribution of this paper is not clear. Some relevant works are as follows:\nCORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark\nPaper2Code: Automating Code Generation from Scientific Papers in Machine Learning\nAgent Laboratory: Using LLM Agents as Research Assistants\nThe AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery\n https://github.com/HKUDS/AI-Researcher\n2.  In the experiments, comparisons with other works utilizing multi-agent systems for generating code from papers should be added.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces SciReplicate-Bench, a novel and highly significant benchmark designed to evaluate Large Language Models (LLMs) in reproducing algorithms from NLP research papers into executable code, a task currently challenging for even state-of-the-art models like GPT-4o. The benchmark is praised for its high quality, clarity, and originality, notably introducing "reasoning graph accuracy" as a new metric alongside execution accuracy. This rigorous, well-structured work fills a critical gap in LLM evaluation, providing a realistic "literature-to-code" scenario that will significantly benefit research in automated algorithm replication and LLM capabilities, despite its current focus on single-function implementation.
Decision: Accept