FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
Authors: Qixing Zhou
JiaCheng Zhang
Haiyang Wang
Rui Hao
Jiahe Wang
Minghao Han
Yuxue Yang
Shuzhe Wu
Feiyang Pan
Lue Fan
Dandan Tu
Zhaoxiang Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Agentic Coding
Benchmark
Large Language Models
Primary area: datasets and benchmarks

Abstract: Agents powered by large language models (LLMs) are increasingly adopted in the software industry, contributing code as collaborators or even autonomous developers. As their presence grows, it becomes important to assess the current boundaries of their coding abilities. Existing agentic coding benchmarks, however, cover a limited task scope, e.g., bug fixing within a single pull request (PR), and often rely on non-executable evaluations or lack an automated approach for continually updating the evaluation coverage.  To address such issues, we propose FeatureBench, a benchmark designed to evaluate agentic coding performance in end-to-end, feature-oriented software development. FeatureBench incorporates an execution-based evaluation protocol and a scalable test-driven method that automatically derives tasks from code repositories with minimal human effort. By tracing from unit tests along a dependency graph, our approach can identify feature-level coding tasks spanning multiple commits and PRs scattered across the development timeline, while ensuring the proper functioning of other features after the separation.  Using this framework, we curated 200 challenging evaluation tasks and 3825 executable environments from 24 open-source repositories in the first version of our benchmark. Empirical evaluation reveals that the state-of-the-art agentic model, such as Claude 4.5 Opus, which achieves a 74.4% resolved rate on SWE-bench, succeeds on only 11.0% of tasks, opening new opportunities for advancing agentic coding. Moreover, benefiting from our automated task collection toolkit, FeatureBench can be easily scaled and updated over time to mitigate data leakage. The inherent verifiability of constructed environments also makes our method potentially valuable for agent training.

Published as a conference paper at ICLR 2026
FEATUREBENCH: BENCHMARKING AGENTIC CODING FOR COMPLEX FEATURE DEVELOPMENT
Qixing Zhou Jiacheng Zhang 22, Haiyang Wang"; Rui Hao', Jiahe Wang', Minghao Han" Yuxue Yang', Shuzhe Wu², Feiyang Pan Lue Fan't, Dandan Tu?, Zhaoxiang Zhang' Institute of Automation, Chinese Academy of Sciences Huawei Technologies Co., Ltd haiyang. wwwg@nuawei.ccom
lue. fww.ia.nn.cc.... on
Code: github. com http://beerrcdeeeeeeetreeeeeeee
Dataset: FeatureBench
Project Page: Beyond bug fixing and ship real features.
ABSTRACT
Agents powered by large language models (LLMs) are increasingly adopted in the software industry, contributing code as collaborators or even autonomous de- velopers. As their presence grows, becomes important to assess the current boundaries of their coding abilities. Existing agentic coding benchmarks, how- ever, cover limited task scope, egg. bug fixing within a single pull request (PR), and often rely non-executable evaluations or lack automated approach for continually updating the evaluation coverage. To address such issues, we propose FeatureBench, benchmark designed to evaluate agentic coding performance in end-to-end, feature-oriented software development. FeatureBench incorporates execution-based evaluation protocol and a scalable test-driven method that automatically derives tasks from code repositories with minimal human effort. By tracing from unit tests along dependency graph, our approach can identify feature-level coding tasks spanning multiple commits and PRs scattered across the development timeline, while ensuring the proper functioning of other features after the separation. Using this framework, we curated 200 challenging evaluation tasks and 3825 executable environments from 24 open-source repositories in the first version of our benchmark. Empirical evaluation reveals that the state-of-the oh-the- art agentic model, such as Claude 4.5 Opus, which achieves a 74.4% resolved rate opportunities for
SWE-bench, succeeds only 11.0% of tasks, opening advancing agentic coding. Moreover, benefiting from our automated task collec- tion toolkit, FeatureBench can be easily scaled and updated over time to mitigate data leakage. The inherent verifiability of constructed environments also makes method potentially valuable for agent training.
INTRODUCTION
Software development is rapidly evolving with the advent of large language models (LLMs) (Sap- kota et al., 2025), marking a shift toward end-to-end agentic coding systems (Wang al., 2025a). Recent advances, such as Claude Code (Anthropic, 2025b) and Qwen Code (Qwen, 2025) exem- plify this evolution by introducing requirement-driven agents that autonomously plan, execute, and interact with external tools (e.g., compilers) to iteratively tackle complex software development tasks (Gong et al., 2025), thereby relegating human intervention to a supervisory role.
Recently, various benchmarks have been introduced to assess this paradigm shift, including SWE- bench (Jimenez et al., 2024), PaperBench (Starace et al., 2025), and GirTaskBench (Ni et al., 2025). While these benchmarks have made significant contributions to task-oriented agentic coding, they are limited either by the narrow focus on bug-level scenarios or by reliance on handcrafted gen- eration pipelines. As agentic coding expands toward more complex settings, such as feature-level
"Equal contribution. Corresponding Author.

---

Published as a conference paper at ICLR 2026
Performance Comparison
Codoc OPI Oodox
12.5
Task Description
LLM Agents
Dteaepp a CPT model lalowis the provided and assure b ciuckty cuasdi..
Chure Cash Clude Opes 4
11.4
Interface of the features to be tested
Generate Callable Solution
Opecliatts e Olade Opuc
16.
frome transformers Amport GP72Wodee
+3000-13 |
Closia GP PPPoll (ros Modulg
Opecltants Desgienind)
5,5
Aggn grack lnpel lengh))
Unit Tests
Carzzi Gartin2-tte Frnvee
5.0
Pre
Pool
Tosbs
Opertlands Chooes
Tex: twrt
+ Codebase (optional)
Openttands Gnant Coder 4000 4308 tratruct
3.5
a) Formulation
b) Resolved of current LLMs
Figure 1: a) The agent must implement directly callable feature based on the task description and interface definitions, either by developing from scratch or extending an existing repository. b) Our benchmark shows that even Claude Opus 4.5 achieves only a 1.00% solution rate.
development, these constraints hinder their ability to fully capture the capabilities of frontier code agents. Therefore, there is a need to build a challenging benchmark that broadens evaluation scope to feature-level scenarios, supported by automated collection toolkits to facilitate its future usage.
Constructing such benchmark poses nontrivial challenges. Effective and ieccution-based evalu- ation of feature-level agentic coding generally depends on clearly defined functional interfaces to resolve ambiguities between the implementation and test criteria. However, these specifications often absent in previous benchmarks. Furthermore, creating an automated data collection toolkit to support the scaling benchmarks introduces additional complexities. Conventional pull request (PR)-based methods (Jimenez et al., 2024; Pan et al., 2025; Jain et al., 2025b) are incffective in capturing complete feature patches, as these often span multiple PRs scattered across the timeline, making them difficult to associate. Moreover, many PRs lack tagging, hindering the reliable iden- tification of feature contributions. Notably, PR-driven methods are inherently tied to the historical trajectory of commit submissions, limiting the tasks to fixed development combinations.
Motivated by these shortcomings, we introduce FeatureBench, a challenging benchmark that targets feature-oriented agentic coding scenarios. It integrates an execution based evaluation pipeline and a test-driven toolkit for automatically collecting instances from Python repositories. As shown in Table 1, our bench provides the following characteristics:
1. Feature-oriented real-world software development. Unlike SWE-bench, which is dominated by bug-fixing issues with only about 18-22% of its instances corresponding to feature requests, our benchmark is explicitly designed to target systematic feature- agentic coding. As shown in Figure 1, given human- like clear requirements (e.g., interface signatures and high-level func- tional descriptions), our task entails the implementation of new capabilities either within an ex- isting codebase or as standalone modules. For example, adapting the Transformers library (Wolf et al,, 2020) for compatibility with Qwen3 (Yang et al., 2025a) or engineering Attentino (Dao ct al., 2022) from scratch.
2. Reliable execution- based evaluation. Highly ambiguous requirements without explicit function signatures often introduce multiple valid implementations that are incompatible with the interface expected by unit tests. This misalignment complicates execution-based evaluation and typically necessitates additional manual inspection or LLM-based judgement (Starace et al., 2025; Seo al., 2025). To mitigate this issue, we adopt test-driven formulation strategy when constructing requirements. Each prompt explicitly specifies the clear interface definitions, import paths, and the descriptions of expected behaviors, and enforees that the solution must be directly callable, as illustrated in Figure 1. This method guarantees that a correct implementation will pass all associated tests, thereby enabling automated execution based evaluation.
3. Scalable instance collection toolkit. To support the extensible creation of feature-oriented, real-
istic evaluation environments with fail-to-pass (F2P) and pass-to-pass (P2P) tests, as introduced in SWE-bench, we develop an automated generation pipeline driven by unit tests. The pipeline be- gins by selecting and executing F2P and P2P tests, followed by the construction of a dependency graph through dynamic tracing. Based on the traced dependencies, the system automatically tracts the implementation of the targeted features while ensuring the integrity other features. The final problem statements then synthesized. This approach enables us generate naturally
2

---

Published as a conference paper at ICLR 2026
Featurc-oricnted Agentic Coding
Exccution-basee Evaluation
Scalable Instance Collection
Continually Updatable
Instance Number
Benchmark
BigCodeBench (Zhuo et al,, 2025)
1140
LiveCodeBench (Jain et al., 2025a)
454
FullStackBench (Cheng
2024)
3374
SWE-bench (Jimenez al., 2024)
500
PaperBench (Starace et al., 2025)
20
Paper2Coder (Seo el ul., 2025)
90
MLEBench (Chan et al., 2025)
72
DevEval al., 2025)
20
GiCTaskBench (Ni ul., 2025)
54
FeatureBench (ours)
200
Table 1: A comparison of FeatureBench with current coding benchmarks reveals that our bench emphasizes feature-level realistic software development. It leverages an execution- evaluation pipeline and integrates a test-driven toolkit for the automatic generation of task instances.
verifiable environments from any Python repository in a scalable and flexible manner, free from the constraints the availability and predefined trajectory of human-written PRs or commits.
4. Continually updatable. Building on our collection toolkit, FeatureBench supports continual supply of new task instances, enabling evaluation on tasks created after their training date, thus mitigating the risk of contamination. Using this pipeline, have curated benchmark with 200 evaluation instances and 3825 verifiable environments, created from May 2022 to September 2025, sourced from 24 real-world GitHub repositories in the first version of our benchmark.
We evaluate multiple sate-of-the-art LMs on FeatureBench and find that they fail to solve all except the simplest tasks. Using the Codex agent framewor, GPT-5.1-Codes (medium reasoning) success- fully completes 12.5% of the task cases. Furthermore, we carried out comprehensive experiments, offering insights into potential improvement directions on our benchmark.
In nutshell, our contributions are three-fold: 1) We introduce FeatureBench, benchmark for agen- tic coding that evaluates L.LMs on solving feature-level, real-world complex tasks through an auto- mated, execution-based evaluation pipeline. 2) We release scalable, test-driven toolkit for instance collection that integrates seamlessly with our benchmark and automatically generates verifiable en- vironments from Python repositories. Using this toolkit, construct benchmark comprising 200 evaluation tasks and 3825 executable environments from 24 open source GitHub repositories. 3) We benchmark state-of-the-arr LLMs, including both open- and closed-source variants, and perform in-depth analysis to identify and highlight remaining challenges.
RELATED WORK
Agentic Coding Benchmarks. The most widely adopted benchmark for agentic coding is SWE- bench (Jimenez et al., 2024), whose verified subset has emerged as standard for assessing LLMs. Although originally highly challenging, its success rate has increased from below 10% to over 70% within a year, reflecting rapid advances in L.-M-based agents (Anthropic, 20256; Yang al., 2025a). Despite its importance, SWE--hench has notable drawbacks. It mainly focuses on bug fixing, with comparatively limited coverage of feature development tasks, which often span multiple PRs. Other benchmarks address narrower domains or predefined workflows. PaperBench (Starace et al., 2025) and MLE-Bench (Chan al., 2025) focus on machine learning problems but rely on expert curation or high-quality cases from Kaggle. GirTaskBench (Ni et al., 2025) broadens task coverage but offers only 54 expert-designed tasks, while DevEval (Li et al., 2025) spans the development lifecycle but enforces fixed workflows with 22 handcrafted tasks. To tackle the above problems, we propose a challenging benchmark specifically designed for feature-oriented agentic coding scenarios. This benchmark integrates execution-based evaluation pipeline and an automated toolkit that collects instances from Python repositories in a scalable manner.
Scalable Collection Pipeline. A verifiable environment is crucial for achieving better agentic cod- ing. SWE-Gym (Pan et al., 2025) follows the pull-request based approach SWE- whereas R2E-Gym (Jain et al., 2025b) derives tasks from commits by synthesizing tests and back-translating code changes into problem statements with LLMs. These approaches mitigate scalability concerns but provide limited guarantees of evaluation quality. SWE-Smith (Yang al., 2025b) synthesizes
3

---

Published as conference paper at ICLR 2026
Real Repositories
Setup Environment
Select Tests
Source Code
models!
readme.md
Developers
Execute and Check Unit Tests
List packages and installation commands
testaa
setup.37
X
Select
Unit Tests
Fall-to-Pass (F2P)
(P2P)
tests/test, bert.p
Docker Creation
testahosst
tests/test, laav
testa/test, dinov2.py
Automatically repository
laststtsttggpt2.pp
tests/test gy
tests/teut_ lava
Code Patch Extraction
Post Verification
Synthetic Tasks
P2P & F2P
Graph Traversal
Pre-solied Codebase
Instance
F2P
Docker Image
X P2P
Problem Text
Codebase
Applying the code patch
Dependency Graph
Node Classification
Func unique to P2P
Func of codebase
F2P
<
Gold Patch
Func unique to F2P
Func of code patch
P2P
Func of both P2P F2P
Dependency
Figure 2: Given GitHub repository, our automated toolkit initializes the development environment via Docker. For each benchmark instance, it validates and selects feilctsfail-to fail-to-pass and pass-to-ped tests. Then, the system performs dynamic tracing to capture runtime behavior and construct an object dependency graph. Leveraging this graph, the toolkit synthesizes code patches, derives correspond- ing poe-solved codebases, and formulates final problem statements. This pipeline has yielded 200 benchmark tasks and 3825 executable environments from 24 GitHub repositories.
tasks from repositories using heuristics such as LLM generation, procedural modifications, or pull- request inversion. SWE-Flow (Zhang et al., 2025) synthesizes data based fail-to-pass tests but neglects pass-to-pass tests and does not ensure the proper functioning of other features in undevel- oped codebases, resulting discrepancies compared actual development settings. Although suc- cessful, none of them can generate tasks that are both feature-oriented and reflective of real-world development scenarios. Our benchmark addresses these gaps by providing test-driven, scalable tool for generating feature-level agentic coding tasks, complemented by rigorous post-verification that ensures the integrity of undeveloped codebases, consistent with real-world scenarios.
3 FEATUREBENCH
FeatureBench establishes a benchmark for evaluating the capabilities of code agents in end-to-ent software development tasks. The benchmark requires agents to interpret high-level goals and their associated code interfaces, autonomously manage execution environments, and synthesize correct and callable implementations either within existing codebases structed with minimal human intervention, the benchmark leverages an automated pipeline that de- rives feature-orienter coding tasks from open-source repositories, thereby extending the scope of agentic coding beyond bug fixing to encompass feature development.
standalone solutions. Con-
3.1 FEATURE-ORIENTER AGENTIC CODING
Task Formulation. As illustrated in Figure 1, each instance in FeatureBench provides the agent with a comprehensive problem statement. This includes a high-level task description, a specified functional interface, blacklist of prohibited URLs to mitigate potential cheating of agents, and dockerfile defining the execution environment. The agent is then tasked with generating a solution that addresses the problem, whether by editing existing code implementing from scratch. Notably,

---

Published as a conference paper at ICLR 2026
to facilitate automated and unambiguous evaluation, the agent's output is required to be directly callable module. Its invocation path, function signature, including input and output variables as well as comprehensive annotations, all explicitly provided within the problem statements.
Difficulty. In realistic settings, software development may proceed either by extending an exist- ing codebase by implementing feature entirely from seratch. FeatureBench reflects these two scenarios with two difficulty levels. Level (I,) consists of incremental development within an existing repository based on task requirements, while Level 2 (L2) requires constructing the same functionality from scratch.
Metric Design. Our evaluation protocol follows the established setup of SWE-bench (Jimenez et al., 2024), where each agent-generated solution is validated by executing its associated fail-to- pass (F2P) and pass-to-pass (P2P) tests. A task is considered resolved when the proposed solution successfully passes all these tests. We report three primary metries: (1) Resolved Rate, the propor- tion of tasks fully solved, like SWE-bench; (2) Passed Rate, the average fraction of fail-to-pass tests passed per task, serving soft indicator of partial correctness; (3) Token I0, the average number of input and output tokens consumed, reflecting the computational efficiency of the agent.
3.2 BENCHMARK COLLECTION
Execution Environment Configuration. To rapidly set up an environment for a given repository, we manually specify installation commands (taking approximately three minutes), rather than rely- ing on the more error- prone and uncontrollable approach of having the agent search for installation methods itself. Automated scripts are then used to configure the environment and package the repos- itory into a Docker image. The benchmark includes 24 widely downloaded PyPI packages across various domains, such visualization libraries and LLM infrastructure. Notably, human interven- tion is required only for this step of the pipeline, and the total human labor required to complete this for all 24 repositories amounts to less than one hour.
Constructing Fail-to-pass and Pass-to-pass Tests, We construct benchmark instances by identify- ing candidate test files in the repository using pytest's collection function, followed by validation through execution. For each instance, n validated test files are designated as fail-to-pass (F2P) tests, as introduced in SWE-bench. These tests fail in the undeveloped repository but succeed once the agent correctly implements the target functionality. To additionally assess incremental development capability, we include m randomly sampled validated files as pass-to-pass (P2P) tests, which are ex- pected to pass both before and after the agent's 's solution. Since a single test file typically corresponds to one functional implementation, n is usually set to one in our setting.
Test-Driven Code Patch Extraction. Obtaining the pre-solved codebase together with the cor- responding code patch requires isolating the functionality linked the F2P tests. However, the inherent ambiguity of functional boundaries in real-world codebases poses significant challenge. Naively extracting relevant code fragments risks inadvertently disrupting other well-established fea- tures. As depicted in Figure 2, our approach mitigates this issue by incorporating P2P tests to accurately identify code modules required by other functions or those serving as foundational com- ponents of the repository. The detailed implementation is as follows:
Construct the object dependency graph. We initiate the process by executing the available F2P and P2P test cases for given benchmark instance. During runtime, we employ Python's 's built- in tracing facility capture function call events and their dependencies. From this trace, we construct an object dependency graph in which each node represents a function and is enriched with metadata, including a unique identifier, source location, a list of dependent functions, and a binary flag indicating if the function was triggered during P2P tests.
Graph traversal and node classification. To distinguish functional components, a large language model analyzes the F2P test files and separates the imported functions related to the target feature from those that serve supporting roles in the testing process. The nodes identified as central to the undeveloped fcature serve as the initial entry points for a breadth-fii first traversal of the graph. During this traversal, nodes are systematically classified: those encountered in P2P executions are designated as remained, while nodes not observed in P2P runs are classified as extracted.
Extracting the code. The traversal process yields a subset of graph nodes identified as relevant the intended functionality. In the final stage, the corresponding segments of source code
5

---

Published as a conference paper at ICLR 2026
Lite
Fall
Scaffold
Model
% Passed
Resolved
Token BO
Passed
% Resolved
Token I/O
Openilands
Qwen3 480B--388833888 Insiruct
38.31
6.7
2.6MI 16k
24.55
3.5
2.0M / 14k
OpenHands
DeepSeek: V3.2
3.IM
Opentlands
Gemini-3 3-Pro-Previeww
45.14
10.0
6.0M 41k
30.08
4.5
6.2M 40k
Gomini-CLI
Gemini-2-Pre-Prroiccs
43.38
10.0
2.6M/ 13k
32.43
5.0
2.5M 12K
Claude Code
Clande Opus 4.5
59.12
20.0
9.0M/ / 35k
43.29
7.5M 34k
Codex
GPT Codex!
60.22
20.0
6.6M / 39%
41.66
12.5
6.3M 39k
Opentlonds
Claude Opus
8.8M/29K
Table 2: The performance of various frontier large models combined with advanced agentic frame- works on the Lite and Full evaluation sets of our benchmark. Models marked with t use low reason- ing, and use medium reasoning.
extracted from the original codehase. This operation produces modified codebase devoid the target functionality and complementary code snippet that realizes the previously absent feature.
Post Verification. To ensure the successful extraction of the target functionality from the code- base without affecting other components, we implement a rigorous verification process. The first step involves validating the pre-modified codebase by ensuring that it passes all P2P tests, thereby confirming its integrity. Simultaneously, must fail all F2P tests, demonstrating that the target functionality has been effectively removed. Following this, we assess the accessibility of all utility functions required for the F2P tests in the modified codebase. This step ensures that the changes made are confined the target functionality and do not inadvertently impact other core depen- dencies. Finally, reapplying the patch to the undeveloped codebase should allow all tests to pass, confirming the patch's correctness.
Problem Statement Generation. By leveraging the extracted code snippet, the pre-modified code- base, and the corresponding unit tests, we automatically generate the problem statement for each instance. This procedure includes the derivation of the feature signatures, which encompass the types of input and output variables, alongside the functional description as inferred from the code docstrings. In the absence of such docstrings, employ large language model to generate them directly from the code snippet. Further details can be found in the appendix.
To this end, our pipeline automatically generates the core components of each instance: natural language problem statement, an undeveloped codebase, verified code patch, and suite of unit tests corresponding to required features. The sole manual intervention required is the specification of the repository's 's installation procedure, a process that takes approximately three minutes per repository.
3.3
BENCHMARK CONFIGURATION
Full Set. Leveraging our pipelines, configured the number of P2P test files five and curated 3825 coding environments derived from 24 Python repositories. To ensure the benchmark mean- ingfully challenges best-performing agents, we restricted inclusion tasks exceeding 100 lines of pending implementation, encompassing at least 10 F2P test points, with test files initially committed after May 2022. This filtering yielded 200 high-qualit instances comprising the full set.
Lite Set. Evaluating LMs our bench can be time- conemmnng and, depending on the model, require costly amount compute or API credits, as illustrated in Table 2, where the average number of input tokens approaches the million-tokee mark. To facilitate wider adoption of FeatureBench. we randomly selected instances from the full set to create streamlined lite set.
4
EXPERIMENTS
4.1
PERFORMANCE ON FEATUREBENCH
4.1.1 BASELINE
To establish strong baselines, we adopt the OpenHands (Wang et al.) framework for software devel- opment agents, which tops the SWE-bench. In the experiments, the maximum of steps per task is

---

Published as a conference paper at ICLR 2026
SWE-bench
Ours
Problem Texts
Length (Words)
195.1
4,08.0
# Lines
32.8
790.2
Gold Solution
# Files
1.7
15.7
Full Set (200)
Functions
29.2
uphin (10)
Fail pass (test points)
62.7
Tests
# Total (test points)
120.8
302.0
sytorch- Sarrb-igghtngggtee
Table 3: Average numbers characterizing different tributes of a SWE-bench task instance. well as our FeatureBench (L1 set).
Figure 3: Distribution of our bench-
mark across 24 GitHub repositories.
SWE-Bench Verified
FeatureBench subset
Model
% Resolved
% Passed
% Resolved OpenHands
Token I/O
mina-StE OpenHands
DeepSeek V3.2
60.00
22.98
0.0
3.8M 25k
Qwen3- Code--orrr-------- Gemini-3-Pra-Prent Claude Opus 4.5
55.40
69.60
23. 46
0.0
2.3M 16k
74.20
30.05
0.0
6.7M 45k
74.40
41.08
5.2
9.7M 34k
Table 4: Compare the performance of the frontier agents on SWE-bench and our FeatureBench, using a subset of our benchmark with repositories shared with SWE-bench for fair comparison.
as 500 by default. Internet access is freely available, while no specifie browser-use tools pro- vided. To ensure the integrity of our evaluation, robust anti-cheating mechanisms are incorporated prevent agents from assessing the ground-truie repositories (see the appendix for details).
We evaluate seven seaffold+model configurations with frontier LLMs, including DeepSeek- V3.2 (DeepSeek, 2025), -anee--Cooo---0--3-------
(Qwen Team, 2025), Gemini-3-Pro- Preview (low reasoning) (Google, 2025a), Claude Opus 4.5 (Anthropic, 2025a), and GPPT-5.1-Coder (medium reasoning) (OpenAI, 2025b) under representative agentic scaffolds (OpenHands (Wang et al., 2025b), Gemini- CLI (Google, 2025b), Claude Code (Anthropic, 2025b), and Codex (Ope- nAI, 2025a)). The results are presented Table 2. As can seen, even the most capable settings, i.e. Claude Code (routing) Claude Opus 4.5 and Codex + 1--T--.--Codxx (medium reasoning), resolve only 11.0% and 12.5% of the tasks on the Full set, respectively. This underscores the highly challenging nature of the feature-oriented development tasks in our FeatureBench, which require agents to write substantial amounts of code and pass comprehensive test suites.
For more nuanced evaluation, further analyze passed rates and token consumption by different LLMs. The passed rates, while remaining at a low level of below 50%, are much higher than the resolved rates. This discrepancy indicates that current agents often produce seemingly plausible lutions with a large underlying gap from truly solving the problem, which accounts for the common need of tedious debugging for Al-generated code. Regarding token consumption, all LLMS consume over one million input tokens. Given the low resolved rates, this reflects the extremely low efficiency of existing agents lackling real-world development tasks, which thus an important topic for fu- ture research. In addition, a high consistency is observed in the rankings of different LLMs across the Lite and Full sets terms of both pass and resolved rates, demonstrating the representativeness of the Lite set.
4.1.2 COMPARISON WITH SWE-BENCH
Compared with the SWE-bench (Jimenez et al., 2024), our FeatureBench introduces more chal- lenging suite of development tasks. It encompasses 16 additional popular repositories apart from repositories originally covered by the SWE-bench, the full list which is shown in Figure 3. Table 3 presents comparative statistics illustrating the task difficulties across the two benchmarks. Specifically, the tasks our benchmark exhibit a substantial increase of complexity in terms of the length of problem texts, number of lines, files and functions to be edited well the number of tests to pass. These enhancements necessiate agents with strong long-con long-context understanding and management capabilities alongside comprehensive problem analysis handle diverse test cases.

---

Published as a conference paper at ICLR 2026
Siops
Passed
Models
% Resolved
% Passed
22.9
Original
Gemini-3-Pro-Preeeee GPT-5,1-Code
10.0
42.4
500
100
10.0
6.7
45.1
43.8
16.7
53.9
50
3.3
28.9
Verified
GPT-5,1 Codex"
Gemini-3-Pro-Pren
20.0
10.0
60.2
Qwens-Coder-t00B Coder-Ro0---SSS A35B-Innstte
100
3.3
30.4
38.3
Table 6:
An ablation study on the max ex- ecution steps of OpenHands with Gemini-3- Pro-Preview and Qwen3-Coder-480--A5 Instruct in Lite Set. Models marked with use low reasoning.
An ablation study to evaluate the ne- cessity of manual verification for the examples generated by our system. Models marked with use low reasoning, and f use medium reasoning.
Table 5:
Without Interface
Visible Unit Tests
Model
% Resolved
% Passed
Token I/O
% Resolved
% Passed
# Token I/O
Semmin---Prro-revvie
3.3 (-6.7)
25.3 (-18.1)
7.0M/ 10K
(+50.0)
80.6 (+37.2)
6.9M/
GPT-5.1-Codex+
16.7 (-3.3)
42.0 (-18.2)
7.6M/ 38K
63.3 (+43.3)
80.9 (+20.7)
8.2M / 46K
Table 7: Performance comparison of lite set with visible unit tests and without interface. Models marked with i use low reasoning, and use medium reasoning.
For more grounded analysis, further compare the performance of agents the SWE-bende and our FeatureBench. To draw a more aligned comparison, we construct a subset of our bench- mark including only repositories shared with SWE-bench. The results in Table reveals stark performance gap between the two benchmarks in terms of resolved rate. Specifically, the most ca- pable Claude Opus 4.5 only resolves 5.2% of the tasks in our FeatureBench subset in contrast the 74.40% on the SWE-bench. This indicates the highly challenging nature of our benchmark, which provides considerable room for future improvement and establishes rigorous testbed measure the upper bound of existing agents.
4.1.3 FAILURE CASES ANALYSIS
We conduct failure case analysis based the results in our full set from the Claude Opus 4.5 model, leading to the following findings.
Limitations in Code Reasoning, As shown in Figure 4, the dominance of NameError suggests that current LLMs still struggle with cross-fil dependency resolution. When a feature spans multiple files, models often focus local edits without consistently re-establishing all necessary references, leading to unresolved symbols and frequent name-related failures. This highlights a key limitation in maintaining coherent program context beyond single file.
The "Idle Habits" of LLMs. We also find that current L.MSs exhibit a tendency toward "laziness". For example, they often resort guessing (even hallucinating) the interface attributes of compo- nents defined across files, rather than performing the actual file reading required to retrieve precise prototypes and members. This behavior leads considerable number of both TypeError and AttributeErros occurrences.
Appropriate Information in FeatureBench. Among the remaining failures, AssertionErroz becomes the most frequent category. This suggests that substantial portion of LLM- L-M--neerated so- lutions can run the assertion checkpoints without earlier runtime crashes. This result underscores that FeatureBench can effectively provide the LLMs with appropriate information to generate com- plete programs.
4.2
ABLATION STUDY
4.2.1
ANALYZING THE QUALITY AND NECESSITY OF OUR BENCHMARK DESIGN.
Without Interface. We performed an ablation study to assess the role of explicit interface speci- fication in agent performance. For controlled comparison, we employed the lite set, systematically removing function signatures and call path annotations from the prompts. As shown in Table 7, this
8

---

Published as a conference paper at ICLR 2026
Dirhouty
Scatiol
Modils
Resolved
Piood
Failure Modes
Opcaffardk
3.
22.4
Ogenblands
DeepSeck-V32
4.8
21X
Ogenitands
Qpes
2000
Ogreltznds
Phol
Count
1500
Clande Code
Ojes 455
436
1000
Ognallards
toaamr Inamer
29
162
Oprallands
herpSeek-V33
126
500
Opzeilands
Osta
422
Opzeitands
Poo
35.6
f
Genini
Piol
Namefiror
AtributeEroor
TypeEroor
AssertonErmat
EaskEroo
ValueError
Clinde Code
Cladde Oyes 4.5
41.9
Table 8: Performance comparison of tasks with different difficulty levels in FeatureBench. Mod- els marked with use low reasoning, and & use medium reasoning.
Figure 4: Failure modes of the Claude Opus 4.5. Models marked with use low reasoning, and use medium reasoning.
removal leads to a marked decline in task success rates. The results confirm that clearly defined interfaces are critical for enabling effective reasoning and program synthesis by LLM-based agents.
Sample Quality. Our automated data generation pipeline yields high-quality, evaluation-ready sam- ples with minimal human intervention, supported by a rigorous post-verification process. To assess the fidelity of these samples, we conducted an ablation study which senior engineer with five years of industry experience in AI infrastructure and system architecture independently revised the prompts in the lite set. The verification details are provided in Appendix Figures 20 and 21. As shown in Table 5, model performance on the manually revised subset is highly consistent with the original dataset. These results affirm the reliability and robustness our automated data pipeline.
Lines of Code and Task Initial Commit Date. Figure explores the relationship between task pass rates, initial commit timestamps, and the number lines of code required for task com- pletion. We observe clear negative correla- tion between pass rate and code length, indicat- ing that tasks involving more lines of code are inherently more challenging for current large models. In contrast, task performance shows minimal dependence on commit time, likely be- cause the task set remains largely unexplored by existing models. To further understand why commit time has little influence, we ana- lyze how feature complexity evolves over time. Specifically, the lower panel of Figure 5 plots the normalized trends of code length and pass rate across commit periods. The two normal- ized curves exhibit highly similar fluctuations, reinforcing that variation in task performance driven far more by feature complexity than by commit time. However, as agentic systems increasingly participate in feature development workflows, the risk of data leakage may become more pronounced and should be monitored in future benchmark design.
Lines
0.6
8.5
3 0.4
0.3
0.2
0.1
200
600
1000
1400
1800
2200
2600
3400
3800
4200
Time of Creation
0.8
Pass Rate
0.4
0.0
<2023
2u23
2021
2023
203
2024
2024
2014
2014
2025
2025
2025
2025
Code Complexity and Pass Rate Time
0.8
0.6
0.2
0.0
-2023
2021
2023
02
2023
03
2023
04
2024
02
2014
2024
2024
Q4
2025
or
2025
2015
2005
Figure 5: The pass rate of Claude Opus 4.5 in our
benchmark varies with the number of code lines and task creation time.
Comparison between T,1 and 7.2 Subset. Comparison between I,, and L.2 Subsets. Our benchmark defines two evaluation settings: L., where new functionalities are incrementally added to an existing codebase, and Lg. where functionalities are implemented entirely from scratch. All conditions are held constant across both settings, except for the presence or absence initial code context.
This distinction leads to notably different levels of reasoning complexity. In the L1 setting, the agent still has access to most of the original codebase except for the functions and classes removed along the traced execution path. This partial repository shows how the feature fits into the surrounding code and gives the agent contextual clues about expected behavior. As a result, L, tasks are more guided, since only the missing implementations need to be completed. In contrast, Toz tasks remove

---

Published as a conference paper at ICLR 2026
Metric
Precision
Recall
Score
Accuracy
Value
81.03%
89.24%
84.94%
91.74%
Table Performance of the LLM classifier for identifying top-level tested objects.
all surrounding code. The agent does not see any part of the original repository and must rely only on the interface to implement the required functionality. Without the structure provided by the existing codebase, the agent has reconstruct the full logic and organization of the feature entirely from scratch, which makes L2 substantially more difficult. As shown in Table 8, the from-scratch (L2) setting is more challenging, with lower resolved rates: performance on L2 varies little across settings, suggesting that removing the codebase structure creates common bottleneck that hampers coherent multi-step reasoning and end-to-end implementation.
Accuracy of LLM--based Top Import Classification.
To validate the reliability of our LLM-based classifier for identifying top-level tested objects in test file, we conducted a quantitative evaluation against expert annotations. Domain experts evaluated all 605 import statements in Lite Set and identified 158 of them as top-level tested objects. The details of the procedure the LLM classifier. These results indicate that LLMs can accurately identify tested objects at scale, supporting the use of LLM-bassed classification in data construction pipeline.
provided in Appendix Figure 19. Table reports the performance of
4.2.2 ANALYZING THE KEY FACTORS IN BUILDING END-TO-END CODEAGENTS
Visible Unit Tests. We conducted an ablation study to assess the impact of providing accurate unit tests on agent performance complex coding tasks. In this setting, the agent was given access ground-truth unit tests alongside the Lite As shown in Table 7, both task success rates and pass rates increased significantly. These findings underscore the importance of high-quality unit test generation as a key factor enabling robust agentic coding.
Longer Execution Steps. Table reports the effect of increasing the maximum number of exe- cution steps on model performance. Increasing the maximum step size from 50 100 results in notable performance gains for both Seminn-3-Pro-PPrviiee and However, beyond this threshold, the improvements become marginal.
A335--Inttrutt.
CONCLUSION
In this work, we introduce FeatureBench. a novel benchmark designed to evaluate the capabilities of LLM-ppoeree agents in realistic, feature-oriented software development scenarios. Leveraging test-driven task extraction and execution-basee evaluation, FeatureBench overcomes key limitations of existing benchmarks by enabling greater task diversity, scalability, and verifiability. Empirical results reveal that current agentic systems face persistent challenges in planning, reasoning, and managing long-horizon tasks. With its extensible and automated design, FeatureBench offers not only a rigorous evaluation framework but also foundation for the development of next-generation agentic coding models.
ACKNOWLEDGEMENTS
This work was supported in part by the Beijing Natural Science Foundation (No. L257015 and No. L257004) and in part by the National Natural Science Foundation of China (No. 62320106010).
REFERENCES
Anthropic. Claude opus 4.5, 2025a. claude- - glaude-opus-4-r 4-5.
URL http:://www.attarriicccooo
Anthropic. Claude code, 2025b. wttphh/tss///odeee...ddddeee///e
10

---

Published as a conference paper at ICLR 2026
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. MLE- bench: Evaluating machine learning agents machine learning engineering. In International Conference on Learning Representations, 2025.
Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijie Geng, Aoyan Li, Bo Li, et al. Fullstack bench: Evaluating Ilms as full stack coders. arXiv preprint wrXi::24/2.00535, 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. In Advances in Neural Information Processing Sys- tems, 2022.
DeepSeek. Deepseek-v3.2, 2025. URL https://api-doss http:///d....deeeeee.......o/// news251201.
Yaxin Du, Yuzhu Cai, Yifan Zhou, Cheng Wang, Yu Qian, Xianghe Pang, Qian Liu, Yue Hu, and Si- heng Chen. Swe-dev: Evaluating and training autonomous feature-driven software development. arXiv preprint arXiv:2505. 16975, 2025.
Jingzhi Gong, Vardan Voskanyan, Paul Brookes, Fan Wu, Wei Jie, Jie Xu, Rafail Giavrimis, Mike Basios, Leslie Kanthan, and Zheng Wang. Language models for code optimization: Survey, challenges and future directions. arXiv preprint arXiv:2501.01277, 2025.
Google. gemini-3.
Gemini 3, 2025a.
URL https://ai..ooglee.de/ /ptps//emmnnaaaa//cccs////
Google.
Gemini cli, 2025b. URL
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Repre- sentations, 2025a.
Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng. Koushik Sen, and Ion Stoica. R2e-gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents. Conference on Language Modeling, 2025b.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? In Interna- tional Conference on Learning Representations, 2024.
Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang. Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et al. Prompting large language models to tackle the full software development lifecycle: A case study. In International Conference Computational Linguistics, 2025.
Ziyi Ni, Huacan Wang, Shuo Zhang, Shuo Lu, Ziyang He, Wang You, Zhenheng Tang, Yuntao Du, Bill Sun, Hongzhang Liu, et al. GitTaskBench: A benchmark for code agents solving roal-world tasks through code repository leveraging. arXiv preprint arXiv:2508 18993, 2025.
OpenAl. Codex, 2025a. URL /ttp:ttsss//pppaa...oo///eee///s
OpenAl. Godde. 2025b. URL /ttp://pp////aaatoo.o...aaaa//oa 101d-..eeedd
Jay Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-Gym. In International Conference Machine Learning, 2025.
Qwen. Qwen code: Research- cli tool for qwen-coder models. https: / //wenl/.. hooll//gg/////// qwen3 -ooeenn-cddr//// 2025.
github. ct/bllg/////oooocoooocooooooooct

---

Published as a conference paper at ICLR 2026
Qwen Team.
2025. URL https://huggiiggfaceee.
Ranjan Sapkota, Konstantinos I Rouueliotis, and Manoj Karkee. Vibe coding vs. agentic coding: Fundamentals and practical implications of agentic ai. arXiv preprint arXiv:2505. 19445, 2025,
Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. Paper2code: Automating code generation from scientific papers in machine learning. arXiv preprint arXiv:2504. 17192, 2025.
Giulio Starace, Oliver Jaffe, Dane Sherbur,, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating Al's ability to replicate Al research. In International Conference on Machine Learning, 2025. URL https: oppprevieeeieveeeeeeeee net/ / forum? crrumide ?id=
XF5PuTLPbn.
Huanting Wang, Jingzhi Gong, Huawei Zhang, and Zheng Wang. Ai agentic programming: A survey of techniques, challenges, and opportunities. arXiv preprint arXiv:2508.11222 2025a.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng. Heng Ji, and Graham Neubig. OpenHands: An Open Platform for AI Soft- ware Developers Generalist Agents. URL https: /tte///aaxae /awxivivvvvv./////////a//////av/ org/ abs 2407 1641.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh. Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for Al soft- ware developers as generalist agents. The Thirteenth International Conference on Learning Representations, 2025b. URL https:://// openreview.eeee..eoeeeeee.eoe.ee net / forum? id=OJd3ayDDoF.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- Art Natural Language Processing. Association for Computational Linguistics, 2020. URL https:/ /wwww//ww.////////////// aalawleeee web. wargg/anthlll. wwwpggg/////// 20000 emn demos. 6.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025a.
John Yang. Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling data for software engineering agents. arXiv preprint arXiv:2504.21798, 2025b.
Lei Zhang, Jiaxi Yang, Min Yang. Jian Yang, Mouxiang Chen, Jiajun Zhang, Zeyu Cui, Binyuan Hui, and Junyang Lin. Synthesizing software engineering data test-driven manner. In International Conference on Machine Learning, 2025.
Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthia Gallé, and Alexan- der M Rush. Commito: Library generation from scratch. arXiv preprint arXiv:2412.01766 2024.
Terry Yue Zhuo Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari. Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li,, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In International Conference on Learning Representations, 2025.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A DETAILED BENCHMARK COLLECTION
This section complements the details of benchmark construction (Sec. 3.2), which contains detailed recipes of the data collection, patch extraction, and prompt design, along with fuller characteriza- tion of the task instances.
A.1 DATA COLLECTION PIPELINE
Environment Setup.
For each selected repository, we manually prepare an environment configuration file (see Figure for example). Empirical observations indicate this procedure can be accomplished within three minutes. Upon completion of environment configuration, our pipeline constructs a Docker image, with all subsequent operations executed within this sandboxed environment. This is the sole stage requiring human intervention. All succeeding stages operate under full automation.
Patch Extraction. The patch extraction process consists of four main steps.
Patch Extraction Step 1: Dependency Gruph Construction. This procedure generates function-level dependency graphs for all test files within the code repository, establishing the foundation for sub- sequent patch extraction operations. We leverage pytest's ' intrinsic test case collection mechanism aggregate all viable test cases the file granularity, where each file contains a potential test case. For each test case, we execute the test within the sandbox environment, selecting test cases that achieve complete success as fail-to-pass (F2P) instances. Concurrent with test execution, we con- struct function-level dependency graphs for each F2P instance utilizing dynamic tracing library.
Patch Extraction Step 2: LLM Classification. For each F2P test file, we employ an LLM to differen- tiate between imported objects serving test targets versus those functioning test dependencies and general utilities. We provide the LLM with the test file's name and content as classification references. Our prompt template for the LLM to classify is illustrated in Figure 9. Objects clas- sified through this methodology are designated as top-level objects, representing directly imported interfaces by the test file.
Patch Extraction Step 3: Pass-to-pass (P2P) Selection. For each F2P instance, we select multiple pass-to-pass cases. These P2P cases executed after coding agents finishing implementations to ensure existing functionalities remain normal. Since the aforementioned top-level objects of F2P cases will be removed from codebases, here the pass-to-pass, cases should not share top-level objects with the F2P cases. For this reason, if we find only few P2P cases have different top-level objects from F2P cases, may indicate erroneous classification of general utilities top-level objects by the LLM. In this circumstance, we will reconsider the top-level objects according to their invocation frequency.
Patch Extraction Step 4: Final Extraction. For each F2P case, we utilize top-level objects entry points and execute BFS according the constructed dependency graph. Node objects belonging to P2P are designated remained, while others are marked as extracted. Nodes marked as extracted are added to the BFS queue for continued traversal. BFS termination occurs upon queue finish or when extracted code lines reach predetermined maximum value, randomly selected between 3000 and 5000 lines per case. Finally, we remove objects marked as extracted from the codebase, yielding complete codebase with F2P functionality eliminated.
Post-verification. For each codebase after code patch extraction, we conduct post-verification to ensure the modified codebase has normal functionality. Specifically, execute F2P within the modified codebase, expecting pass rates below a predetermined parameter. Then we further execute all selected P2P cases, ensuring complete test passage.
13

---

Published as a conference paper at ICLR 2026
Environment Configuration File part 1 of
SPECS SPECS_LITGPA LITGPT
"repository": "Lighttiigg-A/Littttt,
"commit": "22c2a4f", [llone_methodl "https", "base_ur1": None, "base_image" "python318",
"rebuild saapuulddd_ba___mae__e_e False,
"rebuild__instanc__imeee False,
[],
"'rr_installl:: [],
"install": "pip install [[etra]'",
:aip_paccaaes": [],
"docker_see seocker_specs:
"run Brun_args": {
"cuda_visible visible_devices":
devices": "0,1,2,3",
"shm_size": None, "cap_add": [],
},
boustom_docker_are []
},
grest_scanner_cmmm TEST DISCOVEEY_DEEAAEE
"timeout "timeout_scanner" scanner": 300,
"scan iscan_cache", True,
"start_time": None, iii__teet_naat
test_numm:: num": 1,
max "maa__f2pp__ @2p__umm.. -1,
max ceax_p2p_nuut nu_": -1,
Figure 6: Environment Configuration File part of
14

---

Published as a conference paper at ICLR 2026
Environment Configuration File part 2 of
"test_cmd": TEST PYTEST _EEES_, "timeout_run : 1200, "timeout_ "timeout_one": :eteoouutone:::: 10,
"test_cache"" True,
"test cirrt__dnmmiccmmmm cmd": TEST_DYNANNE L____DDI_AAAACEEEEAE_AI TRACE SRACE_DFFAUL, "timeout_dynamie" 1200, "dynamic_cache". True,
1ll__cache", True "batchsize :essiizeeetp::::: 5,
max bexxx_deph__tte_ top": 5,
"min_p2p_files"" 1,
"max _eax_p2p__fai_ hnax_p2pffiles": files": 5,
>>2p_ccche": True,
maxx mmmax_code_____ let___inn_________________ line_lowerr lower bound": 3000,
"max_coder lwwe__upeee_____eeeeee _tp_u_pppeuddd""""" bound": 5000,
"ddtaa_cachee True, "timeout_collect'; "timeout_ collect" 300,
ff2p_pass_raat_tthreee "aatt__trreshod""" 0.3, "f2p_ galn_prompt_for_cas True,
"library_name"s "litgpt", bolack_links http:::://gttuub..m//igggggggn
Figure 7: Environment Configuration File part of
15

---

Published as a conference paper at ICLR 2026
A.2 DATA FORMAT AND PROMPT DESIGN
In this section, we present the essential components included in a qualified example (covering both L1 and L2 tasks), illustrating the test case format of our benchmark, organization our prompts, and the effectiveness of using an LLM to supplement missing docstring entries.
Directory Structure of Generated Instances. Each successfully generated instance includes directory structure containing four main files: problem statement md, patch di ff, test patch... diff and instance json. Specifically, probl em statement. mdd serves as the generated task prompt; patch. diff and test-patch di ff represent the gold patch and test patch, respectively; while instance. json records metadata such as the task ID, source repository, and commit ID.
Prompt Structure and Organization. For each successfully generated instance, we construct mm file as input to the agent. The content of
tailored and detailed proolemmstaaeeme aatollmmstateeenn womeettmmd md consists of two components: Task and Interface Descriptions. All prompts generated automatically without manual labor, following unified prompt template combined with an instance-specifee configuration file via scripting. As shown Figure 12, the Task section provides the agent with overview of the task under the heading "Core Functionality." "Main Features and Requirements" describes the essential code features and requirements. The outlined under "Key Challenges." Finally,
mandatory components that must be implemented the "NOTE" subsection provides specific requirements. In the Interface Descriptions section (as shown in Figure 13), we offer detailed instructions on how construct the code. This includes requirements for file locations and structure, suggestions for implementing interfaces, and specific objectives related to the current task.
Prompt Design: I vs. L2. It is noteworthy that there are subtle differences between the L1 and L2 prompts. In the Task section of the prompt, for L2 examples, we require the agent to independently implement the solution from scratch, without access to the repository's codebase. To enforce this constraint, the agent instructed not to download the repository, and even in the event of its doing so, it is instructed not to install it. This is closely monitored during the testing phase, where have set up mechanisms to check whether the agent engages in any unauthorized actions, such as cheating. Figure 15 and Figure 16 show a specific J, and L2 prompt, respectively, and they both come from the test-layer-norm test in liger-kernel library.
Supplementation for Missing Docstrings. In cases where the source code lacks adequate docu- mentation regarding function interfaces or behavior, we leverage a LLM to infer and complete the missing information. Figure 11 illustrates the prompt for doestring generation. The LLM-generated docstring is exemplified in Figure 18, where the LLM is used to supplement the missing docstring in the functional description.
B DETAILED BENCHMARKING PROCESS
Our benchmarking pipeline is organized into two sequential stages: (1) agent inferring and (2) automated evaluation. The following subsections provide detailed explanation of two phases.
B.1 INFERRING
During the inference phase, after initializing the task environment image, we proceed as follows based on the setting:
For L1 We use patch. diff to remove the target feature from the origin repository within the image and remove the corresponding F2P test file.
For Ly: We completely remove the entire code repository from the image.
Subsequently, the agent is deployed within the environment and provided with problem stat ement........ md as the prompt of the task. Upon completion, we extract the modifications made by the agent
patch. di ff file for later evaluation.
16

---

Published as a conference paper at ICLR 2026
B.2 EVALUATION
During the evaluation phase, we utilize the same base image and reset the repository its pre- inferring state (replicating the setup procedures for LI and L2 described above). And then:
For L1: We apply the agent-generated patch. to the repository and reintroduce the F2P test file.
For L2: We apply the gentt-eeeeattd patch.dif diff to the image and install it as the agent_code library. We then restore original repository and apply the test.patch. dif..iffff..ff
Finally, we employ the pytest framework to perform automated testing and get the pytest report.
B.3 METRICS
The evaluation process produces raw output from the pytest framework, which subsequently processed to extract key statistics, including total, cassed skipped, error, xfail, and xpass.
From these statistics, we derive two primary evaluation metrics: pass_rate and is_solvdd. The pass_rate is defined as the ratio of successfully passed tests to the total number of executed F2P tests. The binary metric is ssovel indicates whether the task is fully solved. It is assigned value of 1 if pytest command exit with (both F2P and P2P), and assigned a value of otherwise.
B.4 CHEATING PREVENTION AND DETECTION
To protect against potential cheating attempts by agents, such as using pip install <package> followed by inspecting the source code of the installed package, implement twofold defense mechanism. First, defensive prompting is incorporated into the task descriptions to discourage such behavior. Second, the evaluation framework conducts automated inspection of the agent's execution logs after task completion to identify suspicious activities.
The log inspection process searches for regular expression patterns that indicate unauthorized tempts to access the source code of installed packages. If any pattern matches, the agent is flagged for potentially dishonest behavior in the evaluation report.
r' "message" *cat /ust//lcall/iib/ptthonn/ command" *cat /usr//ocal//ibb/pttton//// r' "message" *radding file: /ttp:///occ//l////ttaaaaa (d+), \d+r "message" "eeesagge..xeeddng /uss/local/lib/ytttoo ld+). \d+
\d+ Ad+ \d+r
These patterns capture attempts to directly access files within the Python library directory, which is classified form of cheating under the evaluation criteria.
C ANALYSIS OF FALURES OF GEMINI PRO MODEL
In our baseline experiments, we found that the Gemini 3 Pro model performed poorly. Through analysis of the model output logs, we discovered that this may be caused by lack of strict adherence to JSON schemas during tool invocation. Specifically, we observed recurrent pattern parameter key hallucination, where the model substituted valid schema definitions with semantically similar but syntactically incorrect keys.
For example, in case ast astrooy-sstrop wtw.sttooooo..oooooooooo 10dbbdda cDlbbdda.test.basi.... www.:ee....b........oo the model produced the output shown Figure 8, the model attempted to invoke read_file using the argument pa h instead of the strictly defined file file.path. This suggests that while the model understands the intent of the tool, struggles to suppress its internal priors in favor of the provided API constraints.
This situation accounts for the majority cases in our evaluation, This indicates that Gemini 3 Pro seems have certain deficiencies in completing large-scale code editing tasks.
17

---

Published as a conference paper at ICLR 2026
Benchmark
Task Source
F2P/P2P
Real-world soflware development
Agent Eval.
Avg. LoC
SWE-Bench
PR
V
X
32.8
SWE-Dev
Unit Tests
X
X
190
FeatureBench
Unit Tests
790.2
Table 10: Comparison of FeatureBench with SWE-Bench and SWE-Dev.
Benchmark
Full Implementation Realistic Software Development
Scalability
commit0
FeatureBench
Table 11: Comparison of FeatureBench with SWE-Bench and commitO.
D COMPARISON WITH EXISTING BENCHMARKS
This section provides additional comparisons between FeatureBench and two representative bench- marks, SWE-Dev (Du al., 2025) and commitO (Zhao et al., 2024). These comparisons comple- ment the high-level discussion in Section 2 and clarify the distinctions in task sources, construction pipelines, evaluation settings, and scalability.
Comparison with SWE-Dev SWE-Dev derives tasks from unit tests and LLM--genrated problem requirement descriptions (PRDs). Its task formulation and construction pipeline differ substantially from FeatureBench, particularly in how tasks are specified, validated, and filtered. Table 10 provides a concise comparison, followed by brief clarifications of the key distinctions.
Realistic development workflow and stricter construction. FeatureBench preserves the original, well-developed features of each repository and leaves only the target feature unimplemented, closely matching incremental development. This is enforced through precise patch extraction, F2P/P2P filtering, and strict post-verification. SWE-Dev omits P2P verification and does not perform -est-veriiication, allowing patches that unintentionally break existing behavior.
Interface-driven task specification with minimal ambiguity. SWE-Dev uses LLM-generated PRDs, which naturally introduce ambiguity. FeatureBench instead exposes native top-level inter- function signatures and invocation paths extracted directly from the codebase ensuring clear, deterministic, and implementable task specifications.
Agent-based evaluation in this work. SWE-Dev reports results for LLM and multi-LLM settings but does not evaluate coding agents. FeatureBench conducts end-to-end agent experiments us- ing a unified OpenHands scaffold with multiple LLM backends (Claude, GPT, Gemini, Qwen), providing a realistic assessment of agent performance.
More realistic and complex feature-level tasks. SWE-Dev tasks involve roughly 190 LoC across three files. FeatureBench tasks require around 790.2 LoC across more files and substantially more test points, reflecting the multi-file, cross-module modifications typical in real feature de- velopment.
Comparison with commitò commitO studies whether LLMS can reconstruct entire libraries from documentation and high-coverage test suites. This setup differs markedly from FeatureBench, whose focus is real-world feature development with full, from-scratch implementations and scalable con- struction. Table 11 summarizes the key distinctions, with brief explanations provided below.
Real-world development and full implementation. In commitO, only the bodies of functions and classes are removed while definitions and architectural scaffolding remain, making tasks closer to fill-in-the-blank partial completions. FeatureBench removes definitions, imports, and associated logic, ensuring that the target feature fully absent and must be implemented from scratch, better aligning with real development workflows.
Low-cost scalability to new repositories. commit0 requires repositories with well-organized doc- umentation and very high test coverage (>90%), severely limiting applicability. FeatureBench requires only a runnable unit-test suite; after a short configuration step, the rest of the pipeline is fully automated, enabling efficient scaling across wide range of real-world codebases.
18

---

Published as a conference paper at ICLR 2026
E DATASET OVERVIEW AND EXPERIMENTAL RESULTS
The construction of the dataset resulted in 200 evaluation tasks derived from 3825 candidate coding environments across 24 Python repositories. These repositories encompass wide range of domains, including machine learning, scientific computing, visualization, web frameworks and fundamental software engineering utilities. This diversity ensures the dataset captures broad spectrum of real- world coding scenarios.
To promote transparency and reproducibility, the appendix contains two tables that describe the dataset composition. The Table 13 provides overview of each repository, including summary information and licensing details. The Table 12 presents quantitative statistics such as the average number of extracted code lines and the number of test points in the test suite.
To illustrate the dataset structure, we include an example of an individual data entry. Each en- try includes the following fields: instance.id, patch, test.patch, FAIL.TO_PASS. PASS_TOLPASS, image_name, repo, base.commit it, probl meableemsttttemeen and repo.settings. The specific meaning of each field is detailed in Table 14.
Additionally, the experimental results, summarized in seven comprehensive tables (Table 15 to Ta- ble 21), evaluate the performance of multiple large language models across the dataset. Each table reports three repository-level average metrics: Passed, Resolvee ved, and Token IO. These re- sults provide insights into model capabilities, including task pass rates, resolution rates, and token input-output statistics. The results demonstrate the strengths and limitations of current coding agents in diverse scenarios, forming foundation for future advancements in agentic coding research.
Repo
Lines
Files
Func- tions
Test points
pytorch-lightninn
765.4
24.0
35.7
42.0
metaflow
520.0
5.0
31.0
astropy
800.0
15.9
34.6
132.7
fastapi
110.0
9.0
8.0
10.0
accelerate
1057.0
12.0
17.0
33.0
transformers
494.8
13.4
13.7
90.9
879.0
20.0
352.0
Kermel
539.5
15.5
10.3
29.8
matplotlib
232.0
1.0
11.0
31.0
meson
220.0
7.0
14.0
16.0
miflow
732.9
16.9
29.4
25.8
scaborn
518.8
6,4
25.6
97.1
optuna
104.0
12.0
8.0
28.0
pandas
1522.9
19.2
39,2
69.2
pydantic
436.4
24.8
45.6
xаrray
847.8
15.8
37.3
21.5
hatch
847.0
17.0
55.5
16.0
packaging
785.0
3.0
36.0
294.0
setuptools
3283.0
50.0
185.0
30.0
pytest
528.0
3.0
34.5
105.5
mypy
392.0
11.0
5.0
10.0
scikit-learn
900.7
12.0
12.3
89.0
sphinx
1025.9
18.3
47.4
31.2
sympy
979.6
25.2
35.2
33.8
Table 12: Repository statistics.
19

---

Published as a conference paper at ICLR 2026
Repo
Summary
License
pytorch-lightning
Lightweight PyTorch wrapper for high-ppeformance research
Apache-2.00
metaflow
Framework for building and managing real-life data science projects
Apache-2.0
astropy fastapi
Astronomy and astrophysics core library
BSD 3-Clause MIT License
Modern, fast hhigh-peffrmmance) web framework for building APIs Library for running PyTorch training any distributed configuration
accelerate
Apache-2.0
transformers
State-of-the-ar pretrained models for natural language processing and beyond
Apache-2.0
trl
Library for training large language models with reinforcement learning from human feedback
Apache-2.0
Liger-Kerne
deep learning kernels developed
BSD 2-Clause
large-scale distributed training
matplotlib
Plotting library for creating scientific and publication-quality visuals Open source build system meant to be both extremely fast and user friendly
Custom
meson
Apache-2.0
mlflow
Open source platform for the machine learning lifecycle Statistical data visualization library built top of matplotlib
Apache-2.0
seaborn
BSD 3-Clause
optuna
Automatic hyperparameter optimization software framework
MIT License
pandas
Data analysis and manipulation library providing resh-performance data structures
BSD 3-Clause
pydantic
Data validation using Python type hints
MIT License
Xarray
Library for N-dimensional labeled arrays and datasets
Apache-2.0
hatch
Modern, extensible Python project management
MIT License
packaging
Core utilities for Python packaging
Apache-2.0
setuptools
Fully-featured library for packaging Python projects
MIT License
pytest
Testing framework for Python
MIT License
mypy
Optional static typing for Python
MIT License
scikit-learn
Machine learning algorithms and tools Python Documentation generation system for Python projects
BSD) 3-Clause
sphinx
BSD 2-Clause
sympy
Computer algebra system for symbolic mathematics in Python
BSD 3-Clause
Table 13: Summary and licenses for all GitHub repositories that task instances were extracted from.
Field
Description
instance id
(str) Unique identifier for the task.
patch
(str) Git diff showing the implementation.
test patch
(str) Git diff showing test file modifications.
FAIL. TO PASS
(list[str]) List of test files that must pass after implementation.
PASS TO.TTOPAAS.
(list[str]) List of test files that must continue passing.
image name
(str) Docker image containing the development environment.
repo
(str) Source repository (e.g., "owner/repo-name""
caseic
(str) Git commit hash of the base version.
problem statement
(str) Detailed task description and requirements.
repo sepo_settings
(str) Repository configuration settings as JSON string.
Table 14: Description of each field of FeatureBench task instance object.
20

---

Published as a conference paper at ICLR 2026
Model
Repo
% Passed
Resolved
# Token 10
Liger- Kernel
30.7
14.3
4.6M 35k
accelerate
0.0
0.0
16.1M 73k
astropy
34,9
11.8
12.5M 60k
fastapi
65.0
1.0M 14k
hatch
3.8
4.4M 23k
matplotlib
54.0
0.0
2.0M/ 40k
Codex
meson
24.4
0.0
7.0M / 57k
metaflow
100.0
100.0
4.8 37K
GPT-5.1-Codex (medium reasoning)
miflow
51.4
28.6
3.0M/ 20k
mypy
15.0
0.0
59k 2K
optuna
14.3
0.0
5.1M/ 36k
packaging
93.5
0.0
6.0M 45k
pandas
8.3M/
pydantic
50.3
0.0
9.9M/ 40k
pylest
61.8
0.0
3.2M/ 38k
pyterch-lightning
51.7
8.3
8, 8.9M/ 45k
scikit-learm
7.4M/
seaborn
43.7
3.3M/ 41k
setuptools
23,3
0.0
4. 4.7M/ 23k
sphinx
48.4
20.0
8. IM 49k
sympy
9.6M
transformers
24.6
4.4M 34k
trl
38.9
0.0
1.6M / 37k
xurry
46.0
0.0
18.6M TOM//// 85k
Table 15: Performance of Codex + GPT-5.1-Codex on each repository.
Model
Repo
% Passed
Resolved
Token I/O
Liger-Kent Kernel
77.3
3.7M 34k
accelerate
92.3
0.0
12.2M 53k
astropy
25.7
0.0
10 40k
fastapi
35.0
2.0M
hatch
3.8
14.7M 54K
matplotibb
54.0
0.0
2.3M/ / 20k
Claude Code
meson
28.4
0.0
5.4M/ / 33k
metaflow
100.0
100.0
7.7M
(routing)
miflow
50.7
24.5
5.0M/ 27k
-laude-0pus-4.55
mypy
25.0
0.0
569k / 5k
optuna
92.9
0.0
5.8M/ / 37k
packaging
88.1
9.8M 37k
pandas
39.6
13 6. 50k
pydantic
31.6
20.0
13.6M 41k
pylest
67.4
0.0
8.0M/ 42k
py pytorch-lightinng
36.2
0.0
8. 8.8M/ 35k
scikit-leam
6.6M
seaborn
42.6
0.0
5.7M/ 31k
setuptools
20.0
0.0
20. 73k
sphinx
39.0
10.0
10 36k
sympy
9.5M
transformers
35.6
4.4M/ 26k
trl
49.9
0.0
959k 13k
xarray
40.8
0.0
14.9M / 55k
Table 16: Performance Claude Code + Claude-Opus-4.5 on each repository.
21

---

Published as a conference paper at ICLR 2026
Model
Repo
% Passed
Resolved
Token I/O
Liger- Kernel
77.3
4.8M/
accelerate
0.0
0.0
19.1M 51k
astropy
20.2
0.0
11.IM 35k
fastapi
80.0
3.1M
hatch
18.0
1 36k
matplotlib
65.1
0.0
2.3M/ / 27k
meson
29.3
0.0
4.8M/ / 34k
OpenHands Claudc-Opus-4.5
metaflow
100.0
100.0
5.7M
miflow
54.1
20.4
6.9M/
mypy
25.0
0.0
418k / 8k
optuna
14.3
0.0
6.2M/ 23k
packaging
91.2
0.0
9M 16 16.9M 48k
pandas
50.0
10.4M 35k
pydantic
41.9
0.0
12.2M 30K
pylest
59.7
0,0
2 14k
pyterch-lightning
40.2
0.0
10.2M/ 2M / 34k
scikit-learm
45k
seaborn
43.7
6.0M/ 30k
setuptools
20.0
0.0
16.2M/ 42k
sphinx
36.9
0.0
8.9M/ 28k
sympy
50.5
11. 40k
transformers
32.8
5.3M/ 22k
trl
97.9
0.0
5.9M/ 35k
xurry
49.7
0.0
17.1M 47k
Table 17: Performance of OpenHands + Claude-Opus-4.5 on each repository.
Model
Repo
% Passed
Resolved
Token I/O
Liger-Keet Kernel
63.1
42.9
2.9M/ 21k
accelerate
0.0
0.0
3.4M 23k
astropy
10.3
0.0
4.1M/ / 25k
fastapi
35.0
1.9M
hatch
3.4M/
matplotibb
52.2
0,0
1.9M 24k
meson
27.2
0.0
2 / 30k
OpenHands DeepSock-V3.2
metaflow
100.0
100.0
2
miflow
33.3
12.2
2.7M/ 23k
mypy
25.0
0.0
532k / 7K
optuna
7.1
0.0
3. IM 21k
packaging
32.6
4. IM
pandas
17.8
3.5M/ 24k
pydantic
13.2
0.0
4.3M/ 19k
pylest
12.1
0.0
4 4.2M/ 36k
py pytorch-lightning
26.2
0.0
2.9M 24k
scikit-leam
4.8M/
seaborn
27.5
2.8M/ 22k
setuptools
23.3
0.0
114M 51k
sphinx
29.9
0.0
2.6M/ 19k
sympy
7.0M/
transformers
17.9
2.3M/ 19k
trl
81.4
0.0
3.0M / 54k
xarray
29.9
0.0
3.6M 29k
Table 18: Performance of OpenHands + DeepSeek-V3. V3.2 on each repository.
22

---

Published as a conference paper at ICLR 2026
Model
Repo
% Passed
% Resolved
Token I/O
Liger- Kerne
65.9
42.9
1.M / 15k
accelerate
0.0
0.0
2.3M / 12k
astropy
12.6
0.0
7.6M / 24k
fastapi
85.0
50.0
1.0M
hatch
3.8
0.0
L0M/ 5k
matplodib
65.1
0.0
474k / 8k
Gemini CLI
meson
29.1
0.0
730k 16k
metaflow
100.0
100.0
422k
Gemini-3-Pree Prt-----PPo--riiicw
millow
36.5
1.2M/
(low reasoning)
mypy
25.0
0.0
LOM 3K
opiuna
7.1
0.0
828k 6k
packaging
46.6
0.0
1.2M / 11k
pandus
29.2
5.0
4.5M / 10K
pydantic
21.1
0.0
9.4M/ 16k
pytest
60.0
0.0
887k 14k
pytorch-lightn -iightilgg
26.4
0.0
646k / 8k
scikit-learn
81.5
0.0
1.M 13k
scaborn
32.9
582k /
setuptools
0.0
0.0
1,4M/ / 8K
sphinx
35.0
0.0
2.5M / 21k
sympy
37.8
0.0
6.4M / 14k
transformers
22.3
1.4M/
trl
91.2
0.0
442K 17k
xarra
28.0
0.0
4.6M / 17k
Table 19: Performance of Gemini CLI +
on each repository.
Model
Repo
% Passed
Resolved
Token IOO
Liger- Kernel
72.6
42.9
4.5M 41k
accelerate
0.0
0.0
4.9M/ 40k
astropy
10.5
0.0
9.2M / 49k
fastapi
35.0
1.IM
hatch
7.7M/ 58k
malplotlib
50.4
0.0
1.2M/ 45k
OpenHands
meson
28.6
0.0
1 1.6 27k
metaflow
100.0
100.0
8.6M/
Seminni--Pro--Peviiie
miflow
32.2
4.6M/
(low reasoning)
mypy
25.0
479K 10K
optuna
14.3
0.0
4.3M/ 28k
packaging
0.0
12.6M 56k
pundas
10.4 10.4M/ 46k
pydantic
15.3
11.M 45k
pylest
67.7
0.0
16 9M / 81k
py pytorch-lightning
26.5
0.0
11.1M 36k
scikit-learn
8.3M/
scaborn
30.7
2 31k
setuptools
23.3
0.0
24. 24.3// 7lk
sphinx
30.4
0.0
S.0M 43k
sympy
36.9
8.7M
transformers
17.0
3.5M/ 31k
trl
97.3
0.0
M. / 58k
xarray
29.5
0.0
6. 43k
Table 20: Performance of OpenHands + Seemini---Pro-Peeveen on each repository.
23

---

Published as a conference paper at ICLR 2026
Model
Repo
% Passed
% Resolved
Token I/O
Liger Kemel
33.3
14.3
2.1M / 15k
accelerate
0.0
0.0
2.5M / 16k
astropy
5.7
0.0
2.4M / 13k
fastapi
35.0
499k /
hatch
3.8
0.0
1.3M / 12k
matplodib
43.1
0.0
888k 12k
Opentlands
meson
28.6
0.0
1,5M / 15k
metaflow
100.0
100.0
1.2M/
Qwen3-Coder-4800
miflow
36.2
10.2
1.9M 14k
A35B-Instruct
mypy
25.0
0.0
284k / 3K
optuna
7.1
0.0
951k / 8k
packaging
16.7
0.0
5.3M / 39k
pandas
pydantic
13.1
0.0
7M 16k
pytest
52.9
0.0
3. 3.2M / 19K
pytorch-lightning
26.1
0.0
1M 13k
scikit-learn
83.1
2.9M
scaborn
27.6
0.0
200M 15k
setuptools
10.0
0.0
1.M / SK
sphinx
28.9
0.0
1.7M / 11k
sympy
12.6
0.0
4.5M 27k
transformers
8.8
1.7M 12k
trl
70.3
0.0
1.6M 40k
xamay
21.0
0.0
1.8M / 21k
Table 21: Performance of OpenHands O0we33-Coder-480B-AA Instruu on each repository.
24

---

Published as a conference paper at ICLR 2026
Output Analysis of Gemini 3 Pro
"timestamp";
"model": "gemini p-r------------irt "tools":
"functiianDeellaatiions::
"name": "read_
"aescription"; "Reads and returns the content of specified file...",
"parametersusooncceema""
"type": "objest",
"required"; "file_path"
"properties": "file_path"; {
"description": "The path to the filc to read.", "type": "string"
"limit": "offset":
"request"; ...l, "contents":
"role": "model", "parts": [
"thoughtSignaturre" "...",
"functionCal1""
"name"; "read_file"
"args": "path"
"role": "user", "parts":
"funetionResponse"
"name"; "read_file",
"response";
"error": "params must have required property path"
...h,
"response";
"status"; 200,
"content": "**Claritying File Requirement wnlennnntt realized my initial approach was incomplete. Specifically understand now that the 'read_file' tool explicitly needs 'file_path' input. I'm focusing on ensuring that parameter is accurately and appropriately utilized moving forward."
Figure 8: Partial output Gemini Pro model using Gemini CLI framework while completing task astropy -strroy. bbdbbdda. bas.tbaiichhhb 06we927.677777. .v1
25

---

Published as a conference paper at ICLR 2026
Prompt for classifying top-level objects part 1 of
Task:
From a list of candidate objects, identify which ones are tested objects" in the context of Python test file.
**Definition "Tested Object":**
A "tested object" is an object that the test file is specifically designed to test. It represents the core functionality or feature being validated, NOT utility functions, test helpers, infrastructure code.
**Test File Information:**
Test file path: "[test_file)' Test file litest_iile name) name)
*+st File Content:**
'python (cest_file_content)
**Candidate Objects to Classify:**
(candldatee__sctiinn)
Guidelines:**
**Tested Objects (should be selected) ***
Core algorithms, classes, functiens that the test file is designed to validate
Main interfaces or APIs being tested
Key components whose behavior is the primary focus of the test
Non-TTested Objects (should NOT selected) :**
Utility functions from test utilities (e.g,, 'test.utils."" pytest.*')
Common tools defined in the codebase (e.g., Liferr_deiiee ' pessttt_vrrcco_aalclloee
Figure 9: Prompt template for classifying top-level objects part 2
26

---

Published as a conference paper at ICLR 2026
Prompt for classifying top-level objects part 2 of 2
**Examples for Reference:**
**Example Scenario:**
Test file: //tetba//ee///tanoooo/m///tttt/p (testing JSD
algorithm)
++Should Select (Tested Objects)
//testbed/src/11 htec///geeeemel//taaaafereeeer
lierss.. forward
::64' Core JSD implementation
//testbec/src/ligerr aeeetbec/see/leeeeeeeeesttttttrr
init
1:59* JSD class initialization
**Should NOT Select (Non-Tested Objects) ***
::91 Helper function in test file
/aarba///err/lgggggeeee/////g/gp ous. wrappe
1:34' General utility
**Your Task:**
Please analyze each candidate object and determine which tested objects for the given test file.
Provide your response in the following structured format:
## Analysis
For each candidate object, briefly explain whether it should be selected as tested object and why.
## Final Answer
Provide your final selection in the fellowing JSON format: 'ssonn {(
"tested_obe ciitedd_oject_ideee [
"object_id_l",
"object_id_2
"reasoning"; "Brief summary of the selection criteria applied" }}
**Important Notes:**
The 'tested object_jeeeeeeeeceee ids list should contain ONLY the object that are tested objects
If none of the candidates "ttested_bhjoct_dds" Include the full object ID exactly as provided in the candidate list
tested objects, return empty list []*
Unless it's obvicus or you' re pretty sure that a candidate object
is general purpose tool, you need to categori ze it tested object
Now, please begin your analysis for the candidate objects listed above.
Figure 10: Prompt template for classifying top-level objects part 2 of 2
27

---

Published as a conference paper at ICLR 2026
Prompt for completing docstring
Generate detailed docstring for the following Python function. The
docstring should include:
The main function description
2. Parameter description (if any)
3. Return value description (if any)
4. Important notes or exceptions (if applicable)
Funetion qualified name: 'python [Eunotion.cualittee name)
The full file content for reference: 'python
(file_content) ...
You only need to generate the docstring of the { function_qualfieedaaa function, the complete content of the file is for your reference only, and there is need to generate the docstring other functions classes.
In docstring, in order for us to parse it correctly, you are forbidden to use syntax like "'python " which may cause the end result to be confusing. Please only return the docstring content!!!
include triple quotes or other format tags:
Figure 11: Prompt template for completing docstring given to LLM
User prompt part of 3 (Level 1)
## Task
**Task Statement::*
Implement
**Core functicnalitiesrr:
2. **Main features and requirementss*::
3. key challenges**:
Figure 12: Unified prompt template for L part Task.
28

---

Published as a conference paper at ICLR 2026
User prompt part 2 of (Level 1)
+
This test comes from the 'f 1ibrary_name }} 1ibrary, and have given you the content of this code repository under /teatbed I', and you need to complete based on this code repository and supplement the files we specify. Remember, all your changes must be in this codebase, and changes that not in this codebase will not be discovered and tested by us.
We' ve already installed all the envirenments and dependencies you need, you don't need to install any dependencies, just focus on writing the code!
CRITICAL REQUIREMENT After completing the task, pytest will be used to test your implementation, ++YOU MUST.* match the exact interface shown in the **Interface Descriptionsx (I will give you this later)
You are forbiacen to access the follewing URIs: II black_ ||
Your final deliverai ielialeable should be code under the \/testbed/ directory, and after completing the codebase, will evaluate your completion and it important that you complete our tasks with integrity and precision.
The final structure is like below.
/tetbbed dirl/
# all your work should be put into this codebase file1.py
dir2/
## Interface Descriptions
### Clarificatien
The *IInterface Description** describes what the functions we are testing do and the input and output formats.
for example, you will get things like this:
Path: 'I interface code seteeffae_cod__exxmp___e_eeee } "pyyhonn (1 interfacc_codd_xxamge }
The value of Path declares the path under which the following interface should be implemented and you must generate the interface class/Function viven to you under the specí fied path.
Figure 13: Unified prompt template for L₁ part 2, Precautions.
29

---

Published as a conference paper at ICLR 2026
User prompt part 3 of
In addition to the above path requirement, you may try to medify any filc in codebase that you feel will help you accomplish our task. However, please note that you may cause our teat to fail if you arbitrarily modify delete
generic functions in existing files, so please be careful in completing your work.
What's more, in order to implement this functionality, some additional libraries etc. you any libraries, you need think about what dependencies you might need and fetch and install and call them yourself. The only thing is that you **MUST** fulfill the input/output format described by this interface, otherwise the test will not pass and you will get
often required, don't restrict
points for this feature.
And note that there may be not only one *IInterface Descriptiones, you should match all Interfaace Description in)*
### Interface Description
Below is **Interface Description 1**
Path: h/path/toxxxx.p/
" "python
def my function:
n** Implement
Parameters: argl: str,
Returens:
arg2: int,
# <your code>
### Interface Description class MyClass:
# <your codc>
Remember, **the interface template abeve is extremely important**.
You must generate callable interfaces strietly according to the specified requirements, as this will directly determine whether you can pass our tests. If your implementation has incorrect naming improper input/output formats, it may directly result 0% pass rate for this case.
Figure 14: Unified prompt template for L1 part 3, Test and Interface Description.
30

---

Published as a conference paper at ICLR 2026
User prompt for test-layer-norm (Level 1)
## Task
**Task Statement:
Implement Optimized Layer Normalization Function**
Develop a high-performance layer normalization function that:
1. **Core Functionality::: Applies layer normalization input tensors using weight, bias, and epsilon parameters for numerical stability
2. ** Requirementsw::
Integrate with existing Liger kernel optimization framework Support standard layer normalization mathematical operations ( mean centering, variance scaling, affine transformation Handle multi-dimensional tensor inputs efficiently
3. A*Main Challenges**:
Optimize memory usage and computational performance
Ensure numerical stability with configurable epsilen values Maintain compat ibility with transformer model architectures Provice seamless integration with the broader Liger functional interface ecosystem
**NOTE+*:
This test comes from the Miigerkernel library, and have given you the content of this code repository under 1/testbed/', and you need to complete based on this code repository and supplement the files we specify. Remember, all your changes must be in this codebase, and changes that are not in this codebase will not be discovered and tested
We' vo already installed all the envirenments
dependencies you need, you don't need to install any dependencies, just focus on writing the code!
#CRITICAL REQUIREMENT**: After completing the task, pytest will used test your implementationn. *xYOU MUST match the exact interface shown in the v*Interface Descriptions (I will give you this later)
You forbidden to access the following URLS: black_links:
Your final deliverable should be code under the '/testbed/ directory, and after completing the codebase, will evaluate your completion and it is important that you complete our tasks with integrity and precision.
The final structure is like below.
testbed dir1/ all your work should be put into this codebase
file1.py
dir2/"
## Interface Descriptions
Figure 15: User prompt for test-layer-norm (Lz).
31

---

Published as a conference paper at ICLR 2026
User prompt for test-layer-norm (Level 2) part 1 of
|| Task
A A*Task Statement:
Implement Optimized Layer Normalization Functions
Develop herfoomrnoomance layer normalization function that:
1. **Core Functionality4*: Applics layer normalization to input tensors using weight, bias, and epsilon parameters for numerical stability
2. **Key Requi rements**:
Integrate with existing Liger kernel optimization framework Support standard layer normalization mathematical operations ( centering, variance scaling, affine transformation)
Randle milti-dimeensonal tensor inputs efficiently
3. **Main Challenges*4:
Optimize memory usage and computational performance Ensure numerical stability with configurable epsilon values Maintain compatibility with transformer model architectures Provide seamless integration with the broader Liger functional interface ecosystem
This test is derived from the "iigerkeernel 1ibrary, but you are *+NOIE+*:
NOT allowed to view this codebase or call any of its interfaces. It is +VVERY IMPORTANT** to note that if detect any viewing calling this codebase, you will receive ZERO for this review.
CRITICAL:*:: This task is derived from 'Liger-kernel', but you ** MUST** implement the task description independently. It is ** ABSOLUTELY FORBIDDEN similar commands to access the original implementation, and doing so will be considered cheating and will result an immediate score of ZERO! You must keep this firmly in mind throughout your implementation.
'pip install liger-kernel
You now in \/testbed/', and criginally there 'Liger-kernel' under 1/testbed/ that been installed via 'pip install -e .'. However, to prevent you from cheat ing, we' ve removed the code under 1/testbed/'. while you can traces of the installation via the pip show, it's artifact, and "liger-kernel" doean't exist. So you can't and don need to use 'pip install liger-kernel', just focus on writing your 'agent_code and accomplishing our task.
specific
implementation
Also, don't try to 'pip uninstall liger-kernel even if the actual 'liger-kernel' has already been deleted by us, this will affect our evaluation liger-kernel will result in you getting a ZERO because our tests won't run.
you, and uninstalling the residual
We've already installed all the envi renments and dependencies you need, you don't need to install any dependenci es, just focus writing the code!
*CRRIICAL REQUIREXENT**: After completing the task, pytest will be used to test your implenentation. *+YOU MUST** match the exact interface shown in the **Interface Descriptionsx (I will give you this later)
Figure 16: User prompt for test-layer- norm part 2f (L2).
32

---

Published as a conference paper at ICLR 2026
User prompt for test-layer-norm (Level 2) part 2 of
You are forbidden to access following URLS: black_links: attps://githube com htwaa//iiied/iiieeeeeeeeee htt.pp//ggttt...cooo///iaii/eae/
Your final deliverable should be code in the //testbed/agent_cool directory.
The final structure is like below, note that all dirs and files under agent_code/ are just examples, you will need to organize your own reascnable project structure to complete our tasks.
/testbed
agent agent_code/
# all your code should be put into this dir and match the specific dir structure
init .yy
# ggent_code// folder must contain inin -Py
dirl/
init py code1.py
setup.py
# after finishing your work, MUST generate this file
After you have done all your work, you need to complete three
CRITICAL things:
1. You need to generate '_init_.p"' under the 'agent_code/' folder and import all the classes or functions described in the ** Interface Descriptions** in it. The purpose of this is that we will be able to access the interface code you wrote directly through "agent_code..xxaalllllaass ExampleClass in this way. You need to generate /ttestbed/sctup....teo place the following content exactly:
under "/testbed/'
"pyythnn
from setuptools import setup, find_packages
setup(
mame==agent__ode""
version="O.1",
packages-tind_
)
3. After you have done above two things, you need to use 'cd / testbed & pip install command to install your code. Remember, these things are +*VERY IMPORTANT**, they will directly affect whether you pass tests.
# Interface Descriptions
Figure 17: User prompt for test-layer-norm part 2 of 2 (L2).
33

---

Published as a conference paper at ICLR 2026
Example an LLM-generated docstring
class
@staticmethed
def backward (ctx, Agrad_outputt ***
Computes gradients for the fused linear DPO (Direct
Preference Optimization) function during backpropagation
This static method implements the backward pass automatic differentiation in PyTorch's s autograd aystem. It delegates the gradient computation to the parent class's backward method and then filters the results to match the expected number parameters for the DPO function.
Fa Parameters:
ctx: PyTorch autograd context object containing saved tensors and other information from the forward pass Variable-length argument list of gradient tensors flowing back from the loss function, Each tensor represents the gradient with respect to the corresponding output of the forward pass
Returns:
tuple: A tuple containing gradients with respect to the
input parameters of the forward function:
Gradient w.r.t. input tensor (_input)
Gradient w.r.t. weight tensor (weight)
Gradient w.r.t. target tensor (target)
Gradient w.r.t. bias tensor (bias)
None values for parameters that don't require gradients (ref_input, ref_weight, ref_bias, ignore_index, beta, campute_nli_lose compiled, use_ref_model, everage_log__ree chunk_aize, loss_type)
Important Notes:
This method is part of PyTorch's Function interface for custom autograd operations
The method truncates the parent class gradients to the
first elements using [:4] slicing
Additional None values are returned to match the
signature of the forward method parameters The actual gradient computation logic is inherited www.eediieeerrrreeeenneeaaeeeeee
from
This ensures proper gradient flow for DPO loss optimization while maintaining compatibility with PyTorch's autograd system.
<your code>
Figure 18: Example of an LLM-generated docstring.
34

---

Published as a conference paper at ICLR 2026
Human Evaluation Guideline for Top-L.vel Tested Object Classification
Each import statement in test file should be evaluated independently to determine whether represents a top-level tested object or an auxiliary component. The procedure as follows:
Step 1: Understand Test File Purpose
Read the test file to understand its testing objective and scope.
Identify the main functionality or module being validated.
Step 2: Identify All Import Statements
Locate all import statements, including absolute and relative imports.
Step 3:
Filter External Library Imports
Exclude imports from external libraries (e.g., pytest, unittest, torch).
Step 4: Classify Repository-Intes
Imports
Assert statement usage: If the imported object appears in assertions comparing sults, it is likely a tested object.
Name correspondence: the object's name matches keywords in the test filename, it is likely tested object.
Module correspondence: If imported from module matching the test filename, it is likely tested object.
Utility module exclusion: Imports from ut 1s.i testing// www.helppprr.... etc., usually auxiliary.
Frequency and prominence: Objects used extensively across the test file are more likely tested objects.
Classification Decision
Mark each import as either Top Import (tested object) or Non-Top Import (auxiliary). When criteria conflict, prioritize the first three criteria over the last two.
Figure 19: Human evaluation guideline for identifying top-level tested objects.
35

---

Published as a conference paper at ICLR 2026
Expert Verification Guideline for Feature-Level Tasks (part of
Each feature-level task must be manually verified to ensure that (I) the task is structurally correct (objects, imports, masking, etc.), and (2) competent engineer can implement the required functionality using only the prompt and the remaining codebase, without external documentation.
You will typically two resources:
inetadata.outpon
logs and classification results (lists of top objects and spe-
cific objects).
Level-1 task directory: problem_statement md (this is the primary target for verification).
Stage 1: Check Structural Consisteney Get the list top and specific objects
Open the classification summary under metadata..utpputs/ Identify:
Top objects: top interface of the feature being tested.
Specific objects: objects that are functionally related but top interfaces.
Treat these lists checklist for the following steps.
1.2 Check masking of top objects
For each top object, open its source file in the Level-1 directory.
Confirm that the implementation body is removed and that only the signature, doc- string, and minimal scaffolding remain.
If any implementation detail is still visible, manually remove it while ensuring that tests can still import and call the interface.
1.3 Check removing of specific objects
For each specific object, confirm that it does not appear in the remain codebase.
If leftover definitions are found, please remove them.
Figure 20: Expert verification guideline for feature-level tasks part 1.
36

---

Published as a conference paper at ICLR 2026
Expert Verification Guideline for Feature-Level Tasks (part 2f
Stage 2: Check Prompt Completeness and Solvability
2.1 Check the high-level Task Description
Read the Task Description and ask: "If only had this description and the codebase, do I know what to implement?"
Verify that it:
Explains what feature or behavior needs to be implemented. Provides context about where the feature sits in the system.
Mentions any key technical considerations that affect correctness.
If the description is vague or incomplete, rewrite it to make the implementation goal clear.
2.2 Check the Test Description sections
For each Test Description, verify that:
All required top objects are correctly referenced.
Imports match the real file structure under the task directory.
Fix missing interfaces and incorrect module paths as needed so that the agent will know where implemenet them.
2.3 Check Interface Descriptions and docstrings
For each Interface Description:
Ensure the docstring is semantically complete: what the function/class does, pa- rameter meanings, and return values.
Confirm that it is self-contained and does not require external documentation. Keep it concise but readable; something a real engineer would be happy to follow. docstring is too short, ambiguous, or inconsistent with the real behavior, revise it and, if necessary, refer to the original implementation to understand the intended semantics.
When to Mark Task as Verified
task considered verified if competent engineer can implement the required function- ality using only the prompt.md and the remained codebase without external documentation. Concretely, this requires that:
All top objects correctly masked, imported, and documented.
All specific objects that should not remain in the codebase removed.
The Task Description clearly states what to build.
Test Descriptions match the actual file layout and cover all required interfaces.
Interface Descriptions and docstrings are accurate and self-contained.
any of these conditions are met, the relevant parts and re-check the task using the same steps before marking it as verified.
Figure 21: Expert verification guideline for feature-level tasks part 2.
37


Reviews:
[{"review_id": "OfEKVbxC4C", "reviewer": "Reviewer_anv6", "summary": "This paper introduces ACE-Bench, an execution-based and continually updatable benchmark for feature-oriented agentic coding, built via a test-driven, dependency-trace pipeline that yields 212 tasks across 16 repositories and shows frontier agents solve only ~7.5%.", "questions": "Precise distinction from SWE-Dev and commit0.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The benchmark targets feature-level development (not just bug fixes) and pairs each task in two modes—L1 (extend an existing repo) and L2 (from scratch)—a clean formulation that isolates the role of context and raises the ceiling on task complexity.\n\n- The evaluation is execution-based, with explicit interfaces and anti-cheating controls; the pipeline includes post-verification and ablations (e.g., hiding interfaces, step budgets, visible tests), plus clear metrics (Resolved/Passed/Token I/O). These choices make the results reproducible and the failure analysis informative. \n\n- The paper is well structured (pipeline figures, instance layout, and evaluation workflow are easy to follow), and it surfaces useful empirical trends (e.g., performance drops with longer required code; L2 is markedly harder than L1).", "weaknesses": "- Positioning vs. closely related work needs to be sharper. The paper should more directly compare and differentiate from SWE-Dev (feature-driven development on large existing codebases with runnable environments; 14k train / 500 test and developer-authored unit tests) and commit0 (from-scratch library generation with API spec + interactive tests).\n\n- Dataset composition skew. Although spanning 16 repos, the task mass is concentrated (e.g., Transformers dominates), which risks domain bias and may understate generalization across diverse stacks (services, infra, build systems).\n\n- Baseline coverage & fairness details. All agents are run inside OpenHands, which is reasonable, but diversity in agent frameworks (gemini-cli, kimi-cli) would triangulate where the difficulty lies.\n\n```\nSWE-Dev: https://arxiv.org/abs/2505.16975\ncommit0: https://arxiv.org/abs/2412.01769\n```", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eMqnlk7FVS", "reviewer": "Reviewer_erVP", "summary": "The authors propose ACE-Bench a python-only, execution-based coding benchmark to evaluate coding agents' performance on feature development. Similar to related benchmarks, they (manually) create execution environments and extract fail2pass and pass2pass tests to evaluate whether a solution would solve the problem at hand without breaking other functionality. Problem statements are synthesized using a LLM and include invocation path, function signature (including input and output variables), as well as annotations. The authors developed an algorithm to extract functions that are relevant to a test patch from an object dependency graph. In the evaluation of the OpenHands agent with four LLMs shows that compared with SWE-Bench the resolve rates are much lower.", "questions": "* You report 889 executable environments but only 212 tasks, can you explain the descrepancy?\n* You configure the nnumber P2P tests. Doesn't this mean a there may be tests that would fail even though it would pass both P2P and F2P?\n* Is the subset of 30 instances purely random or stratified in some way?\n* In section 4.2.1 you say you conducted a \"professional-level algorithm engineer\" who revised prompts. Can you detail what this title means and how they were revised?\n* I don't quite understand how you arrived at the $L_1$ vs. $L_2$ datasets. Can you elaborate?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "* The authors don't base their dataset on already existing ones but scrape their own data which lessens the risk of data leakage\n* The paper is well written and easy to follow. Visualization illustrate the core aspects of the work well. \n* Assessing feature development capabilities is an important area which is under-explored\n* The dataset is seems to be significantly more complex in terms of gold solution lines, files, functions and number of tests.\n* The graph-based function extraction is novel and seems sound", "weaknesses": "* The authors do not provide a lot of analysis to show that their tasks are truly solvable. Given that the problem statements are LLM generated, this needs to be shown. The authors propose that AssertionErrors indicate problem statements contain sufficient information. However, runnable code does not correlate with solvability of the tasks.\n* The data set is Python only which severely limits to which degree one can measure coding agent performance.\n* Only a single agent (OpenHands) is evaluated hence the claim of \"laziness\" is specific to their scaffold.\n* All \"feature development\" tasks are more feature extensions rather than new features. While this is due to the fact that truly new features are extremely hard to test, this is a major limitation for a benchmark that focuses on feature development assessment.\n* The quality of the generated prompts is hard to quantify, yet the authors claim to have developed a \"high-quality\" data set. \n* Providing invocation path, function signature (including input and output variables), as well as annotations in the prompt seems unreasonable and not typical of a feature development task. More typical are natural language description that are rather vague. \n* You consider SWE-Bench a benchmark that doesn't contain any feature requests. This is not true. As quantified by Rashid et al.[1] (containing 22% feature requests), it actually contains 18% feature requests. \n\n1. Rashid, M. S., Bock, C., Zhuang, Y., Buchholz, A., Esler, T., Valentin, S., ... & Callot, L. (2025). SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents. arXiv preprint arXiv:2504.08703.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "apWJghXzWu", "reviewer": "Reviewer_oMnc", "summary": "This paper proposes ACE Bench, a novel benchmark for evaluating coding agents. While existing benchmarks focus on evaluating PRs or bug fixs, ACE-Bench emphasizes assessing feature-level implementation capabilities of coding agents.", "questions": "- I found it somewhat confusing that there exists another agent evaluation benchmark with the same name ACE Bench [Chen+ 2025]\n\n[Chen+ 2025] \"ACEBench: Who Wins the Match Point in Tool Usage?\"", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Evaluating feature-level implementation is both novel and important. As evidenced by recent SWE-bench leaderboard results, modern coding agents can perform bug-fixing tasks with high accuracy. However, their capability to handle feature-level implementations remains largely unexplored. This paper addresses this gap by providing a benchmark specifically designed to evaluate this capability.\n\n- The benchmark is designed with usability in mind. Given that evaluating the full set requires approximately one million tokens (with associated computational costs), the authors provide a \"lite set\" that reduces evaluation costs. Additionally, the \"Passed Rate\" metric (the average fraction of fail-to-pass tests passed per task) enables partial assessment of feature-level implementation capability.", "weaknesses": "- Allowing unrestricted library usage may enable agents to complete tasks by simply calling existing library functions, essentially testing library knowledge rather than implementation capability (The benchmark allows agents to use pip install to add arbitrary libraries (Figure 13)). While the authors prevent accessing ground-truth implementations via anti-cheating mechanisms,the policy on legitimate library usage remains unclear. The authors should clarify whether the evaluation assesses (a) the ability to select and leverage appropriate libraries as part of software development skills, or (b) pure implementation capability with a fixed set of libraries. This distinction is critical for interpreting what capabilities are actually being measured.\n\n- The benchmark tasks are derived from commits created between May 2022 and September 2025, which overlaps substantially with the training periods of models (i.e. knowledge cutoff). Fig. 5 shows that task performance currently exhibits minimal dependence on commit time. However, as the authors acknowledge, the risk of data leakage may become more pronounced in the future. Therefore, continuous updates to the benchmark will be critical to maintain its validity as an evaluation tool.\n\n- The benchmark only supports Python, limiting its generalizability to other programming languages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e0oBTPSDHe", "reviewer": "Reviewer_3aZ5", "summary": "This paper introduces ACE-Bench, a new benchmark for evaluating LLM-based agents on feature-oriented software development tasks. Unlike existing benchmarks focusing primarily on bug-fixing within single PRs, ACE-Bench proposes more complete coding scenarios. The benchmark comprises 212 tasks and 889 executable environments from 16 open-source Python repositories.\n\nKey contributions include:\n\n(1) A feature-oriented evaluation framework with two difficulty levels (L1: extending existing codebases; L2: implementing from scratch)\n\n(2) Execution-based evaluation with explicit interface specifications to enable unambiguous testing\n\n(3) Empirical results show that more capable models achieve very low success rate, like Claude 4 Sonnet achieves only 7.5% success rate", "questions": "My questions are following several aspects mentioned in weakness:\n\n- How can you ensure the generalizability of ACE-Bench, given various programming languages and real-world scenarios?\n- What are your repository selection criteria, and why do you believe 16 repositories are sufficient?\n- What are the justifications and verifications for LLM classification, filtering criteria, and the number setting to ensure adequate coverage?\n- How would you address the limitations in evaluation, such as the high input tokens and cost, undistinguishable performance, and inadequate analysis?\n- How are you going to address these interface specification dependency problems in ACE-Bench?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**1. Problem Formulation**: This work proposes ACE-Bench that contains complex software engineering tasks. Its problem formulation explains its differences from existing benchmarks in task complexity and dataset construction.\n\n**2. Rigorous Evaluation Protocol**: The execution-based evaluation with comprehensive anti-cheating mechanisms and two difficulty levels provides a reliable performance assessment.\n\n**3. Significant Challenge**: The obvious performance drop between SWE-bench and ACE-Bench demonstrates the limitations of current agentic systems, providing meaningful direction for future research.", "weaknesses": "**1. Limitations In Scale and Diversity**:\n\n- ACE-Bench uses 212 evaluation data from 16 repositories, which can be relatively small.\n- Python-only instances limits generalizability to other programming languages and real-world scenarios.\n- Repository selection criteria are not clearly justified.\n\n**2. Limitations In Methodology**:\n\n- Using LLMs to classify top-level objects introduces potential systematic biases and errors. However, no quantitative evaluation of classification accuracy is provided.\n- The 100-line minimum and \"10 F2P test points\" filtering criteria lack justification.\n- The paper sets m=5 P2P tests per instance, but doesn't justify why this number provides adequate coverage.\n\n**3. Limitations In Evaluation**:\n\n- The average of 1M+ input tokens per task raises concerns about practical applicability and cost.\n- Near-zero success rates on L2 tasks suggest the difficulty may be unrealistically high, limiting the benchmark's ability to differentiate between models and provide meaningful insights for future research.\n- Figure 5 shows minimal correlation between task creation time and performance, but deeper analysis of how feature complexity evolves over time can be valuable.\n\n**4. Limitations In Interface Specification Dependency**:\n\n- Table 7 shows remarkable performance drops without interfaces, suggesting that the benchmark may be testing interface-matching more than general coding ability.\n- Real-world development often involves ambiguous or evolving requirements, while the explicit interface specification may not reflect realistic scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)