Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents
Authors: Jiayi Kuang
Yinghui Li
Xin Zhang
Yangning Li
di yin
Xing Sun
Ying Shen
Philip S. Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Software Engineering
Agent
Large Language Model
Primary area: datasets and benchmarks

Abstract: Large language model-based agents show promise for software engineering, but environment configuration remains a bottleneck due to heavy manual effort and scarce large-scale, high-quality datasets.
Existing benchmarks assess only end-to-end build/test success, obscuring where and why agents succeed or fail.
We introduce the Environment Configuration Diagnosis Benchmark, EnConda-bench, which provides process-level trajectory assessment of fine-grained agent capabilities during environment setup-planning, perception-driven error diagnosis, feedback-driven repair, and action to execute the final environment configuration. 
Our task instances are automatically constructed by injecting realistic README errors and are validated in Docker for scalable, high-quality evaluation.
EnConda-bench combines process-level analysis with end-to-end executability to enable capability assessments beyond aggregate success rates.
Evaluations across state-of-the-art LLMs and agent frameworks show that while agents can localize errors, they struggle to translate feedback into effective corrections, limiting end-to-end performance.
To our knowledge, EnConda-bench is the first framework to provide process-level internal capability assessment for environment configuration, offering actionable insights for improving software engineering agents.

Published as a conference paper at ICLR 2026
PROCESS-LEVEL TRAJECTORY EVALUATION FOR EN- VIRONMENT CONFIGURATION IN SOFTWARE ENGI- NEERING AGENTS
Jiayi Kuang1.2; Yinghui Li;; Xin Zhang Yangning LP44, Di Yin", Xing Sun",
Ying Shen 1..6f Philip S. Yus
Sun Yat-sen University, Tencent Youtu Lab, The Hong Kong Polytechnic University
*Png Cheng Laboratory, UUniversity of Illinois Chicago
GGunggdong Provincial Key Laboratory of Fire Science and Intelligent Emergency Technology
ABSTRACT
Large language model-based agents show promise for software engineering, but environment configuration remains bottleneck due heavy manual effort and scarce large-scale, high-quality datasets. Existing benchmarks assess only end-to- end build/test success, obscuring where and why agents succeed or fail. We intro- duce the Environment Configuration Diagnosis Benchmark, IaConda--Benono which provides process-level trajectory assessment of fine-grained agent ca- pabilities during environment setup-planning, perception-driven error diagno- sis, feedback-driven repair, and action to execute final environment configura- tion Our task instances are automatically constructed by injecting realistic README errors and are validated in Docker for scalable, high-quality evaluation. EnConda-Bench combines process-level analysis with end-to-end executabil- ity to enable capability assessments beyond aggregate success rates. Evaluations across state-of-the-art LLMs and agent frameworks show that while agents can lo- calize errors, they struggle to translate feedback into effective corrections, limiting end-to-end performance. To our knowledge, EnConda -ennch is the first frame- work to provide process-level internal capability assessment for environment con- figuration, offering actionable insights for improving software engineering agents.
INTRODUCTION
Large language models (LLMs) have rapidly advanced Li et al. 2023 Yu et al. 2024 Du et al. 2024 Li et al. [2024c] Huang et al.
2024
Li al. 2022a Zhang et al. 2025 Xu et al. 2025 Li al. Li et al. 2025c di Kuang et al. 2002aa). spurring exploration
2025e Kuang et al. 2225b Team 2025
of challeningg Software Engineering (SWE) tasks with high academic and industrial value (He al.
Wang et al. 2024 Fan et al. 2023 Wang et al.) Zhang et al. 2024b). SWE offers precise, verifiable evaluation systems, making prime domain to study agentic intelligence (Hendrycks) ct al.; Austin et al.) 2021). Numerous code-oriented agents, such as OpenHands (Wang et al.) and -Aeet (Yang et al. 2024). aim to assist with complex project development and mainte- nance. In SWE benchmarks like SWE- BEECH (Jimenez et al. 2024), agents edit and repair code based on a given issue, then submit pull request and validate execution. Within this workflow, configuring a runnable execution environment is the most fundamental and critical first step, yet it remains challenging for both human engineers and current LLMs (Eliseeva et al.), requiring sub- stantial manual effort. This burden constrains large-scale, high-quality dataset production, making rigorous evaluation of agents' environment configuration capabilities essential for progress in SWE.
2025
Most existing environment configuration benchmarks rely on end-to-end success (build and test pass) (Milliken et al. 2025) Bouzenia & Pradel 2025 Eliseeva et al.) Vergopoulos et al., 2025), yielding only coarse outcomes and obscuring process-level capabilities along the configuration tra-
"Work done during the internship at Tencent Youtu Lab.
CCorresoond to Yinghui Li (lebronyhli (tw.hhiittennee.commmm and Ying Shen
Data and code are available in http://gttbb.oooooo/oooooennnete

---

Published as a conference paper at ICLR 2026
want to set up an environment for arepository....
However, meet some prob lerm.......
X
Enore Engital Order Froor
Follow README
instalation
Puthor
Instali
The lnst yolknn so sorvited ianibdediae
The dreciiy forths
Trediressy forths
Instali
inperettemmen
pyftanko|pios1 limage
sumoort,ope wpw.ttpppppppp...oo amp]
commonllooemaal OK
cormmaalooommttte
Instrall
try or
Co Command Syrtax Einor
Broor
Erter 7
Compatiii Ernor
The coecorerane reneoreet prolectzees DNUMall. lnnvibmi bob
supporied projet mosite Pytton 3.8 lata: 11 caci instaiation rallrnes.
Figure 1: The illustration of common problems in environment configuration. When human en- gineers configure the environment, they often encounter various errors. They should first identify the step where the error occurred and then fix the problem before proceeding to the next step, until the configuration is complete. Similarly, intelligent agents performing environment configuration should possess good planning, perception, feedback, and action capabilities.
jectory. For example, it is difficult to locate the specific stages in environment configuration where errors are likely to occur, or to identify which capabilities the agent lacks to perform more precise and effective configuration. They cannot pinpoint failure stages or missing capabilities, limiting deep insights and research directions. In addition, data construction is another bottleneck: high- quality, correctly buildable repositories are scarce; selecting and annotating them demands expert effort. As result, it is challenging for researchers to obtain large quantities of high-quality data for evaluating agent environment configurations.
To address these challenges. we focus on process-level evaluation along the agent's configuration trajectory. Specifically, we investigate: (1) how agents upply planning devise reasonable config- uration steps and strategies given the task requirements: (2) how they use perception to accurately localize the causes of errors when failures occur (e.g., version incompatibilities or missing depen- dencies); (3) how they utilize feedback to analyze the errors and try to fix them; and (4) how they translate precise feedbacks into actions that correct these errors, complete environment configu- ration, and ensure that subsequent code runs and passes evaluation. This process-level trajectory evaluation provides deeper and more valuable references for improving agent capabilities in envi- ronment configuration and for subsequent related studies.
However, directly extracting the planning and feedback segments from agent trajectories, or eval- uating entire long trajectories, is difficult. Inspired by how human engineers configure environ- ments-typically following README steps first, then analyzing the causes of failures and attempt- ing fixes-we consider editing an originally correct README by injecting erroneous commands or confusing steps. As the model configures the environment based on such a README, it must locate and repair these errors. This design enables process-level evaluation along the agent's trajectory and allows us to observe which error types the model more readily repairs and which are harder to detect, providing valuable insights for future agent development.
Motivated by this task schema, we further design an automated data construction framework that scales instance generation and produces agent execution trajectories for training. We (1) select high-quality repositories via strict criteria; (2) employ advanced LLMs to edit key environment READMES with common error types and annotate categories and suggested fixes; and (3) validate and filter for effective errors via an automated framework to obtain high-quality task instances. We then build an evaluation suite supporting both process-level analysis (error localization, repair) and end-to-end executability, along with an automatic data engineering pipeline that generates task in- stances and agent trajectories. To our knowledge, we are the first to enable process-level assessment for agents and to propose an automated data framework in this setting. Empirical evaluation on advanced LLMs and agents shows that, while agents exhibit basic error judgmen//loalizznn they struggle to convert feedback into effective corrective actions, limiting end-to-end performance. We present our contributions and several noteworthy findings:
We propose trajectory- based EnConda-Bench for process-level evaluation of environ- ment configuration SWE, enabling detailed assessment of the capabilities agents exhibit during environment configuration.
2

---

Published as a conference paper at ICLR 2026
We introduce automated data construction pipeline, which reduces manual labor and supplies large-scale training data for agents and LLMs.
Our evaluation across multiple LLMs/agents finds basic error localization/classsfctttoo abilities but limited environmental interaction and feedback utilization, often yielding inef- fective repairs, providing valuable findings and inspiration for future research.
2 RELATED WORK
Agent Methods Early agent attempts to automate environment setup relied on specific heuristics that infer dependencies from source code, offering determinism but falling short on system pack-
ages, version pinning, and platform heterogeneity (Gruber & Fraser [023 Zhang et al., 2024a Li jet al. 2025b Ye et al. 2025 Yang et al. 2025 Guo et al. 2026 Li et al. 2025a Lu et al. 2025a b).
Tool Tagg-augeetted code agents extend LLMs with search, editing, and execution capabilities and show promise (Wang et al. 2024, Wang et al. Zhang et al.) 2024b; Yang et al., 2024 Xia et al. 2024), yet setup remains fragile bottleneck due to sensitivity to external toolchains and long decision chains. Specialized environment agents try to narrow this gap. INSTALLAMATIC targets Python with curated installation context and exemplar Dockerfiles, judging success via tests (Milliken al.) 2025). EXECUTIONAGENT generalizes to five languages with CI-log ground truth, requiring both Dockerfiles and setup scripts, and evaluating build success and test-result deviations (Bouzenia & Pradel 2025). but still needs manual inspection and is comparatively slow. Repo2Run employs
dual-environment architecture, performing configurations in an isolated Docker environment while leveraging an external environment for monitoring and assistance, with rollback mechanism that restores the system to the last known stable state upon command failures (Hu et al., (025). Over- all, the trajectory moves from heuristics to tool-augmented agents to interaction agents, and our approach aims to improve process-lever level, actionable interactions with the environment.
Environment Configuration Benchmarks In early SWE benchmarks, function-level bench- marks (c.g.. HvmanEval, MBPP, APPS) catalyzed progress but are misaligned with real-world build
202T Hendryeks et al.) Li et al., 2022 2024b Austin et al., 2021 Jain et al.) Miao 2025 Chen et al. 2025). Repository-leve efforts better reflect practice (Liu et al. [024b Jain et al.
(Chen al.
jet al. 2024 Jimenez et a (024), but they ignore the environment configuration task by providing manually configured Docker files. Environment setup specific benchmarks are attempting to explore this territory. INSTALLAMATICbeen
curates 40 Python repositories with exemplar Dockerfiles, assessing success via tests (Milliken et al., (2025). EXECUTIONAGENTeene spans five languages with Cl-log ground truth and evaluates build/test success and test-result deviations (Bouzenia & Pradel 2025). For larger-scale data and more languages, recent benchmark EnvBench expands to 994 repositories across Python, Java, and Kotlin projects, while still offering limited visibility into data collection and evaluation strategies (Eliseeva al.)). Thus, automated construction further scales evaluation. SETUPAGENT further automates extraction of installation and testing proce- dures, supports historical states, and collects test-level results, accelerating data generation, though its evaluation remains largely end-to-end (Vergopoulos et al. 2025). Nonetheless, most benchmarks still reduce evaluation end-to-end executability, obscuring where and why setup fails. In contrast, our work provides process-leve trajectory evaluation with an automatic data construction framework for robust evaluation of agents.
3
ENCONDA-
3.1 TASK DEFINITION AND WORKFLOW
As illustrated in Figure EnConda-Bent Bench requires agents to diagnose and repair environment configuration errors. Specifically, when an error arises, the agent should (i) identify the step at which the failure occurs, (ii) analyze the precise error type, and (iii) plan an appropriate repair strategy. Building this, the agent should refine its feedback and corrective actions to ultimately produce an accurate shell script that fully configures the environment. For evaluation, we assess both (a) whether the environment is successfully built and executable, and (b) whether the agent's trajectory demonstrates correct error localization, reasoning, and feedback usage. Concretely, the task design and full pipeline comprise three components: input task instances, agent execution, and evaluation.
3

---

Published as a conference paper at ICLR 2026
Input Yask Instance
Agent Exccution
Process Level Evaluadion
Type
Ero Type ludgemeet Coveard ogg fpot eice
Saccaad Dhaggn oo Syrtor Now
Reponitrry
Image
Enror Desuripioo
Brror Descritionn Judgement
Goldon Answer JSON
conesee un maaatty
De conmaodd entair
"nedow:
woooee besncw supderea
Wroned berture "updene-a
Shell Scrpe
Shell Eealustion
yyou, twalt of bsage of stoe lena
Agent
"yeoe deserptont "The commondd to lenos videeved's froome browee "aaphreetts ecta pir dog"
Devilgaeee
Enchemment Schop Script
bocker
Stell
Myetiic aper Jua ctutal "Sydia coine
"Ax inprecial enplineetiile inpresiii-
croemmnn crett-gtt &
"pelder, rowerr "aan pio Nosel
Figure 2: An example of the overall workflow of our process-level environment configuration task.
Input Task Instances Each task instance includes: (1) Repository. We collect and filter a set of high-quality GitHub repositories to ensure reproducibility and moderate difficulty. To avoid ver- sion drift during evaluation, each repository is pinned to specific revision commit. (2) Dockerfile. Following EnvBench (Eliseeva et al.). we supply a base Docker image with minimal prerequisites (e.g.. Python, Conda). We run the agent to execute the environment configuration inside a Docker container. (3) README. Both humans and agents typically begin from the repository's 's README for environment configuration. Accordingly, each task instance includes the README as the pri- mary guide for the agent's execution. (4) Labeled golden answer JSON. For each task instance, we provide JSON tile to support evaluation, including the golden answers of error types, detailed error descriptions, candidate repair command sets, and the final correct command sequence.
Agent Execution For each instance, the agent leverages its planning abilities to devise sequence of environment configuration steps, guided by the provided README. Leveraging its perception capability, the agent carefully examines the README and repository to identify potential errors. When encountering errors, it employs feedback, and analytically reasons them in detail and formu- lates appropriate repair strategies. Drawing on its action skills, the agent implements the proposed fixes and generates shell script for the environment setup. After execution, we process the trajec- tory and extract error type judgments, repair commands, and the final shell script.
Process-Level Evaluation Given the judgments extracted from the trajectory and the final shell script, we conduct two complementary methods for process-level evaluation. For error diagnosis, we compare the predicted eиror types, descriptions, and fix suggestions with the gold-standard JSON and compute the corresponding metrics. For executability, we pull the Docker and repository. run the agent's 's shell script, and check whether it successfully builds the environment and passes the unit test. This evaluation suite yields a process-level assessment of the agent's capabilities for environment configuration, highlighting which capabilities of agents are weaker, which error categories are more easily detected, and which are more challenging.
3.2 DATA CONSTRUCTION
Repository Selection Although GitHub hosts numerous repositories, many do not meet the re- quirements for reliable environment configuration. If a repository is not reliable (e.g., due to a faulty README missing dependencies), error annotation becomes fabor-intensive and unreliable, and the resulting task may be prohibitively difficult. We therefore retain repositories that satisfy the fol- lowing criteria that indicate higher quality: at least 10 stars, over 1,000 commits, and more than 10 closed issues. Furthermore, we incorporate repositories from existing benchmarks that have under- gone strict human filtering and manual verification of environment setup, using them as the basis for subsequent error synthesis. Details about the repository selection are in Appendix A.
Error Synthesis After collecting high-quality repositories, we edit the READMEs to synthesize realistic, commonly encountered configuration errors. In fact, our initial plan does not involve syn-
4

---

Published as a conference paper at ICLR 2026
Reyos Selection
Error Synthesis
Automatic Validation
Data Filte Filtering
NIADME 2ist five Deceenton
Error Validness s
Centil
Clasdn
Entr lrae
Doctir
Calder strend
ADLOHE
Dopan Sanvy Indalltttninn
mage Systectrs
Run Validation
Kiewr
Manuel check
JSDY
Logiza Srioe
High-auiityy repeadorry
Feer Cemeittcceheneee bray
Variinn Campalibiiy Bror
Figure 3: The illustration of our overall pipeline benchmark construction.
thesizing errors. Instead, consider leveraging existing READMEs by decomposing them into executable steps (to assess whether each step runs correctly) or by annotating intrinsically erro- prone steps. However, this approach is highly labor-intensive. Without step just using tools to conduct evaluation of overall agent trajectories would over-rely on the models themselves, making it difficult extract key steps from long trajectories and explore specific ca- pabilities. In addition, each repository typically contains only a single README, and high-quality repositories are scarce, which constrains the number of available task instances.
annotations,
To address these issues, we treat each executable README as ground truth and inject errors. This enables scalable, automated task generation and supports process-level evaluation of planning, per- ception, feedback, and action during environment configuration. We define six canonical error cate- gories: Dependency Installation Error, Command Usage or Syntax Error, File Path or Missing File Error, Logical Order Error, Version Compatibility Error, and Other Miscellaneous Errors (see Ap- pendix A.2 for detailed definitions and examples). For each README, we prompt claude-4-sonnet and gemini-2.5-pro introduce two errors and produce structured JSON with the error type, de- scription, candidate fixes, and ground truth, while instructing minimal edits limited to the necessary lines, avoiding broad rewrites that could compromise README integrity (detailed settings in the Appendix A.3) Taking strictly filtered original READMES as reference, each case will yield a con- trolled error label, concrete description, and correct fix. From 323 repositories, produce 1,772 erroneous READMES, and each README contains exactly two injected errors.
Automatic Validation We then automatically val- idate the effectiveness of injected errors. An in- jected error considered effective if: (i) follow- ing the erroneous README, the environment setup fails, and (ii) after repairing this error, the setup pro- ceeds through the affected step. For each erroneous README, we use gpt g.t-4.1-min 1-mini to generate a shell script and execute it inside the provided Docker environment (details in Appendix A.4.. If the script succeeds in building the environment and passes the test, the corresponding error is regarded as invalid. We intentionally avoid stronger models this stage because they may implicitly "auto-fix", resulting in scripts that diverge from the erroneous README and thus undermine verifying the error's effective- ness.
(ei The stuterice readinss par raportooyyyy
The diso-ibutin
LLM-Assisted Filtering and Human Validation
Figure 4: Data statistics results.
Automated validation does not guarantee that the in- jected error is valid, we conduct second-pass validation to ensure (a) the error truly impacts configuration, (b) the error is explicitly manifested in the README, and (c) the categorization and proposed fixes correct. To reduce human effort, we employ gpt- 1-mini with predefined criteria (see details Appendix [A.5) to assess. The items that fail in this filtering removed. Human evaluators then review the remaining data under
5

---

Published as a conference paper at ICLR 2026
the same criteria. The agreement between the LLM filter and human judgments reaches 98.5%, sup- porting the feasibility this procedure. We ultimately preserve 1,230 valid erroneous READMEs, each containing two emrors. To diversify difficulty, we further split and merge errors to construct READMEs containing 1-10+ errors, yielding 4,201 READMEs and 9,471 total errors.
3.3 DATASET STATISTICS AND DATA EVALUATION
Data statisties. As described Section 3.2 we complete the benchmark construction. From 323 repositories. we construct 4,201 READMEs, averaging 13 per repository, the distribution shown in Figure aa). We further stratify difficulty by the number of injected errors per README, defining levels 1-10, shown in Figure 4b.. Most READMES fall into level 1 or level 2, which aligns with real-world practice: a README typically contains I-2 issues that hinder environment setup, but rarely many more. This ensures that task difficulty remains moderate for agents. Finally, in the error-type distribution each around 1,600 instances, contributing to balanced dataset. The "Other" category contains only 312 instances, which both preserves coverage completeness and discourages agents from overusing a catch-all class. Compared with other benchmarks in Table benchmark shows great advantages in evaluating the environment configuration capabilities of intelligent agents.
Figure 4c). the five standard error categories are comparable in count,
Table 1 Comparison of enviromment-oonfiuuaaion benchmarks.
Benchmark
Instances
Metric
Process-level
INSTALLAMATIC Milliken et al.
2025
40
Success build Success build & test
x
EXECUTIONAGENT Bouzenia & Pradel
2025
50
x
Success build & test,
EnvBench Eliseeva
994
missing imports Success build & test, Success build & test, error detection and fix
SetupBench Vergopoulos et al. 2025
93
X
EnConda-Bench
4,201
Data Evaluation. Since our task instances are constructed using LLM, these generated errors may not fully reflect the real-world task conditions, we further verify the data quality. Nevertheless, we aim further verify whether our generated data aligns with the difficulty level of real-wordh environment configuration tasks and reflects human cognitive patterns. To this end, we select ex- isting environment configuration benchmarks, whose instances are directly sourced from real-world code repositories, and establish criterion assess the difficulty level of both these benchmarks and our tasks. Difficulty is rated on a scale from (very easy) to 5 (very hard) by human experts (see Appendix B for details). The results shown in Table 2 indicate that the difficulty distribution and average scores of our tasks closely match those the real-world instances, demonstrating that our dataset possesses realistic applicability and high quality, To verify whether our LLM M.M--genrrtted errors reflect real-world distributions, we conducted blind test with three expert engineers. We mixed 50 generated errors with 50 real-world errors collected from GitHub. As shown in Table 3 the rate at which engineers misclassified our generated errors as "Real" (54.7%) is statistically com- parable to the identification rate of actual real-world errors (58.0%). This suggests that our synthetic errors are effectively indistinguishable from human-authores configuration failures.
Table 2: Difficulty scores of the benchmarks, where easy/med/hard corresponds to 1-2/3/4-5. For EnvBench and EnConda- -Bncch, sample 100 task instances.
Benchmark
Instances
Mean Score
Easy
Med
Hard
INSTALLAMATIC Millike et al. 2025
40
3.92
12
21
7
EXECUTIONAGENT [Bouzena (BoUzEe
Pradel 2025
50
3.85
16
26
(erBennch/Elieevaa EnvBench Eliseeva et al.
100
4.08
27
46
27
SelupBenent Vergopoulos al. 2025
93
3.78
34
47
12
EnConda-Bente
100
3.95
30
47
23

---

Published as a conference paper at ICLR 2026
Table 3: Human Evaluation of Error Realism (Confusion Matrix). high "Judged Real" rate for -LL--eeeerated errors indicates high realism.
Actual Source
Judged as "Real"
Judged as "AI"
Real-World Errors -.nM-Geeratedd (Ours)
42.0% 58.0%
54.7%
45.3%
3.4 EVALUATION SUITE DESIGN
After validating benchmark instances, we build an evaluation suite for environment configuration agents. Given the README and repository info, the agent plans and executes, producing trajectory from which we extract perception (error diagnoses), feedback (repairs), and a final shell script for planning and action. Because README may contain multiple errors, we compare the agent's s predicted error types/descriptions to the gold set and report precision, recall, and FI. We then match each predicted error description and fix to the gold answer, and use GPT-4. PPPT-4.1mmnn 1-mini assess consistency and evaluate accuracy. For executability, each script runs Docker container on fixed commit. A run is counted as pass only if the environment is successfully built, the test files execute correctly, and the process exits normally. Additionally, we propose an overall data- synthesis framework that automatically generates and verifies task instances from repositories, runs agents to collect trajectories, and produces evaluations for obtaining final post- and even pre-training trajectory data (more information in Appendix
judge to
4 EXPERIMENTS
4.1 BASELINES
We evaluate advanced LLMs and agent frameworks (detailed setting in Appendix D). For founda- tion models, we include representative open- and closed-source LLMs: GPT-4. 1 Claude-4-sonnec 202505144 14a Gemini2.5-Pra"" and DeepSeek-V3-0324 (Liu al. 2024a). For agent frameworks, consider three settings: (1) Zero-Shot: no additional agent scaffolding. The model receives the task instance, targeted prompting for environment setup, and pointers the repository's README and directory structure, along with evaluation configuration details (e.g.. Ubuntu version), and directly produces the setup. (2) Code Agents: specialized
fetware-engineering agents that benefit from tool use and planning, often trained or optimized on large SWE workloads. We evaluate OpenHands [Wang et al.) and SWE-Agent (Yang et al.) 22024) both strong performers on SWE-bench-style tasks. (3) Environment-setuu Agents: frameworks tailored to environment configuration, includ- ing INSTALLAMATIC (Milliken et al. 2025)) and Repo2Run (Hu et al. 2025).
4.2 MAIN RESULTS
We evaluate agents the environment setup shown in Table Zero-shot LLMs exhibit high recall but low precision in error typing (c.g., GPT-4.1: Rec. score of 90.6 vs Pre. score of 33.4, F1 score of 48.8), with weak fix suggestions and poor end-to-end success performance. This in- dicates broad but noisy error perception and limited agentic intelligence. In contrast, code agents significantly improve error perception and corresponding repair feedback, for instance, OpenHands DeepSeek- V3 achieves Fl score of 58.7 with a description accuracy of 51.9. However, the action and feedback abilities are still under exploration. with fix ACC. score of 33.8 and Pass score of 9.1. Environment configuration agents deliver the largest end-to end-to-end gains, showing better capabil- ity to utilize perception and feedback with execution action. We observe that Repo2Run + Claude-4 reaches Fl score of 60.6, description accuracy of 52.2, fix accuracy of 47.3, and Pass@1 score of 22.9, underscoring the value of environment probing perception and failure handling feedback. Nonetheless, the persistent gap between description and fix accuracy, and between fix accuracy and
hutpss//opeaiicoomm/mddeggg44/
7

---

Published as a conference paper at ICLR 2026
Table 4: Evaluation results across agents and LLMs. Bold: Optimal performance for every setting.
Agent Capability
Perception
Feedback
Feedback Action
Planning and Action
Framework
LLM
Error type
Error description
Fix suggestion
Execution
Pre.
Rec.
F1
ACC.
ACC.
Pss
Base Model
GPT-4.1
33.4
39.6
1.5
Claude-4
37.1
80.6
50.8
45.1
28.5
3.1
Zero-Shot
Gemini2.5-pro
35.2
77.8
48.5
45.2
25.2
1.8
DeepSeek-V2 V3
33.2
65.8
44.2
39,7
22.3
3.3
GPT-5 Codex
40.5
88.2
55.5
49.8
32.4
Agent
GPT-4.1
43.7
83.2
55.3
49,8
30.7
7.2
Claude-4
46.4
85.6
58.2
52.4
34,5
9.4
SWE-Agent
Gemini2.5-pe
45.1
56.8
50.2
7.8
DeepSeek-V2 V3
41.2
70.3
$1.9
44,5
27.8
7.4
Codex
11.5
GPT-411
42.5
53.2
46.0
29.1
8.5
Claude-4
48.0
87.5
60.1
54.2
36.1
10.6
OpenHands
Gemini2.5 pro
45.2
57.5
8.5
DeepSock-V. V3
46.7
93.6
58.7
51.9
33.8
9.1
Environment Configuration Agent
GPT-4.1
37.5
48.9
45.3
5.6
Claude-4
41.9
75.3
53.8
50.7
34.1
7.9
INSTALLAMATIC
Gemini2.5ppe S-pro
39.0
72.1
50.6
47.5
30.8
6.4
DeepSeck V3
40.7
76.8
53.2
49.3
32.5
7.1
GPT4.1
44.2
72.3
54.8
48.5
38.6
14.1
Repo2Run
Sopo
Claude-4
48.6
49.5
79.3
60.1
54,2
52.2
45.6
22,9
17.8
DeepSeek-V3 V3
46.3
74.2
56.8
44.6
41.2
16.2
Gemini CL.I
Gemini 2.5
13.2
Pass@ 1, reveals bottlenecks of the agent translating correct feedback into robust and valid execu- tion actions. To further benchmark the inherent difficulty of environment configuration, conduct an experiment using the original, correct READMES in Appendix E..)] Our process-level evalua- tion of agent trajectories is therefore crucial, which guides targeted improvements rather than only pass rate, which highlights that we need to make better use of feedback information obtained from interactions with the environment, truly enhance the execution capability.
(FT,) Cava De0me//
180
300
1809
300
(a) zero-shot
(b) openhands
(c) repo2run
Figure 5: The statistics of the enror numbers of the golden label and the model's prediction.
4.3 ERROR TYPE JUDGMENT ANALYSIS
For each eтror type, we observe that the total number of predicted errors exceeds the ground truth shown in F[sure more detail in Appendix E.I). indicating a conservative strategy with stricter checks. Sensitivity to specific error types is uneven: most models tend to overpredict the El cate- gory. There are also model-specific differences. For example, DeepSeek- V3 predicts very few E6 than the ground-truth labels, suggesting under-detection for that error type. Fi-
cases, fewer nally, many cases are grouped into the catch-all "other" category. making E8 the highest or second- highest category. This is undesirable users expect precise, actionable diagnoses rather than vague
8

---

Published as a conference paper at ICLR 2026
GPT-4.1
Claude-4
Gemini2.55pp
DeepseekV3
E
Zero-Shot
OpenHands
INSTALLAMATIC
Figure 6: The illustrations of the enror type judgment performance the Fl score.
classifications. Consistent with the above, the overuse of "other" leads to a markedly low FI score on E8 as shown in Figure II to concrete types are incorrectly assigned to E8, inflating false positives for E8 and depressing recall for the true types. This hedging behavior hampers the practical value error perception and feed- back, degrading downstream planning and action. Beyond this, models show small but systematic performance differences across specific types. For example, the results show stronger detection for command usage and syntax error E2 but weaker for categories like file path B4 that often de- pend on the agent's system-level understanding of the entire repository and the interaction with the environment.
see see more detail Appendix [.2]. Many instances that should belong
4.4 EFFICIENCY ANALYSIS
We investigate the relationship between output tokens and performance to provide comprehensive assessment of model efficiency. As shown the Figure 7 (see more detail in Appendix E.3). for the accuracy of error descriptions, most models exhibit clear upward trend the number of output tokens increases. In contrast, for the Pass@) metric, allocating more tokens does not consistently yield improvements. For example, zero-shot Claude-4 uses three times many tokens as zero-shot DeepSeek- V3 yet improves performance by only about 0.2. Notably, in certain agent frameworks (e.g.. Repo2Run), performance scales more favorably with larger token budgets, indicating compar- atively higher efficiency.
Claude 4
Gamini2.5ppo
Daepsseleess swer egentgeettee GPT-4.1 swe-agant-Claude-
ower wewett-teemmeeeeeeee Liemini bro hands Deepseekv3 repo2rur repo2run-Gemhi2 5emi
swe wewean--nngggeeeee Ueepseekv
openhands 4P1-43
openhands Gaudee44 @ Bpemhand--Gmiiitbber
Installmatic-GPP
nstollmatic Claude-4
Installmatic-Gen 2.
Dnstslmaaee-eeppeela
repo2run-GPT-42
repozun-DspsoeeN
30
1000
120
2003
2200
300
340
1200 1400
2000
1200
2400
300
Figure 7: The illustration the statistics of output token and model performance.
4.5 CASE STUDY
Beyond the above analyses, we further explore concrete cases to validate the effectiveness of our framework and support the claims. As shown in Figure observe that models sometimes

---

Published as a conference paper at ICLR 2026
Consistency Error Type and Description
Fix Sugg estion Evaluation
Model output:
Model output:
suggestion:
Errar description: the command pipd install androldviewcllent instals the package fromthe PyPL For uaers who want to deeve should irstall clored repositor.
Lyper
Error, description: install from PyPl, the commend should 1pip3 install undroidviewdiiee -uperadle: To ins tall from a locallyy doned repository, use "pl install projectroct directory.
onswer:
Golden onswer:
Fix suggestion: Change the package name in the command from "culebra" endroickievelient
Error, descriittion: The insta lation command pip install androidvie www.ewewweetttteee uupdatee incorrect. The "-updatee flag is not valld upgrading paikage;
type:
Text providess derailed installation commands, while Test 2 focuses on from 'oulcbraa' "androivviovelllen:
Judg e
uppradde".
changing the the core solution-using correct package name-io the samme."
Judge the Error Correctly but Fail to Fix
output:
Error_iye::E of operations virtual environment packago installation incorred.
suggestioon First, virtualenv 'source anv/bn/activvat" then instail requiramems, debug txt.
Set_up shi A Create and octivate virtual enviromment source enyytiny activate/ # Upgrode pip pip instal uuprrade pip Install Python dependencles irstall requirements
Figure 8: We select some typical cases and provide an analysis of observed phenomena.
rectly judge the error type without actually locating the error commands, which undermines sub- sequent feedback. We also find that some proposed fixes differ from the golden solution, but evaluation protocol can accommodate such variability, focusing on whether the issue is resolved rather than on exact match, thereby supporting the soundness our methodology. Finally, we ob- serve a relatively low pass rate. One key reason is that a single README may contain multiple errors, and models often fail to fix all of them. Moreover, during execution, a model may correctly diagnose an error and suggest an appropriate fix command, but fail to apply the feedback in the shell scripts used for environment setup, may introduce new faults. Notably, such errors are likely to arise during multi-round agentic iteration, extensive edits can introduce additional mistakes.
5 CONCLUSION
Environment configuration remains decisive bottleneck for SWE agents. Beyond end-to-end benchmarks, we introduce a benchmark that focuses on process-level evaluation, including plan- ning, perception-driven diagnosis, feedback-driven repair, and final actions. By injecting realistic errors into READMES and validating their effects in Docker, the automatic data framework we propose generates scalable, high-quality task instances for evaluation and rich trajectories that en- able training. We conduct experiments on advanced agents, and observe that in error perception, the agent demonstrates certain level of capability, but tends to classify uncertain error types as "others". However, when comes to specific repair actions, the agent shows limited performance. We attribute this limitation to the lack of effective interaction and feedback mechanisms within the agent. Although can identify errors, it struggles to plan the feedback information and interact with the real-world environment provide better repair actions. In the future, enhancing the agent's ability to strengthen its interaction with the environment will be an important research direction.
ACKNOWLEDGMENTS
This work was supported in part by the New Generation Artificial Intelligence-Nationn Science and Technology Major Project (2025ZD0123003), the National Natural Science Foundation of China Enterprise Innovation and Development Joint Fund (Artificial Intelligence Field) Key Sup- port Projects (U25B2072), the Basic Research Fund of Shenzhen City JCYJ202408131 12009013.. and The Major Key Project of PCL (Grant No. PCL2025A17).
ETHICS STATEMENT
We introduce a novel benchmark, Ceconda--Bench incorporating a thorough description of repository collection, error synthesis, data validation, and filtering. We emphasize that the dataset's
10

---

Published as a conference paper at ICLR 2026
creation adheres strictly to ethical guidelines. We make sure that all repositories with their respective licenses. Great care has been taken to uphold ethical standards in the dataset, employing anonymization, desensitization, and data cleaning. The samples pose risk to public welfare. For all data sourced from these websites, we obtain permission for data usage. Hence, the innovative research directions and tasks proposed are ethically harmless to society.
comply
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint 10Xiv2208.07720 2021.
Islem Bouzenia and Michael Pradel. You name it, i run it: An Ilm agent to execute tests of arbitrary projects. Proceedings of the ACM on Software Engineering, 20SSTA)):054--0076 2025.
Li-Guo Chen, Zheng Xiao, Yi-Jiang Xu, Rui-Chuan An, Xin Wang. Yang-Ning Li, Ying-Hui Li, Yi- Dong Wang, Zheng-Ran Zeng, Qing Gao, et al. Coderankeval: Benchmarking and analyzing 11m performance for code ranking. Journal of Computer Science and Technology. 40(5): 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 2rXIv:2107.03374. 2021.
Jiangshu Du Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, et al. Lims assist nlp researchers: Critique paper (meta-) reviewing. In Proceedings of the 2024 conference empirical methods in natural language processing, pp. 5081-5099, 2024.
Alcksandra Elisceva, Alexander Kovrigin, Ilia Kholkin, Egor Bogomolov, and Yaroslav Zharov. Envbench: A benchmark for automated environment setup. In ICLR 2025 Third Workshop Deep Learning for Code.
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engi- neering (ICSE-FOSE)) pp. 31-53. IEEE, 2023.
Martin Gruber and Gordon Fraser. Flapy: Mining flaky python tests at scale. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)) pp. 127-131. IEEE, 2023.
Xinshuai Guo, Jiayi Kuang, Linyue Pan, Yinghui Li, Yangning Li, Hai-Tao Zheng, Ying Shen, Di Yin, and Xing Sun. Evoconfig: Self-evolving multi-agent systems for efficient autonomous environment configuration. arXiv preprint ar arXiv:2601.16489, 2026.
Junda Hc, Christoph Treude, and David Lo. Llm-based multi-agent systems for software engineer- ing: Literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology, 34(5):1-30, 2025.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace Hc, Dawn Song, et al. Measuring coding challenge competence with apps. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Ruida Hu, Chao Peng, Xinchen Wang, and Cuiyun Gao. An llm-based agent for reliable docker environment configuration. arXiv preprint 27X11:2502.13681, 2025.
Shulin Huang, Shirong Ma, Yinghui Li, Mengzuo Huang, Wuhe Zou, Weidong Zhang, and Haitao Zheng. Lateval: An interactive ilms evaluation benchmark with incomplete information from lateral thinking puzzles. In Proceedings of the 2024 Joint International Conference Com- putational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 10186- 10197, 2024.
11

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference Learning Representations.
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2E: turning any github repository into programming agent environment. In Forty-first International Con- ference on Machine Leurning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL htttps::/openreviieee
net htttt//oorrmmma=kkkpppeapp
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, net / forum?id?
May 7-11, 2024. OpenReview.net, 2024. URL pattps://openreet VTF VTF8 8yNQ6666
Jiayi Kuang, Haojing Huang. Yinghui Li, Xinnian Liang, Zhikun Xu, Yangning Li, Xiaoyu Tan, Chao Qu, Meishan Zhang, Ying Shen, and Philip S. Yu. Atomic thinking of Ilms: Decoupling and exploring mathematical reasoning abilities. CoRR, 0bs/2509.257255 2025a. doi: 10.48550/
ARXIV
2509. 25725
Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. Natural language understanding and inference with MLLM in visual question answering: A survey, ACM Comput. Surv., 57()):10:11190:36 2025b. doi: 10.1145/371 680.
pecttps://oo..oom
1115/ 1600
Yangning Li, Yinghui Li, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinran Zheng, Hui Wang, Hai-Tao Zheng, Philip S Yu, Fei Huang, et al. Benchmarking multimodal retrieval augmented generation with dynamic vqa dataset and self-adaptive planning agent. arXiv preprint arX092241..02937, 2024a.
Yangning Li, Tingwei Lu, Yinghui Li, Yankai Chen, Wei-Chieh Huang, Wenhao Jiang, Hui Wang, Hai-Tao Zheng, and Philip S Yu. Teaching according to talents! instruction tuning lims with competence-an aware curriculum learning. guistics: EMNLP 2025, pp. 1 11724-11741, 2025a.
Findings of the Association for Computational Lin-
Yangning Li, Weizhi Zhang, Yuyao Yang. Wei-Chieh Huang, Yaozu Wu, Junyu Luo, Yuanchen Bei, Henry Peng Zou, Xiao Luo, Yusheng Zhao, et al. Towards agentic rag with deep reasoning: A survey of rag-reasoning systems in llms. arXiv preprint a5Xiv:2507 250.00757 2, 2025b.
Yinghui Li, Qingyu Zhou, Yangning Li, Zhongli Li, Ruiyang Liu, Rongyi Sun, Zizhen Wang, Chao Li, Yunbo Cao, and Hai-Tao Zheng. The past mistake is the future wisdom: Error-driven con- trastive probability optimization for chinese spell checking. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 3202-3213, 2022.2022.......
Yinghui Li, Haojing Huang, Shirong Ma, Yong Jiang, Yangning Li, Feng Zhou, Hai-Tao Zheng, and Qingyu Zhou. On the (in) effectiveness of large language models for chinese text correction. arXiv preprint arXiv:2307.09000 2023.
Yinghui Li, Zishan Xu, Shaoshen Chen, Haojing Huang, Yangning Li, Shirong Ma Yong Jiang, Zhongli Li, Qingyu Zhou, Hai-Tao Zheng, et al. Towards world writing assistance: A chinese character checking benchmark with faked and misspelled characters. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I; Long Papers), pp. 2024b.
Yinghui Li, Qingyu Zhou, Yuanzhen Luo, Shirong Ma, Yangning Li, Hai-Tao Zheng, Xuming Hu, and Philip S Yu. When lims meet cumning texts: A fallacy understanding benchmark for large lan- guage models. Advances in Neural Information Processing Systems, 37:1 12:1333331122488 2024c.
Yinghui Li, Haojing Huang, Jiayi Kuang, Yangning Li, Shu-Yu Guo. Chao Qu, Xiaoyu Tan, Hai-Tao Zheng, Ying Shen, and Philip S. Yu. Refine knowledge of large language mod- els via adaptive contrastive learning. In The Thirteenth International Conference Learn- ing Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025c. URL https: htttps://openrewe net fert://orumtii fott://oommmmid-HjRRRI1
12

---

Published as a conference paper at ICLR 2026
Yinghui Li, Jiayi Kuang, Haojing Huang, Zhikun Xu, Xinnian Liang. Yi Yu, Wenlian Lu, Yangn- ing Li. Xiaoyu Tan, Chao Qu, Ying Shen, Hai-Tao Zheng, and Philip S. Yu. One example shown. many concepts known! sounterexample-drivee
conceptual reasoning in mathematical Ims.. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp. Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR OpenReview.net, 2025d. URL https: //proceedings. mlx prwss/eeess...... V227 1 5ax. html
Yinghui Li, Shang Qin, Jingheng Ye, Haojing Huang, Yangning Li, Shu- Guo, Libo Qin, Xuming Hu, Wenhao Jiang, Hai-Tao Zheng, et al. Rethinking the roles of large language models in chinese grammatical error correction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics ( (Volume 6: Industry Track), pp. 553-567, 2025e.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL
https://cpeereew/
net wow.oooomddppp11000000
Tingwei Lu, Yangning Li, Liyuan Wang, Binghuai Lin, Jiwei Tang, Qingsong Lv, Wanshi Xu, Hai- Tao Zheng, Yinghui Li, Xin Su, et al. Lsr-mets: Alleviating long range dependency in code generation. arXiv preprint arXiv22504.07433 2025a.
Tingwei Lu, Yangning Li, Liyuan Wang, Binghuai Lin, Jiwei Tang, Wanshi Xu, Hai-Tao Zheng, Yinghui Li, Bingxu An, Zhao Wei, et al. From token to line: Enhancing code generation with long-term perspective. arXiv e-prints, pp. arXiv-2504, 2025b.
Chunyu Miao, Henry Peng Zou, Yangning Li, Yankai Chen, Yibo Wang, Fangxin Wang, Yifan Li, Wooseong Yang. Bowei He, Xinni Zhang, et al. Recode-h: A benchmark for research code development with interactive human feedback. arXiv preprint arXiv:2510.06186. 2025.
Louis Milliken, Sungmin Kang, and Shin Yoo. Beyond pip install: Evaluating Ilm agents for the automated installation of python projects. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 1-11. IEEE, 2025.
Youtu-LLM Team. Youtu-llm: Unlocking the native agentic potential for lightweight large language models. CoRR, abs/2512.24618,, 2025. doi: 1044550/ARXXVV2512244111 URL https:
doi www.orgg11..0 48150/44555/04 / 2512. 2461
Konstantinos Vergopoulos, Mark Niklas Müller, and Martin Vechev. Automated benchmark gener- ation for repository-level coding tasks. arXiv preprint arXiv:2503.07701, 2025.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Repre- sentations.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint 10Xiv:2407.16741,3 2024.
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01480 2024.
Zhikun Xu, Yinghui Li, Ruixue Ding, Xinyu Wang, Boli Chen, Yong Jiang, Haitao Zheng, Wen- lian Lu, Pengjum Xie, and Fei Huang. Let Ilms take the latest challenges! a chinese dynamic question answering benchmark. In Proceedings of the 31st International Conference on Compu- fational Linguistics, pp. 10435-10448, 2025.
13

---

Published as a conference paper at ICLR 2026
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao. Karthik Narasimhan, and Ofir Press. Swe-agent: Agent computer interfaces enable software engineering language models. arXiv preprint arXiv.2005.1579,, 2024.
John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. arXiv preprint urXiv:2504.22 21798, 2025.
Jingheng Ye, Yong Jiang, Xiaobin Wang, Yinghui Li, Yangning Li, Pengjun Xie, and Fei Huang. Productagent: Benchmarking conversational product search agent with asking clarification ques- tions. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Pro- cessing: Industry Track, pp. 383-398, 2025.
Tianyu Yu, Chengyue Jiang. Chao Lou, Shen Huang, Xiaobin Wang, Wei Liu, Jiong Cai, Yangning Li, Yinghui Li, Kewei Tu, et al. Seggpt: An out-of-the-box large language model for open do- main sequence understanding.
Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19458-19467, 2024.
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13643-13658, 2024a.
Weizhi Zhang. Yuanchen Bei, Liangwei Yang, Henry Peng Zou, Peilin Zhou, Aiwei Liu, Yinghui Li, Hao Chen, Jianling Wang. Yu Wang, et al. Cold-start recommendation towards the era large lan- guage models (Ilms): A comprehensive survey and roadmap. arXiv preprint wrXiv:2501.01944 2025.
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT Intermational Symposium on Software Testing and Analysis, pp. 1592-1604, 2024b.
A DATA CONSTRUCTION
A. REPOSITORY SELECTION
We focused on Python repositories given their dominance in the ecosystem (over 21M repositories on GitHub). To ensure benchmark quality, we conduct:
Selection Criteria: We adopted the criteria of 10+ stars, 1000+ commits, and 10+ closed issues based on established practices in software engineering research. These metries serve as proxies for repository maturity, active maintenance, and community relevance, ensuring we benchmark on meaningful projects rather than toy examples.
2. Filtering Pipeline: We strictly filtered for repositories with precise READMES located in the root directory to minimize ambiguity.
3. Rigorous Verification: The "manual checking" involved *3 expert engineers. They manu- ally attempted to set up the environment following the original READMES. Any repository that failed to build or had ambiguous instructions was strictly removed. This ensures that every Ground Truth in our benchmark is guaranteed to be executable.
Beyond the filtering criteria outlined in Section 32 we selected repositories primarily from ex- isting, human-validated, and pre-filtered benchmarks to ensure the originals are reliably buildable. This choice is crucial because our error synthesis procedure treats the unmodified README as the ground truth; consequently, the README must be correct and actionable. To that end, we en- gaged professional annotators perform manual environment setup, removing repositories whose READMEs were themselves erroneous or whose dependencies were incomplete, thereby preserv- ing high-quality baselines. We further verified that all selected repositories carry licenses permitting non-restrictive research use: most adopt permissive licenses (e.g.. BSD, MIT, Apache), while the remainder fall under copyleft licenses (e.g., GPL.), which are compatible with our intended research
14

---

Published as a conference paper at ICLR 2026
scenarios. We also reviewed repositories under custom licenses and confirmed their suitability for the uses contemplated in this study. A promising direction for future work is to broaden repository coverage to include codebases written in wider range of programming languages, extending the evaluation to environment configuration tasks across more language ecosystems.
A.2 ERROR TYPE DEFINITION
Taxonomy of Common Anvironnent-Confguraaa Errors To ensure
taxonomy
ers real-world failure modes, we adopted a multi-source derivation process. First, we surveyed existing empirical studies on configuration errors identify historical patterns. Second, mined common issues from developer communities like StackOverflow and GitHub. Finally, we refined the taxonomy through interviews with five expert engineers (10+ years of experience), merging cor- related types and removing non-typical ones. This process resulted in the six defined error types, with an "Other" category to capture long-tail distributions.
Guided by failure modes frequently encountered when configuring execution environments from software repositories, we define six-type taxonomy intended to cover the vast majority of practical issues while maintaining clear, operational boundaries between categories. The scheme is designed to support consequent error generation, and the set comprehensive routine evaluation and error synthesis.
Table Error taxonomy for repository environment configuration. Identifiers are retained from our internal schema and are non-contiguous by design.
ID
Name
Definition
E1
Dependency Installation Error
Errors related to system or Python dependency installation steps, including missing dependencies, unnecessary dependencies, or version errors.
E2
Command Usage or Syntax Error
Errors caused by incorrect commands, invalid parameters, or improper syntax causing execution failure.
File Path or Missing File Error
Errors where dependency file paths are incorrect or referenced files do not exist.
E6
Logical Order Error
Errors caused by incorrect execution order of installation steps, such as installing pip dependencies before creating a virtual environment.
E7
Version Compatibility
Errors caused by unspecified Python or dependency versions, version conflicts, incompatibilities.
Error
E8
Other Miscellaneous Errors
Other uncategorized errors, such as messy formatting, missing critical explanations, or unclear descriptions.
Examples
E1 Typical symptoms include package-not-found errors (e.g., 404 indexes or channels), missing system libraries (e.g., OpenSSL, GCC toolchains), or failing installers (apt/conda/pip).
E2 Often manifests as immediate termination with usage messages or exit code 2, invalid or deprecated flags, shell quoting/escaping errors, invoking commands from the wrong work- ing directory.
E4 Common signals are "No such file or directory," misspelled filenames, incorrect relative paths, or reliance on artifacts not checked into version control.
Symptoms include installing into an inactive environment, failing to source activation scripts before use, or attempting builds before installing toolchains.
15

---

Published as a conference paper at ICLR 2026
E7 Presents solver conflicts, runtime ImportErrors due to ABI/GLIBC/CUDA mis- matches, or subtle behavior differences across Python or library minor versions.
E8 Catch-all for issues outside the above scopes, such incomplete ambiguous instruc- tions, inconsistent naming, or extraneous formatting that obscures required steps; curators should use this sparingly and prefer specific categories when feasible.
A.3 ERROR GENERATION
We use Claude-4-sonnct and Gemini 2.5-pro together to generate errors. We imput the original README file, the desired types of errors to insert, the number of errors per README, the desired number output README files, and a list of error type definitions, and then have them output the following:
Erroneous README (Markdown or RST file)
List (JSON format):
Readme id
Error type
Error description (natural language)
Candidate fix suggestions (operational tips)
-gronnd truth of fix answer (golden answer)
Here is the instruction for error generation:
Prompt for Error Synthesis
You are professional environment setup engineer and the README text modifier.
You receive the following inputs:
correct README markdown file located path: treadme_path) 1ist of error types to inject, chosen from the follewing:
(error_types_str)
Number of errors to insert per README:
Number of distinct erroneous markdown files cutput:
Inum readmes).
Task:
Please generate (num_roadmess) distinct erroncous README markdown files by minimally modifying the original README, Keep as much of the original README the same as possible, and only inject errors by changing or adding very small number of sentences (usually or 2 to introduce the requested error types.
For each generated erroneous README file, output the following parts:
1. The full erroneous README markdown text, preserving all criginal formatting and content except the minimal injected errors.
2. A JSON metadata block describing all inserted errors, with the structure:
16

---

Published as a conference paper at ICLR 2026
Prompt for Error Synthesis
Jjson
{{
"readme": "readme_(( ((ndexx)),
"errors": 11
"error serro__type:: "<error type code e.g., El>",
"error "error_descti description": "<brief description of the introduced error in this README>", correcionaandddtte::: <candidate fix #1>",
<candidate fix #2>*
"golden_answer", "<precise correction fix the errer>"
1)
Please output results for all (num_readmes) files in this format, separated by a line of exactly three dashes (---):
Erroneous README [[index))
<full markdown text>
'ijson
"readme": "readme_ (index))
"errors":
})
Additional notes:
Ensure JSON blocks are enclosed exactly by triple backticks and labeled as json.
Maintain overall readability and realistic style.
Output the markdown and JSON blocks exactly as specified, with extra text or commentary.
The original README content, not injected errors, must remain unchanged.
End of instructions.
A.4 AUTOMATIC VALIDATION
After generating the erroneous README file, as in the formal evaluation process, we used GPT- 4. 4.i-mini to generate shell script based the given README and the directory structure of the repository. This script was then run within Docker to verify whether the environment could be successfully built. We instructed the model to strictly follow the instructions in the README when generating the script, without making any modifications or corrections, to ensure accurate evaluation results. The specific prompt used is follows:
17

---

Published as a conference paper at ICLR 2026
Prompl for Generating Shell for Validation
You are given the README of Pythen project and the directery structure this repository. Please output ONLY one complete bash shell script that automates environment setup for this project on Ubuntu 22 22.04. Do not include any explanations or markdown, just the script content.
Requi rements for the script:
Create and activate new Python virtual environmen using Kiniconda inside the project directory (e.g., ./env ./venv name).
Install any required system packages and Python packages inferred from the README.
Install Python dependencies (infer from README; if requirements.txt/ pyproject.tom is expected, handle both).
Install the project in editable mede (pip install if applicable.
You must run the test suite to verify setup (pytest, or whatever is indicated/ iadectt///iff)))))))
Start with shebang (#!/usr/bin/env bash) and use 'set -euo pipefail'.
Prompt for Generating Shell for Validation
Attention: You don't t have to ensure that the generated Shell script will definitely configure successfully, but **make sure that the Shell script is totally consistent with the contents of the README**. Do not make any changes!
Assume you execute all commands from the project root directory.
A.5 LLM AS JUDGE TO VALIDATE
We further annotate each enror using GPT-4.1 1-mii ensure that the final generated errors valid. We input the error's ' README, JSON file containing the error annotation, the error definition, and the basic Dockerfile design for environment configuration. We ask GPT-4.1-mm
to check the following: (1) Is the error type classification accurate? If not, suggest a corrected type. (2) Is this error described in the README? (3) Is this error valid? We consider error valid only if truly prevents step in the environment configuration from succeeding, and its fix allows the configuration execute correctly. (4) Is the standard solution for this error correct? After this screening, discard all READMEs corresponding to invalid errors. The specific prompt is as follows:
Prompt for LLM as Judge to Filter the Valid Error README
SYSTEM PROMPT
You are a strict checker for environment-setup Output only N lines of minified JSON objects, one per error, exactly matching the schema below. No explanations, no trailing commas.
prose, code fences, no blank lines,
DEVELOPER PROMPT Task:
RRVELOPPER_PROFT
For each
errors_json errors (in order), decide:
error perp__tte__ju_meeee.ee does error_type match error description per definitions? If salse, set seor__typ__oddice to the single best- matching type; else "",
error_reaaine_or: does the README text actually contain the erroneous content (wrong or a required omission that would be needed to succeed?
wrong venv order)
answer juswer_jjdgmmett would golden_answer answer remove the root cause of this error?
18

---

Published as a conference paper at ICLR 2026
Prompt for L.M as Judge to Filter the Valid Error README
atror_valid; will this break setup until fixed under the given environment
assumptions and success criteria (install command errors, dependency resolution
required Library, unusable envirenment due to verv misuse)? Fadse for cosmetic/aptional issues or it uncertain. Use only:
conflict, ImportError
readme_text
error_definitions
errors_jscn
environment assumptions/sucoess criteria
Be conservative if context is missing, Preserve input order.
Output schema (one line per error) "rodn ind____exs_______" string", "error_reacie_crr :trreerrr"rrrulleeoee fone/falle false, "answer_juuer susewwrrjddgment :truel false, "error_valid falrue]false) Rules
index>",
"error_.yype_.udmmmtttt.e :true false, "error_tyoo_oodi"""" or empty
error error_type__ ttpe. juegment::: compare strictly with provided definitions; if multiple types łit, choose the most specific root-cause category. ertor_readie__rr: true only if the README substantiates the error; for omissions, true only if the missing step/dependency is clearly required to succeed.
answer_judgment golden_ansken answer must directly fix the errer; being in correation_candidatee error_valid: true for invalic flags/eptions, incompatible/blooker pins, miseing essential deps, wrong Saeaandsspaths///ee /mooules, critical veny misuse/order; false for stylistic advice or speculative failures.
sufficient.
Expected model output (one line per error, in order) ["id":"<reacme" id>#0", "error type poooe__udmment""eee "error_ type modišy":"" "error_readme er" :true "answer juugmmett""""""" true, "error valid"
B DIFFICULTY RATE
Because task instances incorporate error injection into existing README files using LLM, wanted to verify whether these generated task instances reflect the challenges encountered in real- world environment configuration, thus possessing characteristies similar to real-world environment setup tasks. Therefore, we selected several benchmarks that collect task instances directly from real- world code repositories. including INSTALLAMATIC Bench, ExecutionAgent Bench. EnvBench, and SetupBench, and compared their difficulty scores with our benchmark. For EnvBench and our benchmark, we sampled 100 task instances each; for the other benchmarks, we scored all available instances. By analyzing the distribution of difficulty scores, we can assess whether the methods used our benchmark align with those used
wellworld task instances. Specifically, we used 1-5 scale, where I is very easy, 2 is easy, 3 is moderate, 4 is difficult, and 5 is very difficult. We invited professional annotators to score the selected task instances, considering factors such as the clarity and completeness of the instructions in the README. whether the commands execute directly, whether additional files or pages need to be consulted, and the number dependencies that need be considered.
C TRAJECTORY TRAINING DATA GENRATION FRAMEWORK
We have automated the process generating environment configuration task instances and de- signed comprehensive evaluation suite that allows agents to execute these task instances, capture
19

---

Published as a conference paper at ICLR 2026
1. Repository Selection
Criteria
Github
10 Star
Execution Check
High-quality Repository
Repository
1000
Error Readme Generation
Type Definition
X
Dependency Instalaation Error
README Repository
Command Usage Syytax firoo
P*
LLMs Synthesize
Error README Instances
File Path Missi Error
Emrrr
Logioal Order Enror
Vension Compod billty Enror
Other Misoolancous Enrra
3. Data Validation
README Instences
Genera to Setup shell
Valid Task Instances
LLM Filter
Valid Task Instances
Execution Dockerfile
Trajectory Generation
Valid Task Instances
Repository
Agent Attemptatt configure envrromment
Execution Trajectory
Eva luate
Trajectory dreing
README
Dockerfile
Figure 9: The overall data synthesis framework of trajectory training data generation.
their execution trajectories, and perform evaluations. Therefore, we can build complete synthetic data framework based on this generate synthetic trajectory data representing both successful and failed agent executions of these environment configuration tasks. This can efficiently produce large amounts of trajectory data for model fine-tuning large-scale pre-training, provided that a sufficient quantity of high-quality original repository data available. The specific data generation process is illustrated in the Figure 9
D EXPERIMENT SETTINGS
When the agent is tasked with configuring the environment, we provide instructions including the repository directory information, README information, and the basic environment requirements. We also outline a feasible workflow for the agent to follow, ensuring that the entire environment configuration adheres to the specified standards. We require the agent to explicitly identify any errors during execution, and to perform unit tests after completing the environment configuration to verify its success. The specific prompt is as follows:
20

---

Published as a conference paper at ICLR 2026
Prompl for Agent to Execute the Environment Configaration
You are an expert Python envi ronment setup assistant Your task analyze README files, detect potential errors envircnment setup instructions, and provide comprehensive solutions.
Given README tile, you shoula:
1. **Error Detection and Analysis**: Carefully analyze the README for potential errors in environment setup instructions, including: El: Dependency Installation Error (missing dependencies, unnecessary dependencies, E2: Command Usage or Syntax Error (incorrect commands, nvalid parameters, or improper syntax) E4: File Path or Missing File Error (incorrect dependency file paths referenced files that do not exist)
versien errors)
E6: Logical Order Error (incorrect execution order installation steps)
87: Version Compatibility Error (unspecified Python or dependency versions, version conflicts, or incompatibilitiess R8: Other Miscellaneon krrors (messy formatting, missing critical explanationsans or unclear descriptions)
2. ++Error Analysis Output**: First, output JSON object containing your error analysis with the following structure: jsooo {
"detected_errors"
{ "error_type": "El |4
"error_descriptionn "Detailed description errer found",
"fix_euggestion"; "Specific suggestion on how to fix this error"
}]
}}
...
3. .<Environment Setup Script.:: After the error analysis, create a comprehensive shell script that:
Fixes all detected
Sets up the environment correctly Handles common Python environment sctup patterns (pip, conda, pootry, etc.)
Includes error handling and verification steps
Your response should contain:
1. The JSON error analysis (wrapped in "Ysson code blocks)
2. The corrected shell script (wrapped in baaahh code blecks)
l'echnical requi rements:
Always start by examining the repository structure to lccate the dependency
Check for Python version requirements and pyenv for version management
Identify the dependency manager (pip, Poetry, etc.) and use it appropriatcly
21

---

Published as a conference paper at ICLR 2026
Prompl for Agent to Execuse the Environment Configuration
Handle eystem-level dependencies with apt-get
Knsure proper virtual envi ronment setup Include veri fication steps to confirm successful installation non-interact commands (e.g., 'apt-get install -y") Install from local repository, net PyPI packages
## Repository Structure: (file_structure)
# README Content: [readme_conten::
Please analyze the above README, file and provide your respense the specific format.
E ADDITIONAL EXPERIMENT RESULTS
E.1 ADDITIONAL RESULTS OF THE STATISTICS OF ERROR JUDGMENT NUMBER
We counted the number of errors in the agent's 's output and compared it with the number of errors in the correct answer. The complete results are shown in Figure 1T0 Overall, the results still indicate that the agent tends to produce more incorrect judgments than the correct answer, employing more stringent strategy.
(a) Zero-Shot
(b)-w-Ageent
(c) OpenHands
(d) INSTALLMAATC
(e) Repo2Run
Figure 10: The complete results of the statistics of error judgment number.
E.2 ADDITIONAL RESULTS OF ERROR TYPE JUDGMENT PERFORMANCE
(6) Zero-Shot
(b) Swe-Agent
(c) OpenHands
(d) Id|NNSAALLACTTIC
(e) Repo2Run
Figure 11: The illustrations of the error type judgment performance on the Fl score.
E.3 ADDITIONAL RESULTS OF THE STATISTICS OF MODEL OUTPUT TOKEN
We analyzed the relationship between the number of tokens generated by the agent model and its performance across various metrics, order to further evaluate the efficiency of different models. The complete statistical results for these different metrics are shown here.
22

---

Published as a conference paper at ICLR 2026
Model
zero-hhi-cllude-4 zero-shtt-Gpmminnibbr pero-shot-Ueeppeevv swe-sgenb-GFi-e
swe-sgen-Glaude-
swe agent- Gemmmi255--p
swe ageeeggenttee DeepseekV
operhannc-GFT-4 GPT-42
operhands Apenhand--Smmini2.5mm S-pro
operhands Despseskv3
Instalmatte-5m-41
instalimttic Claude-4
Iatallmatic-Gmmin22 Dnstoliaati--espsakkk reps2run-PF443
repozrur
repozuu S-pro Dpuprun-Dapsieevs
Figure 12: The complete results of the statistics of model output token.
E,4 IMPACT OF ERROR COUNT (DIFFICULTY).
We analyzed model performance across different difficulty levels based on the number of injected eπrors. As presented in Table [6 the Fl-score exhibits an "inverted-U" trend, typically peaking at Level or At lower levels (L.1-1.3), models tend to hallucinate additional errors, resulting in low precision. Conversely, at high levels (L10), while precision improves, recall drops significantly as models struggle identify every error in highly defective files. This confirms that moderate complexity (L5-L7) represents the "sweet spot" for current agents.
Table 6: Ablation Study on Error Difficulty Levels. Metries are reported as Precision Recall / F1-Score.
Model
Level 1
Level 3
Level 5
Level 7
Level 10
GPT-4.1
37.6/40.2/32.1
38.7/46.4/44 4/42.2
74.2/44.6/55.
84.2/35.0/49.
88.4/20.6/33.4
Claude-4-sonnet
0/33/416//3.4
47.8/28. 8/36.0
24.6/38.5/500
79.8/33.9/47.6
85.7/25.0/33
Gemini-2..5pru
28.0/42.9/33.9
47 $7.6/28. 1/35.4
71.8/35.0/47.1
76.4/47.2/58.3
67.2/23.4/34,
DeepScck-V.
27.5/34.3/30.5
43.0/20.5/27.8
64.7/32.3/4..1
71.0/29.6/41.8
71.4/26.3/38.4
E.5 ADDITIONAL EXPERIMENTS CORRECT README
To further benchmark the inherent difficulty of environment configuration, conduct additional experiment using the original, correct READMES (i.e., without injected errors) under identical set- tings. As shown in Table 7 while the End-to-End Pass rates correct READMES higher than those on the erroneous ones, the overall success rate remains surprisingly low. This observation confirms that automated environment configuration is non-trivial task even when pro- vided with correct instructions.
slightly
Table Comparison model performance on Erroneous vs. Correct READMES. The results show that while correct READMEs yield slightly higher success rates, the task remains challenging. Note that peocess-level metrics (Pre/Rec/F1) are not applicable to correct READMEs as they contain no ground-truth errors.
Type Judgment
Error
Fix Suggestion
End-to-End
Correct README
Framework L.M
Pre
Rec
F-1
ACC
ACC
Pass@1
Pass@1
Base Model Zero-shot
GPT-411
33.4
90.6
48.8
39.6
18.2
1.5
4.5
DeepSeek-VS
Code Agent
SWE-agent
GPT-4.1
43.7
83.2
55.3
49.8
30.7
7.2
9.2
D3epSeck-V3
41.2
70.3
51.9
44.5
27.8
7.4
11.3
23

---

Published as a conference paper at ICLR 2026
F THE USE OF LARGE LANGUAGE MODELS (LLMS)
In accordance with the official policy on the use of large language models, we used LLMs solely as general-purpose assistive tools for grammar checking and minor wording refinement during manuscript preparation. All LLM-suggested edits were manually reviewed and selectively accepted by the authors. Our usage complies with the official requirements, and we disclose here.
24


Reviews:
[{"review_id": "LBjTpixLnK", "reviewer": "Reviewer_x88R", "summary": "The paper presents EnConda-Bench, a new benchmark for evaluating llm code agents. The authors collected various repositories which includes neccessary README files which can be used as a guide to correctly setup the repository environments. The author synthetically introduce errors in the README using LLMs and create the benchmark to evaluate how llm code agents can repair the issues in the README as well as to generate a correct bash script to setup the environment. The paper evaluates several llm agent baselines and provides detailed analysis on the benchmark performance.", "questions": "1. Did the authors evaluate the base error-free results from the benchmark?\n2. How does the agent perform the error classification during evaluation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- tackles a very important and currently understudied problem of using llm agents to build software development environments\n- the benchmark is constructed nicely with detailed descriptions of the pipeline and manual examination, which can be used by future work\n- the authors also evaluate the benchmark already on several important baselines together with detailed analysis", "weaknesses": "Missing key evaluation category:\n- As the author demonstrated in the paper and from prior work, developing a script that can successfully build an environment is non-trival.\n- In the paper the authors focus on the task of repairing a README with errors\n- However, we can also easily use the benchmark without README with errors to evaluate given a correct README what are the performance of generating a correct environment using LLM agents.\n- I think this is an interesting scenario and can allow the authors to further compare and contrast the repair results\n\nUnclearness of the error classification class:\n- From reading the paper it is unclear how the error classification is done.\n- For example how does an agent know what are the different error categories? if they do not know, then do the authors still use llm-as-judge to determine that?\n- I think this is a very inefficient evaluation category as it only shows how good the LLM-agents are with identifying the error type\n\nMinor issues:\n- some of the figures are extremely small with small texts that are difficult to see (e.g., Figure 4)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Dnmd23amQ", "reviewer": "Reviewer_mVvg", "summary": "This paper proposes a new dataset to advance agentic software engineering. This dataset focuses on measuring environment configuration, which is identified as a common weakness for current agents. It consists of 100 problem instances and includes process-level annotations. The dataset is constructed using LLM-assisted filtering as well as human validation.", "questions": "Why not include Codex CLI, Gemini CLI, Jules, and Claude Code? These are specifically optimized to handle novel codebases and deal with configuration.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper identifies an important issue in agentic coding and proposes a targeted dataset and benchmark to enable future research.\n* While there are no technical contributions beyond the dataset and some of the analysis, the evaluation suite does seem to offer some key benefits over previous work, in particular when it comes to the “process-level” evaluation.\n* The dataset creation procedure is thorough and well-explained; I think this will be a useful resource for the community.", "weaknesses": "* There are some notable omissions in the evaluations, such a GPT-5-Codex and Claude 4.5, both of which are considered SOTA base models for coding. Furthermore, for the coding agents, why not include Codex CLI, Gemini CLI, Jules, and Claude Code? These are specifically optimized to handle novel codebases and deal with configuration issues.\n* I’m not sure that ICLR is the best venue for this work. Perhaps a dedicated dataset/benchmark track would be better suited.\n* Some of the figures are too small to be useful, such as Figure 5 and Figure 6. It would be better to focus on a subset of the results in the main text (relagate others to appendix) to better highlight differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cVIv0bvExE", "reviewer": "Reviewer_qXby", "summary": "In this paper, the authors propose a framework for analyzing process-level trajectories of LLM agents in environment configuration. This moves evaluation beyond simple pass/fail build outcomes to diagnose where and why an agent fails. The dataset is created by injecting errors (six defined categories) into valid README files followed by automatic Docker-based validation. However, this methodological strength also possibly limits the benchmark's real-world applicability (discussed more in Weaknesses section).", "questions": "1. Were the six error types derived from any minded empirical study or developer survey of configuration failures?\n\n2. Did you experiment with multiple variants per error type to check whether the evaluation metrics remain stable?\n\n3. How does EnConda-Bench handle repositories with pre-existing errors or ambiguous READMEs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This work addresses a critcal yet unexplored bottleneck, moving from end-to-end pass/fail metrics to process-level trajectory assessment. This is useful to extract actionalble feedback that is useful for agent designers.\n\n2. The decomposition of evaluation into the perception, feedback, and action provides fine-grained diagnostics beyond aggregate success metrics.", "weaknesses": "1. The six error types chosen are said to be \"guided by failure modes frequently encountered in practice\", but no citation or prior empirical study is provided to substantiate this taxonomy. Without grounding in developer-observed data, it is unclear whether these six types cover real-world failure modes, or simply reflect intuitive assumptions.\n\n2. Each erroneous README is created by injecting two errors per file. This raises two concerns: (i) since both synthesis and evaluation rely on LLM behavior, the resulting benchmark may reflect model-specific phrasing or error styles, rather than human-authored configuration errors; (ii) generating only two errors per category constrains diversity, a stronger design would sample multiple error candidates and retain a stratified subset verified by humans.\n\n3. The results lack ablation on error difficulty levels, number of injected errors, and impact of Docker environment variations. These would strengthen the claim that process-level evaluation provides deeper insight than end-to-end metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ip997jCPQ4", "reviewer": "Reviewer_pDzU", "summary": "This paper introduces a benchmark, EnConda-Bench, for evaluating LLM-based agents on environment configuration tasks in software engineering. The key innovation is providing process-level trajectory assessment across four capabilities: environment setup-planning, perception-driven error diagnosis, feedback-driven repair, and action execution. This work constructs 4,201 tasks from 323 GitHub repositories. Their evaluation across multiple LLMs and agent frameworks reveals that. while agents can localize errors reasonably well (F1 ~60%), they struggle to translate feedback into effective corrections.", "questions": "My questions are following several aspects mentioned in weakness:\n\n- How did you validate that your synthetic error distribution matches real-world configuration problems?\n- How did you take partial credit into consideration, as it cannot be effectively assessed by Pass@1?\n- Can you provide more detailed analysis of *why* Repo2Run performs better? Is it the dual-environment architecture, the rollback mechanism, or something else?\n- Through case study, what are the common failure modes or patterns? What are their distributions? What are the implications for future improvements?\n- Can you give more detailed explanation for selection, filtering, and validation criteria in dataset construction?\n- Can you revise the figures to make them clearer and more readable?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper propose a new benchmark with process-level evaluation for environment configuration.\n\n2. From repository selection to filtering and validation, the multi-stage dataset construction pipeline demonstrates rigor.\n\n3. The paper clearly articulates the problem (environment configuration bottleneck), motivation (limitations of end-to-end metrics), and solution (process-level evaluation with synthetic errors).", "weaknesses": "**1. Limitations In Language Coverage**: The benchmark focuses exclusively on Python repositories. Given that environment configuration challenges exist across all programming languages, this can limit the generalizability of the evaluation and conclusions.\n\n**2. Limitations In Synthetic Error Validity**: While the authors validate that injected errors cause failures, there's insufficient evidence that these errors represent the *distribution* of real-world configuration problems. The difficulty comparison (Table 2) shows similar mean scores among different benchmarks, but doesn't validate whether the *types* of errors match real-world distributions.\n\n**3. Limitations In Evaluation Metrics:** Pass@1 metric doesn't account for partial progress (e.g., fixing 1 of 2 errors).\n\n**4. Limitations In Analysis:** The specialized environment agents (e.g., Repo2Run) are evaluated but not deeply analyzed for why they perform better. There are also no ablation studies examining which agent design choices matter most.\n\n**5. Limitations In Data Construction Transparency**: The repository selection criteria (10+ stars, 1000+ commits, 10+ issues) seem arbitrary, while no justification are provided. Also, the process of \"manual checking\" is mentioned but not detailed (e.g., how many annotators? what was the failure rate?).\n\n**6. Limitations In Paper Presentation:** Some figures are hard to read. For example, all the scatter points and X-axis labels and titles are very hard to read in Figure 7, and model names are also very hard to read in Figure 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)