DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle
Authors: Yuheng Tang
Kaijie Zhu
Bonan Ruan
Chuqi Zhang
Michael Yang
Hongwei Li
Suyue Guo
Tianneng Shi
Zekun Li
Christopher Kruegel
Giovanni Vigna
Dawn Song
William Yang Wang
Lun Wang
Yangruibo Ding
Zhenkai Liang
Wenbo Guo
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
Agent
Tool Call
Primary area: datasets and benchmarks

Abstract: Even though demonstrating extraordinary capabilities in code generation and software issue resolving, AI agents' capabilities in the full software DevOps cycle are still unknown.
Different from pure code generation, handling the DevOps cycle in real-world software, including developing, deploying, and managing, requires analyzing large-scale projects, understanding dynamic program behaviors, leveraging domain-specific tools, and making sequential decisions. 
However, existing benchmarks focus on isolated problems and lack environments and tool interfaces for DevOps.
We introduce DevOps-Gym, the first end-to-end benchmark for evaluating AI agents across core DevOps workflows: build and configuration, monitoring, issue resolving, and test generation. 
DevOps-Gym includes 700+ real-world tasks collected from 30+ projects in Java and Go.
We develop a semi-automated data collection mechanism with rigorous and non-trivial expert efforts in ensuring the task coverage and quality.
Our evaluation of state-of-the-art models and agents reveals fundamental limitations: they struggle with issue resolving and test generation in Java and Go, and remain unable to handle new tasks such as monitoring and build and configuration.
These results highlight the need for essential research in automating the full DevOps cycle with AI agents.

Published as a conference paper at ICLR 2026
DEVOPS-GYM: BENCHMARKING AI AGENTS IN SOFTWARE DEVOPS CYCLE
Yuheng Tang': Kaijie Zhu'r Bonan Ruan", Chuqi Zhang?, Zhang",Michare MMccael Yang', Hongwei Li', Suyue Guo', Tianneng Shi", Zekun Li', Christopher Kruegel', Giovanni Vigna', Dawn Song', William Yang Wang', Lun Wang", Yangruibo Ding², Zhenkai Liang", Wenbo Guo
1 UC Santa Barbara, Nattooall University of Singapore, UCC Berkeley, 'Google, SUC Los Angeles
ABSTRACT
Even though demonstrating extraordinary capabilities in code generation and soft- ware issue resolving, Al agents' capabillitess the full software DevOps cycle are still unknown. Different from pure code generation, handling the DevOps cycle in real-world software, including developing, deploying, and managing. requires analyzing large-scale projects, understanding dynamic program behaviors, lever- aging domain-specinf tools, and making sequential decisions. However, existing benchmarks focus isolated problems and lack environments and tool interfaces for DevOps. We introduce DEVOPS-GYM, the first end-to-end benchmark for evaluating AI agents across core DevOps workflows: build and configuration, monitoring, issue resolving, and test generation, plus end-to-end pipeline tasks that require solving problems across all stages sequentially. DEVŌPS-GYM in- cludes 700+ real-world tasks collected from 30+ projects in Java and Go. We de- velop a semi-automated data collection mechanism with rigorous and non-trivial expert efforts in ensuring the task coverage and quality. Our evaluation of state- of-the-art models and agents reveals fundamental limitations: they struggle with issue resolving and test generation in Java and Go, and remain unable to handle new tasks such as monitoring and build and configuration. These results highlight the need for essential rescarch in automating the full DevOps cycle with AI agents.
INTRODUCTION
Software DevOps refers to the end-to-end process of developing, delivering, deploying, and man- aging software projects. It is a critical yet labor-intensive process. With recent advances in LLMs and AI agents, the development part can be largely automated (e.g.. LLMS can automatically gen- erate code, resolve GitHub issues) (Xia et al. 2024| Zhang et al., 2024c Yang et al.] [022a]] Wang
al. 2025a Li al. 2025 Tang al. 2025 However, the subsequent operational stages, such as system building, deployment, and monitoring, still demand substantial manual intervention.
Automating these complex operational tasks presents distinct of challenges that transcend tra- ditional code generation. It requires the ability to analyze system runtime behavior, interact with domain-specific tools, and execute multi-step plans. For instance, diagnosing memory leak ne- cessitates sequence of actions: repeatedly invoking monitoring tools (e.g., using ps to inspect process state, iostat to identify I/O bottlenecks) to track memory resource usage and interpret- ing the output to identify signals of potential anomalies. Such tasks, which depend on complex tool use, dynamic interaction with a live environment, and coherent decision-making, make them a compelling application domain for autonomous Al agents.
While numerous benchmarks exist for evaluating L.MSs and AI agents, they predominantly focus on software development tasks such code generation (c.g.. HumanEval (Chen et al.) (2011)) issue resolving (c.g., SWE-bench (Jimenez et al.) 2024)), and test generation (c.g., SWT-bench (Mündler al. 2)24)). In contrast, work on the operational side remains limited and is often confined to sim- ulated environments (Jha et al. 2025) or narrow infrastructure settings (Chen et al., (222)). Conse- quently, there exists significant gap for benchmark capable of evaluating the end-to-end DevOps
Equual contribution. Contact: (yubengtang, kaijiezhu] (eaiiiezuu@ssbb.... edu Correspondence Wenbo Guo <henrygsb@ucsb.com com>.

---

Published as a conference paper at ICLR 2026
capabilities of agents in realistic software projects. Furthermore, few benchmarks explicitly de- signed for agentic systems, which must execute multi-step workflows that integrate code reasoning and generation with tool usage. Existing benchmarks also lack dynamic exccution environments with tool-calling interfaces that enable realistic agent interaction and evaluation.
To address these gaps, introduce DEVOPS-GYM, the first benchmark designed to evaluate AI agents across core DevOps workflows using real-world repositories and various DevOps tools. Unlike existing benchmarks that focus on single tasks in isolation with synthetic environment, DEVOPS-GYS provides: (1) Coverage of four essential DevOps stages that form the minimum viable pipeline: build and configuration for project deployment, monitoring for runtime problem de- tection, issue resolving for problem fixing, and test generation for patch validation; (2) Real-world tasks sourced from GitHub issues or synthesized tasks that mimic the complex real-world issue patterns; (3) Agentic workflows that require extensive tool use and multi-step planning; (4) A tool- augmented dynamic evaluation environment with standardized tool-calling interfaces and detailed metrics for different types of tasks.
We make several design choices and efforts to address the key technical challenges of constructing DEVOPS-GYN First, we manually analyze large set of real-world issues, particularly for moni- toring, build and configuration stage, to categorize representative issue types and summarize failure patterns for crafting synthetic tasks. Second, we apply rigorous filtering process to prevent data contamination. Third, we invest extensive expert engineering effort to reproduce tasks to ensure their correctness, which required reconstructing environments, dependencies, configurations, and specific inputs. This was multi-round, time-consuming process. Even with coding agent assis- tance, it often exceeds 10 hours of expert work per task, especially with incomplete reports. Finally, we carefully designed our evaluation metrics to enable rigorous and scalable evaluation, and pro- vided standardized tool interfaces in the germinal-benede format (TenhinalBench) 2025). With these extensive efforts, we craft 704 tasks collected from 30+ real-world projects in Java and Go. Addi- tionally, we create 14 end-to-end pipeline tasks that chain all four stages sequentially, testing agents' ability to maintain context across complete DevOps workflows.
Our evaluation of three widely used coding agents across 5 LLMs and agentic frameworks shows that even state-of-the-art systems fall short. The top-performing agent achieves success rates of just 51.85% on build and configuration, 20.56% on monitoring, 23.87% on issue resolving, and 13.87% on test generation. To the best of our knowledge, DEVOPS-GYM, is the first end-to-end DevOps benchmark featuring spenifci tasks alongside a comprehensive evaluation platform, including environments, tool interfaces, and metrics. We will open-source DEVOPS-GYM, together with its evaluation framework and baseline implementations, and will continue to improve to facilitate future research on AI agents for broader software engineering. The key findings are shown below.
Agents frequently fail high-level planning, struggling to formulate correct sequences of actions for multi-step building and monitoring workflows.
Agents consistently fail to use DevOps-specific tools correctly, particularly for building and moni- toring. We hypothesize that these tools dustr--distribuution (OOD) for the base L.MMs, which seldom trained on relevant tool-use trajectories.
Agents exhibit limited ubility purse and reason about dynamic information, such as program states, runtime logs, and system status. They also struggle with long-contest reasoning required for complex tasks such as monitoring.
For issue resolving and test generation, although SOTA agents report strong performance on ex- isting Python-base based benchmarks, their performance drops significantly We believe this may due part data contamination. It also indicates that L.MSs well-equipped to handle non-sert script languages (e.g., Java and Go), which involve compilation, more complex dependencies, and syntax.
Java and Go tasks.
2 RELATED WORK
Coding benchmarks. Function-level coding benchmarks (Chen al. 2021: Majd al., 2019 Jain et al. 2024) Lu et al,, 2023) are widely used as standard benchmarks for evaluating LLMS code generation capabilities. Some recent efforts move toward repository-level evaluation, which
2

---

Published as a conference paper at ICLR 2026
Build and Configuration
Monitoring
Input
Tools
Output
Input
Tools
Output
project
build succ.
buggy env
report
build
config
anormalies
spec
npm
artifacts
evidence
maven, npm, gradie,
top, pprof,
Test Generation
Issue Resolving
Input
Tools
Output
Input
Tools
Output
project
regression assertion
project
patch
buggy fix Lines
issue
issue
coverage
JUnit, Maven,
sed, jrep,,
Figure I: Overview of DEVOPS-GYM. It includes four core stages of DevOps: Build & Configura- tion, Monitoring, Issue Resolving, and Test Generation. Each stage requires Al agent leverage a distinct set of command-line tools to solve realistic tasks.
more complex and relevant real-world applications (Zhang et al.
2023
Zhuo al. 2025
Ding et al. 2023 Li al. 2024b, Liang et al. 2024| Han et al. 2025 Le Hai et al. 2024. Zhuo
et al., 2024 Zhang et al., 2024 For example, RepoCod (Liang et al. 2025 assesses whether
LLMs can handle multi-fi file code generation rather than isolated functions. Another line of work extends general code generation to specific real-world tasks. For example, SWE-bench (Jimenez
et al. 2024) targets issue resolving in real-world repositories for Python, and its follow-ups extend
the effort to multilingual (Jimenez al., 2024). multimodal (Zan al.
2025 Ni al. 2025
live settings (Jain et al. (2024). and security vulnerabilities (Yang et al. 2024b). Beyond issue
resolving, task-specific benchmarks also include infrastructure (Kon et al. [0024] Munshi al. 2025
Srivatsa et al. 2024). backend (Vero et al. 2025). or website and software development (Xu et al,, 2025 Li al. 2024a), database (Li et al. 2023), ML (Nathani et al. 2025) etc. Although offering more realistic evaluations, these benchmarks only cover the development side of DevOps instead of a holistie cycle. Furthermore, their main target is to evaluate model reasoning and generation capabilities rather than agentic systems with sequential decision-making and tool calls.
DevOps-related benchmarks. Beyond code development, there an increasing effort in bench- marking other steps along the DevOps cycle, including test case generation (Zhang al.) 2024b, Wang et al.] 2025b 2024), build and configuration, and monitoring. Specifically, SWT- bench (ünndler et al., (024) evaluates test case generation in Python, with metries that measure functionality and code coverage. For build and configuration, existing benchmarks (Eliseeva et al.) 2025) mainly focus initial project configuration, evaluated by static and compilation checks for the repository. For monitoring. I'T-bench (Jha et al. (025) constructs tasks on monitoring and re- solving issues during project operations. AlOpsLab (Chen al., 2025) focuses on microservice environments and cloud-base based incident detection, localization, and root cause analysis (e.g., net- work issues, authentication errors). Their evaluation metrics focus on the agent efficiency and costs such as the steps and time of actions, as well as the tokens consumed.
Limitations. There no benchmark that covers the end-to-end DevOps cycle on real-world repos- itories. Most existing efforts focus on individual tasks (coding, testing, monitoring). Moreover, few benchmarks are explicitly designed for agentic systems, which have multi-step workflows that combine code reasoning and generation with tool callings. They also lack dynamic execution en- vironments with tool calling interfaces for agent interaction and evaluation. To fill the gap, we construct DEvOrS-Gy enabling the evaluation of AI agents on end-to-end real-world DevOps tasks with proper metrics, tool call support, and execution environments.
3 CONSTRUCTION OF DEVOPS-GYM
3.1 OVERVIEW
Design principle. We construct DEVOPS- -YM based on the following principles. Realism. We aimed to include real-world tasks that DevOps engineers encounter in practice. To achieve this,
3

---

Published as a conference paper at ICLR 2026
collect real-world GitHub projects and construct tasks based actual GitHub issues synthetic failures. The synthetic ones are injected by experts, which replicate common production problems, e.g,, dependency conflicts from version incompatibilities @ Agentic evaluation. We design tasks to be sequential decision-making processes that involve various command-line tools. Solving these tasks requires domain-specific agentic systems capable of analyzing large projects, selecting proper tools and interpreting their returns, and conducting multi-step planning. For instance, to diag- nose performance issues in large project, such abnormal memory or I/O utilization, agents must plan multi- procedure and invoke appropriate tools (e.g., iostat, top). @ Complete DevOps cycle coverage. We distill four critical stages: Build and configuration: configuring and migrating build systems, compiling the project, and fixing build failures; Monitoring: dynamically detecting runtime performance and resource issues during project executions; Issue resolving: debugging and patching bugs in projects; Test generation: creating regression tests that verify bug fixes.
Benchmark overview. We select Java and Go as our target program languages because they repre- sent large-scald scale enterprise SE projects that have standardized, non- non-trivial build systems, as well as robust monitoring infrastructure and tooling. To ensure diverse coverage of realistic DevOps scenar- ios ios-particularly in monitoring and build, which are not covered existing benchmarks- we man- ually analyze over 1, 000 GitHub issues from repositories cited in DevOps technical reports (DORA) 2024) and categorize the tasks for each stage. For build and configuration, we established two task categories: (i) repair tasks that rebuild the project to fix common failures in the general building process; and (ii) implementation tasks that build from scratch to incorporate new functionalities. For monitoring,
cover six anomalies: performance anomalies (I/O bottlenecks, query inefficien- cies) and resource anomalies (memory leaks, disk exhaustion, CPU saturation, handle depletion). We follow Multi-SWE-bench (Zan et al., (025) and SWT-bench (Mündler al.) (024) collect tasks for issue resolving and test generation. We further create end-to-end pipeline tasks that chain all four stages sequentially, requiring agents to maintain context from build repair through moni- toring, issue resolution, and validation- testing their ability to handle complete DevOps workflows rather than isolated problems. Fig. Shows the pipeline of DEVOPS- configuration tasks (20 synthetic tasks and 34 real- -wolld tasks), 34 monitoring tasks (29 synthetic tasks and 5 real-world tasks) 310 issue resolving tasks, 310 test generation tasks, and 18 end-to-end tasks, across 30+ repositories. The process begins with building the project, where the agent must invoke build-related tools to either migrate an existing build or rebuild the project to incorporate new functionalities while resolving issues. After the build stage, the agent monitors the system's 's status and identifies runtime anomalies. When issues are detected, the agent is then responsible for resolving them and generating tests. Detailed benchmark statistics in Appendix Table
comprising 54 build and
Technical challenges and solutions. Besides task selections and collection, which already require extensive expert efforts, constructing SErvOrs--YMa also encounters three key technical challenges. Data contamination prevention. Large-scale pre-training of LLMs poses significant contamina- tion risks to new benchmarks. To address this, we applied a systematic prefix-completion analy- sis (Carlini et al., 202T; Staab et al. (023) to identify and filter repositories potentially present in training corpora. Additionally, we sanitized the repositories by removing git metadata prevent agents from accessing solutions through git version history (Kahn 2025). Details are provided in Appendix B @ Task reproduction. Reproducing real-world failures, especially for monitoring and building, imposes substantial challenges. We need to fully reconstruct the corresponding environ- ments with correct run- run-time dependencies and configurations, as well as find the necessary inputs to trigger the issues. To make it even worse, the natural language issue descriptions typically do not provide the full information necessary for issue reproduction. Resolving this challenge also requires essential expert efforts to conduct extensive trial and error, i.e., it takes one expert SE researcher more than 10 hours to fully reproduce and validate one monitoring or building issue. Although coding agents (e.g.. cursor and Claude code) can facilitate the process with hints, they still cannot fully finish the reproduction due to a lack of capabilities, as well as seeking shortcuts (we observe that instead of reproducing current errors, coding agents tend to inject other casier-to-trigger errors. Similar behavior has also been observed in other SE and eccurity-related benchmarks (Wang et al.) 2025c] Yang et al. 2024b)). 2 Enable rigorous evaluation. After reproducing the issues, designing and implementing rigorous and scalable evaluation is also challenging, especially for building tasks. As detailed in Section 3.2, building tasks necessitate both dynamic execution validation and static configuration analysis. Besides, to provide a standardized interface for agent execution, we provide necessary tool sets (Figure IT for each stage and convert our benchmark format into the terminal bench format (TerminalBench 2025). In summary, the end-to-end task, selection, construction, and
4

---

Published as a conference paper at ICLR 2026
evaluation pipeline requires extensive system design, engineering, and manual efforts that otherwise cannot be accomplished by SOTA automated agents.
3.2 TASK CONSTRUCTION FOR BUILD AND CONFIGURATION
Overall task design. Build and configuration is a critical step during the DevOps cycle, encom- passing code compilation, dependency management, testing, and artifact creation within controlled environments. Also various eπrors can happen here including dependency conflicts, version incom- patibilities, and configuration errors, frequently disrupting development workflows. To concretely understand agents' capability to perform complex build and configuration tasks, we evaluate the following two categories of build challenges that reflect routine DevOps scenarios. Repair tasks that address five prevalent project building error types: dependency version conflicts, build mis- configuration, compilation errors, tool-chain mismatches, and dependency resource unavailability. Agents must diagnose build failures by analyzing enror logs, identifying root causes among these error categories, applying targeted fixes, and rebuilding the project to complete a correct build. @ Implementation tasks that incorporate new functionalities, which include the following scenarios: build system migration between frameworks (e.g.. Maven to Gradle for Java), target release (i.e., re- lease for specific use cases), plugin integration, and dependency version upgrades. Both categories require agents to understand build system semantics and configuration best practices essential to maintain reliable deployment pipelines.
For repair tasks, we follow the workflow of the BugSwarm framework for task collection Tomassi al.] 2019). i.c., mining recent build failure-success pairs from CI logs and filtering for configuration-levell fixes. For implementation tasks, three domain experts synthesize scenarios based on production environment patterns, ensuring coverage of frequent real-world build challenges.
Key technical challenge and solutions. Constructing build and configuration tasks present three key challenges that distinguish this domain from traditional code benchmarks. Issue reproduction: Real GitHub issues provide incomplete environment specifications, requiring careful reconstruction of tool-chain dependencies, compilation configurations, and version control to reproduce the issues. This challenging process requires extensive expert efforts; approximately 40% of initially selected issues required multiple iterations to achieve consistent reproducibility. Synthetic task design: Creating comprehensive synthesis build task presents significant challenges. Implementing alistic build processes requires a deep understanding of repository configurations and compilation mechanisms, along with extensive DevOps experimentation construct meaningful and challeng- ing real-world scenarios. ② Enable accurate evaluation. Different from other steps, build requires a complex evaluation pipeline with multi-dimennsioall evaluation metrics. requires designing dif- ferent metrics for different task scenarios (detailed below). For example, for migration tasks, we need to design unified metrics that involve multiple configuration tools. Constructing an effective evaluation process for each selected task scenario also requires extensive engineering and manual effort for each individual task, e.g., designing unified metries for multiple configurations requires manually analyzing the joint features with domain-specife tools.
Task details. The inputs, expected outputs, ground truth, and evaluation metrics are as follows:
Input: (1) Repository with failing build configuration (repair tasks) or specification for new build setup (implementation tasks). (2) Terminal access with build tools (maven, gradle, npm), text utilities, and package managers.
Expected Output: For repair tasks: patch (in the diff format) fixing build failure; for implementa- tion tasks: complete configuration files meeting specifications.
Ground Truth: For repair tasks: developer fixes from real repositories; For implementation tasks: expert-created configurations validated for correctness (implementation tasks).
Evaluation metrics: At a high level, judging whether build process is successful contains two sub-metrics: 1) the build process is executed without any errors; 2) the built artifacts correctly realize their required functionalities. For repair tasks, as the build command is standardized, as long as the agent executes the project's original build commands without errors, it can be consid- ered as successful build. For implementation tasks, different scenarios have different metrics for ensuring functionalities: migration between frameworks whether the migrated implemen- tation maintains functional equivalence with the original framework while successfully adapting
5

---

Published as a conference paper at ICLR 2026
to the target framework's conventions and capabilities; target release whether the released arti- facts satisfied the required features: plugin integration whether the plugin functions correctly; dependency version upgrades successfully build the upgraded version; config initialization successfully generate the configuration with all the functionalities we want. All these metrics are concretized as whether the built artifacts pass their dedicated testing cases.
3.3
TASK CONSTRUCTION FOR MONITORING
Overall task design. Given a running project application inside a controlled environment (c.g., a container), monitoring requires agents (a) capture the runtime execution and underlying system states by using external command-line tools, and (b) detect potential performance and resource uti- lization anomalies during execution. Such setting mirrors the role of real-world DevOps engineers, enabling agents to demonstrate their ability to diagnose realistic and complex production issues.
Importantly, our tasks focus on performance and resource anomalies rather than system failures or crashes. This is because immediate crashes can be simply identified through console outputs or error logs, limiting the opportunity to evaluate an agent's diagnostic capability. In contrast, perfor- mance and resource anomalies manifest subtle system degradations that require careful analysis to uncover. Take memory leakage as an example. In a file-system server, developers introduce an in-memory cache for large file downloads (e.g.. 22 MB) but neglect to release it, which causes memory leakage when requesting various large files. While this leak eventually exhausts memory, the early symptoms appear as abnormal memory leak patterns relative to normal behavior (i.e., small file requests). Detecting such anomalies requires agents to capture system behaviors across requests and analyze subtle variances between normal and buggy cases.
Specifically, we consider two types of performance anomalics. Resource usage problems: Mem- ory leaks, disk leaks, system handle (e.g., file and socket) exhaustion, and CPU spikes that gradually degrade system reliability. These issues represent the most common resource-related failures in production systems. Moreover, these problems exhibit gradual degradation patterns; for example, memory leaks may take hours or days to exhaust resources, and CPU spikes often manifest inter- mittently under specific load patterns. @ Performance degradations: We select I/O bottlenecks and inefficient SQL query handling that degrade user experience without causing immediate failures. For instance, a project mishandles I/O requests by opening files with O SYNC IO DIRECT without using proper OS-level caching or file-backend memory mappings. This would trigger extremely slow I/O paths when the request I/O payload size is large. Such degradations manifest as increased latency or reduced throughput, c.g., 10x slower request may still complete successfully. We also include cases where the system operates without any anomalies, requiring agents to correctly iden- tify the absence of issues and avoid false positive diagnoses when monitoring healthy systems. Our tasks mix with real-world GitHub issues and expert-injected synthetic anomalies. In total, we collect 30 monitoring tasks, with the distribution of anomaly types shown in Fig.
Task-specific challenges and solutions. Constructing monitoring tasks share similar challenges with build on O Issue reproduction and @ Synthetic task design. Creating realistic expert-injected anomalies demands deep repository understanding to instrument source code without disrupting normal application behavior. We design anomalies that manifest within 5-15 minutes through stan- dard monitoring tools while requiring sophisticated analytical reasoning rather than trivial detection. @ Observability validation: We manually validate that each task ensures anomalies while remaining detectable through monitoring toolsets (e.g, top, free, ps) without access to the source code.
Task details. The inputs, expected outputs. ground truth, and evaluation metrics are follows:
Input: (i) Containerized environment running an application with bugs; (ii) Terminal access (e.g., top, free, ps, netstat), with access source code, configuration files, trigger scripts.
Expected Output: Structured diagnostic report: specific issue (e.g., memory leak), and support- ing evidence with quantitative metrics (e.g., memory growth rate, affected process ID).
For real GitHub issues, we select only closed issues where the problem has been identified and resolved, and then the DevOps experts classify the issue based the description and resolution. For expert-injected anomalies, ground truth is predetermined by the injection methodology as we know the exact problem type and manifestation because we control the failure
Ground Truth:

---

Published as a conference paper at ICLR 2026
injection. To ensure reproducibility and observability, three senior DevOps engineers indepen- dently validate that each problem can be reliably detected using the provided monitoring tools.
Evaluation metrics: The primary metric is binary accuracy, requiring agents to correctly identify the specific type of anomaly. In the prompt, we define the five anomaly types and explicitly instruct the model to write its diagnosis into a specified file in a single line without any explanation. Evaluation is performed using automated pytest scripts that check: (1) whether the diagnosis file exists, and (2) whether the diagnosed anomaly type matches the ground truth.
3.4
TASK CONSTRUCTION FOR ISSUE RESOLVING AND TEST GENERATION
Overall task design. These two stages represent well-established evaluation domains with exist- ing benchmarks, such SWE-bench (Jimenez al.) 2024) for issue resolving, as well SWT- bench (Mündler et al., 2024) for test generation. Issue resolving requires agents to translate bug descriptions into code fixes, while test generation creates regression tests to prevent issue recur- rence and ensure functionality correctness. Both tasks are essential components of the DevOps pipeline: fixes address problems identified through monitoring or user reports, and tests validate solutions before deployment. Following established methodologies, issue resolving agents receive buggy repositories with natural language descriptions and must generate patches that pass fail-to- pass test transitions (Jimenez al. 2024). Test generation agents create tests based solely on bug descriptions to ensure comprehensive validation coverage.
Key differences from existing benchmarks. O DEVOPS- GYM implements comprehensive decon- tamination procedures (Section 3.T to mitigate potential training data contamination, ensuring more reliable evaluation than existing benchmarks. @ Cross-lande loos-language performance gaps. As discussed in Table D LLM agents perform significantly worse on non-Python languages. This degradation likely stems from Python's dominance in training data.
collection pipeline for issue resolving tasks, targeting well-maintained Java and Go projects with compre- hensive test suites. The pipeline filters GitHub pull requests that resolve issues with test-validated fixes, ensuring reproducible fail-to-pass transitions. Test generation tasks derive from the same issue set, creating corresponding validation scenarios. Regarding metrics, issue resolving success requires patches that pass all given test cases, while test generation success demands that generated tests fail on buggy code but pass the patched code. Task details can be found in Appendix C
Data collection and evaluation. We adapt the Multi-SWE-bench Zan et al. 2025
3.5 END-TO-END TASKS
Beyond evaluating individual DevOps capabilities, construct end-to-end pipeline tasks that test agents' ability to maintain context and solve problems across multiple stages. These tasks simulate realistic DevOps workflows where problems cascade through the development pipeline.
Each pipeline task follows a four-stage sequence: Stage 1: Build. Starting with a monitoring- repository, we inject build configuration errors (e.g., missing dependencies, incorrect paths) that prevent compilation. Agents must diagnose and fix these issues to establish a working baseline. Stage 2: Monitoring and Detection. Once the build succeeds, agents deploy and monitor the running system, which contains a latent performance or resource issue from our monitoring task set (3.3). Agents must use system tools to identify anomalies without prior knowledge of the issue type. Stage 3: Issue Resolving. Based on monitoring findings, agents implement code-level fixes addressing the root cause. Stage 4: Validation and Testing. Agents rebuild the system with their fix and create regression tests that verify the issue is resolved. Tests must specifically target the performance characteristic identified in Stage 2, ensuring the problem cannot recur.
This pipeline design evaluates critical DevOps competencies absent from isolated task evaluation: maintaining problem context across tools, propagating diagnostic information between stages, and verifying end-to-end solution correctness. Success requires completing all four stages sequen- failure at any point terminates the pipeline, reflecting worldworld deployment constraints where partial solutions have no value.
7

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation results on DEVOPS-GrV GyM for different agent frameworks and different L.MMs. The best result for each stage are marked as bold.
Agent
Model
Build & Config
Monitoring
Issue Resolving
Test Generation
Qwen3-Coder-308
20.37%
5.89%
13.22%
6.13%
04-mini
24.07%
8.82%
10.32%
8.70%
OpenHands
DeepSeek-V33 V
11.11%
0.00%
14.20%
3.22%
Gemini- Somini-2255-PP 2,5-Pro
16.66%
11.76%
10.96%
2.90%
Claude-4-Soneet
42.59%
14.70%
23.87%
11.61%
mini-SWE-Agent
Claude-4-Sonnet
29.62%
2.91%
5.16%
0.98%
Aider
Claude-4-Sonnet
5.55%
0.00%
9.67%
2.25%
Claude Code
Claude-4-Snnee
51.85%
20.56%
23.87%
13.87%
4 EVALUATION
In this section, we conduct comprehensive evaluation on DEVOPS- -YM using suite of state-of- the-art agentic frameworks and reasoning models to identify their current strengths and weaknesses in automating the end-to-end DevOps lifecycle.
4.1 EXPERIMENTAL SETUP
Agents and Models. We evaluate three best-performing agentic frameworks: OpenHands (Wang
al. 2025a), mini-SWE-agent (Yang al., 2024a), and Claude Code (Anthropic 2025b). To
comprehensively assess the impact of underlying backbone LLMs, we pair these frameworks with different LLMs. Specifically, we evaluated the OpenHands framework with five leading models: Claude-4-sonnet (Anthropic) (222a)), 04-mini (OpenAl (025). Gemini-2.5-PP
(Google) 2025). Deepseek-v3.1, and Qwen3-Coder-30B (Qwen- Team| 2025). This setup allows for direct com- parison of models within a single agent architecture. For the comparison of different agentic frame- works powered by the same underlying model, both mini-swe-agent and Claude Code are evaluated with Claude-4-sonnet their backbone LLM. Our initial evaluation focuses on the most advanced agents and models to establish an upper bound on DEVOPS- -GMM. We will continue to extend it to weaker models and agents to highlight the challenges in DevOps cycle automation.
4.2 RESULTS
Cross-tool builds and configurations present new challenges to LLMs. As a new yet comple- mentary evaluation to existing benchmarks, DEVOPS-GYM provides the unique chance to evaluate agents' ability to reason about heterogeneous build ecosystems and switch across tools (e.g., Maven to Gradle) while repairing failures and implementing new functionalities, which remains largely under-evaluated. As shown in Table I model performance varies significantly during the build and configuration stage. Claude-4-Sonnet outperforms other state-of-the-an models by more than 20%, demonstrating its superior capabilities in large project configuration management and debug- ging. Regarding agent performance, even when using the same powerful Claude-4-Sonne model, Claude-Code performs substantially better than the simpler mini-SWEEAgent, This performance gap indicates that complex build and configuration tasks require agents with sophisticated tool-call capabilities and well-designed architectures manage challenging, multi-step processes effectively. Furthermore, all of the models perform poorly in the build implementation tasks, especially in mi- gration tasks and target release tasks (see Appendix Table From our observations, agents struggle with understanding the internal mechanisms of build tools like Maven and goreleaser (GoReleaser Team 2024), as well as their practical usage patterns in real-world projects, rather than simply pars- ing error logs. This deeper knowledge gap becomes evident when agents attempt to configure or debug complex build processes. Detailed examples can be found in Appendix EE. This is funda- mentally different from fixing bug in source code, where the context is more self-contained. This result demonstrates that while agents are improving at manipulating source code, they are far from capable of managing the software's 's build and deployment environment.
After examining the agent run logs, we identified three common error types for build and configura- tion tasks. First, toolchain and environment instrumentation limitations (33%). A common example
8

---

Published as a conference paper at ICLR 2026
is unused-impon violations. Similar trends appear in files, and XML parsing failures, all of which reflect the agent's 's inability to validate or inspect con- figuration artifacts because the environment does not expose the necessary validators or schema checkers. These kinds of errors happen in the agent Openhands more than in Claude code. Second, multi-step reasoning and sequential planning failures (23%). In multi-step build repairs and build- system migrations, agents often resolve an initial error but lose track of remaining issues, revealing limitations in context retention. Many failures also occur because prompts do not enforce an iterative "fix-run-veer run-verffy" loop, causing the agent to stop after addressing only part of the pipeline remaining failures. Third, domain-specific knowledge gaps (37%). Failures reflect genuine domain knowledge and deep technical understanding requirements that exceed current model capabilities. These in- clude inherently complex tasks such as laven-to-Gradle migrations that require understanding of build-system semantics, dependency resolution, or platform-specific constraints. Other failures arise from misunderstanding Java generics, type compatibility. JVM version differences, or subtle runtime behaviors such as NullPointerExxcetiions. These tasks represent the upper bound of DevOps com- plexity in our benchmark and quantify where agent abilities fundamentally fall short. Notably, 17% of all failures fall into the "inherently difficult" category, confirming that the benchmark captures real-world, high-difficutyy DevOps scenarios.
errors, malformed build
The dynamic nature of system monitoring reveals critical agent failures in processing contin- nous, temporal inputs. As shown by Table [ agents perform exceptionally poorly on monitoring tasks, even reporting 0% with the state-of-the-an models. This failure stems from the dynamic na- ture of monitoring, which poses three fundamental challenges to current LLM-based agents. First, monitoring requires the continuous processing of evolving system state information. Anomalies like memory leaks often manifest gradually over time, producing a long stream of observations. Agents must ingest this constant flow of new tokens, exhausting their context limit quickly even before a discernible issue appears. Second, we observe that agents struggle to consistently focus on monitor- ing, even though we prompt them to. Specifically, as we show in Appendix [.2] agents frequently become distracted, focusing excessively on analyzing carlier observations, which causes them to stop actively monitoring the live system state. This inability to balance long-term analysis with real-time awareness prevents them from detecting the subtle and sudden signals that define many real-world performance and resource issues. Third, agents exhibit poor baseline discrimination, generating significant false positives by misinterpreting normal operational variance as anomalies in healthy system environments. These monitoring failures reveal that current agents lack the temporal reasoning and sustained attention mechanisms essential for dynamic system observation.
Unlike SWE-Bench, LLM agents perform poorly on DEVOPS-Gyn for issue resolving. As we can see from Table I the resolve rate drops significantly when moving from Python repositories, as we have seen in SWE-Bench (Jimenez et al. 2024). to other languages, such as Java and Go. This performance degradation is particularly striking when examining specific configurations: using the same agent and model combination (OpenHands + Claude-4-Sonnet) the resolving rate achieves 70.4% on the SWE-Bench- Verified leaderboard. yet drops dramatically to 23.87% when applied to Java and Go repositories in our benchmark. This indicates that the existing LLMs struggle with the cross-language capability gap, which might be largely due to the dominance of Python code in the training data. Also, results suggest that those complex compilation processes, dependency management, and environment build and configuration in Java and Go pose major challenges to Python-centric agents overcome.
Though similar trends have also been reported by Www---WE--BBehhh/e- et al. 2025). our evalua- tion provides a more definitive validation with more thorough data decontamination (Section B.1 provided by DEVOPS-GYM. Furthermore, by comparing our results using Claude-4 with those re- ported by Zan et al. (0255) using Claude 3.7, we confirm that, despite the significantly improved reasoning and tool-using capabilities of newer models, they still fail to overcome the major chal- lenges in navigating the non-Python ecosystems.
Generating high-quality tests for the issue is even more challenging than resolving the issue. Interestingly, when using the exact same set of issues for evaluation, we find that the accuracy of generating high-quality tests is notably lower than the issue resolving rate. This highlights the gen- eral difficulty of test generation, which requires agents to not only have a static understanding of the whole repository but also the dynamic analysis capabilities that could reason about how the de- scribed bug would be triggered during execution. The agent also needs to reason about how the bug might be resolved, so that the generated tests could not only reproduce the described failure but also

---

Published as a conference paper at ICLR 2026
Monitoring
599
8.8
0.0
11.8
5.9
Monitoring
5.9
2,9
0.0
20.6
Iss Resolving
13.2
10.3
14.2
11.0
23.9
Issue Resolving
23.9
5.2
9.7
23,9
Test Generation
3.2
Test Generation
11.6
(00)
Config
Config
Qwen3 Coder-308 308
o4-mini
DeepSenk-VD V3.1
Gamini-2.5 2.5-n 2.5-Pr
Claude 4-Sonnet
OpanHaands mini-S Mini-SWE-Appan Agent
Alder
Claude Code
(a) OpenHands Performance Across Dife
(b) Agent Framework Comparison (Claude-4-Sonnet
Figure 2: Performance comparison across different agentic frameworks and LLMs.
validate the correctness of the patch. In contrast, generating patch sometimes be accomplished through more straightforward, static code analysis when the issue description potentially discusses the expected fix. These results suggest that while agents are becoming proficient at predicting coding patterns, reasoning about runtime behavior remains significantly more challenging goal to achieve.
Compared to SWT-Bench (Mündler al. 2024). DEVOPS- Gym is more challenging for LLM agents to achieve satisfactory performance (as illustrated in Table T for two reasons. First, DEVOPS-GrMe focuses compiled languages (Java and Go), where reasoning about dynamic pro- gram behavior is inherently more complex than interpreted languages like Python, which SWT- Bench evaluates, since understanding a multi-stage compilation and linking process adds obvious difficulties for agents. Second, while SWT-Bench incorporates code coverage as a soft metric for test effectiveness, we focus on the strict metric that a generated test must precisely reproduce the failure described in the issue and subsequently pass on the patched code.
Analysis Across Different Agentic Frameworks and Models Fig. 2 shows that Claude Code achieves the best overall performance, while Claude-4-Sonnet is the top-performing LLM across most tasks. Claude Code consistently outperforms other agentic frameworks, reaching 58.33% suc- cess in build and configuration. The substantial gaps between different agentic frameworks suggest that agent design is crucial automated software engineering performance.
End-to-end pipeline tasks, designed to test problem-solving across cascading DevOps stages, show the inability of current agents to handle long-horizon workflows. Agents achieve 0% suc- cess rate in completing all four stages any single end-to-end task (detailed breakdown analysis shown in Table 3). While agents can sometimes successfully resolve build configuration errors in Stage 1, they struggle to complete Stage (Monitoring and Detection) due to the monitoring chal- lenges discussed earlier- carlier-inabilig to process continuous temporal inputs and maintain sustained attention system metrics. Interestingly, (Issue Resolving) or Stage 4 (Test Generation) even with incorrect monitoring conclusions, as differ- performance issues often share similar code-level manifestations that enable coincidental fixes. However, the overall pipeline success rate remains 0%, indicating complete failures in executing the full workflow. These results highlight a critical gap: current agents lack the context propagation and multi-stage planning capabilities required to chain individual DevOps tasks into cohesive workflows that mirror real- world deployment pipelines.
observe that agents occasionally succeed in Stage
5 CONCLUSION AND FUTURE WORKS
We present DEVOPS-Gye Gym, a comprehensive benchmark that evaluated agentic systems across the complete DevOp cycle through four critical stages. The evaluation results revealed substantial limitations in current agentic systems, with agents demonstrating particularly poor performance on monitoring and build and configuration tasks, highlighting critical disconnect between current AI capabilities and real-world DevOps requirements. For the future, first, we will follow our proposed methodology to construct more tasks in each stage, covering more error types well as broader projects. Second, we will also extend MAVOPs-GYM to other programming languages and more DevOps stages. Finally, we call for community wide efforts to advance full DevOps automation by enriching DEVOPS- with additional scenarios and metrics, developing more capable agents, and training specialized models optimized for agent workflows.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Anthropic.
Introducing claude 4, 2025a.
URL httts://ww..nnbrrppccccc//eseess
claude-
Anthropic. Claude code.
GitHub, 2025b.
URL http:s///gtttu...mmmmmatttttottt
claude- code
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Vors Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Brlingsson, et al. Extracting training data from large language models. In USENIX security sympusium, 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 1Xiv:2107.03374 2021.
Yinfang Chen, Manish Shetty, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Jonathan Mace, Chetan Bansal, Rujia Wang, and Saravan Rajmohan. AlOpslab: holistic framework to evaluate AI agents for enabling autonomous clouds. In Eighth Conference on Machine Learning and Systems, 2025.
Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances Neural Information Processing Systems, 36:46701-46777 2023.
DORA. 2024 accelerate state of devops report. Technical report, Google Cloud, October 2024. URL httl:tttt///oouulll google com/ dewops ottp://sttttttteeeeeeeeeee Available in multi- ple languages: English, Spanish, French, Portuguese, Chinese, Japanese, Korean.
Aleksandra Eliseeva, Alexander Kovrigin, Ilia Kholkin, Egor Bogomolov, and Yaroslav Zharov. Envbench: A benchmark for automated environment setup. arXiv preprint arXiv:2503. 2025.
Google. Gemini 2.5: Pushing the frontier with advanced reasoning, mwl.iiiniiimmmiiiiiiidiiiii... next generation agentic capabilities, 2025. URL U.LL:ttpse//aaxxrrrrr////////.//
http://wwccrrrttiiiiiiimmiimiiio long context, and
GoReleaser Team. goreleaser Accessed: 2024-09-24.
Goreleaser, 2024.
URL httpe///ggttubb.cmmggrreeeess
Dan Gustield. Algorithmss on Strings, Trees, and Sequences: Computer Science and Computational Biology. Cambridge University Press, Cambridge, 1997.
Hojae Han, Seung-won Hwang, Rajhans Samdani, and Yuxiong He. Convcodeworid: Bench- marking conversational code generation in reproducible feedback environments. arXiv preprint DrXiv:2502.19855 2025.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint XXiv:2403.0797. 2024.
Saurabh Jha, Rohan Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, Noah Zheutlin, Saki Takano, Divya Pathak, Felix George, Xinbo Wu, Bekir O Turkkan, Gerard Vanloo, Michael Nidd, Ting Dai, Oishik Chatterjee, Pranjal Gupta, Suranjana Samanta, Pooja Aggarwal, Rong Lee, Jae wook Ahn, Debanjana Kar, Amit Paradkar, Deng, Pratibha Moogi, Prateeti Mohapatra, Naoki Abe, Chandrasekhar Narayanaswami, Tianyin Xu, Lav R. Varshney, Ruchi Mahindru, Anca Sailer, Laura Shwartz, Daby Sow, Nicholas C. M. Fuller, and Ruchir Puri. ITBench: Evalu- ating AI agents across diverse real- world IT automation tasks. In Forty-second International Conference on Machine Learning, 2025. URL https::/ https://openzevieet net htt./oormmmmmm 1P59rz1b70

---

Published as a conference paper at ICLR 2026
Carlos E Jimenez, John Yang, Alexander Wettig. Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2024. URL httt:tt::/// /tttps::/opeeee
/ /oppnrevie..
d=VTF 8yNQM66
Jacob Kahn, Repo state loopholes during agentic evaluation. GitHub Issue. 2025. URL https:
git cwm/ www-weeeee......e. ToME-bench. issuess/////ss /46 Issue 465, SWE-bench reEbbenchrpposioor
Patrick Kon, Jiachen Liu, Yiming Qiu, Weijun Fan, Ting He, Lei Lin, Haoran Zhang. Owen M Park, George S Elengikal, Yuxin Kang, et al. Iac-eval: A code generation benchmark for cloud infrastructure-as-oodo programs, Advances in Neural Information Processing Systems, 37: 134488 134506, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
VI Levenshtcin. Binary coors capable or "correcting deletions, insertions, and reversals. In Soviet physics-doklady, volume 10, 1966.
Nam Le Hai, Dung Manh Nguyen, and Nghi DQ Bui. Repoexec: Evaluate code generation with repository-level executable benchmark. arXiv e-prints, pp. arXiv-2406, 2024.
Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang. Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et al. Prompting large language models to tackle the full software development lifecycle: A case study. arXiv preprint arXiv:2403.088. 406000 2024a.
Hongwei Li, Yuheng Tang, Shiqi Wang, and Wenbo Guo. Patchpilot: A cost-efficient software engineering agent with early attempts on formal verification. arXiv preprint arXiv:2502.02447, 2025.
Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. Evocodebench: An evolving code gen- eration benchmark aligned with real-world code repositories. arXiv preprint arXiv:2404.0000 00599, 2024b.
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang. Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36:42330 02022333--433....
Shanchao Liang, Yiran Hu, Nan Jiang, and Lin Tan. Can language models replace programmers for coding? repocod says' yet'. arXiv preprint arXiv:2410.21647, 2024
Shanchao Liang, Nan Jiang, Yiran Hu, and Lin Tan. Can language models replace programmers for coding? REPOCOD says 'not yet'. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 24698-24717, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-9997777222 doi: 18.15/33/v// 2025.acl lan..a.lllooo.1244 URL aaclatoo/ccllaahhoooogooooo
act.:/00.......0..._o..... 1204/
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023.
Amirabbas Majd, Mojtaba Vahidi-Asl, Alireza Khalilian, Ahmad Caraani-Dastjern and Bahman Zamani. Code4bench: A multidimensionnal benchmark of codeforces data for different program analysis techniques. Journal of Computer Languages, 53:38-52, 2019.
Niels Mündler, Mark Müller, Jingxuan He, and Martin Vechev. Swt-bench: Testing and validating real-world bug-fixe fixes with code agents. Advances in Neural Information Processing Systems, 37: 81857-81887, 2024.
12

---

Published as a conference paper at ICLR 2026
Sarthak Munshi, Swapnil Pathak, Sonam Ghatode, Thenuga Priyadarshini, Dhivya Chan- dramoulceswaran, and Ashutosh Rana. Acse-eval: Can Ilms threat model real-world cloud in- frastructure? arXiv preprint a551 2025.
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al. MI- gym: A new framework and benchmark for advancing ai research agents. arXiv preprint arXiv:2502. 14499, 2025.
Ziyi Ni, Huacan Wang, Shuo Zhang, Shuo Lu, Ziyang He, Wang You, Zhenheng Tang, Yuntao Du, Bill Sun, Hongzhang Liu, et al. Gittaskbench: A benchmark for code agents solving real-world tasks through code repository leveraging. arXiv preprint arXiv:2508 18993, 2025.
OpenAl. Introducing openai o3 and 04-mini, 2025. URL https:/ http://oppaaa...m/mmmaaaaaa com/index/
introducing and-o4 o4-mini/
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Qwen-Team. Qwen3 technical report, 2025. URL URL.acups.//aaxxx..orr/////////p
Kalahasti Ganesh Srivatsa, Sabyasachi Mukhopadhyay, Ganesh Katrapati, and Manish Shrivastava. A survey of using large language models for generating infrastructure code. arXiv preprint arXiv:2404.00227, 2024.
Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev. Beyond memorization: Violating privacy via inference with large language models. arXiv preprint 1rX02:2310.07298, 2023.
Yuheng Tang, Hongwei Li, Kaijie Zhu, Michael Yang, Yangruibo Ding, and Wenbo Guo. Co- patcher: Collaborative software patching with component (s)-specific small reasoning models. arXiv preprint arXiv:2505. / 18955, 2025.
TerminalBench. Terminal-bench: A benehmark for ai agents in terminal environments, Apr 2025.
URL htt.httts:///gt twu..hhhub.... com/ hatmmmlude-iinste tute / rmi nal -benhh)
David Tomassi, Naji Dmeiri, Yichen Wang, Antara Bhowmick, Yen-Chuan Liu, Premkumar Devanbu, Bogdan Vasilescu, and Cindy Lubio-Gonzáie Bugswarm: Mining and continuously growing a dataset of reproducible failures and fixes. In 2019 IEEF/ACM 41 International Con- ference Software Engineering (ICSE), pp. 339-349. IEEE, 2019.
Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jo- vanović, Jingxuan He, and Martin Vechev. Baxbench: Can Ilms generate correct and secure backends? arXiv preprint arXiv:2502.1 1844, 2025.
Wenhan Wang. Chenyuan Yang. Zhijie Wang, Yuheng Huang. Zhaoyang Chu, Da Song. Lingming Zhang, An Ran Chen, and Lei Ma. Testeval: Benchmarking large language models for test case generation. arXiv preprint arXiv:2406.04 04531, 2024.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma. Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng. Heng Ji, and Graham Neubig. Openhands: An open platform for AI soft- ware developers generalist agents. In The Thirteenth International Conference Learning Representations, 2025a. URL htt.httts: httpt:p:////peneviiiiie wwt.eeewwwwwwweeeeeeeeeeeeeeeeee teeeet//orummiddddddaaaaaaa
Yibo Wang, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S Yu, and Chen Xing. Projecttest: project-level Ilm unit test generation benchmark and impact of error fixing mechanisms. arXiv preprint 2502:22500.06556 2025b.
Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. Cybergym: Evaluating ai agents' cybersecurity capabilities with real-world vulnerabilities at scale. arXiv preprint arXiv:2506.0255 2025c.
13

---

Published as a conference paper at ICLR 2026
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489, 2024.
Kai Xu, YiWei Mao, XinYi Guan, and ZiLong Feng. Web-bench: A Ilm code benchmark based on web standards and frameworks. arXiv preprint arXiv:2505.07477 2025.
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The eight Annual Conference Neural Information Processing Systems, 2024a. URL
https://arxiie org/
holg/abbs abs 20.pb//24000 2405. 15793
Yu Yang, Yuzhou Nic, Zhun Wang, Yuheng Tang, Wenbo Guo, Bo Li, and Dawn Song. Seccodeplt: A unified platform for evaluating the security of code genai. arXiv preprint arXiv:2410.11066, 2024b.
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving. arXiv preprint orXiv2504.026655, 2025.
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Han-Guang Lou, and Weizhu Chen. Repocoder: Repository-Ievel code completion through iterative retrieval and generation. arXiv preprint 2tti::233..2577000 2023.
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07339, 2024a.
Quanjun Zhang, Ye Shang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. Testbench: Evaluating class-level test case generation capability of large language models. arXiv preprint 14XXv:2409.17661 2024b.
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1592-1604, 2024c.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, al. Bigeodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 15877, 2024.
Terry Yue Zhuo Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu. Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Le- andro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference Learning Representations, 2025. URL http:///pppeeeeeeeeeeeeee/ar////
A CLARIFICATION ON THE USE OF LLMS
The authors acknowledge the use of large language models, employed exclusively to assist with grammar correction, proofreading, and minor stylistic refinements throughout the manuscript. The of LLMs was strictly limited to language polishing and did not contribute to the research content itself. All core research, methodology, experimental design, and substantive findings represent the original work of the authors, who retain full responsibility for the content and conclusions presented.
14

---

Published as a conference paper at ICLR 2026
B DATASET CONTAMINATION PREVENTION
To prevent data contamination from pre-training corpora, we implement systematic detection pro- tocol to select repositories for DEVOPS- GYM. For each candidate repository, we sample 20 unique code snippets from CI/CD configurations, build files, and test files, which are domain-speciie artifacts less likely to appear in general pre-training data. Following the prefix completion ap- proach (Carlini et al. 2021; Staab et al.) 2023), we first randomly select a start point, then extract 50-token prefix from each snippet and prompt the model to generate a continuation. We then assess contamination by comparing the model's 50-token prediction with the ground truth suffix g from the original code.
We measure similarity using five complementary metrics: (1) normalized Levenshtein distance ratio 8 (Levenshtein 1966). (2) consecutive matching token count from the prefix t, (3) position of the first token mismatch p. (4) longest common substring length measured in tokens (Gusfield 1997), and (5) sentence-level BLEU score b (Papineni et al. 2002). A snippet is classified as high-risk any metric exceeds its respective threshold: 8 0.7, 30, p > 30, / > 30, or b 0.5. These thresholds are calibrated to identify cases where the model demonstrates suspiciously accurate re- production, indicating potential memorization. The repository-level contamination rate C' is defined as the proportion of high-risk snippets among the 20 samples.
The results for different repositories are shown in Table 2 To ensure evaluation integrity while maintaining sufficient dataset scale, we exclude repositories with C 2 0.2 and retain only those with LOW LOW-MODERATE contamination levels. This protocol ensures that our benchmark measures genuine reasoning and problem-solving capabilities rather than memorized patterns from pre-training. The 20% threshold balances contamination control with dataset scale. Our analysis shows that even the cleanest repositories (e.g.. (ramw-framework exhibit 10% contamination due to unavoidable surface-level similarities in public code (c.g., common API patterns, standard config- urations). The 20% threshold ensures that 80%+ of code snippets show memorization while maintaining sufficient scale for meaningful evaluation.
To prevent data leakage through repository history, implement striet isolation measures. Fol- lowing reports of contamination SWE-bench where agents could access future repository states containing solutions or implementation hints (Kahn 2025), we sanitize all repositories by remov- ing git metadata and provide only the codebase state at the point of issue creation, ensuring agents cannot query future commits and branches containing fixes.
C TASK DETAILS FOR ISSUE RESOLVING AND TEST GENERATION
C.I ISSUE RESOLVING
In typical DevOps workflows, once problems are identified, whether through monitoring alerts, user reports, test failures, or system crashes, engineers must translate these diverse diagnostic inputs into code fixes. This task evaluates agents' ability resolve bugs regardless of their discovery method. While issue resolving has been explored in benchmarks like SWE-bench, its inclusion here is essential for comprehensive DevOps evaluation, as fixing code remains a core responsibility whether the bug was found through sophisticated monitoring or simple error log. Agents must understand existing codebases, locate bugs based on provided descriptions, and implement minimal fixes that resolve issues while preserving the normal functionalities.
Input: (i) A buggy repository; (ii) Natural language bug description; (iii) Development tools (git, grep, sed, awk) and language-specific test runners.
Expected Output: Patch file in unified diff format that resolves the issue.
Ground Truth: developer-providee fixes from merged GitHub pull requests or expert- written patches for injected bugs.
Evaluation: Pass the Fail-to-pas pass test and didn't 't introduce new fails.
Data Collection: We adapt the Multi-SWE-be SWhrbech--hehhhh Zan et al. (2025) collection pipeline for data collection. The pipeline consists of three stages: (1) repository selection: we target
well-maintained Java/Go projects with comprehensive test suites and active development
15

---

Published as a conference paper at ICLR 2026
Table 2: Contamination risk analysis for different repositories across L.Mss.
The highest contami-
nation rate for each repository is marked as bold.
Repository
Max Contamination
Avg Contamination
GPTT-4o
Claude-Sonnet-4
act beego caddy
0.2000
0.1000
0.0000
0.2000
0.5625
0.2812
0.0000
0.5625
0.2667
0.2000
0.1333
0.2667
checkstyle
0,5000
0,4583
0,4167
0,5000
echo
0,2000
0.1000
0.0000
0,2000
eted
0,2222
0.1389
0.0556
0.2222
fastjson2
0.7333
0,5667
0.4000
0.7333
frp
0.2500
0.1250
0.0000
0,2500
fzf
0.1875
0.1562
0.1875
0.1250
gin
0.4000)
0.2667
0.1333
0,4000
go-zero
0.1765
0.1471
0.1176
0.1765
gorm
0.2500
0.1562
0.0625
0,2500
hugo
0.1250
0.0625
0.0000
0,1250
istio
0,3000
0,2000
0.1000
0.3000
junit-framework
0.1000
0,0000
0.1000
0,1000
lazygit
0,2308
0.1154
0.0000
0.2308
logstash
0.2222
0.1944
0.1667
0.2222
mockito
0.4000
0.3000
0.2000
0,4000
spotbugs
0.3333
0.2778
0.2222
0.3333
(commits within 6 months), (ii) PR filtering: we select pull requests that resolve GitHub issues and include test modifications, indicating test-validated fixes, and (iii) execution validation: verify that associated tests demonstrate fail-to-pass transitions, where tests fail on the pre-patch codebase and succeed after patch is applied, thereby ensuring both issue reproducibility and patch correctness.
C.2 TEST GENERATION
The third stage ensures that resolved issues cannot by creating comprehensive unit tests. While test generation has been explored in prior work Mündler et al. (2024), its inclusion here serves the DevOps pipeline where fixes must validated before deployment. Our setting mirrors real-world scenarios where test developers cannot assume patch correctness. Agents must generate tests in- dependently based solely on bug descriptions and source code examination, without access to the proposed fixes. This approach reflects common DevOps practice where unit tests serve as indepen- dent validation patch quality rather than mere confirmation of known solutions. Furthermore, this constraint also requires agents to fully understand the bug's root cause from description alone and create tests that would have caught the original issue, particularly challenging for performance and resouree issues identified through monitoring, which require specialized test assertions beyond functional correctness.
Input: (i) Buggy repository: (ii) Bug description; (iii) File manipulation tools (touch, echo, cat) and frameworks. Following established settings (Mündler et al., 2024). agents receive only the buggy repository and issue description without access to the ground truth patch or fix implementation.
Expected Output: Patch file for the test in diff format (e.g., pytest, JUnit) that cap- tures the bug's behavior.
Ground Truth: Developer-provided tests from GitHub PRs (when available) expert- written regression tests validated to detect the specific bug.
Evaluation: Following SWT- WWT-Benchh Mündler et al. (2024), we measure test effectiveness through fail-to-pass transitions. A test generation attempt succeeds when: (i) at least one
16

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation results end-to-end tasks for different agent frameworks and different L.MMs. The best result for each stage are marked as bold.
RepositoryTaskk
Model
Build & Config
Monitoring
Issue Resolving
Test Generation
Clande-4-onnet
minio epu-usage
GPT-5-mini
Claude Sonnet
Sonnet
minio fll-handle-teak
GPT-5-mini
Claude-4-Sonee
monio..io-lutency
GPT-5-mini
Clande-4-Sonnet
minio memory-leak
GPT-5-mini
Claude Sonnet
Sonnet
pocketbase epu-usage
GPTi--mmiii
Clande-4-Sonnet
GPT-S-mini
Clande-4-Sonet
pocketbase... Beaccbase__oo-lttenee lo-letecyy
GPT-5-mini
Claude-4-Soneet
pocketbase memory-leak
GPT-5-mini
Claude Sonnet
spring-eocclmiic..cpuuuss
GPT-5-mini
Clande-4-Soneet GPT-5-mini
Clande-4-Sonnet
pytelipp memory--edt
GPT-5-mini
Clanee-4-oonet
file-handle-eakk
GPT-5-mini
Claude-4-Sonnet
tidb_ memoommmoyy
GPT-5-mini
Cluude-4-Sonnet
udb Bib__sgl-hue tel-hanlle-taak
GPT-5-mini
generated test fails on the original buggy codebase and passes after patch application (fail- to-pass transition), and (ii) no tests exhibit failures the patched code. This metric ensures that generated tests accurately capture the bug's behavior while avoiding false positives. We report the success rate the percentage of tasks where agents produce valid fail-to-pass tests without introducing spurious failures.
Data Collection: Test generation tasks are derived from the same issues collected for issue resolving phase. For each issue resolving task, we create a corresponding test generation task.
D EXPERIMENTAL SUPPLEMENT
D.1 GENERATION AND ENVIRONMENT.
We use vLLM Kwon et al. 2023) to host Qwwn3-Coder-3000 on our local server, and all other mod- els are accessed via their official APIs. For the hyper-paramen parammerrseeters of model settings, employ temperature of 0.7 and top-p sampling of 0.95, with a maximum context length of 256K tokens. Ad- ditionally, we implement a 60-second timeout with up retry attempts ensure robust inference. All experiments are executed within the isolated and reproducible Docker container environments provided by DEVOPS-GYM.
D.2 ERROR ANALYSIS FOR MONITORING TASKS
We also identified four common error types for monitoring tasks from the agent run logs. First, inadequate monitoring methodology (37%): Agents used one-time commands (e.g., top) instead
17

---

Published as a conference paper at ICLR 2026
Table 4: Performance across five independent runs sampled subset of 50 tasks.
Agent
Model
Round1
Round2
Round3
Round4
Round5
Mean(STD)
Pass@5
Claude Code
Claude-4-Somne
16.00%
16.00%
20.00%
20.00%
22.00%
18.80% (2.40)
26.00%
OpenHands
o4-mini
14.00%
16.00%
18.00%
20.00%
16.00%
16.80% (2.04) 20.00%
of continuous monitoring (e.g., watch -n 1), occasionally yielding coincidental correct results on error-free instances. Second, pemature conclusion (26%)) Agents submitted answers without performing monitoring or completing diagnostic procedures. Third, insufficient temporal granularity (11%)) Agents monitored correctly but used overly coarse sampling intervals (10-60s), missing transient anomalies like CPU spikes. Lastly, interpretation failure (26%): Agents collected metrics correctly but failed to analyze them accurately or provided no analysis.
D.3 STABILITY AGENT PERFORMANCE ACROSS MULTIPLE INDEPENDENT RUNS
To evaluate the stability of agent performance, we randomly sampled 50 tasks from DEVOPS-GY and executed two representative agent-model pairs Claude Code Claude-4-Sonee and Open- Hands + 04-mini-across five independent runs. Table the results. Both agents exhibit consistent single-run performance, with Claude Code ranging between 16-22% and OpenHands be- tween 14-20%. The mean accuracies across the five runs are 18.8% and 16.8% with the standard deviation of 2.40 and 2.04, respectively. indicating that run-to-run variance relatively small. When aggregating over the five trajectories, success rates increase modestly (Claude Code: 26%, Open- Hands: 20%), but this improvement does not materially affect the relative ranking of the agents or the conclusions of our study. These findings confirm that DEVOPS -YMM yields stable results and that model performance is not sensitive to randomness agent execution.
E CASE STUDY
E.I BUILD
This case shows that the agent fails to properly handle test execution failures while still achieving the core migration objectives.
(Task)
You are tasked with migrating the MobArena plugin build system from Maven to Gradle while maintaining functional equivalence.
The environment contains:
Maven project /app/mobarena/ this is your starting point
Build artifactss and logs from the Maven build in happ:/reaults//odbuuidd. and /app/resslltsolld...og
Your task is to:
Start with the Maven project at /app/mobarena/
Create a complete Gradle build configuration (build.graile, propereis,,, etc.)
Migrate all Maven functionality to Gradle equivalents
Ensure the migrated build maintains functional equivalence with the original Maven build
Success criteria:
18

---

Published as a conference paper at ICLR 2026
The Gradle build completes successfully with /gradlew clean build
Al1 tests pass with /graadlew test
The generated JAR is functionally equivalent to the original Maven JAR
Build produces the same core artifacts as the Maven version
(Agent Action):
I'll start by examining the Maven project structure and understanding the current build configuration plan the migration to Gradle.
function-codebase.sean
query="Nhat is the current Maven build configuration and project structure?"
(Agent Action)
New I need to understand the Maven build configuration by examining the pom. xml file see what dependencies, plugins, and build settings need to be migrated.
function=read.fille
[... extensive migration work creating build. gradle, configuring dependencies, setting up Gradle wrapper ...]
(Agent Action)
Now let me run the tests to see if they all pass as expected. function=bash
command="cd /app/mobarena & ./gradlew test --info | grep -E (Test PAASED FAILED SKIPPED BUILD)
The tests are zunning but many are failing. This seems to be related to missing runtime dependencies and mocking issues.
BUILD FAILED in 6s 253 tests completed, 88 failed
* [Comments: Agent successfully migrates the build system and produces correct artifacts, but fails to resolve all test failures. The core issue is that while the agent achieves the primary migration goals (successful build, correct JARS, functional equivalence), doesn't ensure all tests pass, which was explicitly listed
success criterion.)
This case shows that the agent fails to include Caddy-specific build tags in the GoReleaser configu- ration.
(Task)
You are tasked with setting up GoReleaser for the Caddy project to enable multi-plan multi-platform releases.
19

---

Published as a conference paper at ICLR 2026
The environment contains:
A Go project at /app/caddy-projoeeee - this is your starting point
Build artifacts and logs from the basic Go build in /app/results/oldauuidd and /app/results/ole--
GoReleaser is pre-installed and available in the PATH
Your task is to:
Start with the basic Go project at /app/cadyy-poojett
Create .goreleaser.yn configuration file with modern GoReleaser syntax
Configure multi multi-platform builds for Linux, Windows, macOs, and FreeBSD
Set up cross-compilatienn for multiple architectures (amd64, arm, arm64, s390x, ppc6lle, riscv64) with appropriate platform exclusions
Configure archive generation with proper platform naming
Configure build tags and Idflags for optimal binary generation The configuration should include:
Cross Cross-platform builds with CGO disabled for static binaries
Build configuration for cmd/caddy/main.go structure
Archive replacements for clean platform naming (macos, Linux, Windows)
Windows-specific ZIP format override
Changelog generation with proper filtering
Checksum configuration with SHA512 algorithm for release integrity
Release configuration with draft mode enabled to prevent accidental releases
Success criteria:
GoReleaser configuration validates successfully with goreleaser check
Snapshot build completes with goreleaser build --saphhott --clean
Multiple platform binaries are generated (Linux, Windows, macOS, FreeBSD)
Generated binaries maintain original functionality
Archive naming follows CoReleaser conventions
(Agent Action):
need to start by analyzing the codebase structure to
understand the Go project layout and then examine any existing build artifacts to understand the current build process.
20

---

Published as a conference paper at ICLR 2026
cd /app/caddy-project & find -type f -name "*.go" -20
head
(Agent Action):
Good, I can see this is a Caddy project with the main entry point at cemd/caddy /m/im. Now 1et examine the project structure more thoroughly and check if there are any existing configuration files.
cd lapp//addy-projett & 1s -la & echo "=== Checking for existing config files ===" find -name "*, "*.yml" -o -name "*.yaml" -0 -name "oo" -o -name "Makefile" | head -10
(Agent Action):
Now I understand the project structure and version handling. Caddy uses Go modules for version information, I don't need custom idflags for version injection. Let me create comprehensive GoReleaser configuration file that meets all the requirements.
cd /app/caddy-proaee project & cat <<> BOF' goreleaser.ym.
GoReleaser confiquration for Caddy
This configuration enables milti-platform releases with proper cross-compilatie
project.name:
caddy
before:
hooks:
Clean up any previous builds
go mod tidy
go mod download
builds:
id::
caddy
Build from the main entry point
main:
./mmd/caddy/main,
binary:
caddy
Disable CGO for static binaries
env:
CGO ENABLED=0
Build flags for optimization
flags:
-trimpath
Linker flags for smaller binaries
1dflags:
- w
21

---

Published as a conference paper at ICLR 2026
-extldflags "-static"
Mclti-platform builds
goos:
linux
windows
darwin
freebad
Multi-architecture builds with appropriate exclusions
goarch:
amd64
arm
arm64
s390x
ppc641e
riscv54
BOF
(Comments: The agent creates a comprehensive GoReleaser configuration but fails to include Caddy-specific build tags (nobadger, nomysql, nopgx) that are essential for proper Caddy builds. The confiquration validates syntactically but is functionally incomplete for the Caddy project.]
E.2 MONITORING
This case shows that the agent fails to monitor the system continuously.
(Task):
You are tasked with monitoring server program execution identify any potential unusual patterns and potential issues.
Your environment contains:
A server setup: a server is running.
Standard system monitoring tools (curl, 1s, top, ps, htop, watch, etc.)
Your task is to:
1) Monitor the system behavior (e.g., server network requests, system resource usage) time,
2) Use standard system monitoring tools to observe system behavior,
3) Identify any unusual patterns
22

---

Published as a conference paper at ICLR 2026
(Agent Action):
I'11 monitor the minio server system behavior over minutes to idertify resource usage patterns and potential issues. Let me start by setting up a comprehensive monitoring approach.
function=Todwwiite content="1. Check initial system state, 2. Setting up continuous monitoring for CPU, memory, and disk usage, 3. Monitor network connections and file descriptors, 4. Monitor I/O operations and latency."
(Agent Action):
Now let me observe the system.
1s -la /proc/14/fd
cat /proc/14/io
* [Comments: Agent should monitor the system continuously,
e. e.g., watch -n cat /proc/14/io}
(Tool Execution Outputs):
This demonstrates that agent wrongly get final conclusion due to limited context length.
(Task):
You are tasked with monitoring a server program execution on to identify any potential unusual patterns and potential issues.
Your environment contains:
server setup: server is running.
Standard system monitoring tools (curl, 1s, top, ps, htop, watch, etc.)
Your task is to:
1) Monitor the system behavior (e.g., server network requests, system resource usage) over time,
2) Use standard system monitoring tools to observe system behavior,
3) Identify any unusual patterns
(Agent Action):
I'll monitor the server system behavior.
watch -n 1 "echo $ (date) ===;; ; top -b -n 1 | head -20" >> /monitor cpu.log
watch -n 1 "echo --sort=-$mem head -10" >> /monitor.memorrt log
$ (date) __=²; free -h; echo '; ps aux
watch -n 3 "echo -== $ (date) ==='; ; echo 'Open FDs:'; 1s /proc/1/Ed 2>/dev/null >> /monitor fd.
-1; isof -p 2>/dev/nul -1"
23

---

Published as a conference paper at ICLR 2026
Table 5: DevOpsBench Dataset Statistics
Stage
Languages
# Tasks
Repo Num
Avg. Repo Files
Avg. Human Time(min)
Build & Config
Java&Go
48
26
1579
53
Monitoring
Java&Go
30
3343
21
Issue Resolving
Java&Go
310
16
2162
35
Test Generation
Java&Go
2162
Table Evaluation Results on different error types of build implementation
Agent
Model
Migration
Target Release
Plugin Integration
Version Upgrades
Qwen3-Coder-30R
1/8
0/5
1/3
04-mini
1/8
0/5
1/2
1/3
OpenHands
DeepSeek-V3. V3.1
0/8
0/5
Q/2
1/3
Gemini-2.5-PP
0/8
0/5
0/2
1/3
Claude-4-Someet
0/5
1/3
mini-SWWE-Agen
Claude
2/8
1/5
1/2
2/3
Claude Code
Claude-4-Sonec
1/5
2/3
(Agent Action): : Now 1et me check the CPU log file to detect potential system anormalies.
cat /monitor.cpu.log
(Tool Execution Outputs):
(CPU log files)...
(Agent Action):
This session is being continued from previous conversation
that ran out of context. The conversation is summarized below:
(Agent Action):
According to the summary can see that CPU activity is abnormal. So the answer is cpu usage.
+ (Comments: Agent should check other log files. It successfully monitor the system but fail to analyze the results.)
24

---

Published as a conference paper at ICLR 2026
Memory Leaks (17.1%)
Normal Cases (14.3%)
Disk Leaks (8.6%)
SQL Inefficiencies (14.3%)
Handle Exhaustion (14.3%)
I/O Bottlenecks
(14.3%)
CPU Spikes
(17.1%)
Figure 3: Monitoring anomaly distribution.
25


Reviews:
[{"review_id": "8tysegM8Ff", "reviewer": "Reviewer_uo9F", "summary": "This paper presents DEVOPS-GYM, a benchmark created to test how well AI agents can handle the full software DevOps cycle, including building and configuration, monitoring, bug fixing, and test generation. It goes beyond previous benchmarks that focus only on coding or bug fixing by including real DevOps tasks that require tool use, environment setup, and step-by-step decision making. The dataset contains 698 tasks collected from over 30 open-source projects in Java and Go, combining real GitHub issues with tasks designed by experts to cover common DevOps problems.\n\nThe benchmark was built with attention to realism and fairness. The authors manually recreated build environments, monitored runtime behavior, and avoided data contamination from existing model training data. Each task type evaluates a different part of the DevOps process: building tasks test if the agent can fix dependency or build errors, monitoring tasks test detection of issues like memory leaks or CPU spikes, issue-resolving tasks test whether agents can produce correct patches, and test-generation tasks check if agents can create regression tests that fail on buggy code and pass after the fix.\n\nResults show that even strong AI agents and large models still perform poorly on DevOps tasks. The best agent achieved around 58% accuracy in build and configuration, but less than 25% in monitoring, issue fixing, and test generation. The study finds that current systems struggle to plan multiple steps, use DevOps tools properly, and reason about dynamic system states. It concludes that existing models are far from being able to automate the entire DevOps cycle.", "questions": "1) The benchmark includes 698 tasks from 30+ Java and Go projects, but DevOps typically spans far more diverse environments and languages. How confident are the authors that these two languages and this limited set of projects represent the broader DevOps space, including cloud-native and containerized systems?\n2) The paper claims to prevent contamination using prefix-completion tests, but no quantitative contamination rates are reported. Could the authors provide clearer evidence or benchmarks for how effective this process was across repositories?\n3) The benchmark covers four stages but omits key aspects of DevOps such as deployment, CI/CD automation, rollback, and infrastructure management. Do the authors plan to extend to these in future work, or do they believe these tasks require fundamentally different agent capabilities?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper’s main strength lies in its clear motivation and solid contribution to an underexplored area. It identifies that while many existing benchmarks test AI on programming or debugging tasks, none assess performance across the complete DevOps pipeline, which involves building, monitoring, fixing, and testing software systems. By defining this broader scope, the authors push evaluation toward real-world DevOps scenarios that require understanding of environments, tool interaction, and sequential task execution. This is a meaningful extension beyond static code generation benchmarks and provides a realistic testbed for future research.\n\nAnother strong point is the benchmark’s detailed and careful construction. It covers nearly seven hundred real-world tasks from more than thirty Java and Go projects, including both real GitHub issues and synthetic cases designed by experts to represent frequent DevOps problems. Each task is clearly defined with inputs, expected outputs, and evaluation criteria. The inclusion of four DevOps stages—build/configuration, monitoring, issue resolving, and test generation—ensures comprehensive coverage. The benchmark also integrates command-line tools commonly used in DevOps such as Maven, Gradle, JUnit, and iostat, allowing agents to operate in realistic conditions instead of simulated text environments.\n\nThe authors use contamination checks to prevent overlap with training data, manually reconstruct real-world project environments, and validate that each task is reproducible. They also test multiple well-known agent frameworks and models, providing a broad and transparent performance comparison. Their analysis goes beyond simple accuracy reporting; they identify specific failure patterns such as agents’ difficulty with tool usage, temporal reasoning, and multi-step planning. This combination of realism, technical rigor, and analytical depth makes the benchmark a valuable and trustworthy foundation for studying AI in software DevOps automation.", "weaknesses": "The paper, while ambitious and valuable, has several weaknesses that limit its overall contribution.\n\nFirst, the scope is too broad relative to the experimental depth. Although it claims to evaluate the entire DevOps cycle, the number of tasks for some categories—such as monitoring or build/configuration—is relatively small compared to the hundreds of issue-resolving and test-generation cases. This uneven distribution makes the evaluation appear unbalanced, and it is unclear whether the benchmark fully captures the complexity of real-world DevOps workflows. Some parts, like deployment or CI/CD automation, are also missing, which weakens the claim of “end-to-end” coverage.\n\nSecond, the dataset construction process is heavily manual and lacks scalability. The paper mentions that each task can take over ten hours of expert work to reproduce and verify, suggesting that expanding the dataset or maintaining it over time will be difficult. The strong reliance on expert labor reduces the reproducibility and accessibility of the benchmark. In addition, while the authors describe contamination checks, they provide limited quantitative details or validation of how effective these checks are. Without transparency on contamination metrics or repository overlap, it’s hard to judge the fairness of the evaluation.\n\nThird, the evaluation analysis is somewhat surface-level. While the results show that agents perform poorly, the discussion of why these failures occur remains descriptive rather than diagnostic. For example, the paper reports that agents fail in tool usage or temporal reasoning but does not analyze whether this is due to prompt design, insufficient model context, or poor environment instrumentation. There is also limited analysis of error types or task difficulty variations. The work lacks ablation or controlled experiments that could explain which aspects of the benchmark are truly challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SuVvo52HRG", "reviewer": "Reviewer_EL6m", "summary": "This paper introduces DevOps-Gym, a benchmark for evaluating AI agents across the full DevOps cycle, covering build & configuration, monitoring, issue resolving, and test generation. The benchmark includes 698 tasks from 30+ real-world Java and Go repositories. Evaluation of state-of-the-art agents reveals poor performance across all stages, with particularly severe failures on monitoring (0-23.33%) and build tasks (12.5-58.33%). While this addresses a genuine gap in existing benchmarks, the execution has significant methodological weaknesses that limit the contribution's impact. There are some concerns outlined below.", "questions": "1. **Statistical Rigor:** Why no confidence intervals, multiple runs, or significance tests? This is standard practice for benchmark papers. With small sample sizes (30 monitoring, 48 build tasks), single-run results are highly unreliable and prevent drawing meaningful conclusions about agent capabilities.\n\n2. **Synthetic Task Validation:** Do you have some evidence that expert-injected bugs match real-world distributions? The work relies heavily on synthetic tasks for monitoring and build stages but doesn't provide validation that these reflect actual production issues. Without this validation, it seems incorrect to claim the benchmark represents \"real-world\" DevOps challenges?\n\n3. **Ground Truth Agreement:** It might be good to include inter-annotator agreement scores. The paper mentions \"three senior DevOps engineers independently validate\" monitoring tasks and use multiple experts for other stages, but report on agreement metrics (Cohen's kappa, etc.). If experts disagree, then the validity of the canonical answer is questionable, which limits the benchmark credibility.\n\n4. **Contamination Threshold:** Why allow up to 20% contamination instead of requiring something much smaller? The paper talks about decontamination procedures but then also includes repositories with substantial contamination. The standpoint is a bit confusing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Addresses Real Gap:** First benchmark attempting end-to-end DevOps evaluation\n2. **Realistic Task Design:** Docker environments with real tools better simulate practice than synthetic environments, and covering diverse DevOps stages (though incomplete)\n3. **Extensive Manual Effort:** Authors invested significant time in task construction and validation\n4. **Negative Results Are Valuable:** Showing current agents fail badly on DevOps tasks is an important finding\n5. **Practical Relevance:** High potential impact for industrial applications", "weaknesses": "1. **No statistical rigor**: Missing confidence intervals, significance tests, multiple runs; single-run results unreliable given tiny task counts.\n2. **Small dataset**: Only 30 monitoring and 48 build tasks, which is far smaller than benchmarks like SWE-bench, so this limits generalizability.\n3. **Limited analysis**: The paper reports binary accuracy for monitoring, but there is no partial credit or false-positive analysis provided.\n4. **Limited baselines**: Excludes major frameworks (Devin, Aider, etc.), so it's unclear if the results are significantly better than SOTA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xGsIOm55mM", "reviewer": "Reviewer_NdRZ", "summary": "This paper presents a benchmark, DevOps-gym, for evaluating agents on end-to-end DevOps workflows. The authors curate a benchmark of nearly 700 tasks from 30+ repositories in Java and Go languages. The main contribution of the paper is in covering the entire spectrum of DevOps operations - build, monitor, fix and test. The benchmark comes with pre-set runtime environments and tools. The experimental results on a few agents and multiple LLMs show that they succeed to varying degrees across the task categories, and generally well-below the success rate on popular benchmarks like swe-bench.", "questions": "- How many tasks out of 698 are real and how many are synthetic? What is the distribution by problem types (e.g., memory leaks, build failure, etc.) between real vs synthetic tasks? What is the reason those cannot be found naturally in the wild if your task distribution is to capture the real-world distribution?\n- How would the performance of the LLMs change if DevOps-specific agent harness is prepared and made available to them?\n- How does the performance of the agents change with test-time scaling, e.g., sampling multiple trajectories?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper makes a comprehensive contribution, covering all phases of DevOps operations. The benchmark is based on real-world repositories and targets languages common in industry like Java and Go (compared to benchmarks that target Python). The experimental setup is well-designed.\n- The paper is written very nicely. The key aspects of design of the benchmark are explained properly, showing both the diversity of sub-tasks within each task categories, and clearly specifying input, output and evaluation metrics. The analysis of results is also insightful.\n- The experimental evaluation is supportive of the value of this benchmark, by demonstrating that the current agents and LLMs are not very successful on this benchmark and their performance varies across task categories.", "weaknesses": "- The benchmark curation involved synthetic data generation where faults are injected by experts. The paper claims that they are inspired by real-world scenarios, but the adequate details are not provided.\n- The paper uses OpenHands and mini-SWE-Agent as harnesses. Though they may allow access to the terminal, their tools are primarily designed for assisting agents in GitHub issue resolution. No effort is made to design an agentic harness that specifically aids the agent in DevOps tasks.\n- It is well-known that test-time scaling can improve the performance of agents substantially. The paper does not conduct scaling experiments.\n- The benchmark does not cover the CI/CD aspects of DevOps tasks and does not throw light on how these agents could operate in CI/CD pipelines with platform-specific tools, e.g., GitHub actions.\n- Typo: interpreting their returns", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bq7r1dMyxO", "reviewer": "Reviewer_jCaQ", "summary": "The paper proposes a benchmark for evaluating SWE Agents on four different activities frequently encountered in the software development cycle: build/configuration, monitoring, bug fixing and test generation. They source and vet tasks for Java and Go and evaluate across popular coding agents and models. Their analysis shows that current systems fall short on configuration and monitoring tasks.", "questions": "1. What are guards against agent finding trivial ways to resolve tasks. For example in the build/configuration task, what if the agent modifies the build scripts to pass trivially? \n\n2. How is evaluation performed for the monitoring task. The paper mentions that the LLM has to identify the type of anomaly? How is the response from the LLM evaluated to determine correctness?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents a novel perspective to evaluating coding agents. The tasks presented in the paper go beyond popular bug-fixing setting and are sourced from Java and Go projects as opposed to the popular python setting. Especially the monitoring tasks which requires the system to interact with the environment over time. The difference in performance between agents also indicates the tasks to be a good discriminator. Evaluation with this benchmark maybe useful to practitioners working on industrial codebases that require maintenance beyond fixing bugs.", "weaknesses": "1. Two of the four tasks are not novel: bug-fixing and test generation are well explored in existing literature.\n\n2. While the paper presents useful contributions to practitioners who may want to evaluate SWE-agents for DevOps use cases, the paper offers limited insight into the behaviour of current agent frameworks or models, limiting usefulness to agent and model developers and hence relevance to this venue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)