ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks
Authors: Saurabh Jha
Rohan R. Arora
Yuji Watanabe
Takumi Yanagawa
Yinfang Chen
Jackson Clark
Bhavya Bhavya
Mudit Verma
Harshit Kumar
Hirokuni Kitahara
Noah Zheutlin
Saki Takano
Divya Pathak
Felix George
Xinbo Wu
Bekir O Turkkan
Gerard Vanloo
Michael Nidd
Ting Dai
Oishik Chatterjee
Pranjal Gupta
Suranjana Samanta
Pooja Aggarwal
Rong Lee
Jae-wook Ahn
Debanjana Kar
Amit Paradkar
Yu Deng
Pratibha Moogi
Prateeti Mohapatra
Naoki Abe
Chandrasekhar Narayanaswami
Tianyin Xu
Lav R. Varshney
Ruchi Mahindru
Anca Sailer
Laura Shwartz
Daby Sow
Nicholas C. M. Fuller
Ruchir Puri
Venue: ICML 2025 oral (2025)
Keywords: Benchmark
GenAI
Agents
IT Automation
Primary area: applications
TL;DR: Benchmark for IT automation tasks

Abstract: Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. IT-Bench includes an initial set of 102 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios, 25.2% of CISO scenarios, and 25.8% of FinOps scenarios (excluding anomaly detection). For FinOps-specific anomaly detection (AD) scenarios, AI agents achieve an F1 score of 0.35. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast. IT-Bench, along with a leaderboard and sample agent implementations, is available at https://github.com/ibm/itbench.

ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks
Saurabh Jha
Rohan Arora
Yuji Watanabe
Takumi Yanagawa'
Yinfang Chen
Jackson Clark?
Bhavya Bhavya'
Mudit Verma'
Harshit Kumar' Hirokuni Kitahara Noah Zheutlin
Saki Takano'
Divya Pathak' Felix George' Xinbo Wu?
Bekir O Turkkan Gerard Vanloo
Michael
Nidd
Ting Dai
Oishik Chatterjee' Pranjal Gupta' Suranjana Samanta Pooja Aggarwal' Rong Lee'
Jae-wook Ahn
Debanjana Kar!
Amit Paradkar' Yu Deng' Pratibha Moogi Prateeti Mohapatra
Naoki Abe!
Chandrasekhar Narayanaswami Tianyin Xu Lav R. Varshney2 Ruchi Mahindru'
Anca Sailer
Laura Shwartz'
Daby Sow! Nicholas C. M. Fuller' Ruchir Puri!
Abstract
1. Introduction
Modern IT systems are driving many facets of our economy. They have grown significantly in complexity with the adop- tion of cloud computing and agile development practices (Harvard Business Review Research Report, 2022; Trask, 2025). Effective management of these systems is becom- ing extremely challenging as corporations struggle to keep with this growing complexity. Various IT personas- ranging from Chief Information Officers to Site Reliability Engineers and Security and Compliance officers-and and IT engineers in general are struggling to ensure resiliency, reli- ability, security, and cost effective operations of IT Systems.
Realizing the vision of using Al agents to au- tomate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, framework that offers systematic methodology for bench- marking AI agents to address real-world IT au- tomation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design en- ables AI researchers understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable met- rics. ITBeench includes an initial set of 102 real- world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 1 1..4% of SRE scenarios, 25.2% of CISO see- narios, and 25.8% of FinOps scenarios (excluding anomaly detection). For FinOps-specifie anomaly detection (AD) scenarios, AI agents achieve an Fl score of 0.35. We expect ITBench to be key enabler of Al-driven ITT automation that is correct, safe, and fast. ITBench, along with a leaderboard and sample agent implementations, available https://github..co//mm//ttttttet
The recent CrowdStrike outage highlighted these challenges as it brought down our society's 's most critical systems- from hospital services to air travel-and was estimated to cost US Fortune 500 companies a staggering S5.4 billion (Kerner, 2024). This incident underlined the critical need for intelligent IT incident resolution, with compliance and risk management capabilities, topic also addressed in the Digital Operational Resiliency Act (DORA) in Europe (Par- liament and the Council of the European Union, 2024).
The rising popularity of Al agents and their projected ability to handle intricate tasks have increased the demand for AI agents managing IT systems (John, 2024; Miguel Carreon, 2024; Pujar et al., 2023). Given the complexity of IT tasks, major hurdle for this research is establishing systematic methods to assess the effectiveness of AI agents prior to their production deployment (Bogin et al., 2024; Kapoor et al., 2024). Consequently, there is an urgency to develop methods for evaluation of AI agents based on real IT tasks and their corresponding environments.
Eqqual contribution Urbana-Chai Champaign. <Saurabh.Jha ha @ @aa@@ibm.co com>..cm>.
'BM University of Illinois at Correspondence to: Saurabh Jha
This paper addresses this critical need and presents ITTBench, first-of-its-kndd framework that is both comprehensive and visionary for benchmarking real-life IT automation tasks. The goal of ITBench is to measure the performance of AI
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ITBench
SRE Ensures app resilience and performance
CISO
FinOps Manage IT spend
Manage threats and assess policies
Resolve "High error rate service
Assess compliance posture for "new control rule detected for RHEL 9." Assess and report critical risks across the failing controls.
A
Resolve "IT' spend exceeded the budget." "
order-management." Backup directory 'foo' to data lake.
Report return investment per application.
Sample personas and IT tasks. Bell icon represents event-triggered tasks, Information icon represents other tasks such data analysis. preventive maintenance tasks, continuous optimization.
Figure I:
agents across a wide variety of complex and real-life IT tasks across personas, including Site Reliability Engineering (SRE), focusing on availability and resiliency; Compliance and Security Operations (CISO), ensuring compliance and security of IT implementations; and Financial Operations (FinOps), enforeing cost efficiencies and optimizing return on investment, among others (as shown in Figure 1).
tive problems, including incidents diagnosis and resolution, compliance assessments in regulated environments for new controls, and cost management events. In the future, we plan to expand on benchmark evaluation capabilities and include new benchmarks for additional IT processes. Cur- rently, ITBench comprises an initial set of 102 scenarios spanning across SRE (42), CISO (50), and FinOps (10), with respective successful scenario handling rates of 13.8%, 25.2%, and 25, 25.8% (refer Section 4).
ITBench aims to advance innovation and establish new stan- dards in the field. Our contributions can be summarized along the following three axes:
We believe that, similar to the highly influential SWEBench (Jimenez et al., 2024), our new lTBeech framework- which encapsulates and measures the ability of Al agents to auto- mate complex, real-world IT wasks-will spur a comparable acceleration in the performance of real-world IT Al agents.
Reflecting the real world: ITBBench addresses the IT automation requirements that are relevant and prevalent in production settings. SRE scenarios are based on real- world incidents observed our own SaaS products. CISO scenarios are based on CIS benchmark (for Internet Secu- rity, CIS), FinOps scenarios are identified by the FinOps Foundation (Foundation, 2025a) through key business outcomes.
2. Related Work
ITBench targets a comprehensive set of tasks for wide range of personas within IT automation. The initial release of ITBeecch focuses on evaluating scenarios within IT Op- erations (ITOps). Figure 1 illustrates currently targeted personas and exemplar tasks that they are routinely facing. There is clearly rising interest in developing benchmarks to evaluate AI and ML techniques in ITOpps with specific focus SRE, CISO, and FinOps.
Being open and extensible with comprehensive IT cov- erage: We view ITBBench as a central hub for benchmark- ing Al-driven solutions across diverse IT automation use cases. To support this, we provide IT benchmark suites and framework for vertical expansion (i.e., adding more scenarios) and horizontal expansion (i.e., adding more per- sonas), ensuring extensive coverage of IT tasks. ITBench is an open-source framework built with open-source tech- nologies, while allowing organizations with proprietary technologies to use it for developing and benchmarking their solutions.
TrainTicket Ticket (Zhou et al., 2018) provides 22 scenarios col- lected through an industrial survey of real-world incidents, using hardcoded faults in the TrainTicket application to fo- cus on fault localization. AIOpsLab (Chen et al., 2024a) provides 10 SRE-focused scenarios (referred as "prob- lems") utilizing a real environment (system) integration that allows interactive access text, time series, and tabular data, InsightBench (Sahu et al., 2024) provides 100 scenarios to analyze ticket data using static tabular data and synthetic scenarios. TSB-AD (Liu and Paparrizos, 2024a) focuses on anomaly detection with 40 synthetic scenarios.
Enabling automated evaluation with partial scoring: ITBench is designed to provide constructive feedback to drive improvements in the design of agentic solutions for IT problems. It includes a comprehensive evaluation framework and leaderboard that provide feedback to users at various stages of their agents' reasoning process.
ITBench provides push-button deployment and tooling for setting up environment, runtime agent, guardrail engine, as well as authorization and authentication. It allows develop- ers and researchers to build novel solutions for managing complex IT systems. Currently, ITBench addresses reac-
CIS- Benchmark (CIS, 2024) provides best practices for se- curing IT infrastructure. Despite the name of "benchmark," it offers only recommendation policies; it provides exper- imental platform. Recently, Cloud Native Compute Foun-
2

---

ITBench
Table 1: Comparison of ITBench with related benchmarks
Automated Evaluation
Benchmark
# of Scenarios
Personas and Tasks
Resolvable
Environment
Leaderboard
SRE: Incident Resolution,
CISO: Compliance Assessment,
ITBench (ours)
102
Real Env.
(verified)
FinOps:
Cost Management
Ticket
22
SRE:
Incident Diagnosis
X
X
Real Env.
x
AIOpsLab
10
SRE:
Incident Resolution
X
Real Env.
(unverified)
InsightBench
100
Ticket Data Analysis
X
Synthetic
TSB-AD
40
Anomaly Detection
Synthetic
CIS
1000+
Compliance/Sccurity Focal
X
n/a (info. only)
X
Note:
We are not aware of related benchmarks in the FinOps domain that go beyond scorecards.
dation (CNCF) Sandbox project (OSCAL-compass, 2024) released an SDK to support the translation of the CIS human readable formats into OSCAL (OSCAL, 2024). OSCAL was developed by the National Institute of Standards and Technology for programmatic usage compliance automa- tion. ITBench CISO automation leverages this technology to assess policy requirements.
vant for the CISO persona involves assessing the compliance posture for a "new control rule detected for RHEL 9." A fundamental challenge is to emulate such problems in a man- ageable testbed environment. A scenario environment an operational testbed in which a specific problem(s) occurs.
A scenario p generally corresponds a problem to be solved in ITBench. We formalize p as tuple < M, E,E,TD >, where the variables are as follows:
FinOps Foundation (Foundation, 2025a) provides bench- marks that compare cloud financial performance across or- ganizations and departments, focusing on KPIs such as re- source utilization efficiency, contract coverage, and cost apportionment. These benchmarks help assess cloud effi- ciency by evaluating internal and external metrics, fostering structured, collaborative approaches to cloud optimization.
Scenario Specification. M represents metadata and deploy- ment descriptors, for each scenario, which is stored in the Scenario Specs database ITBench (see Figure 2). Exem- plar metadata elements per scenario include scenario_ _ame, scenario _deecpppioiiciii scenario _oomain scenario_domain, scenario_clane scenario erenario_complextne complexity, and ble 2), which are defined below:
catarrio_rrunurutthh (see Ta-
While existing benchmarks are valuable resources for spe- cific tasks and use cases, and highlight the critical need for systematic benchmarking, they are limited in reflecting real-world IT problems, covering broad landscape, and automating evaluation. These limitations are addressed in ITBench, shown in Table
scenario_mand name is name given to a scenario. For example, a scenario in ITBench has the name "Recommendation Service Cache."
scenario description describes the scenario. An exam- ple of description of the scenario is "Recommendation Service in Astronomy Shop has cache failure."
3. ITBench
aeenario_domain scenario domain represents different personas within IT cntomation-namele "SRE," "CISO," "CISO, and "FinOps." scenario_class is used to group similar scenarios, such as "Kyvemo-opa,"" "Kyverno-update" "CacheFailure," "HighCPU," and "CorruptImage."
ITBeech a systematic benchmarking framework and run- time environment designed to evaluate Al agents tasked with automating IT operations, incorporating a robust archi- tecture (see Figure 2) comprising the AI Agent, Scenario Specification and Environment, Evaluator, and Leaderboard to facilitate comprehensive performance assessment.
ceenario_compleen captures the difficulty of a problem and is defined using domain knowledge. Figure 4a shows the breakdown of SRE, CISO, and FinOps scenarios in the bench. Figure 4b, 4c, and 4d shows scenario complexiit distribution for SRE, CISO, and FinOps, respectively. SRE scenarios are developed based on real-world inci- dents observed in our own SaaS products. CISO scenarios are based on CIS benchmark (for Internet Security CIS). FinOps scenarios were developed based "Domains" and "Capabilities" identified by the FinOps Foundation (Foundation, 2025a) to describe key business outcomes. ccenario_gronnding records task-specific outcomes that
Here, we present a brief overview of the key components: 1) Scenario Specification and Environment, 2) AI Agents, and 3) Leaderboard. More details are in Appendix B.
3.1. Scenario Specification and Environment
The bench incorporates a collection of problems that we call scenarios. For example, one of the problems in ITBench is to resolve "High error rate on service managemmang in Kubernetes environment. Another example that is rele-
3

---

ITBench
ITBench
Scenario Environment
Leaderboard
A Aplication Stack
Rank Publish
Agent Builder
Build & Register
Evaluator
o
Agent
Observability Stack
Metnc ljMemis 2)
Metric
Setup/Cleanup
Action
Info.
Sy steem state
per scenario
Agent result
B Benchmark Builder
Develop & Register
Scenario Specs
Benchmark Runner
Figure 2: ITBench automation framework.
the Evaluator uses to compare against the agent's expected output. For instance, incident resolution for SREs, the ground truth for the Diagnosis task includes a list of entities involved in the fault propagation chain, the actual fault propagation chain(s), and fault conditions, while for the Mitigation task, it captures plausible mitigation actions.
call
Agent f(o: 0 az-1)
APIs
Toolbox
output
SRE/FinOps
CISO
Nb2c
Generatak yverno
NL22Tiaess
GenersteO Rego
Environment
cmd
Me.2Metrics
GenieratPlaabbok
9t g(
NL2Logs
RunKubeect ubeett
0, h(s,)
info
NI.2A
Rego RunFlaybook
NL225cript
Environment, E represents an an operational testbed where the problem occurs. Components within the environment expose APIS observe and control the environment. When the Agent Builder registers the agent for benchmarking, the Benchmark Runner (see Figure 2) randomly selects set of scenarios, which may be optionally filtered based on the agent_type and agent_level, Next, the Benchmark Runner iterates through the set scenarios, and for each scenario it instantiates a testbed. An example of an environ- ment is Kubernetes cluster installed with OpenTelemetry Astronomy Shop Demo application (Community, 2024), ob- servability stack including Grafana (gra), Loki (lok), Jaeger (jae), and Prometheus (pro), along with mechanisms that induce problem(s) in the environment.
Figure 3: Agent and environment as a POMDP. Agents interact with the environment via the APIs exposed by IT- Bench's toolbox.
Scan assessment posture tasks. FinOps persona focuses on the cost management, where sample tasks include Identify inefficiency and Mitigate inefficiency. During evaluation, each step (task) is assessed independently and is measured using well defined metrics; see Table 3.
The goal of ITBench is to evaluate Al agents on broad range of real-world IT automation tasks that are otherwise performed by SREs, FinOps, and CISO personas.
In this paper, an Al agent is defined an autonomous or semi-autonomous software program that uses LLM to plan, make decisions, interact with the target environment, and execute actions to achieve goals. An AI agent is ex- pected to successfully handle any of the scenarios in the Bench, by interacting with the environment.
Triggering Events. T is a set of triggering events that occur due to manifestation of a specific problem the environ- ment. Tools are configured to observe the environment and raise triggering events on problematic conditions. An exam- ple of triggering event is "High Error Rate adservice," which may be triggered in the environment due to cache failure problem.
As shown in Figure 3, agent and environment form a Par- tially Observed Markov Decision Process (POMDP), where the state the snapshot of the environment. The state tran- sitions are determined by the environment, which are then (partially) observed by the agent.
Desired Outcome. defines the automation objective and represents the ultimate goal. For instance, in case of SRE incident resolution, the ultimate goal to clear T in the E.
Given scenario instantiated in an environment E, an agent probes the environment via one of the tools and re- ceives an observation 0g E O, based on which, it decides the next action:
3.2. AT Agents
In IT automation, the different personas are focused on a specific desired outcome, which defines their automation goals. For SREs, incident resolution the primary objective. Achieving this can involve multiple steps, such as diagnos- ing an incident, or single step, like generating diagnosis report. CISO persona focuses on the regulatory controls posture assessment process, including Collect evidence and
d2 fo, (1
Here f is the agent's decision function. o,- is the sequence of observations up to time t - 1 and ā is the sequence of actions taken up to 1.

---

ITBench
Table 2: Exemplar scenario classes and complexities in ITBench across 102 scenarios.
Scenario
Scenario
Scenario Class
Technologies
Domain
Complexity
CacheFailure: Create a memory leak due to an exponentially
SRE
Medium
K8s, Redis, MongoDB
growing cache
HighCPU: Trigger high CPU load in target service
Medium
K8. Host, Pods K8s, Image registry
Corruptimage: Deployment uses wrong Docker image
Easy
HTTPRequestBody TaaoeerFauar Modify HTTP Post request
Medium
K8s, ingress/egress
between services
UTTPReqqestAAoorFall Interrupt HTTP connection between
Medium
K8ss, ingress/cgress
services
MemoryResourceLiiit Reduce memory limit on target service
Easy
K8s, Host, Pod
CISO
New K8s De-bhenchmmrks Kyverno
Easy
K8, Kyverno
New DiShhenchmarks on OPA
Medium
K8s, OPA, Kubectl
New RHEL9 CiS-henchmarks on Ansible-OPA
Medium
RHEL9, OPA, Ansible
Update K8s SIS-benchmarks on Kyverno
Hard
K8s, Kyverno
MootAllltsicooofuuaatiiooo Alert threshold is too low, causing false alerts
FinOps
Easy
K8s, HPA
AutoscalerMissconiguaaioo Horizontal pod autoscaler thresh- olds are misconfigured. creating excess pods
Hard
K8s, HPA
Data Insights Generation: Analyze cloud bills and retrieve data based on natural language query
Easy, Medium, Hard
Natural Language SQL
Anomaly Detection & Ranking: Identify overspending events in cloud bill with regard forecasted spend amounts, and rank
Anomaly detection, forc- casting, data query from database
Hard
anomalies based on user-specified criteria
Scenario complexity depends on the characteristics scenario and is independent from agent capability. See appendixes details. K8s refers to Kubernetes (kub). Here, "technologies" denotes the tools/systems a domain expert must understand to perform the task.
9.8%
Hard
Hard
FinOps
Hard
20.0%
50.0% 00.0%%
CISO
49%
Easy
24.0%
Easy
20.0%
Easy 20.0%
$2.0%
60.0%
30.0%
Medium
Medium
Medium
(a) Scenarios by persona.
(b) SRE scenario complexity.
(c) CISO scenario complexity.
(d) FinOps scenario complexity.
Figure 4: Characterization of ITBench scenarios.
Initially, On may be a triggering event showing a problematic state 80 of the environment. Given state 8t-1 and action a;-1, the environment transitions the next state:
An agent reflects on the result to guide its next action, contin- uing until the final goal is achieved. Given set of scenarios that the agent works on, it targets to maximize the success defined follows:
St = g 8t- a+-1)
(2)
Epr 0: =()))
(5)
The observation or is determined and is in general proxy for the environment state 8t, hence the formulation can be thought of as POMDP:
function of the state
where Il an indicator function comparing the terminating state with goal state and is the distribution of scenarios.
= h(s))
3.2.1. 1. BASELINE AI AGENTS
The set A of actions is defined as QUI+}, where Q is the set of tools and 1. represents the "stop action" by the agent. We define t as the time when the agent stops:
We developed baseline agents SRE-Agent for SRE, Com- pliance Assessment Agent for CISO, and FinOps-agent for FinOps. Each of these agents uses state-of-ther art agentic techniques such as ReAct-based planning (Yao et al., 2023), reflection (Shinn et al., 2023), and disaggregation (Xu et al.,
t" min 1}
(4)
5

---

ITBench
Table 3: Personas, tasks, and metrics in ITBench.
such as git facilitate GitOps workflows for code manage- ment, and 4) deploy and execute the generated policies to accomplish the assessment task.
Personas
Tasks
Metrics
SRE
Diagnosis
pass 1, Fault Local- ization, Fault Propa- gation Chain, Mean Time Diagnosis pass 1, Mcan Time to Repair
3.3. Leaderboard
ITBench includes a leaderboard to promote reproducibility and comparative analysis, following the AI common task framework (Donoho, 2019; Varshney et al., 2019). The leaderboard offers a predefined, extensible set of perfor- mance metrics designed to provide clear insights into agent performance relative to the evaluation criteria.
Mitigation
CISO
Collect evidence Scan assessment posture
pass
pass@1, 1, Time to Pro- cess
FinOps
Identify inefficiency
pass@1 1 pass @ 1, Hourly infra cost, Efficiency
Mitigate inefficiency
ITBecch devises scoring methods for partially correct so- lutions to provide meaningful feedback for summative as- sessments. This comprehensive approach establishes new standard for evaluating and advancing AI-driven solutions in ITT automation. For each scenario that an agent works on, upon task completion, the ITBench records the final system state, which then used at the end of all scenario runs along with the pre-defined ground truth data to validate how well the agent performed across all the scenarios.
Data Insights
pass @ Token uti- lization F1 Score, rank score
Anomaly Detection
2023). Reflection techniques include syntax checking/lint- ing, semantic validation (Xie et al., 2024a), and Ilm-as-a- judge (Zheng et al., 2023).
We are open-soureing a small subset (11 out of 102) of scenarios along with the baseline agents to help the com- munity become familiar with ITBench through practical examples. We reserve the remaining scenarios in ITTeecch to benchmark and evaluate the submitted agentic solutions.
We use the open-source CrewAl framework (cre) to create and manage agents. The agents can be configured to use various I.MSs either through watsonx, Azure, or vL..M. Each agent is initialized with prompt that describes its goal, the context, the tasks, and the expected output format. In-context learning examples are included to guide the agent and demonstrate tool usage. Agents use tools to interact with the environment for information gathering.
4. Results
4.1. Evaluation Setup
Logs, traces, and metrics collected during the diagnosis process would overwhelm the context window of any LLM currently available due to large volume of data. Therefore, agents targeting the SRE FinOps persona are equipped with specialized tools to interact with the environment (refer to Figure 3): 1) NL2Traces to extract trace data in struc- tured format, 2) NL22Metrics to analyze key system metrics, 3) NL2Logs to parse log data effectively, 4) Nu2Kubecll to perform Kubernetes specific operations, and a summariza- tion tool to condense extensive data into actionable insights. For example, the agent may the Nub2Kueetl tool to "list all of the pods in the default namespace." In turn, the NL2Kubeetl tool uses an LLM to transform the utterance into an executable command: "kubectl get pods -n default." "
To understand the impact of reasoning and planning ca- pabilities of LLMs on ITBench scenarios, we instantiate our agents using different LLM models, both for natural language reasoning and code generation. Specifically, we employ GPT-4o (checkpoint version 2024-11-20), Llama- 3.3-70B-instrucc,
and Granite-3.1- 8B BB-instruct for tasks that rely natural language under- standing and reasoning. For code-focused use cases, we utilize GPT-4o-mini, 1aama--.1-405b-inse
and Mixtral- 8x7b-instruct. All models use a context window of 128K tokens, enabling them to process more extensive input se- quences.
We conduct our experiments primarily on AWS EC2 in- stances (m4.xlarge), although ITBench can also be readily deployed on consumer-grand -rade laptop using pleudo-cluster, thus making it easier to develop Al agents (Appendix C.4. 1)
Similarly, the compliance assessment required for new reg- ulations and technologies, with the evidence and diverse policy languages, would be overwhelming if submitted di- rectly to LLMs. The compliance agents designed for CISO compliance assessment automation are equipped with spe- cialized tools. These tools include capabilities to 1) generate policies such as Kyverno or OPA Rego Policy as Code start- ing from natural language specifications, 2) generate scripts for the collection of evidence, 3) access code repositories
Below, we provide an overview of our bascline agents' per- formance across ITBench scenarios for SRE, CISO, and FinOps. Our findings indicate that both open-source and proprietary models often struggle with real-world tasks, un- derscoring the importance of benchmarks that push the lim- its of reasoning and planning in foundation models. For
6

---

ITBench
Table 4: Evaluation of SRE-Agent on SRE scenarios
Diagnosis
Mitigation
Models
pass@1 (%)
FL (
FPC (NTAM))
MTTD (s)
pass@ (%)
MTTR (s)
Frrante-331-8B-innttre
3.57 0.94
0.16 + 0.02
0.19 0.02
259.92 65.01
0.24 = 0.25
845.50 +
-tnss-e.--B--nstruuct
0.99 + 0.51
0.07 = 0.01
0.08 + 0.01
57.50 i 2.05
1.98 0.68
245.13 40.66
1lama-333-70B-imstuue
3.10 + 0.84
0.16 1 0.02
0.16 1 0.02
191.85 1 31.34
3.33 1 0.90
776.27 1 27
gpt-4o
13,81 1 1.67
0.39 1 0.05
0.34 1 0.03
72.44 1 4.71
11.43 1 1.52
282.47 1: 30.04
runs per scenario per model. 1pass0i I values are shown as percentages. indicates missing (NTA)- Normalized topology-aware metric for root cause, FPC (NTAM) Normalized topology-awan metric for fault propagation chain (valuc between O and 1.0), MTTD Mcan time to diagnosis (scconds). MTTR Mcan time to repair (scoonds). Bold: the best performance. Details of NTAM are available Appendix C.6.3
442 scenarios (2) scenarios with traces and without traces).
data. std error for each metric is listed.
more comprehensive results and detailed scenario-level dis- cussions, please refer to Appendix C (SRE), Appendix D (CISO), and Appendix E (FinOps).
CISO. We measure the efficacy of our agents across the four scenario classes introduced in Table 2. Each scenario c_cllssslsssss imposes a distinct set of CIS- IIS-benchmarks requirements (e.g.. "minimize the admission of containers wishing to share the host network namespace"). has specific level of complex- ity (e.g., Easy, Medium, or Hard), and generates scenario- specific code artifacts.
4.2. Overall Results
Table 4, Table 5, and Table 6 show the performance of SRE-agent, CISO-agent, and FinOps-agent respectively.
The efficacy of CISO-agents is measured based on the abil- ity to detect artifact misconfigurations (aka non- non-compliance, e.g., no minimum count of containers sharing namespace, or the count is above the threshold), or confirm proper config- urations (aka compliance), within the varied environments of the scenario classes randomly injected with misconfig- urations. Notably, GPT-based models dominate on both pass @ I and Time to Process metrics. The pass @ is nearly two times better than second- best performing model, while the TTP shows a handling of the scenarios in the minimal time across our scenario classes
SRE. We measure the efficiency of SRE-Agent on its ability to diagnose and mitigate production incidents (e.g.. "a high error rate on frontend service").
Diagnosis efficiency is measured using pass @ I(Chen et al., 2021) (i.e., identifying the cause as mentioned in ground truth), NTAM (Normalized Topology-Aware Metric) for root cause and fault propagation chain, and time to diagno- sis.' Mitigation efficiency is measured in terms of pass @ (i.e., whether the alert was cleared) and mean time to repair.
As shown in Table 4, across all SRE scenarios, GPT-4o consistently outperforms the other models, achieving the highest pass@1 scores for diagnosis (13.81% (%) and mitiga- tion (11.43%), as well as the highest score on NTAM (FL and FPC) metrics. L.lma-3.3-70B ranks second overall, trailing GPT-4o on most metrics. The SB models have lower mitigation success rate. Surprisingly, Granite-3.1-88 (with- out any specialized finetuning) achieves higher accuracy than Llama-3.3-70B on the diagnosis task.
FinOps. In addition to the standard event-driven scenarios, ITBench was extended support non-alert-driven scenarios for the FinOps persona, demonstrating its extensibility. In particular, we added data insights and anomaly detection scenarios to ITBench. Table 6 presents our results in all FinOps usecases. We report pass @ score for data insights, diagnosis, and mitigation tasks, and Fl score and rank score for anomaly detection. Fl score measures the precision and recall abilities of the agent identify anomalous costs with regard to the ground truth. The rank score measures the relative ranking of the anomalies as determined by the agent with regard to the ground truth ranking. GPT-4o consistently outperforms all other models, achieving a 33% pass rate for diagnosing the origin of the cost increase alert, 29% accuracy in data insights scenarios, and 0.6 Fl score in anomaly detection. Refer to Appendix E.5 for futher details.
Removing trace data can drastically reduce success rates (see Table 20 and Table 21 in Appendix). For instance, GPT- 4o's pass@1 1 in diagnosis falls from 13.81% % with traces to 9.52% without them, and mitigation plummets to 2.86%. This highlights the critical role of system observability in SRE, which ITBench can evaluate under varying conditions. Because there is perfect observability in practice, how to guide SRE-agents to collect new observability data and to help SRE-agents reason about failures with incomplete observability is an important but open problem.
4.3. Impact of Scenario Complexity
NTAA is Normalized topology-awar aware metric that measures the quality of the predicted root cause and fault propagation chains using a system and application topology. Refer to Appendix C.6.3.
SRE. We categorize scenarios as Easy, Medium, or Hard based on factors such as fault propagation chain length, number of resolution steps, and the diversity of technolo-
7

---

ITBench
Table 5: Evaluation of CISO Compliance Assessment Agent on CISO scenarios
Scenario pass @1 (%) f
Models
O/A pass@1 1 (%) f
TTP (8)
kyverno
k8s-opa
rhel-opa
kyverno-update
granite-3.1-8B-intruee
7.84 + 3.84
0.00 + 0.00
0.00 + 0.00
1.59 1.58
1.71 0.76
197.03 + 2.52
mixxtral-k27-inntrueet
7.35 +: 3.19
1.43 1.42
0.00 0.00
1.29
4.34
3.94 1.03
120.63 1 3.77
-aama-31-88--nstteee
8.57 +: 3.37
0.00 1 0.00
0.00 + 0.00
7.46 1 3.23
3.59 + 1.07
121.49 1 3.00
1lama-3.--7BBinsttrree
18.46 4.94
0.00 0.00
1.43 1: 2.88
8,06 + 3.50
9.32 1 1.67
189.61 1 2.71
mistral-large-2
6.56 = 3.20
7.23 + 2.88
10.45 3.77
11.55 = 1.95
167.98 = 3.42
1lama-331-405B-instruue
16.22 + 4.32
20.83 + 4.86
8.75 + 3.26
3.17 2.22
12.46 1.98
178.89 = 3.37
gpt-4o-mini
16.18 + 4.54
43.10 + 6.99
30.38 = 5.43
9.43 - 4.08
25.19 + 2.80
102. 102.40 = 3.70
gpt-4o
40.28 +: 5.99
39.34 + 6.55
7.61 + 2.81
17.74 1.92
24.74 + 2.61
101.29 + 3.81
50 scenarios. "8 runs per scenario per model.
percentages. "TIP to process (scconds). kyverno = New K&s CIS benchmnrkks on Kyvemno easy scenario class: kBs-opa New K8s CIS benchmarks on OPA, medium scenario class; rhel-opa = New RITEL9 hTS-henchmarks Ansible-OPA, medium scenario class; kyvermo-apdate- Update CIS-henchmarks on Kyvemn, scenario
pass values
Table 6: Evaluation of FinOpsAgent on FinOps scenarios.
Non-Alert-Driven Scenarios
Alert-Driven Scenarios
Models
Data Insights
Anomaly Detection
Diagnosis
Mitigation
p
F1 Score f
Ranking
pass @1 (%) 1
pass (%)
granite-3.1-SB-istruus
0.4 0.07
0.3 = 0.00
0
0
laama-3.1-S-instruuct llama3.3--0B-insttrr gpt-4o
0.4 0.03
1 0.00
0
29
0.0
0.0 + 0.00
16.6
0
29
0.6 + 0.00
0.5 : 0.00
33
0
@a values as percentages. agent's model with temporature zero.
we fixed dataset configure Diagnosis and Mitigation evaluations include only two sccnarios, making variance nogligible.
Data- Data-Tnsight evaluations exhibit zero variance
gies involved, as described in Equation (6). Our results show that success rates (pass@1 1) clearly decline as the crenario_complett increases. Even the best perform- ing model, GPT-4o, diagnosed only 36%, 7.73%, 5% of Easy/Medium/Hard cases (Table 18) and mitigated just 21%, 12.27%, 0% (Table 19). None of the models could mitigate the Hard scenarios, even though over 50% of Easy scenarios were mitigated. Notably, GPT-4o is the only model that successfully diagnosed multiple Hard scenarios.
5. A Case Study on SRE-Agent Failures
Understanding the decision process of LLM-based agents is challenging due to the complexity of agentic systems but is feasible through detailed agent trajectory logging and structured prompting framework. We log each input and out- put for planning agents and tools, including the ReAct-style "Thought" step, enabling us to distinguish between high- level reasoning errors (e.g., flawed strategy) and low-level tool errors (e.g., malformed commands), enabling practical analysis and guiding design improvements.
CISO. The complexity of the CISO scenarios is di- rectly mapped to scenario classes. For example, sce- mario_ corp__oomlexitty of Kyverno scenarios is Easy, sce- nario cirtiooxceexttyy of k8s-opa and rhel-opa is Medium, while scenario complexity of ayverroouudate scenarios is Hard. All models struggle, as expected, as the difficulty of the scenarios increases from the Easy kyverno class to the Hard kyverto-upadate class.
5.1. Analyzing Lower-Level Tool Calling and Execution
Figure 5 shows tool usage and failure types across models. NL2Kubectl dominates usage, suggesting overreliance. En- couraging the agent to make more balanced use of other tools, such as NL2Traces, could be useful, especially when kubectl commands alone are insufficient. Smaller models (e.g., granite-3. - -----iisttt 1lama-3.1-$B-instiii. more invalid tool calls, syntax errors, and repeated invoca- tions, indicating lower accuracy and efficiency.
FinOps. Currently, IT'Bench includes 2 Easy, Medium, and 5 Hard scenarios. None of the models were able to resolve the Hard scenarios. GPT-4o performs better in anomaly detection and alert-driven scenarios, while the 1LaMAA-.3-70B-Innee model achieves comparable per- formance to GPT-4o in data insight scenarios.
show
5.2. Quantitative Analysis High-Level Reasoning
We quantify reasoning by aligning each exploration path with the ground-truth prott-propggatio chain. An effec- tive agent is expected to focus its exploration around this chain, while significant deviations may signal reasoning
8

---

ITBench
inva tool
getalerts
niZtraces
niZkubectl
ml2meriiss
niZlogs
Number of Actions
1500
1000
500
granite-3 2.1.-88-iiin. -88-inst.
Haama 1-88-inst.
got.4o
Ilama-3. 708-inst. inst.
granite-3 1.1--.1 1-88-4 -8-inst.
liama 3.m-3.11-5 3.1--8--inn ---8-inst.
gpt-Ao
llama-3 3, 3- 3--stttnst.
granite 1--.--8----- ge-inst.
ilamaa 3.1 1--8--nnst 18-inst.
get.4o
Ilama-3 3 3- 708-inst.
granite: (ranite-3.1--8. 88-inst. llama 1-188-inntt llama-3, 3-708- 7os-inst. inst.
4pt.40
Syntax granite-3 1-----8--- 88-inst.
liama 1-88-inst.
gpt-4o
Ilama-3, 3. 3-708--ing 708--nstt.
granite-3 1-88-1 1-88-inst.
Itama-., faama-3.1-88 3.188-innst -8--nst.
get
Ilama- 3, 3. inst.
Total
Incorrect Argument
Repeated Usage
Execution Fallure
Successful Execution
Figure 5: SRE-Agent Tool Usage Distribution
flaws. Based on this insight, we introduce two evaluation metrics: (i) Detoured Services: [Vissted Vigl. the number of visited services that are not on the ground-truth chain (smaller - more focused search); and (ii) Relative Cov- ered Services: VVisited n Viil the fraction of ground-truth Vggll
6. Discussion and Conclusion
We presented ITBench, the first framework and experimen- tal platform to benchmark AI Agents for IT automation tasks. ITBench strives to capture the complexity of modern IT systems and the diversity of IT tasks. The reproducibility of ITBench ensures the community-dee -riivnn effort despite inherent nondeterminism of large-scale IT systems.
services visited (closer to 1 better coverage).
As shown in Figure 6, successful diagnosis trajectories show fewer detours and higher coverage than unsuccess- ful ones, validating the metrics. Among successful tra- jectories, GPT-4o shows detours (Kolmogonov-Smiroo p-value 20.123) and coverage (p-value20.089) that are comparable to other models (i.e., Graniit-331-88B-Inssuu Mauma-3.1-8BIInttruue and Llama-3.3-70B Instruut,,,,,, while achieving higher coverage than 1 1.---B...--BBBB-- Ins-ruut (p- value=0.024). This suggests that successful agents tend to follow similar reasoning patterns. For unsuccessful trajecto- ries, GPT-4o significantly surpasses all baselines, showing both fewer detours (p-value=0.000) and greater coverage (p-value< 1). These results underscore ITBeenchss util- ity in revealing insightful patterns in agent reasoning and overall performance.
One of the key design principles of ITTBench is ensuring its flexibility to support diverse areas of different IT systems and its extensibility to new scenarios. While the current scope of ITTBench is comprehensive and representative, we plan to further enrich the benchmark suites by adding other important processes essential to modern automation. Fur- thermore, we plan to expand our benchmarking beyond event-triggered scenarios. We actively working pand scenario coverage for the supported processes and pro- mote growth through open- open-community contributions. We invite the community to reproduce their real -oold-ord--nspereedd incidents a synthetic sandboxed environment leveraging the ITBench. We expect that everyone contributing can bring their expertise to the table.
We expect ITBeench drive the innovations of Al agent- based techniques with direct impact on the safety, effi- ciency, and intelligence of today's IT infrastructures. With ITBench, we are starting to explore many deep, exciting open problems: How to develop domain-speciinn Al agents that specialize certain types of IT tasks? How to orches- trate multiple agents with various expertise collaborate on bigger projects? How can we ensure safety of agent-driven solutions? How while developing diverse adaptive agents? We invite every- one to participate in answering these questions and realizing the vision of using AI agents to automate critical IT tasks.
1.75
17.5
granite-311-8--nstan
granite: 3.1 8B instruct
15.0
llama 3.1 8B instruct
1.50
Ilama-3.1 10ira-31-8B-innstuuct
Ilama-3.3 1imma-3.-700-Insttuec
1.55
1lama-..3-708-ine 708tinsttuutt
12.5
10.0
4G
1.00
gpt-4o
7.
50
0.50
25
0.25
Successful
Unsuccessful
0.00
Successful
Unsuccessful
effectively use human-in soaman-in-the-loge -hop
(a) Detoured Services
(b) Relative Covered Services
Figure 6: Quantitative Analysis of Hige-Level Reasoning

---

ITBench
Impact Statement
that legal experts and policy-making entities can experi- ment with to shape the future of what Al-driven end-to-end software engineering could look like.
Ethics & Broader Impacts
This research presents a novel benchmarking framework to measure the performance of Al agents across a wide variety of complex and real-life IT tasks, which has the potential to be a key enabler for AI-driven IT automation that is correct, safe, and fast. While the primary focus is on advancing the field of machine learning, as this effort is an open source framework built with open source technologies, it allows organizations with proprietary technologies to use it for developing and benchmarking their solutions more effec- tively. It also encourages mindsharing in the community and lowers the barrier innovate in IT domain.
Reproducibility
To help the greater community reproduce the results pre- sented in this paper and build on the ITBench, we open source all of our resources that were created for this project. The source code for the interactive pipeline, context man- agement logic, command implementations, interface design, and everything else is entirely available in a GitHub repos- itory. We provide extensive text and video documentation describing how to run and modify different parts of the codebase. Practitioners should be able to easily recover our findings by running the agent with simple scripts. The re- sults presented in the main and supplementary parts of this paper can be fully obtained by following instructions in the repositories. Finally, we also maintain an active online help forum assist with any reproduction problems or questions about how build on ITBeech.
Agents that interact with the system pose several risks. We identify three main risks that could arise when building and using a ITBench and associated agents, then discuss how we incorporates measures that mitigate such problems.
First is the security risks that come with executing LM- generated code/commands on the system. Examples include executing commands like kubecti delete node and rm -rf asset/ To defend against this, we containerize the agents and also provide a self-contained Kubernetes environment to create various scenarios.
Acknowledgements
We would like to thank everyone at IBM and the University of Illinois at Urbana-Chaan Champaign who supported this project but are not on the author list. In particular, we are grateful to Ravishankar K. Iyer and Deming Chen for their guid- ance and encouragement, well to all who shared their excitement, provided feedback on early prototypes, and col- laborated with the core team across various aspects of this work. We gratefully acknowledge the core contributors- Pavankumar Murali, Paulina Toro Isaza, and Xi Yang- for their decisive role in addressing reviewer feedback. Pa- vankumar designed the FinOps ios, while Paulina and Xi carried out the agent-trajectory analyses. The work is supported in part by the IB.M-Ilinooi. Discovery Accelerator Institute (IIDAI) and National Sci- ence Foundation (NSF) CNS-2145295.
Second, if the wider community develops an interest in IT- Bench and associated agents and builds upon it, is also pos- sible that illegitimate evaluation datasets or infrastructure can be used to inject testing devices with malicious code or instructions to generate malicious code. For instance, an un- official repository claiming to host an inference/evaluation harness for I'TBench and associated agents could include a task instance with an issue description that tells the LM agent to build key logging functionality and store it in a hidden folder. To eliminate confusion and reduce the pos- sibility of such an event, we provide clear guidelines listed on our GitHub repositories, data stores, and websites indi- cating the official repositories and channels that we actively maintain. We also encourage third parties to incorporate any improvements into our codebase and help with integrating such contributions.
scenar-
In addition, we acknowledge the support of our colleagues in IBM Instana: Ameet Rahane, Marc Palaci-Olgun, Guangya Liu, Brad Blancett, Chad Holliday, Arthur De Magalhaes, Ragu Kattinakere, Chris Bailey, Isabell Sippili, and Danilo Florissi; IBM Granite and Data Model Factory: Hui Wu and Bing Zhang; IBM Emerging Technology Engineering: Carlos Fonseca, Aditya Gidh, Mike Sava, Bill Rippon, and Danny Barnett; IBM UX Research: James Sutton, Connor Leech, and Justin McNair; IBM Research: Michal Shmueli- Scheuer, Lilach Edelstein, and Roy Bar-Haim.
Lastly are the consequences of ITTeench agents being de- ployed in the real world. Prior works have conceptualized and put forth prototypes of agents that can carry out offen- sive security measures. It is also not difficult to imagine that a system like SRE-Agent can be incorporated into pipelines, resulting in the production of malicious code and libraries. The strong performance of agents on ITBench implies that future AI systems will likely be increasingly adept in the aforementioned use cases. Releasing ITBench agents open source agents can support research toward designing sound, effective constraints for what software engineering agents are permitted to do. It can also serve as a system
References
CISO Compliance Assessment Agent. https:
//github. www..c/////./////.../.. LLL.M///bbcch-tteeecc cirscchh-iioooo ciso- caa-
10

---

ITBench
agent?tab rtab=readme a. 2025-01-30.
Accessed:
T. Ahmed, S. Ghosh, C. Bansal, T. Zimmermann, X. Zhang, and S. Rajmohan. Recommending root-cause and mit- igation steps for cloud incidents using large language models. In 2023 IEEE/ACM 45th International Confer- ence on Software Engineering (ICSE), pages 1737-1749, 2023a. doi: 10.1009/CSEE8811222223.
CISO Compliance Assessment Agent.
https:
// wwomm//t//tbenncssamaaat
scenarion httpr///teeeeeeee/eeeeeee////e/e b.
Accessed:
2025-01-30.
T. Ahmed, S. Ghosh, C. Bansal, T. Zimmermann, X. Zhang, and S. Rajmohan. Recommending root-cause and mit- igation steps for cloud incidents using large language models. In 2023 IEEF/ACM 45th International Confer- ence on Software Engineering (ICSE), pages 1737-1749. IEEE, 2023b.
Ansible.
https://ww.redhat. com/en/ Sechnologies /ecciooooogssss//annnneeeeetterp Accessed: 2025-01-30.
OPA Gatekeeper.
201 http:///0066/ppaggatteeeeeeeeeee governance for kubereeeeeete/, b. Accessed: 2025-01-30.
B. Arzani, S. Ciraci, T. Loo, A. Schuster, and G. Outhred. Taking the blame game out of data centers operations with netpoirot. In Proceedings of the 2016 ACM SIGCOMM Conference, pages 440 453, 2016.
Kyverno, Policy as Code, Simplified! https://
kyverno. io, c. Accessed: 2025-01-30.
S. Ashok, Harsh, B. Godfrey, R. Mittal, S. Parthasarathy, and L. Shwartz. Traceweaver: Distributed request tracing for microservices without application modification. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 828 842, 2024.
Open Policy Agent. https://githis com/open- caliccy-gent/opa, d. Accessed: 2025-01-30.
NIST Cybersecurity Framework 2.0 Quick-Start Guide for Creating and Using Organizational Profiles. https:
/ nist. www.:niitt.o.ooo hott.//////bbeerrawwwwk.k Accessed:
A. Avizienis, J.-C. Laprie, and B. Randell. Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Transuctions on Dependable and Secure Computing (TDSC), 1(1):1-23, Jan. 2004.
2025-01-30.
Crew ai. 2025-01-30.
com/. Accessed:
F. Bagehor. J. Rios, S. Jha, R. Filepp, L. Shwartz, N. Abe, and X. Yang. A fault injection platform for learning aiops models. In Proceedings of the 37th IEEE/ACM Interna- tional Conference on Automated Software Engineering, pages 1-5, 2022.
Datadog.
http::://ww.cctaadgggg..llll Ac- cessed: 2024-11-30.
Dynatrace. cessed: 2024-11-30.
Ac-
L. A. Barroso, U. Hölzle, and P. Ranganathan. The Data- center as Computer: Designing Warehouse-Scale Ma- chines. Morgan and Claypool Publishers, 3 edition, 2018.
Finops sample data.
Open Cost -and-Usage Spec/FOCUS-
Sample-Data. Accessed: 2025-05-30.
B. Beyer, N. R. Murphy, D. K. Rensin, K. Kawahara, and S. Thome Site Reliability Workbook: Practical Ways to Implement SRE. O' O'Reilly Media Inc., Aug. 2018.
Grafana. https://grafamaa.com Accessed: 2025- 01-30.
Instana.
https://wwx ibm. compproducts// instana. Accessed: 2024-11-30.
B. Bogin, K. Yang, S. Gupta, K. Richardson, E. Bransom, P. Clark, A. Sabharwal, and T. Khot. Super: Evaluating agents on setting up and executing tasks from research repositories. ArXiv, 0bs/24009004440, 2024.
Jaeger. https://www. http://www...ggettraciin.gin. Ac- cessed: 2025-01-30.
L. Boisvert, M. Thakkar, M. Gasse, M. Caccia, S. D. Chezelles, Q. Cappart, N. Chapados, A. Lacoste, and A. Drouin. Workarena++: Towards compositional plan- ning and reasoning-based common knowledge work tasks, 2024.
Kubernetes. https:://ubernetees 1o/. Accessed: 2025-01-30.
Grafana Loki. https///gthubb.comgggeaaann
httptthuuu.oommeeeeeeeeeeeee wwtp////////////////////....aaat loki. Accessed: 2025-01-30.
C. Boulton. Rising cloud costs leave CIOs seek- ing ways to cope. https: http::://www.cio// www.cio.cmm article/3496500
Prometheus. https: https:://prometheusse io/. Accessed:
2025-01-30.

---

ITBench
leave-cios seeking to html. Accessed: 2025-01-30.
C. Di Martino, F. Baccanico, J. Fullop, W. Kramer, Z. Kalbarezyk, and R. Iyer. Lessons Learned From the Analysis of System Failures Petascale: The Case of Blue Waters. In Proceedings of the 44th Annual IEEE/I- FIP International Conference on Dependable Systems and Networks (DSN'14), 2014.
K. Budhathoki, L.. Minorics, P. Blöbaum, and D. Janzing. Causal structure-based root cause analysis of outliers. In International Conference on Machine Learning, pages 2357-2369. PMLR, 2022.
Y. Diao, D. Horn, A. Kipf, O. Shchur, I. Benito, W. Dong, D. Pagano, P. Pfeil, V. Nathan, M. Narayanaswamy, and T. Kraska. Forecasting algorithms for intelligent resource scaling: An experimental analysis. In Proceedings of the 2024 ACM Symposium Cloud Computing, pages 126-143, 2024.
B. Burns, B. Grant, Oppenheimer, E. Brewer, and J. Wilkes. Borg, Omega, and Kubernetes. Communi- cations of the ACM, 59(5)50-557, May 2016.
S. Chakraborty, S. Garg, S. Agarwal, A. Chauban, and S. K. Saini. Causil: Causal graph for instance level microser- vice data. In Proceedings of the ACM Web Conference 2023, pages 2905-2915, 2023.
D. Donoho. Comments on Michael Jordan's Sadaan'sea essay "the Al revolution hasn't 't happened yet". Harvard Data Science Review, June 2019.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, and J. K. ct. al. Evaluating large language models trained on code, 2021.
A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. D. Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquez, Chapados, and A. Lacoste. Workarena: How capable are web agents at solving common knowl- edge work tasks?, 2024.
Y. Chen, H. Xie, M. Ma, Y. Kang, X. Gao, L. Shi, Y. Cao, X. Gao, H. Fan, M. Wen, et al. Empowering practical root cause analysis by large language models for cloud incidents. arXiv preprint arXiv:2305. 1..000..5777. 15778, 2023.
H. Fang, T. Tao, and C. Zhai. Diagnostic evaluation of information retrieval models. ACM Transactions In- formation Systems (TOIS), 29(2): 2011.
Y. Chen, M. Shetty, G. Somashekar, M. Ma, Y. Simmhan, J. Mace, C. Bansal, R. Wang, and S. Rajmohan. Aiopslab: A holistic framework for evaluating ai agents for enabling autonomous cloud. November 2024a.
Y. Fangkai, L. Wang, Z. Xu, J. Zhang, L. Li, B. Qiao, Cou- turier, C. Bansal, S. Ram, Z. Ma, I. Goiri, E. Cortez, T. Yang, V. Ruehle, S. Rajmohan, Q. Lin, and D. Zhang. Snape: Reliable and low-cost computing with mixture of spot and on-demand vms. In Proceedings of the In- ternational Conference on Architectural Support for Pro- gramming Languages and Operating Systems, Volume 3, pages 631-643, 2023.
Y. Chen, H. Xie, M. Ma, Y. Kang, X. Gao, L. Shi, Y. Cao, X. Gao, H. Fan, M. Wen, J. Zeng, Ghosh, X. Zhang, C. Zhang, Q. Lin, Rajmohan, D. Zhang, and T. Xu. Au- tomatic Root Cause Analysis via Large Language Models for Cloud Incidents. In Proceedings of the 19th Euro- pean Conference on Computer Systems (EuroSys'24), Apr. 2024b.
B. Feng, Z. Ding, and C. Jiang. Fast: A forecasting model with adaptive sliding window and time locality integration for dynamic cloud workloads. In IEEE Transactions on Services Computing, pages 1184 1197, 2023.
Y. Chen, H. Xie, M. Ma, Y. Kang, X. Gao, L. Shi, Y. Cao, X. Gao, H. Fan, M. Wen, et al. Automatic root cause analysis via large language models for cloud incidents. In Proceedings of the Nineteenth European Conference Computer Systems, pages 674-688, 2024c.
R. Fonseca, G. Porter, R. H. Katz, and S. Shenker. (X- Trace]: A pervasive network tracing framework. In 4th USENIX Symposium on Networked Systems Design & Implementation (NSDI 07), 2007.
CIS. Center for Internet Security Benchmarks List. https: / // www.. i secur www.rrtt......e.ee..ee www.euuttttttttoreeoeoeee wtt.:r///ibbbbeccaarr....... 2024. Accessed: 2025-01-30.
C. for Internet Security (CIS). CIS Benchmarks. Center for Internet Security, 2024. Technical guidelines for secure system configurations.
O. Community. demo. /ttps://opentellee demo/, 2024. Accessed: 2025-01-30.
Opentelemetry astronomy shop
D. Ford, F. Labelle, F. I. Popovici, M. Stokely, V.-A. Truong, L. Barroso, C. Grimes, and S. Quinlan. Availability in Globally Distributed Storage Systems. In Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation (OSDI' 10), Oct. 2010
J. Dean. Designs, Lessons and Advice from Building Large Distributed Systems. In Proceedings of the the 3rd Large Scale Distributed Systems and Middleware (LADIS'09) Oct. 2009.
12

---

ITBench
F. Foundation. Using efficieney metries to evalu- ate cloud optimization and value between parts of the organization against industry peers to inform decision- makiing and align finops with business objec- tives. https: /tmpp::/wwwrr wwww oog...org//// /////rrmme framework/ gppaablittiesbenncneee cessed: 2025-01-30.
Harvard Business Review Research Report. Tam- ing it complexity through effective strategies and partnerships. 2022 e e effect ieective-ie strategiesssse and -nd-taadteeershheeee 2022.
2025a. Ac-
P. H. Hochschild, P. Turner, J. C. Mogul, R. Govindaraju, P. Ranganathan, D. E. Culler, and A. Vahdat. Cores that don't count. In Proceedings of the 18th Workshop on Hot Topics in Operating Systems (HorOS'21), 11), June 2021.
F. Foundation. Finops kpis. https:/ / /www//wwwwoooo finops.
org/wg/ http://gfinossspp..... 2025b. Accessed: 2025- 01-30.
J. Huang. X. Chen, S. Mishra, H. S. Zheng, W. Yu, X. Song, and D. Zhou. Large language models cannot self- correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023.
J. Gao, N. Yaseen, R. MacDavid, F. V. Frujeri, V. Liu, R. Bianchini, R. Aditya, X. Wang, H. Lee, D. Maltz, et al. Scouts: Improving the diagnosis process through incident routing. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technolo- gies, architectures, and protocols for computer communi- cation, pages 253-269, 2020.
L. Huang, M. Magnusson, A. B. Muralikrishna, S. Estyak, R. Isaaes, Aghayev, Zhu, and A. Charapko. Metastable Failures in the Wild. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI'22), July 2022.
S. Ghosh, M. Shetty,
C. Bansal, and S. Nath. How to fight production incidents?: an empirical study on a large-scale cloud service. In Proceedings of the 13th Symposium Cloud Computing (SoCC'22), Nov. 2022.
IDC. Storm clouds ahead: Missed expectations in com/
cloud computing. https://blogs (024//0/2//stommmcoooeeee wwwp:oudds-aaaaadmiimamee----- expectat innetaaiiiinniiii ietessiincilleee www- Blog post, published on 28 October 2024.
2024.
H. S. Gunawi, M. Hao, T. Leesataporwongsa, T. Patana- anake, T. Do, J. Adityatama, K. J. Eliazar, A. Laksono, J. F. Lukman, V. Martin, and A. D. Satria. What bugs live in the cloud? study of 3000+ issues in cloud systems. In Proceedings of the 5th ACM Symposium on Cloud Computing (S)CC'14), Nov. 2014.
A. Ikram, S. Chakraborty, S. Mitra, S. Saini, S. Bagchi, and M. Kocaoglu. Root cause analysis of failures mi- croservices through causal discovery. Advances in Neural Information Processing Systems, 35:3 118-311181.700 2022.
H. S. Gunawi, M. Hao, R. O. Suminto, A. Laksono, A. D. Satria, J. Adityatama, and K. J. Eliazar. Why Does the Cloud Stop Computing? Lessons from Hundreds of Ser- vice Outages. In Proceedings of the 7th ACM Symposium on Cloud Computing (SoCC' 16), Oct. 2016.
S. Jha, S. Cui, S. S. Banerjee, T. Xu, J. Enos, M. Showerman, Z. T. Kalbarczyk, and R. K. Iyer. Live forensics for hpc systems: A case study on distributed storage systems. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
H. S. Gunawi, R. O. Suminto, R. Sears, C. Golliher, S. Sun- dararaman, X. Lin, T. Emami, W. Sheng, N. Bidokhti, C. McCaffrey, G. Grider, P. M. Fields, K. Harms, R. B. Ross, A. Jacobson, R. Ricci, K. Webb, Alvaro, H. B. Runesha, M. Hao, and H. Li. Fail-Slow at Scale: Ev- idence of Hardware Performance Faults in Large Pro- duction Systems. In Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST" 18), Feh. 2018
C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. SWE-bench: Can language mod- els resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
A. John. Idc predicts 80% of cios to leverage ai and au- tomation for business agility and insights by 2028, 2024.
E. Jonas, Schleier-Smith. V. Sreekanti, C.-C. Tsai, A. Khandelwal, Q. Pu, V. Shankar, J. M. Carreira, K. Krauth, N. Yadwadkar, J. Gonzalez, R. A. Popa, I. Sto- ica, and D. A. Patterson. Cloud Programming Simplified: A Berkeley View on Serverless Computing. Technical Report UCB/EECS-2011 2019-3,, University of California at Berkeley, Feb, 2019.
C. Guo, L. Yuan, D. Xiang, Y. Dang, R. Huang, D. Maltz, Z. Liu, V. Wang, B. Pang, H. Chen, Z.-W. Lin, and V. Kurien. Pingmesh: A large-scale system for data
center network latency measurement and analysis. SIG- COMM Compur. Commun. Rev. 45(4): Aug. doi: 10. 45/2829988.27874999
2015. ISSN 01 0146-4833.
13

---

ITBench
S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan. Ai agents that matter. https: arxiv. org/ aba/ /407, 01502, 2024.
B. Maurer. Fail Scale: Reliability in the Face of Rapid Change. Communications of the ACM, 58(1 49, Nov. 2015.
M. G. Kendall. The treatment of ties in ranking problems. Biometrika, 33(3):239-251, 1945.
T. Mclissaris, K. Nabar, R. Radut, S. Rehmtulla, A. Shi, S. Chandrashekar, and Papapanagiotou, Elastic Cloud Services: Scaling Snowflake's 's Control Plane. Proceed- ings of the 13th ACM Symposium Cloud Computing (SOCC'22), Nov. 2022.
S. Kendrick. What Takes Us Down? USENIX :login:, 37 (5):37-45, Oct. 2012.
S. M. Kerner. Crowdstrike outage explained: What caused it and what's next. https::/www.techtarggg com//haaccs//aattuuee///ppaaaaaa
Microsoft and contributors. Dowhy: A python library for causal inference. https: /www. http://www//tt.. pywhy. www.
dowby/ver /tt.vv111////// 2023. Version 0.12.
Targeett-TT-ootagg-----hhatroort
M. D. L. C. Miguel Carreon. Idc: 80
20comssdderee2200heeeerttttt2222 20companies$20222
N. Murphy, Beyer, C. Jones, and J. Petoff. Site Re- liability Engineering: Monitoring Distributed Systems. O' O'Reilly Media, 2024. Accessed: 2024-11-07.
4tw0b11101b11111 0atp:aaneeee2222255221011111.... 2024.
Y. Koh, Lo, L. Jang. V. Duvvur, M. C. Lim, P.-Y. Huang. G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried. Vi- sualwebarena: Evaluating multimodal agents realistic visual web tasks, 2024.
NIST 800-53, NIST Special Publication 800-53 Revision 5.
https: /ttp::://wlpbbs/// nist .essstgoor www/ngstpubbs..s.ooo..o. SpecialPubli ccattctttiiiii wat.icaiiinssiii /NISIISTTTTTTT. -000- 53r5, pdf, 2020.
J. B. Leners, H. Wu, W.-L. Hung M. K. Aguilera, and M. Walfish. Detecting failures in distributed systems with the falcon spy network. In Proceedings of the Twenty- Third ACM Symposium on Operating Systems Principles, pages 779,294,77 2011.
A. Nodari, J. Nurminen, and C. Fruhwirth. Inventory theory applied to cost optimization in cloud computing. In Pro- ceedings of the 31st annual ACM symposium on applied computing, pages 470-473, 2016.
H. Liu, S. Lu, M. Musuvathi, and S. Nath. What bugs cause production cloud incidents? In Proceedings of the 16th Workshop on Hot Topics in Operating Systems 19), May 2019.
N. OSCAL. Open security controls assessment language. https: /ttp//paaass../ nist gov/OSCAL/, 2024. Ac- cessed: 2025-01-30.
OSCAL-compass. Oscal-compass enef sandbox project. https: /ttp://gtttbb/// cwwmooccalccmppaaaccccccoacaecaa 2024.
M. Liu, L. Pan, and Liu. Cost optimization for cloud storage from user perspectives: Recent advances, taxon- omy, and survey. ACM Computing Surveys, 55(13s):1-37, 2023a.
Accessed: 2025-01-30.
P. Osypanka and P. Nawrocki. Resource usage cost opti- mization in cloud computing using machine learning. In IEEE Transactions on Cloud Computing, pages 2079- 2089, 2020.
Q. Liu and Paparrizos. The elephant the room: Towards reliable time-series anomaly detection benchmark. In NeurlPS 2024, 2024a.
L. Pan, M. Saxon, W. Xu, D. Nathani, X. Wang, and W. Y. Wang. Automatically correcting large language mod- els: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv: 03188888,......... 2023.
Q Liu and J. Paparrizos. The elephant in the room: Towards reliable time-series anomaly detection benchmark. In The Thirly-eight Conference Neural Information Pro- cessing Systems Datasets and Benchmarks Track, 2024b.
N. Papanikolaou, S. Pearson, and M. C. Mont. Towards Natural-Language Understanding and Automated En- forcement of Privacy Rules and Regulations the Cloud: Survey and Bibliography. Secure and Trust Computing, Data Management, and Applications, 2011.
Z. Liu, C. Benge, and S. Jiang. Ticket-bert: Labeling inci- dent management tickets with language models. arXiv preprint arXiv:2307.00/100 00,8, 2023b.
L. Ma. T. He, A. Swami, D. Towsley, K. K. Leung and Lowe. Node failure localization via network tomogra- phy. In Proceedings of the 2014 Conference on Internet Measurement Conference, pages 195-208, 2014.
E. Parliament and the Council of the European Union. Digi- tal operational resilience act for the financial sector and amending regulations. https: : /urr/euuuruee lex. europa. eu/EN/1 1egalleglll www.tonntt//oeeooecco /wtt///mmmarrraaam /iigittalll
14

---

ITBench
forecee-fo-theee the- financial-secoe .tmml, html, 2024.
J. Storment and M. Fuller. Cloud FinOps. O' O'Reilly Media, 2nd edition, 2023, Chapter 1.
D. Patterson, A. Brown, P. Broadwell, G. Candea, M. Chen, J. Cutler, P. Enriquez, A. Fox, E. Kiciman, M. Merzbacher, D. Oppenheimer. N. Sastry, W. Tetzlaff, Traupman, and N. Treuhaft. Serovery-Oriented Com- puting (ROC): Motivation, Definition, Techniques, and Case Studies. Technical Report UCB//CSD-00-1122 University of California Berkeley, Mar. 2002.
X. Sun, R. Cheng, J. Chen, E. Ang, O. Legunsen, and T. Xu. Testing Configuration Changes in Context to Prevent Production Failures. In Proceedings of the 14th USENIX Symposium Operating Systems Design and Implementation (OSDI'20), Nov. 2020.
C. Tan, Z. Jin, C. Guo, T. Zhang, H. Wu, K. Deng, D. Bi, and D. Xiang. (NetBouncer): Active device and link fail- ure localization in data center networks. In 16th USENIX Symposium Networked Systems Design and Implemen- lation (NSDI 19), pages 599-614, 2019.
I.. Pham, H. Ha, and H. Zhang. Root cause analysis for mi- croservice system based causal inference: How far are we? In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 706-715, 2024.
C. Tang, T. Kooburat, P. Venkatachalam, A. Chander, Z. Wen, A. Narayanan, P. Dowell, and R. Karl. Holistic Configuration Management at Facebook. In Proceed- ings of the 25th ACM Symposium Operating System Principles (SOSP'15)) Oct. 2015.
S. Pujar, L. Buratti, X. Guo, N. Dupuis, B. Lewis, S. Suneja, A. Sood, G. Nalawade, M. Jones, Morari, and R. Puri. Automated code generation for information technology tasks in yaml through large language models. https: //arxiv. , wrgg abs/ / 2005 02 78,, 2023.
C. Tang, K. Yu, K. Veeraraghavan, J. Kaldor, S. Michelson, T. Kooburat, A. Anbudurai, M. Clark, K. Gogia, I.. Cheng, B. Christensen, A. Gartrell, M. Khutomenko, S. Kulkarni, M. Pawlowski, T. Pelkonen, A. Rodrigues, R. Tibrewal, V. Venkatesan, and P. Zhang. Twine: A Unified Clus- ter Management System for Shared Infrastructure. In Proceedings of the 14th USENIX Conference on Operat- ing Systems Design and Implementation (OSDI'20), Nov. 2020.
B. Qiao, F. Yang, C. Luo, Y. Wang, J. Li, Q. Lin, H. Zhang, M. Datta, A. Zhou, T. Moscibroda, S. Rajmohan, and D. Zhang. Intelligent container reallocation at microsoft 365. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Sympo- sium on the Foundations of Software Engineering, pages 1438 1443, 2021.
G. Quattrocchi, E. Incerto, R. Pinciroli, C. Trubiani, and L. Baresi. Autoscaling solutions for cloud applications under dynamic workloads. In IEEE Transactions on Services Computing, pages 804-820, 2024.
L. Tang, C. Bhandari, Y. Zhang, A. Karanika, S. Ji, I. Gupta, and T. Xu. Fail through the Cracks: Cross-System Inter- action Failures in Modern Cloud Systems. In Proceedings of the 18th European Conference on Computer Systems (EuroSys' 23), May 2023.
D. Roy, X. Zhang, R. Bhave, C. Bansal, P. Las-Casas, R. Fonseca, and Rajmohan. Exploring l1m-based agents for root cause analysis. https: attps://arxiv.ork waw.iv.xxvvv.///// abs/2403 04 123, 2024.
M. Taylor, H. Zaragoza, N. Craswell, S. Robertson, and Burges. Optimisation methods for ranking functions with multiple parameters. In Proceedings of the 15th ACM international conference on Information and knowl- edge management, pages 585-593, 2006.
G. Sahu, A. Puri, J. Rodriguez. A. Abaskohi. M. Chegini, A. Drouin, P. Taslakian, V. Zantedeschi, A. Lacoste, D. Vazquez, et al. Insightbench: Evaluating business analytics agents through multi-step insight generation. arXiv preprint arXiv:2407.064433 2024.
S. Trask. State of FinOps: 2025 report. https:// f Einops.org/ 2025.
Salesforce. Pyrca: A python machine learning library for root cause analysis. https: htttp://github. com/ salesforce/Pyee
2023. Version 1.0.1.
H. Tupsamudre, A. Kumar, V. Agarwal, N. Gupta, and S. Mondal. Al-Assisted Controls Change Management for Cybersecurity in the Cloud. In Thirty-Fourth Annual Conference on Innovative Applications of Artificial Intel- ligence (IAAI-22). 2022.
N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephen- son, M. Plakal, D. Beaver, S. Jaspan, and C. Shanbhag. Dapper, a large-scale distributed systems tracing infras- tructure. 2010.
L. R. Varshney, N. S. Keskar, and R. Socher. Pre- trained AI models: Performativity, mobility, and change. arXiv:1909.0333 003990 [cs.CY]., Sept. 2019.
15

---

ITBench
K. Veeraraghavan, J. Meza, S. Michelson, S. Panneerselvam, A. Gyori, D. Chou, S. Margulis, D. Obenshain, S. Pad- manabha, A. Shah, Y. J. Song, and T. Xu. Maelstrom: Mitigating Datacenter-level Disasters by Draining Inter- dependent Traffic Safely and Efficiently. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI'1)) Oct. 2018
on the Foundations of Software Engineering, pages 50- 61, 2024.
A. Ychoshua, I. Kolchinsky, and A. Schuster. Cco - cloud cost optimizer. In Proceedings of the 16th ACM Inter- national Conference on Systems and Storage, pages 137- 137, 2023.
C. Zhai and S. Massung. Text Data Management and Anal- ysis: A Practical Introduction to Information Retrieval and Text Mining, volume 12. Association for Comput- ing Machinery and Morgan & Claypool, 2016. ISBN 9781970001174.
Q Wang, J. Rios, S. Jha, K. Shanmugam, F. Bagehorn, X. Yang, R. Filepp, N. Abe, and I.. Shwartz. Fault injec- tion based interventional causal learning for distributed applications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15738-15744, 2023.
Zhang, T. Mittal, C. Bansal, R. Wang, M. Ma, Z. Ren, H. Huang, and S. Rajmohan. Flash: A workflow automa- tion agent for diagnosing recurring incidents. October 2024.
J. Xie, K. Zhang, J. Chen, T. Zhu, R. Lou, Y. Tian, Y. Xiao, and Y. Su. Travelplanner: A benchmark for real-world planning with language agents. https:
// //arxiv... /rg/ aaaaaba/22400 01622, 2024a.
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang. Z. Lin, Z. Li, D. Li, E. P. Xing. H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. htttp::/arxiv.org abs / 2306..0000. 05685.66. 2023.
Z. Xie, Y. Yhenng I.. Ottens, K. Zhang, C. Kozyrakis, and J. Mace. Cloud atlas: Efficient fault localization for cloud systems using language models and causal insight. https:/ // arxiv. org/ abs / 2407 0869489444 94, 2024b.
X. Zhou, X. Peng, T. Xie, Sun, C. Ji, W. Li, and D. Ding. Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study. IEEE Transactions Software Engineering, 47(2):243- 260, 2018.
B. Xu, Z. Peng. B. Lei, S. Mukherjee, Y. Liu, and D. Xu. Rewoo: Decoupling reasoning from observations for effi- cient augmented language models. https:// /arxiv. 2tg/abs/2335 2305. 18323, 2023.
T. Xu, J. Zhang, Huang, Zheng, T. Sheng, D. Yuan, Y. Zhou, and S. Pasupathy. Do Not Blame Users for Misconfigurations. In Proceedings of the 24th Symposium Operating System Principles (SOSP'I3) 13), Farmington, PA, Nov. 2013.
Z. Zhu, C. Lee, X. Tang, and He. Hemirea: Fine-grained root cause analysis for microservices with heterogeneous data sources. ACM Transactions on Software Engineering and Methodology, 33(8):1-25, 25, 2024.
J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe-agent: Agent- computer interfaces enable automated software engineer- ing. arXiv preprint arXiv:2405.1577 2024a.
X. Yang, R. Arora, S. Jha, C. Narayanaswami, C. Lam, Leichter, Y. Deng, and D. Sow. Optimizing it finops and sustainability through unsupervised workload char- acterization. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 22990-22996. 2024b.
S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Confer- ence on Learning Representations, 2023.
Z. Yao, C. Pei, W. Chen, H. Wang, L. Su, H. Jiang, Z. Xie, X. Nie, and D. Pei. Chain-of-event: Interpretable root cause analysis for microservices through automatically learning weighted event causal graph. In Companion Proceedings of the 32nd ACM International Conference
16

---

ITBench
Table of Contents
Appendix
A Related Work
18
A.1 Site Reliability Engineering
18
In the appendix, we provide additional analyses and more extensive discussions about ITBench, individual personas (SRE, ComplianceOps, FinOps) and agent performance. Data, code, and leaderboard "link anonymized."
A.2 Compliance
18
A.3 FinOps
19
ITBench
B.1 Benchmark Registration
20
B.2 Agent Registration
20
B.3 Leaderboard
C Site Reliability Engineering
23
C.1 Background
23
C.2 Real-world Incident Example
C.3 ITBench Architecture
24
C.4 Characterizing ITTBench incidents
26
C.5 SRE-Agent
C.6 ITBench Evaluation
32
Chief Information Security Officer (CISO) and Benchmarking the Compliance Assess-
ment Agent
42
D.1
Background
42
D.2 Real- World Benchmarking
D.3
ITBench Architecture for handling
CISO Tasks
D.4
ITBench Real-World CISO Scenarios
45
D.5
CISO Scenario Classes and their Complexity
46
D.6
CISO ITBench Evaluation
E Financial Operations
54
E.1
Background
54
E.2
Motivating Example and FinOps Scenarios
55
E.3
ITBench Architecture for Construct ing FinOps Scenarios
E.4
Evaluation
56
E.5
Results
59
E.6
Example Trajectories
Normalized Topology-aware Match
62
F.1
Notation
62
F2
Overall Score: NTAM
E3
Fault Localization: FL (NTAM)
64
F.4
Parameter Tuning
64
17

---

ITBench
A. Related Work
the variability in success rate exists because of difference in realism of these datasets. This highlights the urgent need for standardized and open source benchmarks to evaluate and improve the efficacy of Al methods on incident resolution tasks effectively.
LM and agents for resolving IT automation tasks. There a surge in use of AI/ML for handling IT automation tasks.
We describe related work for each persona.
A.1. Site Reliability Engineering
SRE-focused Benchmarks The benchmarking landscape for IT operations (TTOps) tasks is still in its early stages, with few existing efforts addressing specific aspects of the domain. AIOpsLab (Chen et al., 2024a) focuses on resolving IT incidents only for SRE personas, covering ten distinct problems created in a real environment. It does not follow SRE best practices for system and application observability, e.g. using an alert management system, lacks comprehensive coverage analysis, and a leaderboard for systematic automated evaluation.
IT incident² resolution encompasses tasks such as detec- tion (e.g., identifying anomalies or outages) (Guo et al., 2015; Leners et al., 2011; Sigelman et al., 2010; Fonseca et al., 2007), diagnosis (e.g.. pinpointing root causes through metrics and logs) (Tan et al.. 2019; Jha et al., 2020; Ma et al., 2014; Salesforce, 2023), and mitigation (e.g.. op- erational fixes or code changes). These efforts often rely on supporting tasks like ticket analysis and routing (Gao et al., 2020; Liu et al., 2023b; Arzani al., 2016), anomaly detection (Liu and Paparrizos, 2024a), topology extrac- tion (Ashok et al., 2024; Chakraborty et al., 2023; Pham et al., 2024; Yao et al., 2024), causal (Budhathoki et al., 2022; Microsoft and contributors, 2023; Ikram et al., 2022; Chakraborty et al., 2023) and interventional (Wang et al., 2023; Bagehorn et al., 2022) analysis using IT data. Clearly, there is significant research in this area, fully automating incident resolution or providing actionable insights to hu- mans remains elusive due to the complexity of real-world systems, the variability of incidents, and the challenge of incorporating contextual knowledge into Al systems (Jha et al., 2020). Recent advances in language models (LMs) have led to their adoption of ticket data analysis and diagno- sis tasks (Roy et al., 2024; Ahmed et al., 2023a; Xie al., 2024b; Chen et al., 2023; Zhang al., 2024). Most notable examples include Cloud Atlas use LLMs for causal graph construction (Xie et al., 2024b), RCACopilot for ticket anal- ysis (Chen et al., 2023) with the aim to diagnose and mitigate incidents. However, they achieve poor performance com- pared to other techniques. For example, (Roy et al., 2024) shows that chain-of of-thought only achieves accuracy of 35%. More recently, LMs are used in agentic workflows, engaging with real virtual environments, using several tools their disposal, for tasks like web navigation (Drouin et al., 2024; Boisvert et al., 2024; Koh et al., 2024), system control (Sahu et al., 2024; Zhang et al., 2024; Chen et al., 2024a), and code generation (Yang et al., 2024a). However, the initial results of these works show a high variability in the success rate- -35% in InsightBench (Boisvert al., 2024) and the ReAct-based agent for ticket data analysis (Roy al., 2024) to 100% in Flash (Chen et al., 2024a) for incident resolution despite the fact that it is a much harder task than identifying planted insights in tabular and ticket data. Our own results in this work suggest that LLMs and agents struggle to con- sistently complete incident resolution tasks. We assert that
InsightBench (Sahu et al., 2024) targets the analysis of Ser- viceNow ticket data, a critical supporting task for incident routing and finding relevant past incidents, but its reliance on synthetic data and the lack of a real environment limit its applicability to agentic workflows. Similarly, TSB-AD (Liu and Paparrizos, 2024b) is designed for univariate and multi- variate anomaly detection, core task for incident detection. However, it is limited to synthetic data and focuses only on anomaly detection.
A.2. Compliance
Compliance automation software is emerging to help busi- nesses streamline and automate compliance processes, re- ducing the need for manual monitoring and tracking of regulations. This ensures continuous adherence to laws. In particular, compliance as code is a very recent development in the IT industry motivated by companies and audit agen- cies shifting from annual audits to expectations of continu- ous and automated measurement of compliance maintain control of their regulated environments' posture and risks for cyberattacks.
Recent works (Papanikolaou et al., 2011; Tupsamudre et al., 2022) have applied AI/ML techniques to speed up these tasks, focusing on mapping regulatory requirements to stan- dard control frameworks such as NIST 800-53 (NIST 800- 53). Our agentic automation in the current ITBeenh solution pioneers this type of effort to author compliance artifacts through AI / ML by bridging compliance as code into policy as code. Policy engines have a longer history in the IT in- dustry compared to compliance as code; however, emerging general usage policy engines such as (Int, d) try to address the need for a common framework for continuous compli- ance. We are not aware of any effort -albeit critical and needed- related to benchmarking of compliance automation software, whether with or without agentic support.
"W use the term scenario broadly to refer to failures, perfor- mance problems, compliance issues and, cost anomalies.
18

---

ITBench
A.3. FsnOps
their overall cloud spend. (Fangkai al., 2023) addresses the issue of helping customers make trade-offs between cost and resource availability in the presence of offerings such as spot VMs which are cheaper than on-demand VMs but have reduced availability. They propose a framework that uses constrained reinforcement learning to optimize cost and availability by identifying an optimal mix of on-demand VMs and spot VMs. Papers such as (Diao al., 2024; Feng et al., 2023; Quattrocchi al., 2024) propose forecasting al- gorithms to scale cloud resources for service level objectives, contributing to the broader field of FinOps-driven cost op- timization. (Osypanka and Nawrocki, 2020) uses anomaly detection, machine learning, and particle swarm optimiza- tion to achieve a cost-optimal cloud resource configuration. (Liu et al., 2023a) analyze the process of using cloud storage to explore opportunities, motivations, and challenges of cost optimization from user perspectives. (Nodari et al., 2016) focuses on finding the optimal combination of on-demand and reserved instances, such that the demand is satisfied and the costs minimized. They model this optimization problem as a stochastic inventory control problem.
The area IT cost management encompasses multiple dis- ciplines, namely FinOps, IT Financial Management (ITFM), Technology Business Management (TBM) and Porfolio Businesss Management (PBM). At present, the FinOps do- main typically deals with cloud costs (Storment and Fuller, 2023; Yang et al., 2024b), which includes compute nodes, memory, other storage, networking, etc., that are incurred with one of the hyperscalers. ITFM includes on-prem infras- tructure, licensing, labor, procured services, tech support, etc. The TBM Council provides standard taxonomy de- scribe cost sources, technologies, IT resources (IT towers), applications, and services. In addition, there are industry- specific extensions to the taxonomy, such as for healthcare, banking, etc. In essence, this taxonomy provides a generally accepted way of categorizing and reporting IT costs and other metrics. PBM refers to the practice of managing collection of projects and programs within organization, ensuring alignment with the overall business strategy and maximizing their collective value by allocating resources efficiently. The FinOps Foundation has indicated that over time it will include elements from ITFM, TBM, and PBM.
(Yehoshua et al., 2023) introduces a scalable cost optimizer that determines the most cost-effective deployment strategy for workloads on public or hybrid clouds, considering re- source requirements and constraints to minimize costs. In FinOps, there is an urgent need to move beyond comparative scorecards and broad taxonomies to specific use cases that test the ability of automated agents to optimize IT invest- ments and reduce resource waste. To our knowledge, no benchmarks exist for use cases like forecasting, anomaly detection, or cost optimization, nor are there standardized methods to evaluate these techniques with or without agentic support. We are confident that ITBench will unite research and development communities to tackle real-world prob- lems through the power of Al and optimization.
Currently, for FinOps, there no benchmark that fits the definition of benchmark that we using in this paper. However, over the years, the FinOps Foundation (Founda- tion, 2025a) has compiled several KPIs that can form the basis for use cases and scenarios for a FinOps benchmark. Current FinOps Foundation KPIs include:
Usage or Spend Apportionment Validation
Total Unpredicted Variance of Spend
Percent of Compute Spend Covered by Commitment Dis-
counts
Effective Savings Rate Percentage
Percentage of Commitment Discount Waste
Percent of Unused Resources
Auto-scaling Efficiency Rate
Forecast Accuracy Rate (Usage, Spend)
Percentage of Unallocated Shared CSP Cloud Cost
Percentage Variance of Budgeted vs. Forecasted CSP Cloud Spend
Percentage of CSP Cloud Costs that are Tagging Policy Compliant
Percent Storage on Frequent Access Tier
Percentage of Legacy Resource
With the advent of the cloud, the academic and industrial research communities have also been active in investigating ways to optimize costs while balancing multiple objectives. Recent works in the space of FinOps have focused on apply- ing machine learning and mathematical optimization tech- niques (Qiao et al., 2021; Yang et al., 2024b) to better serve customers' cloud infrastructure needs while offering them insights and recommendations on how they could optimize
19

---

ITBench
B. ITBench
B.2.1. AGENT REGISTRATION
ITBench framework, as shown in Figure 7, supports two main phases corresponding to two personas as follows: (i) benchmark registration phase, where the target is the Benchmark Submitter persona, and (ii) agent registration phase, focusing on the Agent Submitter persona and the
During Agent Registration, the Agent Submitter specifies the agent metadata as shown in Table 8.
Table 8: Agent Metadata and Examples.
actual runtime benchmarking execution and evaluation.
Field
Example
Agent Name
B.1. Benchmark Registration
Agent Type (predefined)
CISO, SRE, FinOps ... Beginner, Intermediate, Ex- pert
This phase comprises two main steps: (i) scenario develop- ment and registration, and (ii) tasks and evaluation metrics registration.
Agent Level
(maps to scenario complex- ity: Easy, Medium, Hard) For CISO: rhe19 w/ OPA; Ku- bernetes w/ Kyverno; Kuber- netes w/ OPA, Kyverno up- date
Scenario Development and Registration
Scenario Class
Our scenarios are designed instantiate real-works IT prob- lems in realistic and manageable environments. Each sce- nario comprises of two core components: (i) an environment specification, and (ii) scenario specification metadata. The Benchmark Submitter persona then registers these scenar- ios with ITBench, which stores them its database. Each scenario is described using the metadata shown in Table
Once the agent has been registered, the Agent Submitter selects the agent, and the corresponding benchmarks are re- trieved from the database using the agent_lype, agent_level, and scenario ceatariooclasssss specified during registration for the Agent. The Agent Submitter subsequently receives the tasks that the agent must complete to meet the designated objec- tive, each of which has pre-defined evaluation metrics.
Table 7: Scenario Metadata and Examples.
Field
Example
Type
CISO, SRE, FinOps
B.3. Leaderboard
Name
For CISO: k8s CIS-b Minimize con- tainers w/ shared net namespace For CISO: Minimize the admission of containers wishing to share the host network namespace Easy, Medium, Hard For CISO, this is defined based on the technology (c.g.. k8s w/ Kyverno; k8s w/ OPA; Rhel9 w/ OPA).
Effective benchmarking of IT automation tasks, especially when selecting LLMs tailored to an organization's 's specific needs, requires consistent tracking and comparison of agent performance. The Leaderboard facilitates this need by offer- ing predefined, extensible of performance metrics that provide clear insights into agent performance relative to the evaluation criteria.
Description
Complexity Class
The Leaderboard supports both API and UI interfaces, en- abling a streamlined benchmarking workflow. Users must register the agent endpoint via the Leaderboard's UI or API. The agent can then query the Leaderboard retrieve and de- play benchmark scenarios before reporting their operational status. The scenarios can be deployed either automatically by the ITBench, as described above, in its hosted environ- ment, or manually outside the Leaderboard, in the user's s hosted environment, in which case both agent and environ- ment can still leverage the same Leaderboard API endpoint to publish status updates.
Tasks and Evaluation Metries Registration For each scenario type, the Benchmark Submitter registers a well- defined set of tasks that form the basis for the Agent per- formance evaluation. Table 2 summarizes the ITBench cur- rently supported IT automation tasks. Moving forward, we plan to extend ITBench to incorporate additional tasks (e.g., threat analysis and resource optimization) and to broaden its applicability to other domains (c.g., DevOps).
The end-to-end workflow for the agent benchmarking pro- cess, after its registration by the Agent Submitter, is illus- trated in Figure 7, and summarized in the following.
B.2. Agent Registration
During this phase, the Agent Submitter first registers as a user on the platform, then follows with the Agent Registra- tion.
1. New benchmark jobs are stored in the Benchmark Queue for processing.
2. The Benchmark Runner fetches benchmark scenario
20

---

ITBench
Timelhe
Banch Runser
San
10 When somus apdats
Bengimant Oueut
2001
"Depoyed get poat Inpoiion
"Tudirpped"" Hooie Should sibdevaai)
Deauadis (0/9)
IM fun
tMl samveteenn
bunctra minair
Bunderath D5
Agyn Server
3000
runder
penau
pei maniker
pee utana
per. JEGL
Daned
Depoyed
Fauhiaced
Dre00
depicy
evaluste
delete bundle
Board
bundle
lault
Rundee
Crekm
DoMee
EnWorneee
seed
cadentui
STARTTC)
FARE bandio
Agait
foiled agent
Agent
Figure ITBench leaderboard workflow.
for a particular agent from the Benchmark Queue.
status via the Agent API Server's get_bstatus API. Once the status becomes Deployed, it injects fault into the environment by executing the inject_ fault function.
The Benchmark Runner provisions the environment as per the benchmark scenario specification. The scenario's environment is the set of systems required for the ecution of specific IT task. The Agent interacts with (and can potentially modify) the environment to solve the given IT automation tasks. A benchmark evaluation measures the Agent's performance based on whether it successfully completes the tasks in the given environ- ment. The environment could be, for instance, Kuber- netes cluster rumning target application host with specific configuration to be validated. The environment is under the direct control of the Agent and therefore may be subject to destructive actions (in case of faulty performance), thus functioning as sort of "playground."
(d) The Benchmark Runner continues to monitor the en- vironment's status using the jee_bssttus API. Once the status reaches FaultInjected, it updates the man- ifest's 's status in the Benchmark DB to Ready, in- cluding key details such as Benchmark ID, Scenario ID, cluster credentials, and URLs in the manifest. This allows the Agent access and retrieve this manifest for working with the environment.
RHEL 9
(e) Once the manifest status is Ready, the Agent re- trieves it. The manifest contains URLs and creden- tials required to launch the Agent. Before start- ing the Agent, the Agent calls the post_staate API of the Agent API Server to report its status STARTED.
For each scenario included in the benchmark run, the Benchmark Runner and the Agent execute the following steps:
(a) The Agent continuously polls the get manifest API monitor when a new manifest enters the Ready state
After the Agent completes its execution, the post status API is called again to report the Agent's completion its status FINISH.
(b) Benchmark Runner deploys the scenario's environ- ment by executing the deploy scenario function. Each environment reports its status the Agent API Server using the post_ bstatus API.
(g) Benchmark Runner starts the evaluation and exe- cutes the delete_scenario function.
5. Once the evaluation results for all the scenarios in the benchmark are ready, Benchmark Runner aggregates them and publishes the results to the Leaderboard.
(c) The Benchmark Runner monitors the environment's
21

---

ITBench
CISO Agent Leaderboard
Thepremos Leeda hock desurertaties
Agent Benchmark
Search
AgenCrarees
Superata mulipple quatics with?:
Hag gasto 3224-11-3
agenti
Seteet Columees Hepplyy
>age pobre -1 321---22--11113 Jwapa sais, 2004 3003-022--200222202. 212-2222222--22-2220020000
Agenk(
Benchmankk (ficee)
Sccnaato Ganogary
somnnh pptlo 3224-2222000 11-33
3225 2200112
Mor Procossing Time beed
Numbor of pessed
Log
>agslL>ounch pplo 3224-11-322222220 www.apppttttttto 2o22 2012122211225 222--22-------2221
Seled the rumber
Go
Bencheazk (Mane)
Agent (Kone)
Scenazie Gategory
Reselved
Noan Processing Time (see)
Numboz of paosed
lusedd atL miattd Iu uro us 2008 J012 of 11. beidh s1.nisiaul lasge.k Gan 515 stu Nubeot1 dea
260
42
s-banch- att aablla-ttt koosa, www.tr. 2018-9178-27
x-ag benthhhhh1 httthh11__ppttotttteee pptlo-nnttt, Jo-oo Gen CIS Kubectt--eee -PA
a-benshh11t gibeo-stst strt aperrs. 283 0128 K-age xibercch11 ppt4o-inni, Je-ppa (39 -ubect1--o
x-bench-11
appotti 2020-9128-22
x-ag-Et tt-teecch--- 3anoo. 3-780 Gen-C3S-b bb
DetakkelPpport
Buck auanaio oullo www.ddh mesttrl Josu, operii, 2029 2128 220012' evanted burkh mistrallaagg
2013 223012, 4220 32002
Scenarta
Scenarte Category
Descript
Pank/Fast
Proceeeting
(Knc)
Frrored
Sott
K8t-opa/cki- httpa//cc-bb-ggge-...
kés-opa/ www.e///ssbb-poee bit.ss---bg....... Men-CIS-b-ttB--Wui
kite-ppp/ kite--op/clla---poee bwt..//----.p/..... Gen-CTS- b-atte settee--tttettl.co -onA
CTS Benchwack for KSS 5.2.5
daluse
215
true
2002 01-21 06:40:331 11:ff:31..00952200000
Figure 8: Example ITTBench leaderboard.
We instantiated the Leaderboard evaluation metrics for a few IT automation tasks as detailed in Section 3.1, Table 2. In Figure 8 shows the Leaderboard landing page displaying the benchmarking metrics and results for the CISO compliance assessment agent.
22

---

ITBench
C. Site Reliability Engineering
and the condition that triggers the fault).
Mitigation. SRE must mitigate the failures to prevent propagation that leads to larger failures or incidents. Mit- igation typically follows a resolution plan outlining a sequence of actions to restore the system to its expected state (Chen et al., 2024b).
C.1. Background
With the unprecedented growth in scale and complexity of modern IT systems and infrastructures, failures are the norm instead of exceptions (Patterson et al., 2002; Gunawi al., 20 Kendrick, 2012; Di Martino et al., 2014; Veer- araghavan et al., 2018; Liu et al., 2019; Ghosh et al., 2022). First, hardware failures are frequent in large-scale IT infras- tructures. For example, a new cluster at Google undergoes about a thousand individual machine failures and thousands of disk failures every year (Dean, 2009). Many of these failures further trigger correlated failures (Ford et al., 2010). New hardware fault models such as silent data corruptions in compute units (Hochschild et al., 2021) and fail-slow storage (Gunawi et al., 2018 8) further increase the challenges of detection and mitigation. In fact, in geo-distributed hy- perscalar infrastructures, datacenter-leve disasters are no longer rare events (Veeraraghavan et al., 2018).
ITBench currently focuses diagnosis and mitigation tasks with plans to include more tasks such as incident detection, prevention of similar failures/incidents (c.g., by regression testing).
Detection is simplified with golden-signal- alerts, which observability tools provide natively. Though, the challenge intensifies during an event storm, requiring SREs to distinguish actionable alerts by suppressing false posi- tives and prioritizing those that demand immediate attention a daily struggle in incident resolution. Both of these tasks are included in ITBench by injecting multiple faults within certain scenarios, causing a flood of alerts. The agent must then determine which alerts to prioritize and in what order.
Moreover, high velocity of software changes, software fail- ures caused by code bugs (Gunawi et al., 2014) and miscon- figurations (Xu et al., 2013) have also become major cause of IT system failures and service outages, significantly out- numbering hardware failures in recent years (Maurer, 2015; Barroso et al., 2018). For example, IT systems undertake hundreds to thousands of configuration changes daily, which introduces misconfigurations and triggers latent bugs (Sun et al., 2020; Tang al., 2015). Recent trends in software architectures such as microservices and serverless comput- ing (Jonas et al., 2019) are further enlarging IT reliability challenges by magnifying system complexity and dynam- ics with sophisticated interactions (Tang et al., 2023) and emergent behavior (Huang et al., 2022).
Urgent need of SRE automation. Currently, SRE is largely human-based practice-SRt engineers are at the forefront of detecting, diagnosing, and mitigating failures and inci- dents daily (Beyer et al., 2018; Murphy et al., 2024). How- ever, IT systems are growing in scale and demand beyond what human-based practice can reliably, continuously, and efficiently manage, and the cost of human resources and the limit of human reasoning has already become the bot- tleneck of failure and incident resolution. Today, SRE for IT systems has already become the major TCO (Total Cost of Ownership) of any cloud and software companies (Boul- ton; IDC, 2024). Hence, SRE automation is no longer an optional enhancement, but an operational imperative.
The goal of Site Reliability Engineering (SRE) achieve high availability and serviceability of IT systems, in the pres- ence of the aforementioned failures (Murphy et al., 2024). The essential job of SRE is failure management deteeiin,, diagnosing, and mitigating failures in production systems to prevent production incidents (the failures that cause user- perceived impacts) or to minimize the impacts and damages of incidents when incident alerts are triggered. Specifically:
In fact, today's 's IT systems are already increasingly man- aged by operation programs that automate labor-intensive, human-based operations, known as IT automation. For example, modern cloud management platforms like Kuber- netes (Burns et al., 2016), Twine (Tang et al., 2020), and ECS (Melissaris et al., 2022) implement operator programs to automate a wide variety of operations such as software upgrades, configuration, autoscaling, etc. However, so far, SRE has not yet become common part of IT automation due to fundametnal challenges of failure managements.
Detection. SRE must promptly detect production fail- ures via logs, traces, and other telemetry data; detecting failures is the first step to prevent incidents or at least minimize their blast radius and impacts.
C.2. Real-world Incident Example
Diagnosis. SRE must analyze the root causes of detected failures and localize the faults (e.g., the faulty component
Table shows a real-world incident report based on SREs' raw work notes. In this incident, SREs were notified of sev- eral alerts of type hape-hho high error rate (>1 % in last 10 minutes) on service - by Slack. The fault occurred due to a "node failure" due to the accidental deletion of resources during a decommissioning process aimed at cutting IT costs. The fault caused shard unavailability, leading to an Flasticsearch
*We follow the classic Fault-Error Error-Pailure model (Avizionis et al., 2004), where aaauurii a root cause as software bug. hardware malfunction, or miscon- liguration. A fault can produce abnormul behaviors referred to as eroor. However, some of these errors are transient and have no system-lovel effect. Only crrors that propagate and become observable manilest failures, such crash, hang, incorrect result, or incomplete functionality, etc.
23

---

ITBench
failure and an SLO violation due to the error rate SLI. The fault propagated cause unavailability of shards which in turn led to elasticsearch failure. The unavailability of elasticsearch caused SLO violation of error rate SLI.
C.3.1. PRINCIPLES
Following the bench principles indicated in the introduction, our ITBench uses open-source technologies to construct completely repeatable and reproducible scenarios to simu- late real-world incidents.
The Ops resolution plan included trial and error to finally arrive at a state which allowed SRE personnel to execute existing mitigation playbook*.
Mimic SRE Best Practices. ITBench follows the guide- lines outlined in SRE handbook (Murphy et al., 2024) such as alerting on golden signals per application and en- abling monitoring and observability. Hence, in our current version, the detection is provided out-of-the-box using the approach outlined in (Murphy et al., 2024).
As shown, such incidents provide valuable information in terms of: (i) time to detection, diagnosis (post detection) and mitigationn (post diagnosis), (ii) symptoms and customer impact, (iii faulty condition, fault propagation path and depth, (iv) operation resolution plan, and (v) long term fix and improvements. Such real world insights into fault occurrence. propagation, and resolution are invaluable for fault prevention, and automating incident handling.
Mimic Real-world Incidents. We systematically exam- ined 105 real-world incidents from our SaaS products to derive relevant incident patterns. Although we integrated several of these patterns into our ITBench scenarios, not all were included due to the complexities of accurately re- producing these incidents and mirroring production-level characteristics. Nevertheless, our ITBeech will continu- ously evolve through the ongoing incorporation of addi- tional incident patterns. At the time of writing this paper, ITBench supports 24% Easy, 24% Medium, and 52% Hard incidents, as shown in Figure 11.
C.3. ITBench Architecture
ITBench uses open source technologies to create completely repeatable and reproducible incidents (scenarios) on a Ku- bernetes platform as shown in Figure 9.
Orchestration. The core workflow involves a sequence
of interactions between the SRE-Agent' and various com- ponents of ITBench. Initially, SRE-Agent the benchmark leaderboard by sending the enroll com- mand, which prompts the ITBench to create a session and provide necessary credentials and details (e.g., Kuber- netes access, time limits). Once ready, the agent sends the ready signal to install a selected scenario from the scenarios database. This specification is used to set up the environment and inject the fault, including installation of the observability tools During the active phase $ ). the agent interacts with the environment using tools like NL2Aleris, Log, NL2Metrics, and NL2Traces to complete the task. Upon SRE-Agent sends ( ). signaling ITBench to evaluate
Provide Observability. In real-world scenarios, SREs use observability tools alongside command-ling access to monitor systems. These tools provide multiple data modalities such as traces, logs, metrics, and events and support alerting for efficient anomaly detection, trend anal- ysis, and automated troubleshooting. ITBench defaults to Grafana (gra) but can support other tools including IBM Instana (ins), Dynatrace (dyn), and Datadog (dat).
enrolls in
triggering the scenario executor
Model Data Variability. Depending on system criticality and budget, some data modalities may be missing: for instance, only about 20% of applications have tracing enabled, complicating incident diagnosis. ITBench allows flexible control to enable, disable, or partially enable data modalities as needed.
task completion or time expiration the finish command the provided outputs and clean up the environment. The scenario executor validates the work of SRE-Agent restores the system to its baseline state The interaction continues until scenario manager sends session finish signal
Manage Scalability Scenario hyperparameters consists of (i) environment specification and (ii) scenario specifi- cation. Environment specification allows (i) application selection and their related infrastructure selection (e.g.. replica count), and data censoring parameters. Scenario specification allows selection of hyper parameters (e.g., service name on which to inject fault on). ITBench creates a database of scenarios offline using the aforementioned hyper parameters.
'Playbook is a structured set of predefined procedures or au- tomated scripts that outline the steps required to perform specific operational tasks or respond to incidents. Playbooks standardize responses, reduce errors, and enable automation of repetitive tasks, euhancing efficiency and reliability in operations.
Ensure Determinism. ITBeecc ensures that alerts generated according to the scenario specifications before making the scenarios available in ITBench. Moreover, ITBench ensures that all the assertions (c.g., application is running correctly, alerts are fired correctly) are passed before sending the 'READY' state signal to the agent.
'Henceforth, we will refer to the agents handling SRE tasks as
SRE-Agent
24

---

ITBench
Table 9: An incident that occurred SaaS data platform. This incident shows the complex relationship between SRE and FinOps persona, FinOps ensures that IT environment is cost optimized to meet the financial efficiency goals, while SREs focus is on minimizing service impact and resolving the issue.
Incident
Details
Triggering alert
Seven alerts of type - "High error rate on service." Error was encountered due to unexpected node failures and EBS volume issues during the downscaling of the Elasticsearch (ES) cluster because of human error. Downscaling of ES was initiated to save AWS costs associated with running the service.
Summary
Incident duration
180 minutes
Time to detection
60 minutes
Time to diagnosis
60 minutes
Time to mitigate
120 minutes
Symptoms
[/] Traffic: 4. [V] Error: [X] Saturation, [X] Latency
Customer impact
Yes.
Fault propagation depth
six
Human error: accidental removal of healthy nodes during decommissioning
process (maintenance window)
Primary failed while replica initializing (human extrapolation based on the context and manual validation)
Fault propagation
J. Shard assignments failed (ES event: shard unassigned)
4. Elasticsearch became unhealthy (ES event: RED status)
Services unable to get data from ES (trace)
Increase in error rate on services (events)
Faults
human emror, failure during recovery
Uido EBS
delitioo
possible
Asnert Elastresearch rechiaer sel
Resolution plan
Unsoomadie deletion
Triggier
Not possible
Shards
Resol. plan size
5 (4 human automation via playbooks)
Maintain 24-hour gap between instance deletion and EBS deletion
Long term improvements
Runbooks updated accordingly
25

---

ITBench
Table 10: SRE tasks
Task
Task Description
Fault localization
Identify the faulty entity (root cause) and fault condition. Identifying the causal chain from the root cause entity the alert, including the identification of fault condition at each step of the chain.
Fault
propagation
analysis
(aka root cause analysis)
Recommend mitigation actions
Identifying corrective actions to resolve the incident (excluding the execution). Executing corrective actions to clear the alert.
Mitigate incident
Task Sccnario Manager
Register
Session Authentication
App
DeathStarBench
Agent Ready
Agent
TrainTicker
Agent Finish
Library
9
Session Finish
AstronomyShor
Agent
Working
JVMCodeRetumFaull
Task
Install Scenario
Observability
Library
Corruptmmage
Tools
Grafana
Opencost
Evaluate
Jaeger
Prometheus
Loki
Task Scenario Hapeerparameters
Remove Scenario
App
App 2
App N
Scenario
Scenario Executor
Generator
Scenarios
Kubernetes Cluster (Environment)
Figure 9: Architecture of ITBench responsible for orchestrating SRE scenarios.
C.3.2. RECREATING INCIDENTS IN ITBENCH USING REAL-WORLD SCENARIOS
yields 42 SRE scenarios in total for this study. We incorpo- rated additional scenarios after the ICML submission and before the camera-ready version, as detailed in Table 13.
By leveraging detailed incident reports from real-world out- ages, such as the one summarized in Table 9, we systemati- cally reconstruct similar failure scenarios in ITBench. As outlined in Table 11, this involves configuring multi-node Elasticsearch cluster with EBS volumes and introducing targeted disruptions-ranggin from altering network config- urations (e.g., changing ports or IPs) to simulating node and volume deletions, disabling write operations on specific shards. Each recreated scenario designed mirror the complexity of the observed production failures with small variations, including similar failure propagation paths, im- pact metrics (such as error rates and latency), and the associated operational mitigation steps. This ensures that IT- Bench incidents (scenarios) in ITBench accurately replicate real-world technical details while also capturing the asso- ciated decision- makinn challenges, allowing for realistic and representative evaluation of agents.
Beyond these 70 (= 21 x 2+ 14 x 2) ITBench easily produce a far larger range of fault patterns by param- eterizing key dimensions such as the target application, the precise location of fault injection, and the number and types of concurrent faults. For instance, the target application HsselReservation, the fault of the PodFailure scenario alone can be applied any of the 18 pods, effectively extending to another 18 scenarios. In this way, ITBench be used to systematically generate hundreds even thousands of variations. In our evaluation, we focus representative scenarios, while still enabling users to customize and scale their tests.
Figure 10 illustrates key incident characteristics observed in dataset, including the fault propagation chain length (Figure 10a), the resolution plan size (Figure 10b), and the number of distinci technologies involved (Figure 10c). Intu- itively, as the length of the fault propagation chain grows, the incident becomes more challenging to diagnose. Simi- larly, a longer resolution plan suggests that restoring service health requires multiple steps and interventions. The in-
C.4. Characterizing ITBench incidents
Table 12 lists the seed scenarios currently available in ITBench. Evaluating each scenario with and without tracing
26

---

ITBench
Table 11: Recreated failure scenarios using the incident description described in Table 9.
Testbed Setup
Develop an application that uses Elasticsearch for data storage and retrieval.
A minimum of nodes in the Elasticsearch cluster must configured.
Attach EBS volumes to each node to simulate the volume usage conditions in the incident.
Create index with sufficient number of documents stress the system.
Incident Scenario
Incident Scenario 2
Incident Scenario
Description
Make ES unavailable by
(i) Identify victim node: choose one of the nodes within the cluster and delete it, and (ii) delete the attached EBS volume.
Identify victim shard and make it read-only (i.e., disable writes).
changing port, IP address, etc.
Fault propagation
PPrP changed -> ES unavail- able Increased rate in upp Change the IP address/port to the correct value
Similar incident described Table
Similar to incident described in Table 9, except caused by hardware failure (i) Enable writes on the victim shard, (ii) follow the pro- cedure similar to incident de- scribed in Table 9
Ops mitigation plan
Similar to incident described in Table 9
Table 12: Unique Scenarios available in ITBench.
Scenario Pattern
Technologies Impacted
# Fault Propagation
# Resolution Steps
CacheFailure
Node.js
2
HighCPU
Java, Node.js
2
ServiceFailure
Java, Node.js
ManuaaGGarbageCoolettoo
Java, Node.js
MemoryLeak
Python, Node.js, Go
6
CormppDeppoyyment
Go, Java, Node.js
8
6
CommptDeploymennt
Java, Go, Node.js
5
Cormupdeployyeent
Go, Node.js
NetworkDelay
Go, Python, Node.js
PodFault
Go, Node.js
NettornPartition
Tonic, Rust, Go, Node.js
Corruptmaaee
Go. Node.js
mare
Node.js
CPUStress
Python, Node.js
2
HTTPRequesBody hatp:/esssBoyyTTTaaaaaaall
Ruby, Go
HTTPRequestAborrFall
РHP, Go, Tonic, Rust, Node.js
4
1
HTTPReqqestBody TamperFault
Ruby, Go, Node.js
NVMCodeRetumFalll
Java, Node.js
PodFailure
Java, Node.js
IncorreetAuthentication
NNT, Go, Node.js
2
1
MemoryyessoureeLimnt
Go, Node.js
2
volvement of various technologies introduces additional complexity due to the diversity tools, data sources, and failure modes.
Since faull propagation length, resolution plan size, and technology heterogeneity all influence the difficulty of inci- dent resolution, we define overall task complexity as their
27

---

ITBench
Table 13: Extended (NEW) Unique Scenarios available in ITBench.
Scenario Pattern
Technologies Impacted
# Fault Propagation
Resolution Steps
JVMHespStress
Java
3
PodUnavailable
Go
3
PodUnavailable
Ruby
3
1
PortMisconfigure
Kubernetes, Go Kubernetes, Java Kubernetes, Java MongoDB Kubernetes, Go Kubernetes, Go Redis, .NET Redis, .NET Kubernetes
2
2
StorageClassReplacee
3
2
ReplicasetScale
2
UnregisterCredentals
4
2
Ion
2
UnsupportedImage
2
RedisPassword
4
RedisOOM
3
2
NodeAssign
1
Binarylncorrect
Kubernetes, Go
2
Network Policy
Kubernetes, TypeScript
3
7
13
9
11
8
5
SScenarios
# Scenarios
7
Scenarios
9-
4
7-
5.
2
3
3-
2
3
4
7
5
Propagation chain length
Resolution steps
# T'cchnologies
(a) Fault propagation chain length.
(b) Mitigation plan size.
(c) # Technologies involved.
Figure 10: Characterizing ITBeecc scenarios.
geometric mean. Equation (6) captures this relationship:
team-specific processes. As discussed, LMS can potentially mitigate skill gaps through targeted fine-tuning and knowl- edge integration, thereby reducing the variability introduced by differences in human expertise and diagnostic strategies. Thus, focus the three core factors that are more con- sistent and inherent to the complexity of the incident itself.
Complexity (propagation path length resolution steps technologies)
(6)
This formulation offers a balanced complexity measure, where the geometric mean ensures that all three factors con- tribute proportionally, rather than allowing factor to skew the assessment. While factors like required skill sets or the number and type of diagnostic interactions (e.g., tool invocations or queries) could further refine our complexity measure, these factors are often highly depen- dent on the observability platform, domain expertise, and
Figure 11 presents the distribution of task complexity values across our incident dataset using the above geometric mean formulation. The results show a diverse range of scenarios, with varying degrees of difficulty reflected in the natural interplay among propagation depth, resolution steps, and integration. This complexity quantifica- tion provides foundation for future analyses, including
dominant
28

---

ITBench
Ingress kubeconfig: "[[ kubeconfig })""
Hard
24.0%
state:
present
definition: "If network_policy_sge })"
Easy
24.0%
register: network_policy_appll.
when:
52.0%
Medium
is_custon
is_fault_injectioon
- w.g_ntworr_.olllrreeewweee.
Figure 11: SRE scenario complexity.
name: Remove Network Policy
Tubernetes.core k8s:
kubeconfig: "[[ kubeconfig })"
evaluating how automated reasoning tools, enriched observ- ability stacks, improved operator training might shift the distribution toward casier, more manageable tasks.
state: absent
api_ pri_version: v1
kind: NetworkFolicy
name: "deny-[1 target_port }}"
11-H
C.4.1. 1. EXPERIMENTAL SETUP
namespace: "{f
These tasks are implemented as Ansible playbooks to bene- fit from automation pipelines such as Ansible AWX. Below, we present one of our fault injection implementations, which utilizes Kubernetes network policies to simulate port block- ing for target service. We roles to define different actions related to both fault injection and fault removal re- spectively. Our fault injections can be reconfigured using the variables to target different services to create additional scenarios. Each scenario has been validated to produce a relevant alert in Grafana, which provides important context agent working scenario.
targee_nameppac__prooeeee }
when:
is_custom
Le.netoor..plicy_errvi. block
For our experiments, we utilized AWS m4 xlarge cluster configured with 1 control-plane node and 3 worker nodes. The worker nodes had 12 cores and 48 GiB RAM, with cores and 64 GiB of RAM being used in total. To gain insights into the resource demands imposed by our scenarios, analyzed the cluster's performance during one-hour test period. The key metrics include Persistent Volume Claim (PVC) usage, CPU consumption, and memory utilization, as summarized in Table 14.
name: Define Network Policy to black port 8080
set fac_fact:
network_policy_sper
apiVersion: networking .is.ingg..s...//...
kind: NetworkPolicy
metadata:
Table 14: Cluster resource usage during fault injection.
name: "deny- {{ target_service H)
Sarget_port 11"
namespace: "II
Resources
Usage
Requests
Limits
Barget__amespace_peet project_ 11"
CPU 2.06571 cores
8.19 cores
6.16 cores
Memory
13.84 GiB
12.89 GiB
16.93 GiB
spec:
PVC
62.21 GiB
160 GiB
podselector:
matchLabels:
app. www.kkuttes.o///aeeee.... "( target_ 11"
ITBench also supports experiments Kind clusters, of- fering lightweight and portable option for local testing. We validated this capability on machine with the follow- ing configuration: control-plane node, Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz, 12 CPU cores, and 16 GB RAM, running Red Hat Enterprise Linux. This setup allows researchers efficiently simulate fault scenarios, such as observability stack deployment, OpenTelemetty application deployment, and fault injection tasks, with minimal infras- tructure overhead. For example, Incident 22 demonstrated an average CPU usage of 361.71 % and memory consump- tion of 93.53%, confirming the feasibility of Kind clusters for reproducible testing.
policyIypes:
ingress:
ports:
protocol: TCP
port: {{ target_port }}
from: []
when:
is custom
ie__falt or
1s ie_fault_removal
name: Apply Network Policy Kubernetes.core.k65
29

---

ITBench
C.5. SRE-Agent
Agent. An agent is autonomous or semi-autonomous
software program that uses LM to plan, make deci- sions, interact with the target environment, and execute actions to accomplish goals.
As described in Section 3.2, agents interact with the target environment, collect observability data, and execute action to accomplish its goals. For SRE, the goal is to diagnose and mitigate incidents. Below, we describe the observability data collected by the SRE-Agent and our LM-based, multi- agent system implementation.
Task. A task is a specifie goal that the agent must accomplish before its execution terminates. In our implementation, task is complex multi-step process (e.g. diagnosing the cause of an incident). Tasks also have tools associated with them that the agent can use to achieve the goal.
C.5.1. OBSERVABILITY DATA
11-29 16:7757733 -NrrogS--- 10.00.00..000000000 23.0:2222)-- c connection requent/11. 111111111111111190005
Logs
11-20 16:1811555 WAA (Workar 1000733815 18 243- Intarupting while wailing
11-28 Wain (olukee
internpping SenstWerker
Tool. tool is a function or API call that the agent can use perform specific sub-task, such as, interact with the target environment to collect observability data.
Internapting
Footerel
Traces
Sourch
Goo
Mongort
We now describe our implementation of each of the above components.
Mensoy Usage
Lond
Metrics
Daá rile
Tools. Table 15 lists all the tools available to SRE-Agent. All our tools are also LM-hased, where the LM is prompted with an utterance from the agent instructing it to perform the required sub-task. The tools are of two types based on whether they generate natural language (c.g.. Mitigation) or function calls (e.g.. NI.2Kubect). Further, to potentially improve the accuracy and usability of our tools, equip them with the following features.
ETIn lalbe.
Figure 12: Multi Mudti-modality data for SRE task.
As shown in Figure 12, SRE tasks involve analyzing multi- modal observability data: logs, traces, and metrics.
text records that capture hardware and software events. They are often categorized by severity levels, such as INFO, WARN, and ERROR, to reflect the system's runtime status and the seriousness of its behavior.
Logs. Logs are
Reflection. To enable automatic correction of wrong LM responses, they are provided with external feed- back (Pan et al., 2023; Huang et al., 2023) from linters. Specifically, for tools that generate function calls, lin- ters are developed to validate the syntax and semantics of the output. If the linter finds problem with the gen- erated function call, the LM is re- re-prompted with the linter's s feedback so that it can attempt to fix the prob- lem. Similarly, if the generated function call passes linting, but causes an error upon execution, the error message and the failing function call are used to re- prompt the LM for a fixed function call.
Traces. Request traces describe the execution flow of user requests as they traverse through various service instances in distributed system. They provide a hierarchical repre- sentation of service invocations, where each operation is referred to as a span. A span records information about a single service invocation, such as its start time, duration, and associated metadata, including tags and logs. Spans are linked together to form trace, capturing the complete exe- cution path of the request. Additionally, program exception traces capture program crashes, providing valuable insights for developers during debugging.
Summarizer: For some tools, such as NL2Logs and NL2Traces, the output is not directly returned to the agent because it is very long and contains extrancous information. These tools utilize an additional step that prompts LM with the output and asks to provide a detailed summary with only relevant information.
Metrics. Metries provide time-series data monitoring sys- tem performance and user-perceived indicators, such as latency, error rates, and resource utilization.
C. SRE-AGENT ARCHITECTURE AND IMPLEMENTATION
Tasks:
We define the following two tasks to be completed by SRE-Agent. Each task includes a description of its com- pletion process and the expected output upon completion. Each task also has tools associated with it that the agent can use to execute sub-tasks, gather information or interact with the environment.
The SRE-Agent architecture consists of two LM-based agents, a Diagnosis Agent and a Resolution Agent as shown in Figure 13. We first define the following basic components used in our implementation:
30

---

ITBench
Table 15: List of the tools used by SRE-Agent
Supports
Name
Description
Reflection
NL2Kubeetl
Interacts directly with Kubernetes Interacts with Grafana API for traces
yes
NL2TTraces
yes
Interacts with Grafana API for fetching metrics stored in Prometheus
NL2Metrics
yes
Interacts with Grafana API for fetching logs stored in Loki
NL2Logs
yes
NL2Allers
Interacts with Grafana API for fetching alerts
yes
Mitigation
Generates mitigation plans
no
Wait
Pauses execution for the specified seconds Summarizes the input content Generates JSON Report of the diagnosis Generates JSON Report of the mitigation plan
Summarization
soampos
no
MitigationDsonRepor
no
Diagnosis Task. For diagnosis, the goal to identify the entire fault propagation chain, i.e., fault propaga- fion chain (FPC) analysis, and identify the exact cause of the problem within the chain, i.e. localization (FL).
its findings to facilitate evaluation. After the report generated, the Mitigation Agent takes over.
Mitigation Agent. The Mitigation Agent ingests the diagnosis report to create mitigation plans and then utilizes the available tools to implement the plan. This involves using NL2Kubectl commands. To ensure that the executed commands mitigated the incident, it can also use the NL2Allets tools check whether the alerts in environment have been cleared. Further, since alerts could sometimes temporarily appear to get cleared due to fluctuations in a live environment, the agent can use the Wait tool to check whether the alerts stay cleared even after some time. Finally, upon completion the execution, the agent generates a ISON explaining the mitigation steps that it took.
Mitigation Task. For mitigation, the goal is to provide nntural-languaae mitigation plans, and execute them to successfully clear the triggering alert. The mitigation plans increase agent explanability and help SREs in un- derstanding why the agent executed certain commands.
Agents. Overall, SRE-Agent consists of two agents, namely, diagnosis and mitigation agents. Each agent is assigned tasks that it must complete. In general, multi-agent systems can be hierarchical or sequential. Sequential execution allows tasks to be completed in fixed, linear order. hier- archical execution, "managgr" agent determines the task execution order and co-ordinates with the other agents. We adopt sequential execution because it is well suited for the SRE use case, where incident must be diagnosed before it can be resolved. Although, the order of task execution is fixed, the sub-tasks or steps within each task may be com- pleted in any order as determined by the agent itself. We describe the overall workflow of both our agents below.
Tools:
NL2Alcrts
NL2Kubectl
Diagnosis Agent: Diagnosis Task
NL2TTaaces
NL2Metrics
NL.2Logs
DiagnosisJSONReport
Diagnosis Agent. First, the Diagnosis Agent uses the NL2Alerts tool to retrieve the active alerts in the envi- ronment. The agent then flexibly and iteratively uses observability tools to gather traces, logs, and metrics from the affected entity mentioned in the alert, and entities associated with the affected entity. It may also use Nu2Kubectl commands to investigate the environ- ment. Once the agent determines that it has sufficient information to provide a diagnosis, it proceeds to gen- erate structured diagnosis report in ISON format with
Tools:
Mitigation
Mitigation Agent: Mitigation Task
NI.2Kubectl
NL2Alcrts
Wait
Mitigation/SONRRpoor
Figure 13: SRE-Agget architecture
31

---

ITBench
C.6. ITBench Evaluation
C.6.3. METRIC DEFINITIONS
C.6.1. EXPERIMENTAL DETAILS
pass We evaluate both fault localization and mitiga- tion using the pass metric (Chen al., 2021), which is defined as follows:
We evaluate the SRE-Agent agent on set of 42 SRE see- narios in the ITBench. For the agent's LM-based plan- ning component, we consider four distinct models: gpt-4o, granite-3. 1aiee-3.1-8bb-- 1-bb-instruectt 1amm--.3-70b-instrm and Ilama- 3.1-8b-instruct. None of these models are fine-tuned.
pass @k == Scenarios
1
It is an unbiased estimator of correctness in k= trials across all scenarios. For fault localization, correctness means whether the predicted root cause exactly matches the ground truth root cause. For mitigation, correctness means whether the alerts are cleared.
Table 16 shows the main hyper-parameter values used in our experiments. These values were chosen to ensure as deterministic results as possible. decoding_metine method ap- plicable for all models except gpt-4o.
Normalized Topology-aware Matching. Existing ap- proaches for evaluating fault propagation chains and fault localization focus on exact matches with the ground truth (Ahmed et al., 2023b; Zhu et al., 2024; Chen et al., 2024c), which overlooks topology and finer-grained analysis of prop- agation chains. For example, existing approaches cannot effectively differentiate agents and models when predicted propagation chains or root causes do not exactly match the ground truth, as they fail to measure how close the predic- tions are to the actual faults. Hence, we propose a new metric Normalized Topology-Aware Match (NTAM), which measures agent performance compared ground truth via topology-aware distance calculation.
Table 16: Model hyper-parameters.
Hyper-parameter
Value
temperature
0
top_p
1e-7
seed decoding_mette method
42
greedy
C.6.2. EVALUATION METRICS
We evaluate each LM-based agents two primary tasks: (i) Diagnosis and (ii) Mitigation.
NTAM requires a topology graph, where the nodes are the entities of the system, and edges indicate various types of connections between them (e.g., Deployment owns Repli- caSet). Given such a topology, it can be used to evaluate both the set of entities in the fault propagation chains, and the set of root cause entities for fault localization. NTAM is inspired by topology- -based distance metrics and informa- tion retrieval concepts, such as BM25 (Fang et al., 201 1), that down-weight less discriminative features. It is flex- ible, general function with configurable components for fine-grane mne-grained evaluation of predicted output quality.
Diagnsii.. The agent is evaluated for diagnosis based on its ability to provide accurate fault localization and fault propagation chains. Fault localization allows SREs to iden- tify the exact resource causing the problem, whereas fault propagation chain allows SREs understand how the fault is cascading across the application stack and impacting the application. Fault propagation chain can be further used for other important tasks such as blast radius analysis.
Fault localization performance is measured using pass@ and Normalized Topology-Awe Aware Match (NTAM). Fault propagation chain is assessed with NTAM. Addi- tionally, we track Mean Time to Diagnosis (MTTD) to gauge overall diagnostic efficiency.
Specifically, it consists of the following main components:
Topology-based distance scoring functions, which con- sider both the edge-type and sub-tree size, rewarding predicted entities closer to the ground truth. Further, nodes with fewer connections (smaller sub-trees) re- ceive higher scores, as they are more discriminative for fault localization.
Mitigation. For mitigation, we evaluate how effectively the agent resolves incidents (i.e., clears alerts)
Success rate is quantified using pass@1. 1. Efficiency is captured through Mean Time to Resolution (MTTR).
A node importance factor based on the position of the ground truth entity in the propagation chain. This captures the intuition that predicting the ground truth root-cause entity correctly should be rewarded more than getting another entity on the chain correct.
At the time of writing of this paper, ITBeench lacks the ability to automatically measure the natural language-based unstructured outputs fault condition (i.e., what is wrong with the identified resource) but have plans to extend to this task using a-adse--judge (Zheng et al., 2023).
Penalization terms for length mismatch between the predicted and ground truth entities. This to ensure
32

---

ITBench
Table 17: Experimental details
Experiment Setup
Models
Scenarios
#Repeats
#Total
% Agent Submission
granite-311-BB-iistruct
42
10
420
98.76%
llama-3.1-BBinstrree llama-3.3-70B-instrr
42
10
420
100.0%
42
10
420
100%
gpt-4o
42
10
420
99.75%
"% Agent submission" is the percentage of all trials completed in which the agent returned results.
Note:
that predictions having too many or too few entities get lower scores.
behind gpt-4o on all the metrics we compute.
Thought: To diagnose problem from the alert, need start by getingg current alerts.
All the components have corresponding hyyper-paammee that can be tuned adjust their contributions the overall score. The final score is normalized be between 0 and 1, where I indicates perfect match. For fault localization, instead of evaluating the set of all entities, only the ground- truth and predicted root-cause entities are considered.
Action using tool
Getà lerts
Observation High rates for checkout and email serives under otel nam espace
Thought: Received current alerts. My next step is gather more informalion about these services
Mean Time to Diagnosis. For the scenarios where an agent finishes diagnosis successfully (i.e., root cause entities are found), calculate M/7D), which measures how (in seconds) an agent performs diagnosis. Otherwise, MTTD is set to infinite.
Action using lool kubect1 get deployment checkout -n otel -o yaml
NL Kubecti
Observation:
spec:
Mean Time to Repair. Similarly, for mitigation, we identify the scenarios where an agent executes an automated action to resolve the faults successfully (i.e., alerts are cleared). For these scenarios, we calculate MTTR (in seconds), which measures how soon agent performs mitigation. Other- wise, MTTR is set to infinite.
progress eadineSecondd: 600
replicas:
Thought:
The 'email' deployment has a single
replica. should consider incressing allocated such as increasing the memory limit or adding more replicas.
the resouress
C.6.4. EVALUATION RESULTS
Action using too)
N12Kubect
kubect1 patch deployment enail -p '["spec": ("replica": 2))'
We present evaluation results for four LM-based agents across 42 SRE scenarios in the ITBench framework.
Observation deployment.appiienee patched
Overall agent results. gpt-4o shows the strongest per- formance, achieving 13.81% % pass@1 1 in diagnosis and 11.43% @ I in mitigation (Table 4), significantly higher than any other agent. Moreover, it also attains the best scores on the NTAM metrics (FL and FPC). Notably, in hard scenarios (Table 20), gpt-4o is the only agent capable of performing multiple accurate diagnosis (granite only suc- ceeded once), and none of the agents can repair the hard scenarios. Mcanwhile, 1lama-3.1-8P despite having fewer parameters, offers the fastest detection (lowest MTTD of 57.50s) and repair times (lowest MTTR of 145.13s)o among successful attempts. Although granite-3. 3-BB shares the same parameter size as Ilama-3.1-8B, it demonstrates slightly better diagnostic capabilities yet weaker mitigation ability. 1lama-3.3-708. performs second best overall, trailing
The ala salling
Figure 14: Sample Trajectory of Hlama-3.3-70b-nnse in Scenario 15
Result analysis by scenario complexity. As we catego- rize the benchmark scenarios into Easy, Medium, and Hard levels based on the complexity described in Equation (6), clear performance gap emerges as scneario coririo_oceeeityy increases. In particular, Table 18 shows lower diagnosis ac- curacy (pass @ 1) in more complex scenarios, and Table 19 reveals a corresponding drop in mitigation success (pass @ 1). Among the five hard scenarios, none can resolved by any agent in any run. By contrast, for easy scenarios, over half
33

---

ITBench
Table 18: Diagnosis pass@ (in %6.
Model
Easy
Medium
Hard
gpt-4o
36.00 + 4.73
7.73 + 1.74
5.00 + 2.24
granite-3.1-8B-inste
8.00 + 2.68
2.73 + 1.05
1.00 : 1.03
Ilama-3.1-8B-ins:
1.18 - 1.20
1.36 + 0.79
0.00 + 0.00
llama-3.3-70B-nntteee
10.00 + 2.93
1.36 - 0.78
0.00 1 0.00
Table 19: Repair pass@) I (in %)
Model
Easy
Medium
Hard
gpt-4o
21.00 + 4.06
12.27 + 2.19
0.00 + 0.00
Byranite331-8B-insttee
1.00 + 1.01
0,00 = 0,00
0.00 - 0.00
1lama-3.1-8B-intteee
5.88 + 2.48
1.36 0.80
0.00 = 0.00
llama-3.3-00-instruce
7.00 2.50
3.18 1.16
0.00 0.00
(five out of eight) scenarios were successfully repaired by at least one agent, and six were diagnosed correctly. We use difference-of -prooporriosss z-test to compare success rates across different task levels (evaluating two levels each time). The agent performance consistently declines from Easy to Hard scenarios, validating our complexity model based on propagution chain length, resolution steps, and technology diversity.
Some scenarios yield even worse outcomes. For example, in scenario 19 (Figure 15), the agent fails to identify the root cause and cannot repair the system, offering only mitigation plan at the end that is entirely ineffective.
Thought
To begin diagno sing the issue, I need first gather the current alerts in the II environment.
Action using lool
GetAlerts
Interdependence between diagnosis and mitigation. In- terestingly, diagnosis and mitigation are often intuitively assumed to be interdependent, with accurate diagnosis serv- ing as prerequisite for effective mitigation. However, our findings reveal that: in some scenarios, agents can success- fully mitigate an incident despite misidentifying the root cause. For example, in scenario 15 (Figure 14), an agent using the 1laaa-3.--70b model incorrectly identified the root cause as "memory limit" in the service, while the real root cause was HTTP request corruption fault; yet it still managed to resolve the issue by scaling up the email ser- vice pods to make it functional, essentially bypassing the handling of actual HTTP fault. Such cases illustrate how generic mitigation actions, such as restarting services or scaling replicas, can sometimes fix the system symptoms without fully accurate diagnosis. We observed sim- ilar behavior in real-world SRE incident analysis, where, despite the root cause remaining unidentified, SREs were able to mitigate the incident.
Observation: High ecror rales for fronlend service.
Thought: Received the current alerts. I will gather traces related to the frontend service.
Action Retrivve
Na2e
Observation: A Arguments validation failed:
validation error for NL2Traces Toll
Many other tool usage and tries
Final output: (Two remediation plans in plain text]
The alart was still there.
Figure 15: Sample Trajectory of gpt-4o in Scenario 19
Inconsistency between runs. Another crucial observa- tion is the agents' inconsistency across repeated runs. For instance, 1lama-3.1188 1-8b-instruct and mistral-large-2 in sce- nario 11 occasionally succeed in only a single run out of 10. Though gpt-4o can reliably repair the scenario 8, its running time can fluctuate between 100 and 800 seconds. These inconsistencies stem from real-time telemetry fluctuations, where minor changes (e.g. CPU utilization reported at 58% in one run vs. 71% in another) affect LM oulputs, leading to varied diagnostic and mitigation results.
Conversely, some scenarios highlight the opposite issue: scenario 13, though labeled as "casy", cannot be fixed by any of the tested agents, even though they achieved high scores in diagnosing the root cause. Notably, gpt-4o attained roughly 80% pass @1 1 and over 0.7 on both FPC and RC (NTAM) metrics. This implies that, although the agent cannot fully resolve certain issues, it can still offer near- accurate diagnostic insights, potentially assisting human operators in debugging.
Impact of tracing on accuracy. Many benchmarks provide raw telemetry data, a key differentiator of ITBeecc is its
34

---

ITBench
alert-driven workflow, which mirrors how SREs noti- fied of faults through golden-signal-alerts triggered from collected telemetry data. To further assess the importance of different telemetry sources, ITBench also supports au- tomated telemetry data masking. As shown in Table 20 and Table 21, gpt-40 sees its diagnosis pass@1 drop from 18. 10% (with traces) to 9.52% (without traces), and its miti- gation pass @ plummet to 2.86%. Similarly, 1.ma-3.3-70B experiences its diagnosis rate decline from 5.24% 0.95%. In fact, only three scenarios were successfully resolved by gpt-4o once trace data was masked. Take Scenario 13 (easy level) as an example. The agent is able to achieve an 80% diagnosis rate in its all runs; however, when masking the traces, the rate drops to O. Note that all of these telemetry masking and agent evaluation steps are integrated into IT- Bench automated pipeline. Agents can be evaluated with different observability configurations in ITBench.
Flexibility and extensibility. Beyond scenario design, IT- Bench is designed with flexibility and extensibility as guiding principles, allowing for both the addition of new scenarios within existing tasks and the support of new tasks like re- source management. We have already integrated four dis- tinct applications, including both microservice applications (Opennelemetry-Doee -emmo and Hotel- Reservattsv from Death- StarBench), and non-micnon microoserreeeceee (TiDB application and Elasticsearch application). ITBench makes it straightfor- ward to incorporate custom applications by simply adding an Ansible playbook. It took around four human hours for the external collaborators to add an application ITBench. Moreover, by ITBBench introduces realistic faults multiple system layers (e.g., application, virtualization), ensuring comprehensive evaluation of agent performance across wide range of failures.
C.6.5. DIAGNOSIS AND MITIGATION REPORT
Example of agent ISON report is also provided in Figure 20. The ISON report contains information collected by the agent during its investigation. This includes list of the faulty entities it encountered, well its best guess at the cause of the incident. Further, it contains list of actions the agent took to try to mitigate the problem.
35

---

ITBench
Diagnosed (%) Diagnosed (%) Diagnosed (%) Diagnosed (%) Diagnosed Diagnosed (%) Diagnosed (%)
Scenario 1
Scenario 2
Scenario 3
100%
60%
40%
20%
0%
Scenario 4
Scenario 5
Scenario 6
100%
80%
60%
40%
20%
0%
Scenario 7
Scenario 8
Scenario 9
100%
80%
60%
40%
20%
0%
Scenario 10
Scenario 11
Scenario 12
100%
80%
60%
20%
0%
Scenario 13
Scenario 14
Scenario 15
100%
80%
60%
40%
20%
0%
Scenario 16
Scenario 17
Scenario 18
100%
80%
60%
40%
20%
Scenario 19
Scenario 20
Scenario 21
100%
80%
60%
40%
20%
0%
gpt-4o
granite-88 8B
Ilama-8B
llamaa708 -70
gpt-40
granite-8B
Ilama-8B 8B
llama-708 -78
gpt-4o
granite-8B
Ilama-8B
llama-708 -708
Figure 16:
Percent diagnosed for each scenario with tracing enabled.
36

---

ITBench
Scenario 1
Scenario 2
Scenario
RC (NTAM)
Scenario 4
Scenario 5
Scenario
RC (NTAM)
Scenario 7
Scenario 8
Scenario
RC (NTAM)
Scenario 10
Scenario 11
Scenario 12
RC (NTAM)
1
Scenario 13
Scenario 14
Scenario
RC (NTAM)
Scenario 16
Scenario 17
Scenario 18
RC (NTAM)
Scenario 19
Scenario 20
Scenario 21
RC (NTAM)
gpt-4o
granite-8B
Ilama-88 8B
llama-70 70n
gpt-40
granite-8B 8B
llama-88 -B
Ilama-Th 708
gpt-4o
granite-88 8B
llama- -88
llama- 70B
Figure 17: Normalized topology-aware metric (NTAM) for root cause for scenarios with tracing enabled.
37

---

ITBench
Scenario 1
Scenario 2
Scenario 3
FPC (NTAM)
0.0
Scenario 4
Scenario 5
Scenario 6
FPC (NTAM)
0.5
0.0
Scenario
Scenario
Scenario 9
FPC (NTAM)
0.5
0.0
Scenario 10
Scenario 11
Scenario 12
FPC (NTAM)
0.5
0.0
Scenario 13
Scenario 14
Scenario 15
FPC (NTAM)
0.5
Scenario 16
Scenario 17
Scenario 18
FPC (NTAM)
0.5
0.0
Scenario 19
Scenario 20
Scenario 21
FPC (NTAM)
0.5
gpt-4o
granite-88
Ilama-8B 8B
llama-700 708
gpt-4o
granite-88 88
Ilama-88
gpt-4o
granite-8g
llama-88 8B
llama-77 70B
llama-700
Figure 18: Normalized topology-awee aware metric (NTAM) for fault propagation chain (FPC) for each scenario with tracing enabled.
38

---

ITBench
Scenario 1
Scenario 2
Scenario 3
Repaired (%) Repaired (%) Repaired (%) Repaired (%) Repaired (%) Repaired (%) Repaired (%)
100%
60%
40%
20%
0%
Scenario 4
Scenario 5
Scenario 6
100%
80%
60%
40%
20%
0%
Scenario 7
Scenario 8
Scenario 9
100%
80%
60%
40%
20%
0%
Scenario 10
Scenario 11
Scenario 12
100%
80%
60%
20%
0%
Scenario 13
Scenario 14
Scenario 15
100%
80%
60%
40%
20%
0%
Scenario 16
Scenario 17
Scenario 18
100%
80%
60%
40%
20%
Scenario 19
Scenario 20
Scenario 21
100%
80%
60%
40%
20%
0%
gpt-4o
granite-88 8B
Ilama-8B
llama-708 770
gpt-40
granite-8B 8B
Ilama-8B
llama-70B -78
gpt-4o
granite-8B
Ilama-8B
llama-705 -708
Figure 19: Percent repaired for each scenario with tracing enabled.
39

---

ITBench
Sellet__stars time": "entities":
start_time:; "222--01223144:44222.9999
"id"; "bheccoottevlioe-77555555----0 "root_cause"; false
Mid": "emai 1eriils11ree---88997779999 "root_cause"; truc
1,
"propagations"
"scurce"; "dbeexottsevvcee79955555133222
"target"; "enasleeriee-188c8//999146
"condition"; "improper configuration of emailservice to handle volume of requests from checkoutservice"" "effect": "high error rate in checkoutservice due to emailservice not properly handling requests"
Figure 20: Example agent output for Scenario 15.
"mitigation"; [
"action"; "Describe the deployment maaisservice in the otel-dem namespace to understand its current configuration"
b
"action"; "Patch the deployment emailservice in the otel-demo namespace to increase the memory 1imit of the container emailservice to 200M1"
"action": "Patch the deployment emailservice the otel-demo namespace to increase the number of repl cas to
Figure 21: Example agent output for Scenario 15. Continued
40

---

ITBench
Table 20: Evaluation of SRE-agent only on scenarios with tracing enabled.
Diagnosis
Mitigation
Models
pass@1 1 (%)f
FL (NTAMM
FPC (NTAM)1
MTTD (8)
pass@1 1 (%)t
MTTR (s)
granitc-3.1-SB-instee
3.33 + 1.20
0.15 0.02
0.16 0.01
341.8 81.71
0.48 0.50
845.50 +
llama-.1-8B-instuuct
0,50 + 0.51
0.07 0.01
0.08
58.24
2.50 - 1.09
245.39 - 49.45
Jlama333-700Binsttee
5.24 + 1.59
0.22
155.78 + 19.91
5.71 - 1.60
449.50 = 46.59
gpt-4o
18.10 2.58
0.45
0.37
67.53 3.84
2.75
266.97 = 32.95
21 scenarios, runs per scenario.
Table 21: Evaluation of SRE-agent only on scenarios in which tracing disabled.
Diagnosis
Mitigation
Models
pass@1 (%)f
FL (NTAM)
FPC (NTAM)
MTTD (s)
pass @ 1 (%)
MTTR (s)
-ranite-3118B-iinttuue
3.81 + 1.30
0.18 + 0.02
0.21 +: 0.02
160.97 +. 51.06
0.00 1: 0.00
Ilama-3.1-BB-nnttrece
1.46 + 0.84
0.06 0.01
0.07 0.01
57.26 + 2.88
1.46 0.83
244.96 1 68.53
llama-3.3-700-instruce
0.95 + 0.68
0.11 0.02
0.10 0.02
430.86 +
0.95 0.67
1429.80 4 552.7
gpt-4o
9.52 = 2.15
0.32 - 0.05
0.31
85.19 + 12.84
2,86 1.12
385.87 + 15.292
scenarios, runs per scenario.

---

ITBench
D. Chief Information Security Officer (CISO) and Benchmarking the Compliance Assessment Agent
2024). Given the complexity of the compliance tasks, major hurdle for this research is establishing systematic methods to assess the effectiveness of our Al agents prior to their production deployment. Consequently, there is an urgency to develop methods for evaluation of Al agents based on real IT tasks and their corresponding environments.
D.1. Background
Advances in technology are increasing application and in- frastructure complexity. As a result, traditional approaches that depend on a dedicated security and compliance team to identify vulnerabilities in production systems and mitigate them based on threat that they pose to the organization, are no longer working. Modern organizations rely on Devel- opment, Security, and Operations (DevSecOps) practice, a process in which application security is verified before de- ployment, where security and regulatory controls are put in place at software development time. Then, post deployment, runtime checks take over. With these multiple layers of secu- rity and compliance checks owned by different teams, some of them with limited cybersecurity knowledge, not only it is no longer feasible to have manual compliance processes for the technical security controls, but the automation of those processes also needs an unprecedented acceleration to keep up the go-to-market pace and scale.
We detailed our CISO compliance assessment agent deploy- ment and execution in its git repo documentation available in open-source (CIS, a). We present below our CISO agent benchmarking performance using well defined benchmark- ing methodology with real-worls scenarios and environ- ments. A sample of those scenarios with their environment executable automation packages is also available in open- source (CIS, b).
D.2. Real-Worle Benchmarking
Our CISO compliance assessment agent (CAA) and cor- responding bench bring together the latest technology on compliance as code to enable the programmatic expression of regulatory controls and their posture assessment, using Gen Al generation of code to fulfill these tasks. Our agent aim is to empower a compliance team in accelerating the adoption and operation of new regulatory programs by au- tomating the generation of code for the evidence collection and for its posture validation against the requirements, based on compliance requirements described in natural language. Our benchmarking experiments cover the end-to-end agen- tic workflow from the discovery of the policy assessment engine in the benchmark scenario, the generation of assess- ment policy as code and its real-time git PR management, deployment, execution, and posture generation. Finally, the results are evaluated, rated, and reported in our ITBench solution leaderboard.
The overall process starts with CISOs, security administra- tors, regulators establishing and authoring the relevant body of compliance recommendations, typically in natural language, for specific mission critical environments. Then they rely on dev teams or security focals to collect status as evidence across those environments, and to validate it against the recommendations in view of obtaining the autho- rization operate other certifications. The sought after benefit from automating the evidence collection and its val- idation is to enable scalability, both in handling complex environments and supporting frequent scans -daily or on demand per system fix or update- for posture measurement and reporting. Examples of validation automation tools are policy engines such as Kyverno (Int, e), OPA Gatekeeper (Int, b) for Kubernetes, Ansible (Int, a) for PaaS, or Cloud Security Posture Management (CSPM) solutions for the cloud. Figure 22 illustrates the dichotomy of compliance authoring on the left versus compliance validation via pol- icy scripts and their diverse programmatic languages on the right.
D.2.1. TERMS AND NOTATIONS
We define the following key terms used hereafter to describe the main aspects of the agent framework and benchmarking methodology:
1. Agent: An agent is an AI driven software that au- tonomously acts on behalf of a persona to solve a given task. We group the agents by Agent Types that reflect the IT operations personas, for example CISO, FlnOps, or SRE type.
Automating the translation of natural- recommen- dations into policy scripts requires an unprecedented level of trust and synchronization across domains and experts typically in different business units. Additionally, it also demands an unprecedented level of technical knowledge for the compliance teams typically focused on legal and IP matters.
2. Task: A task is a specific job corresponding to the role of a persona that the agents aim to automate. Typi- cal tasks for CISO are to collect evidence and assess compliance controls posture.
The rising popularity of Al agents and their projected ability to handle intricate tasks have increased the demand for Al agents managing IT systems (John, 2024; Miguel Carreon,
3. Scenario: A scenario is a real-life occurrence of a task in a given setting. For CISO, for instance, each Kubernetes CIS-benchmark requirement instantiated
42

---

ITBench
Regultr defines compliance requirements, procedures, and specifications
System administrators
PVP for infrastructures
Compliance Requirement
Policy1
PVP1
Result1
Policy2
PVP for platforms
Iterate every
PVP2
Result2
update
1AM
Policy3
Compliance Result
Result3
PVP3
PVP for services
CISO/Auditor review compliance results, write reports and certificates, and/or take remediation actions
Compliance Administration
Policy Administration
Figure 22: Compliance Authoring and Administration vs. Policy Validation Point Engines
OPA unique scenario. The scenarios can be grouped in classes.
of the system. The agents first step towards moving the environment into the goal state is by reasoning over the outcomes of several observational actions to determine the next optimal step. With strong hypothesis for what that is, agents seek to find and execute strategies to move towards the goal state in the most efficient way.
4. Scenario Class: A scenario class is class of real- life scenarios that are grouped together expecting the same behavior and outcome from the corresponding persona. Examples of scenarios classes are the sel Kubernetes EIS-benchmarks on OPA engine, the set of RHEL9 CIS CIS-benchmarks Ansible engine, or set of Kubernetes CIS US--benchmarks updates on Kyverno engine.
Our CISO compliance assessment tasks are coupled with pre-defined scenarios for assessing the effectiveness of the agents delivering the automation in standard manner. We detail in the next section these pre-defined CISO tasks before detailing our ITBench and CISO scenarios.
5. Scenario Environment: A scenario environment is the part of the scenario that specifies the deployment set- tings.
D.2.2. CISO COMPLIANCE ASSESSMENT TASKS
The compliance assessment tasks include various activities aimed comparing the actual state of the systems with the desired state described in English in the policy. Based this comparison, the system provides "pass" or "fail" posture with respect to the policy.
6. Environment State: An environment has countable set of states that consider to mark particular condition at specific time. Example of states are an environment initial deployment state, an environment failure state after fault or non-compliant configuration injection, or an environment remediated or compliant state after mitigation.
Identify Evidence Collector (IEC): Acquiring ev- idence requires selecting collection mechanisms ap- propriate the target system's characteristics. For instance, evidence about the state of an application in a Kubernetes cluster necessitates access to the Kuber- netes API, often through tools like "kubectl" command. For host configuration evidence, tools like Ansible Playbooks are suitable. This IEC task and associated agent or tools identifies the collector used the envi- ronment in view of generating the script for its corre-
7. Goal: A goal is the desired state for the environment known as the goal state.
Agents are tasked to transition environments from their initial state to their goal state the most efficient manner. At their disposal are environment actions, including requests for observations actuation attempts to affect the state
43

---

ITBench
sponding language and interface.
(CSF) (NIS) for CISOS and security teams to improve their organization's cybersecurity follows: (1) Identify critical data, systems, assets, and capabilities to protect; (2) Protect via security measures that limit the impact of incidents; (3) Develop a strategy to detect non-compliance with clear procedures and tools; (4) Respond via plans to quickly eliminate threats and mitigate damage; (5) Design recovery policy to support timely recovery to normal operations. Our first ITBench release covers the CISO's CSF first three activities by evaluating the following com- pliance tasks: (1) Identify and collect evidence from the systems in the selected scenarios; (2) Implement the poli- cies recommended in the scenarios; (3) Assess the policies posture to detect failure into non-compliance. The remain- ing two CISO's CSF activities of respond and recover will make the topic of future ITBench release that will sup- port agents collaboration, namely the leverage of the SRE agent for mitigation.
Identity Policy Assessment Tool (IPA): Evaluating evidence against policies requires selecting suitable policy engine. For general scenarios, the industry is using the open source policy engine Open Policy Agent (OPA) (Int, d) with its specific programmatic language Rego. Alternatively, for Kubernetes-ype specific configu- rations, Kyverno policies prevalently used along OPA. This IPA task and associated agent or tools identifies the appropriate policy engine for the scenario's policy at hand, in view of generating policies code accord- ing to the policy engine's 's programmatic language and interface.
Collect Evidence (CE): This CE task and associated agent or tools is responsible for the actual evidence col- lection, including the generation of code, management of code, deployment and execution of the evidence collection code acquire the actual evidence from the environment. Proper placement and execution of the code are necessary to achieve this in reliable and scalable manner.
Mimic CISO real-world problems. We studied and used in ITBench the real-world Cloud Internet Security (CIS) Benchmarks (CIS, 2024) which are a set of best practices for securing the IT cloud infrastructure. They are rec- ognized worldwide as the cloud security standards. We used those TIS-benchmarks to create CISO compli- ance policies scenarios of various complexity levels: 25% Easy, 50% Medium, and 25% Hard policies (see Figure 24).
Scan Assessment Posture (SAP): This SAP task and associated agent or tools is responsible for generat- ing the posture whether the evidence does or does not satisfy the scenario CIS-benchmark requirement. It includes the generation of validation code, manage- ment of code, deployment and execution of code the policy engine to assess the evidence and produce the compliance posture.
Provide observability. Cloud Native Compute Founda- tion (CNCF) recent Sandbox project (OSCAL-compass,
2024) released a compliance as code SDK to support the machine readable compliance as code standard (OSCAL, 2024) of the National Institute of Standards and Technol- ogy for programmatic usage in compliance automation. IT'Bench CISO automation leverages this methodology to represent the EIS-benchmarks requirements, detect the events of creation or update of requirements, and trigger the creation or update of evidence collection and valida- tion code.
Table 2 summarizes the CISO tasks initially supported in our ITBench, namely CE and SAP. The other will be cov- ered in subsequent releases. These tasks are executed in IT- Bench against predefined, standard scenarios and compared to the ground truth expected assessment posture "pass"/" fail" stored in the ITBench under each scenario environment spec- ification.
Ensure Determinism. ITBench enforces the scenarios and their environments are generated as per the specifica- tion, while the environment cleanup after each scenario ensures a clean slate for the next run.
D.3. ITBench Architecture for handling CISO Tasks
ITBench uses open source technologies to create repeatable and reproducible scenarios and environments for the CISO tasks, on a Kubernetes cluster as shown in Figure 23.
D.3.2. ITBENCH ARCHITECTURE
The environments for the benchmarking scenarios comprise Kubernetes Cluster and virtual machines (VMs). Each CISO scenario pre-defined in the ITBench described in the section hearafter, is managed by a deployable stack, a software component responsible for handling the bench- marking process and environment in a real run-time envi- ronment.
1. PRINCIPLES
Following the bench principles indicated in the introduction, ITBench uses open-source technologies construct completely repeatable and reproducible scenarios that simu-
late real-world incidents.
Mimic CISO best practices. ITBench follows the guidelines outlined by the National Institute of Stan- dards and Technology (NIST) Cybersecurity Framework
The deployable stack manages various tasks, including preparation cedeply__evironnane fault injection ("in-
44

---

ITBench
1.Register
Task Scenario Manager
Cradentials
Agent
3. Agent Finish
Task Senarios
Scenarios
Scenarios
(New-K3s-CISS-b
(NeN-RHELL-CII-b AnaibleOPAA
Agent Working
OPARego)
Scenario Executor
Install Scenario
Evaluate
Scenarios (New-RHELL-CIS-
Scenarios (Update Kes--------------
Hemove Scanario
RHEL9 VM
(Environment)
Ansible-OPA)
Kyvern)
Ansible
OPA
Kyverno
Kubecti
Kubernetes Cluster
(Environment)
Figure 23: Architecture of ITBench responsible for orchestrating CISO scenarios.
ject fault")" agent performance evaluation ("evaluate") and environment cleanup ("delete_envinee "eneeote_niir for each benchmark scenario run. Each deployable stack is specifi- cally designed for particular CISO compliance assessment scenario, ensuring the necessary tools, configurations, and policies are in place. Environment administrators define these scenarios deployable stacks and configure the required software, which can involve setting up policy engines, tools, creating conditions that simulate violations of specific compliance requirements.
Hard
20.0%
Easy 20.0%
Medium
Figure 24: CISO scenario complexity.
marking, the scenario environment, including Kubernetes Clusters and VMs, is prepared by the Agent Submitter. The setup and metric measurements are automated using a tool called Mini-Bench. Benchmarking results, including Task Metrics, are registered with central Bench Server via API. These results are displayed the Leaderboard the Bench Server. This setup allows benchmarking to proceed uni- formly, whether using the Agent Submitter's 's local environ- ment or a remote environment, as the same API interactions are employed in both cases.
A deployable stack may for instance intentionally exhibit misconfiguration settings to mimic a violation partic- ular compliance standard. the execution of a scenario, agent is presented initially with the natural language description of that scenario compliance requirement. Based the description, the agent generates the necessary arti- facts, including scripts for evidence collection and policies for evidence evaluation. The run-time scenario environment is deployed and made accessible to the agent. The agent ac- cesses the environment to retrieve evidence, deploy policies, and work toward achieving the specified automation goal of assessing the compliance posture, in this case as "fail" or "not-satisfied".
D.4. ITBench Real-World CISO Scenarios
We used in ITBennch the real-world Cloud Internet Security (CIS) Benchmarks (CIS, 2024) standard to create our CISO compliance assessment scenarios. The technologies that we have considered as playground CISO agent are Kubernetes and RHEL9, however, any other technology available in CIS can be leveraged with their CIS-benchmart infusing the policies scenario in ITBench.
During the scenario process, the agent notifies the IT- Bench of the start and completion of its tasks. Upon receiv- ing the completion notification, the ITBench accesses the environment to measure the benchmarking metrics. Once the metries for all predefined scenarios are collected, they are aggregated and displayed on the ITBench Leaderboard. Fig. 25 illustrates the end-to-end benchmarking process for the CISO scenarios.
benchmarking our
Table 22 and Table 23 illustrate examples of the typical CIS-benchnn be-benhhmarks recommendations, Each scenario rendered on ITBench is designed to mirror the complexity of the recommendation. This ensures that ITBBench replicates real-
In the context of our CISO compliance assessment bench-
45

---

ITBench
CISO Leaderboard
For every agent, do steps 1-8 for all relevant scenarios
Update leaderboard
Pick scenario
5. Kick agent Y with report
CISO Scenarios
IT Bench
CISO Agent
Deploy scenario X
6. Evaluate (success/fail, MTTR, etc.)
3. Inject fault (issue)
7. Clear deployed scenario
Find report
App
Reporter
Kubernetes Cluster (kind / OCP)
Host (RHEL9)
Figure 25: CISO Compliance Assessment Agent end-to-end Benchmarking Process.
world compliance requirements thus allowing for realistic evaluation of agents.
IPA: Evaluated by confirming that the policy suc- cessfully generated for Kyverno.
D.5. CISO Scenario Classes and their Complexity
CE: Measured by verifying whether a Kyverno report generated in the predefined location.
D.5.1.
KYVERNO
SAP: Determined by whether the posture reported in
New-KSs-CIS-b-Kyyern represents the Easy scenario class in the ITBench. The 10 scenarios in this class are prepared based the CIS Benchmark for Kubernetes, specifically focusing the Pod Security Policy. This scenario assumes Kubernetes cluster with pre-configured Kyverno policy engine. Within the cluster, certain misconfigurations related to Pod Security Policy are present, but the Agent unaware of their exact locations.
the Kyverno report matches the expected value.
D.5.2. NEW-KSS-CIS---OPAREE
is categorized under Medium complexity scenarios. This benchmark comprises 10 scenarios derived from the CIS Benchmark for Kuber- netes, specifically focusing on Pod Security Policies.
The requirements for the misconfigurations that need to be addressed are communicated to the Agent. Based on these requirements, the Agent generates a Kyverno policy and deploys it to the cluster. Subsequently, the Agent collects the report from the cluster. The accuracy of this report is verified by checking whether successfully identifies the misconfigurations. If the policy correctly generated and deployed, the report should indicate the appropriate posture. Conversely, if errors occur, incorrect posture will be reported.
The foundational background for this scenario closely re- sembles that of Wer---&--IS---Kyeeerre sumption of a Kubernetes cluster as the operating environ- ment and CIS Benchmark for Kubernetes. The key differ- ence lies in the dual output for Open Policy Agent (OPA) policy engine: the generation of both an evidence fetcher script and a policy checker code.
sharing the
A fetcher script is designed to gather the required ev- idence from the target cluster by executing kubectl commands.
In this scenario, the four Compliance Assessment Task are evaluated as follows:
A policy checker verifies the collected evidence for compliance based on predefined rules. This is im- plemented using the Open Policy Agent (OPA) and defined through OPA Rego policies.
IEC: Assessed by verifying whether the correct config- uration is reflected in the Kyverno policy.
46

---

ITBench
Table 22: Kubernetes Center for Internet Security Benchmarks (sample)
Section #
Recommendation #
Title
Assessment Description
Status
1.1.1
L1L11
Ensure cramfs kernel module is not available
Automated
The 'cramfss filesystem type is a compressed read-only Linux filesystem embedded in small footprint systems. A 'cramfs' image be used without having to first decom- press the image.
1.1.1
1.1112
Ensure freevxfs kernel module is not available
Automated
The 'freevxfs' filesystem type a free version of the Veri- tas type filesystem. This is the primary filesystem type for HIP-UX operating systems.
L.I1
1.1.1
Ensure hiss kernel mod- ule not available Ensure hisplus kernel module is not available
Automated
The 'hfs' filesystem type is hierarchical filesystem that allows you to mount The 'hfsplus' tilesystem type is a hierarchical filesystem designed to replace 'hfs' that allows you to mount Mac OS filesystems.
filesystems.
1.11
1.114
Automated
L.I.
1.1.1.5
Ensure jffs2 kernel mod- ule not available Ensure ush-storage ker- nel module is not avail- able
Automated
The jffk22' (journaling flash filesystem 2) filesystem type a log-structured filesystem used in flash memory devices. USB storage provides means to transfer and store files en- suring persistence and availability of the files independent of network connection status. Its popularity and utility has led to USB-based malware being simple and common means for network infiltration and first stop to establish- ing persistent threat within a networked environment. Filesystem kernel modules be dynamically loaded into the Linux kernel extend its filesystem capability, or so-called base kernel, an operating system. Filesystem kernel modules are typically used to add support for new hardware (as device drivers), or for adding system calls.
1.1.1
1,1.1.8
Automated
1.1.1
11119
Ensure unused filesys- tems kernel modules are not available
Manual
picces of code that
The goal of the agent this scenario to generate two out- puts: 1) a script executing kubectl commands (fetcher), 2) an OPA Rego policy for compliance verification (checker).
This approach provides structured evaluation of the agent's capability to generate effective scripts and policies for Ku- bernetes cluster compliance assessment.
The verification process evaluates the outputs generated by the agent as follows: The fetcher script, which consists of kubectl commands, is executed against real Kubernetes cluster to collect evidence. The collected evidence then assessed using the generated OPA Rego policy and the OPA policy engine verify whether the results align with ex- pected compliance outcomes.
D.5.3. NEW.RHHELL-CISS.AAAAIIIL
New-RHELOP CINEL9--I----Ansiblle b-Ansible---b.A. belongs to the Medium complexity scenario class and comprises 20 scenarios based on the CIS benchmark for RHEL9 OS. This sce- nario shares common characteristics with New-K&s-CIS-b- Kubectl-OPAReego including the generation of two codes (a fetcher script and checker policy), and the of OPA Rego Policies for compliance verification. The primary distinction lies in the target system: unlike New-K8s-CIS-a Kubectl-OPARee
The scenario is assessed on the following four Compliance Assessment Tasks:
which focuses on Kubernetes clusters, this scenario targets RHEL9 hosts. Consequently, instead of using kubectl as the fetcher script, New-RHEL9-CIS Ansible-OPe employs Ansible playbooks. The objective of this scenario generate Ansible playbooks fetcher scripts and OPA Rego Policies checkers.
IEC: Determine whether fetcher script, incorporating kubectl commands, is successfully generated.
IPA: Verify if the checker, implemented as an OPA
Rego policy, is correctly generated.
CE: Check whether evidence can be successfully col- lected by executing the fetcher script against the clus- ter.
The verification process evaluates the outputs generated by the agent as follows: the fetcher script (Ansible playbook) is executed against a real RHEL9 host to collect evidence. The collected evidence is subsequently analyzed using the gener- ated OPA Rego policy on the OPA policy engine, assessing whether the results align with the expected compliance
SAP: Verify that the OPA Rego policy evaluates the collected evidence as expected, producing the correct compliance assessment.
47

---

ITBench
Table 23: Red Hat Enterprise Linux Center for Internet Security Benchmarks (sample).
Section #
Recommendation
Title
Assessment Description
Status
1.1.2
1.1.2.3
Ensure noexec option set on /tmp partition Ensure nosuid option set / tmp partition Configure /var
Automated
The 'noexec' mount option specifies the filesystem cannot contain executable binaries. The 'nosuid" mount option specifies that the filesystem can- not contain 'sctuid' files.
1.1.2
1.1.2.4
Automated
1.1.3
'/var' directory used by daemons other system services to temporarily store dynamic data. Some directories created by these processes may be world-writable. The 'nodev' mount option specifies that the filesystem can- contain special devices.
1.1.3
1.1.3.2
Ensure nodev option set /var partition Ensure nosuid option set on /var partition Configure /var/tmp
Automated
1.13
1.1.3.3
Automated
'nosuid' mount option specifies that filesystem can- not contain 'setuid' files.
1.1.4
The '/var/tmp' directory is world-writable directory used for temporary storage by all users and some applications. Temporary files residing '/var/tmp' are be preserved between reboots.
1.1.4
1.1.422
Ensure noexec option set on /var/tmp par- tition
Automated
The 'noexec' mount option specifies that the filesystem cannot contain executable binaries.
1.1.4.3
Ensure nosuid option set on /var/tmp par- tition
Automated
'nosuid' mount option specifies that filesystem can- not contain 'setuid' files.
1.1.4
1.1.4.4
Ensure nodev option set /var/ tmp partition Configure /var/ /log
Automated
The 'nodev" mount option specifies that the filesystem can- contain special devices.
1.1.5
The '/var/log' directory used by system services to store log data.
11.5
1.1.22
Ensure nodev option set //var/voog partition Ensure noexec option set on /var/log par- tition
Automated Automated
The 'nodev' mount option specifies that the filesystem can- not contain special devices. The 'noexec' mount option specifies the filesystem cannot contain executable binarics.
1.1.5
1.1.5.3
1.1.5
1.1.5.4
Ensure nosuid option on /var/log par-
Automated
The 'nosuid' mount option specifies that the filesystem contain 'sctuid" files.
tition
1.1.6
Configure
The auditing dacmon, 'anditd', stores log data in the '/var/log/audit" directory. The 'noexec' mount option specifies that the filesystem cannot contain executable binaries.
/var/ / audit
1.1.6
1.1.622
Ensure
Automated on
option set
/var/log/aws audit
partition
1.1.6
1.1.6.3
Ensure nodev
Automated
The 'nodev" mount option specifies that the filesystem not contain special devices.
option
/var/log/aun
partition
1.1.6
1.1.6.4
Ensure
nosuid
Automated
The 'nosuid' mount option specifies that the filesystem can- not contain 'setuid' files.
option
/var/log/ audit
partition
1.1.7
Configure /home
Please note that home dircctories could be mounted any- where and are necessarily restricted '/home", nor re- stricted single location, way. Checks can made by looking 'Jeccppasswd' look- ing over the mounted file systems with 'mount' or querying the relevant database with 'getent'.
the restricted in any
48

---

ITBench
comes.
and Granite-3. 1-8-iinsttuct for tasks that rely on natural language understanding and reasoning. For code-focused use cases, we additionally utilize GPT-4o-mini, Llama-3.1 1- 405b-instruct, and Saarral-xxb-isstruc. All models use context window of 128K tokens, enabling them to process more extensive input sequences.
In this scenario, the agent's performance is evaluated on the following four Compliance Assessment Task:
IEC: Does the agent generate a fetcher script in the form of an Ansible playbook?
Does the agent generate an OPA Rego policy for the checker?
IPA:
D.6.1. EVALUATION METRICS
The efficacy of our CISO agents is measured based on the ability to detect artifact misconfigurations (aka non- compliance, e.g., minimum count of containers sharing namespace, or the count is above the threshold), confirm proper configurations (aka compliance), within the varied environments of the scenario classes randomly injected with misconfigurations.
Can the Ansible playbook successfully execute against an RHEL9 host and collect relevant evidence?
CE:
SAP: Can the generated OPA Rego policy evaluate the collected evidence and produce the expected compli- ance results?
We evaluate how effectively the agent detects the (non)compliance using the following metrics:
This scenario is designed to assess the agent's performance compliance evaluation tasks for host management envi- ronments other than Kubernetes, specifically focusing on RHEL9 systems.
Success rate is quantified using pass 1.
Efficiency captured through Time to Process (TTP).
D.5.4. UPDATE- BAAANE-KKS-OEE ABS-CIS-BB BELISB-KYYVIE KYVERNO
D.6.2. METRIC DEFINITIONS
falls under the scenario class with a complexity level classified as Hard and currently includes 10 scenarios. Unlike New-K&s-CIS---Kyeemo,, which generates new Kyverno policies based on new re- quirements specified for that environment in the goal, this scenario involves a different objective. Specifically, it takes an existing Kyverno policy input, along with instructions detailing modifications to the original requirements, and generates an updated policy to meets the revised require- ments. The updated policy is then deployed (or updated) as the final output.
Update-K8s-CISS
pass We evaluate the agent proper assessment of the posture "pass"/"fail" using the pass@ metric (Chen et al., 2021), which is defined as follows:
pass @ k Eeenarios
It an unbiased estimator of correctness in k= trials across all scenarios.
Time to Process. We identify the scenarios where agent identifies the posture successfully (i.e., misconfiguration results in a "fail" posture, while compliant configuraiton results in a "pass" posture). For these task scenarios, we cal- culate TTP. which measures how soon an agent performed the assessment. Otherwise, TTP is set infinite.
The validation process for this scenario is consistent with the methodology used in New-K&s-CIS C-S----KV--eKooo..
D.6. CISO ITBench Evaluation
We conduct our experiments primarily on AWS EC2 in- stances (m4.xlarge), although ITBBench also be readily deployed on consumer-grade laptop using pscudo-cluster, thus making it easier to develop AI agents.
D.6.3. EVALUATION RESULTS
Overall agent results.
Overall our results in Table 5 and Figure 28 show the GPT- based models dominate on both pass @ I and Time to Process metrics. The pass @ 1 is nearly 2x better than second-bes models (alternating between llama-3. 1lama-3.1-405b-ine 1-405b-insttrut and mistral large-,, while the TTP shows a handling of the scenarios in the minimal time across our scenario classes.
We measure the efficacy of our CISO compliance assess- ment agent on a set of 50 scenarios across the four scenario classes introduced in Table Each scenario class imposes distinct set of CIS-benchmarks requirements (c.g., "mini- mize the admission of containers wishing to share the host network namespace"), each class has specific level of com- plexity (e.g., Easy, Medium, Hard), and generates scenario- specific code artifacts.
Impact of Scenario Complexity
The complexity of the CISO scenarios is directly mapped to scenario classes. For example, Kyverno scenarios are of Easy complexity, k8s-opa and rhel-opa are of Medium complexity, while Kyverno-update scenarios are of Hard
In our evaluation we considered a variety of LLMs, such as GPT-4o, Llama-3.3- 300 70B-instruet,, Llama-3.1 1-8--instruct,
49

---

ITBench
complexity. As expected, all the models struggle, as pected, the difficulty of the scenarios increases from the Easy kyverno class to the Hard kyverno-upadate class.
The task requires checking, Rego policy for given input file. The policy and input file need to be prepared first. The goal is restricted to authorized
Thought:
D.6.4. CISO COMPLIANCE ASSESSMENT AGENT TRAJECTORIES
RHEL host. To begin, Ansible playbook needed collect necessary configuration from the RHEL host.
We examine two sample trajectories from our experiments highlight the agent's is thought patterns and actions. These include scenarios, one successful and one unsuccess- ful, derived from the NEW-CIS- B- AAEELL- experiments with LLaMa 405B, focusing the CIS Bench- mark for RHEL9 (control section 5.1.8 and 5.2.19).
Action: using tool
GenenttePlabookoToll
Input: ("sentence";
"Collect cron allow and deny configuration from RHEL host...
Observation:
name:
Collect Cron Allow and Deny Configuration
In the successful scenario, the playbook is successfully gen- crated, tested independently, leading to successful data col- lection, followed by the successful creation of OPA, and ultimately detecting issues through evaluation in the bench.
content: "If collected datal' "content]] b64decode quote }) cron deny. quote }3" dest: collected data.json
In the unsuccessful scenario, the playbook is successfully generated and tested independently, leading to successful data collection, but the OPA cannot be properly created.
Thought: New that have the playbook generuted, need to run it to collect the configuration from the RHEL host.
This results in repeated syntax error corrections and retries, eventually reaching the maximum retry limit.
Action: using tool
RunPlaybookTool
Input: ("playbook :ile":"/ /agent/28250127771428 htt.50127777eee//paaabooooooo
Following this, we present the trajectories for both cases. In the successful scenario the agent is tasked with performing compliance assessment based on the following request:
Observation:
PLAY (Collect Cron Allow and Deny]
rhe19
o
changed-4
Ensure is restricted authorized users. The sys- tem must: Allow only authorized users schedule jobs using cron. Specific requirements:
unreachable-0
failed=0
skipped-0
rescued=0
ignored=1
The playbook has been nun successfully, and the configuration has been collected. Next, we need to generate an OPA Rego policy to check if the collected configuration meets the requirements.
Thought:
The file /etc/cron.allow must exist and specify the list of authorized users.
The file /etc/cron.deny must not exist.
Action: using tool
GenerateOP PineraateOPARggull Rog
Unauthorized users (e.g., userl, user2) must not be listed in /etc/cron.allow.
Input: ("sentence":
"Ensure cron is restricted to authorized "input file":" "/ttm///entt///
20250127111481/c01eeee data. .jon"...o"....
To check the condition, do the following steps. collect configuration from the RHEL host and save it locally, you can use ansible- ansible-playbook playbook to do that. cheek if the condition met by using rego policy with the input given by the step above.
Observation:
The generated policy is below:
package check
not contains(input, "rool")
for those steps, you need to create ansible playbook "playbook.yml and OPA rego policy 'policy.rego'. If you can fix the generated code, do it and run the fixed code again. Once you get a final answer, you can quit the work.
This policy file has been saved at tump'agemt22250127711 401 /policccy
Thought: Now that we have the Rego policy, we can use it to check the collected configuration.
Report filepaths that created in the previous task.
The agent runs this task, provided with the target environ- ment credentials and a working directory for saving the generated files to submitted. The Figure shows the trajectory that the agent takes achieve this request.
Figure 26: Sample Trajectory of LLaMa 405B in Success Scenario
In the next scenario (CIS Benchmark control section 5.2.19).
50

---

ITBench
the agent is given different compliance requirement. This time, the focus is ensuring that SSH login sessions adhere to security best practices. However, unlike the previous scenario, the agent encounters a critical issue: it fails to correctly write a syntactically valid Rego policy, causing repeated OPA errors during execution.
First, need check input ready. The input file should contain SSH configuration from RHEL nol, need create an Ansible playbook
Thought:
The compliance requirement for Scenario 5.2.19 is fol- lows:
the
Ensure SSH LoginGraceTTime is set one minute or less
configuration and run
Action: using tool
GeneratePlaybookTood
To check the condition, do the following steps. collect configuration from the RHEL host and save it locally. you can use ansible-playbook to do that. cheek if the condition is met by using rego policy with the input given by the step above.
Input: {"sentence":
"Collect configuration host", "playbook file":
...Sucessfuully generate playbook...
Thought: Now that have the configuration collected, I need generate the OPA Rego policy to check if the SSH LoginGraccTins is set minute less.
for those steps, you need create ansible playbook "playbook.ymi and OPA rego policy 'policy.rego'. If you fix the generated code, do and run the fixed code again. Once you get final answer, you can quit the work.
Action: using tool
GencratoOPARogoTooo
Input: ("sentence":
"Check 1f LoginGraceTine minute less",
The agent run this task, provided with the target environment credentials and a working directory for saving the generated files be submitted. The Figure 27 show the trajectory that the agent takes achieve this request.
Observation:
The generated policy is below:
result := false if
contains(input, "LoginGraceTTme Im") and not contains(input, "LoginGraveTine 0m!")
After repeating this process three times, the maximum retry limit was reached, and only the currently saved play- yml and policy.rego were used at the evaluation. At the evaluation, this rego is syntactically wrong so the evaluation did not pass.
Thought: Now that have OPA Rego policy and input ready, can run the policy check.
Action: using lool
RunoPARegoTool
Input: ("policy file":
/
D.6.5. CISO COMPLIANCE ASSESSMENT AGENT DOURCE
Observation:
We invite reader get more detailed introduction to our CISO compliance assessment agent deployment and execution in its open-source git repo documentation (CIS, a). The CISO agent comes with a sample of scenarios and their environment executable automation packages, available as well in open-source (CIS, b).
encountered error while trying This was error: failed "opa command; error details:
tool.
STDOUT:
not contains(input, "LoginGrace Time Im and not contains(input, "LoginGrace Om")"... ,...
ReReating the same failures...
After maximum retry limit, report the currently filepaths that created in the previous task.
Figure 27: Sample Trajectory LLaMa 405B in Failure Scenario
51

---

ITBench
kyverno-1
kyverno-2
kyverno33
kyvemo-1
lyverno-5
100%
Assessed 1%6
8019-
kyverno-6
kyverno-7
kyverno-8
kyverno-9
kyverno-10
1000%
kBs ape
käs opa 2
kBs apo
k8s opa
kBs ope 5
2016
0%
kBs-apa
opa-7
opa
opa-9
1000%
60%
429
2096
0%
rhel-opa-1
mel-opa-2
rhel-opa
rhe
10096
8095
80%
42%
2096
09
rhel-opa-6
mel-opa-7
thel-opa-8
mel-opa-- 9
mel-opa-10
100%
0016
Assessed
607-
ftellopa-11
rhel-ope-12
thel-opa-11 13
rhel-ope-11 14
rhel-opa 15
100%
0%
rhei ops
rhel opa 17
rhel-cpa ops 18
rhel ope
rhel upa 20
42%
20%
09
lyvemo-update-1
kyverno update
kyvemn-uudetee
lyveroo-uddae
kyvemo update
100%
80%
60%
43%
20%
0%
kyvema update-
kyverno
kyvern update
kyverno
100%
80%
80%
20%
2096
0%
mutrrl
misrrl
Heme 708
get see-4torne yerrs
de
pranite 90
micrral
mattre
Bareer 108
Bano 4t55
got Ao-mini
upt
pranite at
mistrel
lismta st0
rniarnll
flema T1B
Rumat 2-355 ant yairs gat-
prestee Bi
citrral
mustrll
lame-708
sanaz 4rito
ger- ao--oootit
mutrel
misrral
liena 7I8
foma 2-aitt
get- nini god
Figure 28: Percent pass@1 for each scenario.
52

---

ITBench
Table 24: Experimental details
Experiment Setup
CISO/Agent: Models
Scenarios
#Repeats
#Total
% Exp. Completion
% %Agent Submission
granite-3.--8-instree
50
8
400
91.41%
88.48%
mixtral-&x7B-instruuc
50
8
400
91.02%
94.67%
llama-3.1-8-instrue
50
8
400
93.21%
85.71%
Jlmma-33-70B-iistruce
50
8
400
92.82%
89.1 %
mistral-large-2
50
8
400
94.48%
85.23%
Hamaa331-00SS-nnsttuutt
50
400
92.43%
84.50%
gpt-4o-mini
95.25%
85.71%
gpt-4o
50
8
400
90.83%
90.54%
Note: "SExp. Completion" is the percentage of experiments that were started and finished by the bench runner correctly.
Note: "Trials Agent Submitted" is the percentage of all trials completed in which the agent returned results.
Table 25: CISO/Agent: Assessed Results
Scenario pass@1 (%) t
Models
Avg. pass@1 @ (%)
MPR (8)
kyverno
k8s-opa
rhel-opa
kyverno-upadate
-ramite-3.1-8B-instee
7.84 3.84
0.00 + 0.00
0.00 + 0.00
1.59 + 1.58
1.71 + 0.76
197.03 2.52
Mixxtral-xx7-innstrue
7.35 3.19
1.43 1 1.42
0.00 1 0.00
1.29 1 4.34
3.94 1: 1.03
120.63 3.77
llama-3.1-BB-instuuct
8.57
0.00 1 0.00
0.00 1 0.00
7.46 3.23
3.59 1: 1.07
121.49 3.00
1lama-3.3-70B-instruuct
18.46 4.94
0.00
3.50
9.32 1.67
189.61 2.71
mistral-large-2
6.56 3.220
22.73 1 5.32
7.23 2.88
10.45 3.77
11.55 1.95
167.98 3.42
llama-3.1-405B-instteee
16.22 - 4.32
20.83 + 4.86
8.75 + 3.26
3.17 2.22
12.46 + 1.98
178.89 3.37
gpt-4o-mini
16.18 = 4.54
6.99
30.38 5.43
9.43 4.08
25.19 + 2.80
102.40 3.70
gpt-4o
40.28 5.99
39.34 6.55
7.61 2.81
17.74 4.92
24.74 2.64
101 29
+ 3.81
"pass@1" 1" values in percent. @ is calculated defined in Codex (Chen al., 2021)
"MPR" mean processing time
kyverno= New K8s CIS-henchmarks on Kyvemo,, k8s-opa = New K8s CIS-benchmarks on OPA, rhel-opa = New RHEL9 CIS-henchmarks on Ansible-OPA, kyverno-update = Update K8s CIS-henchmarks on Kyvemoo
53

---

ITBench
E. Financial Operations
million dollars year in public cloud, 20% spending more than $100m/yr and over 20% spending $1 bn/yr. Similar to the observation made in the 2024 State of FinOps Report, workload optimization and waste reduction remained key priorities across the board. Management of cloud discount programs (such as Savings Plans and Reserved Instances), and accurate forecasting of spend remained high on the list. Other areas such increased Automation, Al costs, and sustainability registered growing interest. new observa- tion made this year was that Al spending is now managed by the majority of respondents (63% up from 31% % last year) and is expected to impact nearly all FinOps practitioners.
E.1. Background
FinOps (Finance + Operations) is an operational framework and cultural practice which maximizes the business value of cloud and creates shared financial accountability. One of the primary objective is enable timely data-driven deci- sion making by fostering collaboration between engineering, finance, and business teams. FinOps comprises of three it- erative phases Inform (Visibility & Allocation), Optimize (Rates Usage), and Operate (Continuous Improvement & Usage). Importantly, success in FinOps hinges on making iterative changes using real-time insights with data gathered from Application Performance Management (APM), Appli- cation Resource Management (ARM), and Finance (Cloud Cost Management) (Storment and Fuller, 2023).
E. E.I.1. I. KEY TERMS
FinOps performed by working iteratively on the Frame- work Capabilities through three phases (Foundation, 2025b) namely, Inform, Optimize and Operate, which descrihed
Contrary to common belief, FinOps is about maximizing business value and not just about reducing operating costs. The 2025 State of FinOps Report (Trask, 2025) gathered data from 861 respondents representing $69M in public cloud spend, with 31 % currently spending more than $50
below:
(1) The Inform phase involves identifying data sources for cloud cost, usage and efficiency data. This data is then used for budgeting, allocation, forecasting, analysis and reporting.
(2) The Optimize phase identifies opportunities to improve cloud efficiency using the data and capabilities devel- oped in the Inform Phase.
(3) The Operate phase implements operationalizes FinOps using the data and capabilities developed in the Inform and Optimize phase.
Some the common KPIs in the FinOps that are amenable optimization (full set is listed in http::/wwwww.nnoossoggggwg//////
include
(1) Percentage Resource Utilization This is the amount of resources utilized as percentage of the the total capacity allocated.
(2) Total Unpredicted Variance of Spend Measures the unpredicted variance of cost associated with CSP Cloud usage recorded over given period of time.
(3) Auto-scaling Efficiency Rate Maximum capacity cost of running workload to meet workload demand Cost of running workload with auto-scaling to meet same workload demand.
(4) Forecast Accuracy Rate (Usage) Compares forecasted vs. actual cloud usage (vCPUs, Memory, etc) over specific period (e.g., day, month, quarter).
(5) Forecast Accuracy Rate (Spend)- This metric compares forecasted vs. actual cloud spend over a specific period (e-g., day, month, quarter).
54

---

ITBench
(6) Percent of Compute Spend Covered by Commitment Discounts- Measures the percentage of compute cost (excluding Spot) covered by commitment discount for specific time period.
false alerts. Agent finds out that the application is healthy and recommends updating budget alerts based on the new load level. Similarly, Table 29 demonstrates a sample sce- nario where a budget overrun alert has been generated due to increased replicas. However, in this scenario application services are scaled up significantly despite low utilization in containers. Agent finds out autoscaler scaled up the applica- tion at low utilization thresholds and analyze the autoscaler configuration It detects low thresholds for scale up policy and recommends updating autoscaler accordingly.
(7) Percentage of Commitment Discount Waste The per- centage of commitments not applied to on-demand spend.
(8) Percent of Unused Resources- - Measure of unused cloud resources, e.g., unattached/orphaned storage volumes, load balancers, EIPS, Network gateways, snapshots.
FinOps practitioners need to analyze cloud bills as part of their daily tasks. However, existing reporting tools rely on predefined set of reports and analyzing data outside of these reports is not trivial. We defined new data insight generation scenarios based on the common KPIs listed in E. Each scenario starts with a natural language query and retrieves, processes, and summarizes the related data. Table 26 presents description of scenarios and the corre- sponding KPIs.
(9) Percentage of Unallocated Shared CSP Cloud Cost This measurement refers to expenses that cannot be directly attributed to specific project, team, or depart- ment within organization.
(10) Percentage Variance of Budgeted vs. Forecasted CSP Cloud Spend Measures the difference between bud- geted costs and the forecasted costs for using CSP cloud services
(11) Effective Savings Rate Percentage- Actual Spend with Discounts/ / Equivalent Spend at On Demand Rate
Anomaly detection and management (abbreviated as "AD") is a key FinOps capability that allows practitioners to con- tinuously monitor cloud spend and flag unexpected spend events immediately. The lifecycle of cloud cost anomaly detection includes cost monitoring, forecasting, identify- ing drift, outliers and anomalies, identifying root causes, and defining mitigation or resolution strategies. Under AD, we consider two FinOps scenarios anomaly detection and anomaly ranking. Due to the complex topology of FinOps business dimensions and tags that could be mapped a hierarchical structure, identifying the subset of dimensions where anomalies exist, their temporal impact, persistence and relevance to the user, an agentic solution is beneficial in utilizing the FinOps knowledge base and data to perform the task of anomaly detection and prioritizing them.
(12) Percentage of CSP Cloud Costs that are Tagging Policy Compliant - Total Costs Associated with Tagging Policy Compliant CSP Cloud Resources During a Period of Time / Total CSP Cloud Costs During a Period of Time.
(13) Percent Storage on Frequent Access Tier Number of GB in Standard (or "frequently accessed" tiers vs. total GBs stored)
(14) Percentage of Carbon Associated with Untagged CSP Cloud Resources
There are several analogous KPIs related to carbon foot- prints instead of dollar costs. In the current version of the bench our scenarios have used an alert based on variance in spend. The remaining KPIs offer a rich basis for formulating many additional scenarios.
The anomaly detection scenario takes in a natural language query to first gather the relevant data from a database, sub- ject to a time period of interest, and provided values for other relevant dimensions. From here, the data may need to be aggregated (e.g.: hourly or daily) according to the query. Next, an anomaly detection tool is employed to identify anomalies. The output of the tool is summarized by the agent and provided as the final L.M output.
E.2. Motivating Example and FinOps Scenarios
We have analyzed common FinOps scenarios in three main categories; incidents with cost alerts, data insights gener- ation using natural language queries, and cost anomalies in cloud bills. Incident scenarios are triggered by a cost alert and require diagnosis of the root cause of the alert and remediation steps to clear the alert. Table 28 shows an exemplar budget overrun alert-driven incident with the steps for diagnosis and resolution. In this incident, FinOps practi- tioners were notified of an unusuul cost increase (>20% more than last week's 's average)- alert by OpenCost. The increase is mainly caused by the increase in replica counts as a result of an observed load spike. The autoscaler increased the number of replicas to serve the demand as expected. However, budget thresholds are not updated which causes
Similarly, in the case of anomaly ranking, a user-specified scoring of anomalies is provided as part of the natural lan- guage query. Example scoring mechanisms include nor- malized product of anomalous cost and the duration of the anomaly. In this case, the agent is expected to gather the relevant data from the database, perform aggregation, pass it to the anomaly detection tool, compute an importance score for every identified anomaly and rank order anomalies as per the criteria specified in the query. Since the agent does not accurately identify all anomalies (that is, precision <
55

---

ITBench
Table 26: FinOps Data Insight Scenarios
Scenario
Natural Language Prompt
Related KPI
Scenario 1: Cost by re- source type
need summary report on the total cost for services compute and storage categories between 09-01-2024 and 09-30-2024. Please calculate the total cost for each category.
Workload Optimization and waste reduction
Scenario 2: Highest Cost Contributors
want to find out which applications contribute the most between 09-01- 2024 and 09-30-2024. Give me the list of top applications with highest cost.
Workload Optimization and waste reduction
Ignore costs without an application information.
Scenario 3: Cost Allo- cation Performance
What is the percentage of allocated/total cost ratio per cloud provider for September 2024. Consider cost allocated if there is an application attribution. Ignore records with negative costs for the calculation.
Full Allocation of spending
Scenario 4: Peak/Aver- age Cost Ratio
List top 5 applications with highest peak to average ratio in terms of daily total cost during September 2024, For cost calculation do not include negative costs. Calculate maximum daily cost to average daily cost ratio per application and give the list with application, peak cost, average cost, ratio, and peak cost date details.
Workload Optimization and waste reduction
Scenario 5: Cost Vari- ance
Find the top applications with highest daily cost variance hetween 09-01-2022 and 09-08-2024. Use population variance for ranking. Do not include records with negative costs.
Workload Optimization and waste reduction
Scenario 6: Commit- ment Discount Cover- age
Cloud providers regularly adjust bills by providing credits the accounts after the initial charges. Calculate the discount coverage each sub account including tax. Find five sub accounts which have total bill grater than 0 and have the maximum discount percentage. Return sub account information and discount coverage pereentage.
Rate Optimization
1.0 and recall < 1.0), comparing ranking performed by the agent the ground truth ranking, using techniques such as Normalized Discounted Cumulative Gain (NDCG) is not viable. Hence, we rely on comparing the importance score computed by the agent to the ground importance score, iden- tifying the relative rank of an anomaly and computing the fraction of the anomalies with the correct relative rank. This called the "rank score". Table 27 presents description of scenarios and the corresponding KPIs.
For data insights and anomaly detection use cases, we have leveraged sample FinOps data from FinOps Foundation (foc) in FOCUS format. Dataset includes more than 5 mil- lion cost records of 107 accounts for the services of AWS, Microsoft Azure, Oracle, and GCP. We provide natural lan- guage to SQL tool to retrieve the data from the database. The provided tool is responsible for linting the SQL code generated by the agent and if the code syntactically cor- rect, it executes the SQL query and return query results to the agent. Similarly, for anomaly detection scenarios, agent is responsible to generate the proper SQL code to retrieve the related data based on given natural language prompt. Then, the agent is responsible to execute anomaly detection algorithms on the retrieved data and summarize the results.
E.3. ITBench Architecture for Constructing FinOps Scenarios
For alert-driven scenarios, we have extensively leveraged the set up that we established for the SRE scenarios. We employ OpenCost to monitor costs and raise an alert when the predefined budget and efficiency thresholds are crossed. OpenCost is an opensource tool which distributes the cost of a virtual machine/node to the Kubernetes deployments running on it based on the allocated resources. It selects higher of utilization or request numbers of each container and distributes the cost using a load distribution policy. In our experiments, we have forced custom pricing model which mainly includes hourly CPU cost rate for single core and memory cost rate per GB memory. Other pricing components such as networking cost and spot instance pric- ing are not the main scope of our evaluated scenarios. Thus, we have included negligible costs for these components. We mimic real world scenarios as explained in SRE scenarios by including cost variation alerts.
E.4. Evaluation
We have evaluated agents for each FinOps scenario category independently. For alert driven incident scenarios, we have used the same evaluation framework used in SRE-bench by including OpenCost and cost alerts in the scenarios. Data insights scenarios evaluated based on the retrieved data accuracy and total token count consumed by the agent.
E.4.1. EVALUATION METRICS
For alert driven incident scenarios, we evaluate each LLM- based agent for two scenarios: (i) diagnosis and (ii) mit- igation. Data insights scenarios are evaluated based on accuracy of the retrieved results.
Anomaly detection scenarios are explicitly evaluated on
56

---

ITBench
Table 27: FinOps AD Scenarios
Scenario
Natural Language Prompt
Related KPI
Scenario 1: Anomaly
Generate a detailed report consisting of the distinct "ChargePeriodStart" val- ues with corresponding anomalies in "BilledCost" during 09-01-2024 and 09-30-2024 for the service 'Amazon Elastic Compute Cloud" and column 'PricingUnit' being equal to 'Hours'. Summarize the number of anomalies found.
Workload Optimization and waste reduction
identification
Scenario 2: Anomaly ranking
Please create a report with a rank ordered list of the hours with anomalics in billed cost during 09-01-2024 and 09-30-2024 for the Amazon Elastic Compute Cloud service category after filtering on column Pricing being equal to 'Hours'. Compute importance score for each anomaly based on the cost and the duration for which the anomaly persisted. Finally, rank these anomalies the descending order of their importance score.
Workload Optimization and waste reduction
Table 28:
Optimization Use Case: Increased Cost Alert Increasing Demand
Optimization Use Case
Details
Triggering Alert
Cost increase alert for an application "Foo". Cost alert was seen on application "Foo". The increase was 20% higher than the expected budget. Investigations reveal that the application is healthy and cost increase is caused by load increase. Client budget is flexible and thus cost alert updated accordingly increase the threshold. Long term fix requires additional check in CI/CD pipeline and possible automation of budget adjustments.
Summary
Time to detection
7 days (Current practices to observe utilization metrics for cost analysis 60 minutes
Time to diagnose
Time to mitigate
15 minutes
Event Type
An alert generated to show there more than 20% increase in expected cost.
Cost Overrun
20% increase in cost
Checked infrastructure size changes. e.g. Increase in replica counts in Kubernetes cluster.
Found replica counts are greater than historical average.
Checked whether there a legit increase in application load.
Diagnosis Steps
Found there is a stable increase in the application load and the cost increase is acccptable.
Checked budget constraints of the application.
Found application budget is fiexible for scaling up.
Calculated the cost alert thresholds. Update cost alert budget thresholds accommodate new stable load level to prevent false alerts.
Resolution Plan
Long Term Improvements
Created playbooks to ensure such handling such adjustments are automated.
the agent's 's performance in identifying anomalies within a time period of interest at a granularity (hourly, daily etc.) of interest for data subset defined by additional business dimensions such as department, cost center, application, region etc. Given the nature of the sub-tasks, the agent's s performance is implicitly evaluated terms of its ability to gather the right dataset based on the prompt and to produce solution obtained from the anomaly detection algorithm as the final LLM output. Fl-scores are computed to evaluate the performance of the agent on the anomaly detection see- nario. For ranking anomalies, we compute the rank score of the true positive anomalies, as identified by the agent, with the correct importance score, as per the importance defini- tion specified by the user in the prompt. Misinterpretation of the prompt or hallunications by the I.I.M will lower the
agent's performance.
Diagnosis. The agent is evaluated on diagnosis by its ability to provide accurate root cause of budget alerts. Diagnosis includes the analysis steps that the agent follows to identify the root cause of the budget alerts. Agent performance in diagnosis is measured using pass scores that indicate the accuracy the root cause analysis provided to the SREs.
Mitigation. Mitigation involves recommending resolution steps for incidents to clear the alerts and optimizing the cost and efficiency of the application. We evaluated agents with the success rate using the pass@1 score and using proximity scores to analyze the performance of agents to achieve the optimal cost and efficiency metrics defined in Section E.4.2. Both diagnosis and mitigation evaluations
57

---

ITBench
Table 29: Optimization Use Case: Increased Cost Alert - Faulty Auto-scaler
Optimization Use Case
Details
Triggering Alert
Cost increase alert for an application "Foo".
Cost alert was seen on application "Foo". The increase was 20% higher than the expected budget. Investigations reveal that auto scaler was misconfigured. SREs manually updated the auto scaling configuration by changing the scale up policy. Long term fix requires additional check CI/CD pipeline.
Summary
Time to detection
7 days (Current practices to observe utilization metrics for cost analysis 60 minutes
Time to diagnose
Time mitigate
15 minutes
Event Type
An alert generated show there more than 20% increase in expected cost. 20% increase in cost
Cost Overrun
Checked infrastructure size anomalies, e.g. increase in replica counts in Kubernetes
cluster.
Found replica counts are greater than historical average.
Checked whether there a legit increase application load.
Found an increase in load and pending containers.
Diagnosis Steps
Checked utilization of containers
Found low utilization
Checked autoscaler for scaling policy
Found low threshold for scale up rules
Contigure auto scaler to update scaling policies. Manually delete extra replicas.
Resolution Plan
Long Term Improvements
Created playbooks ensure such misconfigurations do not happen for future deployments.
can be expanded in the future to automatically measure the performance of agents using LLM-as-aa L-M--a---judgee similar to SRE-bench evaluations.
Proximity metries. To analyze the performance of agents in achieving optimal cost and efficiency, we defined prox- imity scores for hourly CPU cost, hourly memory cost, workload CPU efficiency, and workload memory efficiency. Proximity scores indicates the performance by calculating the proportional absolute difference between observed and optimal values for the measured cost or efficiency metric. We subtract the proportional value from such that hav- ing proximity score of I indicates achieving the optimal performance. Proximity scores are calculates as follows:
Data Insights. Agents are evaluated in the accuracy of retrieved results using pass@1 I scores. Partially correct answers despite generating similar SQL codes to the ground truth are considered failed for the scenario. Efficiency of the agents are evaluated by calculating token utilization.
Anomaly Detection. Agents are evaluated for their ability identify anomalies by using Fl score.
Anomaly Ranking. For ranking of anomalies, we first compute anomaly importance score Ia for anomaly as: Cuxó, Maz.e x6.5. where A represents the set of anomalies, C. represents the cumulative billing cost associated with an anomaly and 6, represents the duration for which anomaly persisted. Next, we compute the rank score of the anomalies, that were correctly identified by the agent, also having the correctly computed importance score.
proximity ; = jobserved_ optimal optimal
where represents the experiment trial, observed is the retrieved value for the measured metric, and optimal is the value given in the ground truth for the same field.
Hourly CPU cost: The cost of allocating a single CPU core for an hour. We used custom pricing policy through opencost ensure the ground truth values are not affected by changes in pricing between different cloud providers and on-premise deployments.
E.4.2. METRIC DEFINITIONS
Metrics for Alert Driven Incident Scenarios.
pass@ @ 1. We use the same metric defined in Section C.6.2 for both diagnosis and mitigation steps recommended by the agents.
Hourly Memory cost: The cost of allocating GB of memory. It also uses custom pricing policy.
58

---

ITBench
Workload CPU efficiency: We take the average CPU efficiency of all containers of the application. We calculate CPU efficiency by dividing the CPU utilization of a container by the request amount. For instance, if the CPU request is 100m in the deployment configuration but the container uses 50m, the efficiency is calculated as 50.
agents.
The results for the anomaly detection scenarios are con- tained in table 31. The first observation is that once again larger models, GPT-4o in this case, outperforms the smaller Granite-3.1 1-8bb-nnsttucct and LJama-3. Jlama-3.1-8b-inee. 1-8bbiinstruct. An ex- ception to this is the was unable to translate the natural language query accu- rately to an SQL query. It also misinterpreted the query and thought that it was expected detect anomalies without ac- cessing the AD tool provided to it. Due to this, the columns required by the AD tool were not part of the SQL query due to which the AD tool failed to run. The end result of this misstep was that the anomalies identified by the agent were 100% incorrect. Since the anomaly ranking scenario needed anomalies be identified first, the agent failed totally in those scenarios as well when using this LLM model. For the remaining 3 models, during data query well as during summarizing the output of the AD tool, we noticed that the agent hallucinated, thereby lowering both the Fl-score and the rank score metrics.
3-0b-iisttttt where the agent
Workload memory efficiency: It is calculated in the same way with CPU efficiency by dividing the utilization by requested memory amount.
We take the arithmetic mean of all runs for each metric and calculates the standard deviation. We present our results in Table 6.
Metrics for Data Insight Scenarios.
pass@1. 1. We have compared the generated results with ground truth values for the given prompts. Only requested fields in the prompt are evaluated. The results with identical values with ground truth data are considered a "Pass" and all other results are evaluated as a "Fail".
E.6. Example Trajectories
Token Utilization. Agents require different number of iter- ations until reaching the final answer. For each run, we cal- culated total number of tokens during an experiment run for all iterations. Token utilization indicates how efficiently the agent optimizes between iterations and finalize its thoughts. This is used in evaluating the agent's 's performance for the anomaly detection scenarios as well.
Data insights and anomaly detection scenarios require un- derstanding of the prompt, retrieving data, and processing it. In our evaluations, we have found that agents commonly hal- lucinate about the column names in the table, which results in execution error in SQL queries. The performance of the agents in correcting these errors in the following iterations vary significantly. Similarly, agents are unable to consider the version of the database server to use the proper function names or syntax.
Metrics for AD Scenarios.
Fl score. Treating anomalies identified by an anomaly detection algorithm as the ground truth data, we compute the Fl score to account for both the precision and recall of the anomalies identified by the agent.
FinOps issues firing alerts are highly connected to used in- frastructure and deployment configurations and policies for Kubernetes deployments. Incidents at infrastructure or con- figuration can trigger cost alerts. Agents commonly need to use the same tools as SRE-agent to understand the status of underlying infrastructure and deployment configuration of the application to identify the root cause of a FinOps concern. Similarly, to investigate budget variations accu- rately, they need analyzing load and utilization patterns to avoid diverging to irrelevant resolution recommendation. Figure 29 shows trajectory for Scenario 37 where agent starts retrieving the alerts and observing CPU hourly cost has exceeded the budget threshold. It continues checking de- ployment details using Kubectl tool and retrieve replica counts for each deployment. Due to high number of replicas in adservice, it considers scaling down the adservice de- ployment. Missing utilization checks and not analyzing the deployment details caused agent an inaccurate diagnosis and led to scaling down recommendation despite the high load for the application. In a correct trajectory, an agent would analyze the utilization and decides the high load in
Rank score. This denotes the fraction of the true positive anomalies, as identified by the agent, with the correct impor- tance score, per the importance definition specified by the user in the prompt.
E.5. Results
Table 30 presents the evaluation results for data insights scenarios. It shows that larger models perform better in both accuracy and average token utilization. The token utiliza- tion is mainly depending on the performance of updating the generated SQL queries based on natural language to SQL tool responses. Larger models understands tool feedback better and update code accordingly. However, all models largely failed to generate the correct queries for scenarios that require multiple data processing steps, can be seen in the results of Scenarios 4, 5 and 6. We found data hallucina- tion, making up function or column names based on prompt, and ignoring database version as common problems for all
59

---

ITBench
Table 30: Evaluation of Data Insights Scenarios: Pass @ %
Sceuario
Scenario
Scenario
Scenario
Scenario
VinOpa/Agent:: Models
Pass@1 1
#Token
Pass@1
#Token
Pass
#Token
Pass@1
#Token
Pass@1
#Token
Pass@1
#Token
granite-3. Seanite-331-SP- 3.1-----issttre
0
51 1096
100
10304
10210
5778
41,38.0
0
9514
lama-31199B-imsrruc
0
90424
76178
99769
131882
69447
0
60531
3.mma-3.-770B-eae instruut
D
7347
100
6410
100
8107
8029
0
6350
0
7666
gpt-4o
Note: "# Token is the average token utilization over ten runs of each scenario.
(Pass values shown percentage.)
Table 31: Evaluation of AD scenarios
Anomaly Detection
Anomaly Ranking
FinOps/Agent: Models
F1-score
#Token
Rank score
#Token
granite-3.-SS-iistrrme
0.4 + 0.07
22024
0.3 0.0
205042
1amma--.1-8B-insteee
0.4 1 0.03
14461
0.4 1 0.0
166614
llama-3.-770B-inste
gpt-4o
0.6 0.0
550
0.5 + 0.0
164313
the system and would recommend changing the budget alert thresholds if total budget allocation allows or analyze the deployments which could be scaled down without hurting the application performance.
Thought: To diagnose the problem from the alert, need to start by getting the current alerts.
Action: using tool
GetAlerts
Observation: High cost spend vCPU and memory resources otel-demo namespace.
Thought: Received the current alerts. My next step is to gather more information about deployments..
using kubect1 get deployment adservice -n otel -0 yam1
Action:
NL2Kubeetl
Observation:
spee:
progessaDeadiineScconddd 600 replicas:
The "adservice" deployment has many replicas. should consider descrcaing resources allocated such as decreasing epu and memory requests or deleting replicas.
Thought:
Action: using tool
NL2Kubeeth
kubect1 patch deployment adservice -n otel -p '("spec"; ("replica": 2))'
Observation: deploymennt patched
The alert was not cleared after scaling the replicas.
Figure 29: Sample Trajectory of unusual cost variation use case.
60

---

ITBench
Table 32: Evaluation of Alert Driven FinOps scenarios.
Diagnosis
Mitigation
Models
pass @ 1 (%) f
pass@ (%) f
Proximity to Optimal CPU Cost 1
Proximity
Proximity to Optimal CPU
Proximity to Optimal Memory Efficiency
to Optimal
Memory Cost
Efficiency 1
t
granite-3.1-B-iinttrucc -anta-t.-SB-instruut
0
0
0.47 0.01
0.48 + 0.06
0.53 = 0.04
0.94 = 0.01
0
0.49 0.01
0.46
0.07
0.56 0.08
0.96 =
llama-33--78--nnstuee
16.6
0
0.47 0.01
0.49 0.05
0.53 - 0.03
0.96 = 0.02
gpt-4o
33
0
0.48 + 0.01
0.51 0.02
0.63 - 0.07
0.92 - 0.08
pass percentages. from represents sub optimal performance.
Proximity values close observed optimal One represents achieving optinal and any deviations
61

---

ITBench
F.I.I. GROUND TRUTH
A5
The ground truth G is composed of set of unordered fault propagation chains (FPCs):
A2
A3
A4
A1
G = {FP, IpE [I....,,P)).
A6
Each chain PP contains / fault entities (gi..... .9
A7
Every entity 9i has a level, denoted level(g:) = k. which describes its position in the propagation chain (A: E fI,..., K}). Typically, each chain has single root cause entity at level K. But, in general, each unordered fault propagation chain in the ground truth can have U root cause entities.
Figure 30: NTAM sample illustration. Node A2 is the ground truth root cause. Orange arrows represent edges of 'owns' type and black arrows represent edges of 'calls' type. Dotted area represents ground truth fault propagation chain.
Rp {gu).. u E .I.....U) For example, in Figure 30, A2 is root cause and A2, A3, A4 is the propagation chain.
We also define an entity importance measure, GT(g:), to capture that not all entities are equally informative:
F. Normalized Topology-aware Match
GI(g.) ( K (x~-1)²²
Existing work for root cause analysis has used general metries including lexical and semantic metrics like BLEU, ROUGE-L, and BERTScore (Ahmed et al., 2023b), and metrics like FI, ranked list evaluation metrics like hit rate, (Zcu al., 2024; Chen et al., 2024c). However, these metrics are not accurate and granular enough. For example, the root cause "adservice pod", "adservice deployment" a better prediction than "recommendation pod" but lex- ical metrics do not reflect that. Similarly, rank- evaluaaaln metries only capture whether the ground truth the pre- dicted list not. There is need develop domain-specific metrics that accurately measure the quality of the predicted root cause and fault propagation chains.
where k = level(g;). and is a positive parameter. Entities closer to the root cause (higher k) get larger importance.
F.1.2. PREDICTED OUTPUT
Similarly, the model output O a set of Q unordered fault propagation chains:
O ( 1 (1, ( Q1)-
Each predicted chain FP consists of J fault entities (o1,...:00}) The agent also identifies V root cause en- tities, per propagation chain. R² = {o E (1....,V)
To this end, we propose NTAM (Normalized Topology- Aware Match), inspired by:
F. DISTANCE
Distance- metries, rewarding predictions that are "closer" in a topology graph to the ground-trutt entities.
Let T be a topology graph whose nodes are the entities in our system. We define a distance [g(gi, Oj) that accounts for:
Information-rrtrieva concepts such as BM25 (Fang et al., 2011), where features that appear very commonly carry lower weight. By analogy, our scoring function deighweights highly connected nodes (i.e., large sub- tree), so that nodes that appear in many possible paths are less rewarded.
The type of each edge along that path. In typical Kubernetes-base -based system, illustrated in Figure 30, we define two main edge types:
owns (e.g., Deployment owns ReplicaSet, Repli- caSet owns Pod),
In the following sections, we describe NTAM that can used both for scoring the predicted fault propagation chain and fault localization.
calls (indicating actual invocation request from one microservice to another).
Let cost(edge type) be positive realev function. We define:
F.1. Notation
We first introduce notation used NTAM.
cost(owns) = 1, cost(calls) Ç.
62

---

ITBench
For example, in Figure 30, if A2 is the root cause pod and A1 is the corresponding deployment, intuitively, predicting Al as the root cause is better than predicting A6 because A6 is part of another deployment, even though both are one hop away on the undirected graph. So, ( is set to be greater than 1.
F.2.1. GI-WEIGHTED INVERSE TOPOLOGY DISTANCE
(g. (l x (D( oj) +
This is based on the following intuitions:
The sub-tree size of each node along shared path in
the graph. We are inspired by information retrieval scoring functions, such as TF-IDF or BM25 (Zhai and Massung, 2016), where, terms that appear in many doc- uments have lower inverse document frequeeeffreuee frequency weight, as they are less discriminative. Analogously, if a pre- dicted node has large sub-tree size in the topology graph T (i.e. it is connected to many other nodes), it is less informative for pinpointing the fault's true loca- tion.
A predicted entity closer to the ground-truth node on the topology should get a higher score.
Further, the farther you move from a ground-truth en- tity, the relative penalization should decrease. In other words, to avoid significantly large impact on the total score as the distance increases, the penalization should saturate. The above two are ensured by the 1/TD term. DDis, 0j) is large (involving many high-subtree- size nodes or high-cost edges), then IT D (gs, oj) becomes small, reducing the credit for matching Di with 0j. ō is parameter controlling how quickly penalties grow for larger distances
For example, in Figure 30, predicting A5 is more dis- criminative than predicting A6 because A6 calls both A2 and A7, whereas, A5 calls only A2.
Combining the two factors above, we define edge cost be- tween nodes x, in T as follows:
Closer to more important ground-truth entities is better. This is ensured by the GI term.
F.2.2. GI-WEIGHTED INVERSE FD
cost(1; y) wtweeeddg__ypeer,,roeee v)) x subtree_size()
1 Σ ITDwalg:
TFD
Finally, TD(ge 02) is then defined as the shortest distance between I: and 0j in weighted T, where weights are the edge costs.
9.EFP,
A predicted entity closer (on average) to all entities in the FPC is preferred. Analogous intuitions described for ITD above hold for LFD wt as well.
To avoid penalizing exact matches, cost(x, x²) =
In addition T D(g; oj) between oj and single entity gi in FPp. we also define FD(FPp p: oj) as the average TD between 0j and every entity in F.p.
F.2.3. SCORING ONE GROUND-TRUTH CHAIN VS. ONE PREDICTED CHAIN
Consider single ground-truth chain FPp of size I and a single predicted chain F FP, of size J. We define:
These distance quantities guide our scoring function by rewarding predicted entities that lie close to the ground- truth nodes on the topology graph.
1
S(FP,, FP
Σ
S1 x S2
(7)
F.1.4. TASK SCORE
[1--1|+ 1)
i=1
The overall task is to score the quality of O compared to G, defined by S(G, O), where higher score means better quality.
where,
$1 (max, ITD mi (g., o,)))" S2 (max,) [FF (FP, o; and a. 3, c 0 parameters.
Fault localization is a special case of this task, where only subsets Rp and R' are compared instead of entire F FP, and FPr.
We select the best matching predicted output entity based on distance to a single ground truth entity, and to the whole propagation chain. The denominator (IJ-I|+1) penalizes large mismatches in the chain length. This is based on the following two intuitions:
F.2. Overall Score: NTAM
We now describe all components of NTAM below.
63

---

ITBench
If adding extra predicted entities that farther away does not add information, they should hurt the score.
Table 33: Key NTAM hyper-parameters.
Symbol
Meaning
Chosen Value
Conversely, adding extra predicted nodes reduces distance ground-truth entities, do not over-penalize.
Exponent for the max ITD
term
8
Exponent for the max IFD
1
F.2.4. COMBINING ALI. CHAINS: S(G,0)
term
7
Power for entity impor- tance G/
Given all ground-truth chains (FPI,...,FPP) and pre- dicted chains
we define:
5
Distance penalization expo- nent
0.289
P maxq S(FPp, FP;)
Chain-length mismatch
0.5
S(G,0) = Σ (IP QI 11
penalty exponent
Number-of chains mis- d
0.5
match penalty exponent Ratio of cost of edge type calls of edge type owns
6
2.1
Again, the best matching output propagation chain se- lected for each ground truth propagation chain. The term (IP- Q) + 1) penalizes a mismatch in the total number of predicted chains vs. ground-truth. Same intuitions as described for scoring a single chain above hold.
In practice, one can use grid-search or numeric optimiza- tion method (Taylor et al., 2006) to select the parameter values that best align with expert preferences. For example, we could measure how well the ordering induced by NTAM matches expert judgments, using standard correlation coeffi- cients such as Kendall Tau-b (Kendall, 1945).
Normalized TAM (NTAM). We normalize S(G, O) to fall within [0, 1] by dividing by the ideal score achieved when the prediction matches ground truth exactly:
S(G,0)
NTAM
Currently, all values except 5 are not tuned and simply chosen to give uniform and equal weights to all terms. is determined analytically leveraging domain knowledge- based assumptions. As the distance increases by one hop, we want ITD (non-Gi-ve -eeeghhewe))))ed to decrease by 50%. Further, on average, Kubernetes graphs have a subtree size of 10. Based these two assumptions and 1, get 8 0.289.
S(G.G)
Thus, NTAM = indicates a perfect match
F3. Fault Localization: FL (NTAM)
For fault localization, only the root cause entities are con- sidered. Let Rp denote the root-cause()) in ground-truth chain F,, and R. the root-cause(s) in predicted chain F FP Adapting Equation (7) only these subsets:
V
S3 x
S4
S(R,, R2)
Σ
- V 1)
where,
S3 MouxxvIITDa (Jus ou)))
S4 (maxy IIF (FP, on)
Summing over all chains, then normalizing:
Σ P maxq (L)
p=1 (18-Q1+1)*
FL (NTAM) Epen maxp S(p,, Rp))
This measures how well the model predicted the root-cause nodes alone.
F.4. Parameter Tuning
Key used in NTAM are summarized in Table 33.
64


Reviews:
[{"review_id": "fZYw6iaMq9", "reviewer": "Reviewer_G6ic", "summary": "This paper presents IT-Bench, a framework that benchmarks AI agents for IT automation across roles including Site Reliability Engineering, Compliance and Security Operations and Financial Operations. It offers 94 real-world scenarios with automated, partial scoring evaluation and a leaderboard to ensure reproducibility. The framework models each scenario as a tuple of metadata, environments, triggering events, and desired outcomes, and benchmarks the agent's performance. Evaluations using various LLMs reveal modest success rates with FinOps unresolved. This demonstrates the challenges in automating IT complex tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.1 Complexity and infrastructure demands may hinder accessibility**\n>The framework’s complexity is abstracted from the agent interface, which is designed for accessibility, similar in principle to SWE-agent. AI researchers in the broader community have been able to use ITBench. Environment setup is automated (“push-button”) using provided scripts, masking infrastructure details. The framework runs on laptops (≥16GB RAM, Linux/MacOS) for smaller tasks, allowing developers to quickly pull and work on tasks. Workstations or cloud VMs are needed for larger problems.\n\n>ITBench aims to enable research on complex real-world problems. Reducing the inherent complexity is a non-goal; modeling it is necessary to rigorously evaluate agents on realistic IT infrastructures. Simplifying would impair task fidelity and evaluation validity. Our principle is to provide an accessible interface while preserving the necessary complexity for meaningful evaluation, balancing scalability and resource efficiency.\n\n**Q1.2 On extensibility**\n\n> Extensibility is a first-class design principle of ITBench and is treated seriously. We promote and welcome open contributions from researchers and practitioners. Unlike the agent interface, extending the benchmarks requires expertise in IT infrastructure to maintain accuracy and realism.\n \n> We provide clear guidelines in our repositories (anonymized for the double-blind policy) to standardize the addition of new tasks based on their required extensions. The main effort comes from (1) ensuring the reproducibility of the problems in the tasks, and (2) defining tasks-specific criteria for partial scoring (the other criteria is automated based on whether the alarms are resolved by the agents). In our experience, the setup is rarely a problem as it is largely automated, and the reproducibility is verified through an automated Continuous Integration (CI) pipeline.\n\n**Q2. How partial scoring is implemented**\n> Partial scoring is fundamental to our benchmark for systematic, fine-grained assessment of agent reasoning in IT tasks. It values intermediate steps when perfect solutions are hard to achieve, necessitating novel metrics tailored to specific IT domains. We exemplify partial scoring for root-cause diagnosis for SRE scenarios. Given large topologies (100K+ nodes), exact identification is difficult, but recognizing topologically close components demonstrates valuable reasoning. To quantify this, we developed the Normalized Topology Aware Match (NTAM) metric using expert-validated principles: topological proximity, node importance within the fault chain, effective search space reduction, and output length constraints. Inspired by information retrieval ranking (like BM25), NTAM measures prediction relevance and features tunable hyperparameters (see Appendix F).\n\n> Crucially, partial scoring methods are domain-specific. For FinOps, we supplement NTAM with other proximity metrics evaluating alignment with optimal cost/efficiency values by measuring relative difference, rather than requiring exact matches. This tailored approach ensures nuanced performance evaluation across diverse task types.\n\n**Q3. LLM for NL-to-code: Agent's or fixed?**\n>We use the same LLMs for both the planner and the tools (e.g., NL2Kubectl). Exploring hybrid LLM configuration is our future work. For example, we can potentially use small models for NL-to-code tools, but our experience shows that small models are not yet good at generating code or using tools effectively.\n\n**Q4. Token utilization**\n>  Average token consumption varies by model and task type; GPT-4o uses ~675k ± 205k for SRE,  ~208k ± 263k for FinOps and  ~23k ± 32k CISO tasks .\n\n**We also respond to the following important comments.**\n\n**C1. FinOps contains only 2 tasks**\n>ITBench is an evolving benchmark. FinOps tasks increased from 2 at submission to 10 currently, and we continue adding tasks. FinOps initially had fewer tasks as it's a less established field; we are actively defining and adding scenarios. We evaluated agents on these 10 tasks; smaller models struggle, while GPT-4o achieved a 0.2 pass@1 score. We acknowledge the current scarcity and lack of statistical representativeness for FinOps (thank you) and will clarify this in the updated paper. Overall, the benchmark is growing, e.g., SRE tasks increased from 42 to 98 since submission via community contributions.\n\n**C2.  Overall reliability assurances of agents in IT is required**\n> We agree that safety and reliability are critical, and we will add deeper discussion as suggested. We are also enhancing ITBench for finer-grained safety feedback within the SRE, FinOps, and CISO contexts. Furthermore, ITBench already models real enterprise IT settings where SRE, FinOps, and Compliance tasks are interlinked. An agent's action (e.g., an unsafe SRE command) can trigger cross-domain issues (such as Compliance violations or FinOps costs) that ITBench measures. However, this is subject of future work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claims are proposed in this paper.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper presents a novel direction for benchmarking LLM agents in real-world IT tasks, extending their application far beyond SWE applications. IT-Bench provides a comprehensive framework that covers multiple IT personas and reflects the complexity of actual IT operations.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper introduces a comprehensive benchmark that reflects real-world IT automation challenges. It unifies multiple IT roles into one framework, ensuring broad applicability and practical relevance. Moreoever, the use of real-world scenarios and industry benchmarks enhances its authenticity. Drawing on actual incidents and standards, the benchmark offers a realistic testbed for AI agent performance.\n2. The evaluation pipeline is automated and systematic. It proposes several well-defined metrics with real-world implications. The design of a leaderboard can provide performance insight.\n\nWeaknesses:\n1. The domain coverage is imbalanced. FinOps contains only 2 tasks. A success rate of 0% cannot support any claims in difficulty due to the limited dataset size.\n2. The framework's complexity and infrastructure demands may hinder accessibility. The detailed environment setup and the challenge of integrating new benchmarks could restrict broader adoption and ease of extension.", "other_comments_or_suggestions": "I still have reservations regarding the motivation for employing AI agents to address IT challenges. While the paper cites the CrowdStrike incident to demonstrate the need for intelligent IT resolution, it is not clear to me how deploying agents will prevent such failures in practice. In production-grade environments where errors can have significant consequences, how to ensure the reliability of AI agents is crucial. For example, what if an alarm fix inadvertently triggers a cascade of additional errors? I believe a deeper discussion on the built-in safeguards, error mitigation strategies and the overall reliability assurances of agents in IT is required.", "questions_for_authors": "1. In the abstract, you mention that the benchmark can be easily extended through community contributions. Could you elaborate on the process for how one can add new tasks to IT-Bench? Given that task scenarios often involve complex, task-specific setups and requirements, how do you ensure that integration is manageable for contributors? Is there any guidelines designed to standardize the addition of new tasks to IT-Bench?\n2. The abstract claims that IT-Bench supports partial scoring. Could you clarify how partial scoring is implemented during evaluation? Specifically, how are partial scores computed and used to assess agent performance?\n3. For the natural language to code tools in the SRE/FinOps settings (e.g., NL2Kubectl), which LLM backbone is used to translate natural language into specific code? Is this backbone the same model used for the agents, or is it fixed to a particular model?\n4. What is the average token consumption for running the agent on one task?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cRPVE4fqjN", "reviewer": "Reviewer_B5Mk", "summary": "This paper is a benchmark paper. It evaluates the recent LLM agent systems in three IT domains: (1) Site Reliability Engineering (SRE),\n(2) Compliance and Security Operations (CISO), and (3) Financial Operations (FinOps). The main contribution of this paper is preparing three benchmarks and thoroughly evaluating candidate systems. Besides that, this paper also implements several IT agents following the traditional design. In the experiment section, the paper summarizes the key observations that the current agent systems still can not perform well on these real IT tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. unique challenge of IT tasks compared with the broader SWE tasks**\n> IT tasks are more diverse than SWE. Consider SRE, closer to SWE than FinOps/CISO. SRE involves distributed systems (multi-machine, full stack: app, platform, OS, hardware & their integrations). SWE focuses on single programs.\n\n> Key differences between SWE and SRE include:\n> * Complexity/Scope: SRE systems are larger-scale and more diverse than single programs. Root causes are broader than just bugs, including hardware/network faults, misconfigs, overload.\n> * Diagnosis: SRE diagnosis differs from SWE debugging. SRE issues are often hard to reproduce (dependencies, scale, non-determinism) & lack source code. SWE debugging assumes reproducibility with source/debuggers (GDB).\n> * Context/Agent Fit: IT systems' scale/complexity makes full model context infeasible (vs. SWE). Agents are needed; multi-agent/multi-modal designs handle diverse IT data (metrics, traces, logs).\n> * Goals/Actions: SRE goal: production service reliability/availability. SWE goal: program correctness. SRE mitigation prioritizes immediate service restoration (rollbacks, feature flags, etc.), beyond just fixing bugs (SWE focus).\n> * Environment/Safety: SRE is production; SWE is development. Safety is paramount for SRE agents (unlike SWE agents trying anything for tests); unsafe trial-and-error unacceptable. SRE actions need risk/impact assessment.\n> * Evaluation: Evaluating SRE agents is harder than SWE. SWE uses public data (GitHub); replicating production SRE systems is difficult (scale, proprietary). This motivates ITBench for enabling AI in this complex domain.\n\n**We also respond to the following important comments.**\n\n**C1. LLM agent systems are black-box**\n\n> We acknowledge the challenge of understanding LLM agent decision processes. However, ITBench allows us to empirically investigate agent's behavior. \n\n> Agent's behavior can be empirically evaluated because of the following\n> * Detailed Trajectory Logging: We record comprehensive logs for each step, including the specific tool used, the exact inputs (including full prompts), and the resulting action. This provides necessary data for analysis.\n> * ReAct Framework: Our agent utilizes the ReAct framework, which prompts the LLM to output its reasoning (\"Thought\") before acting. This captures intermediate reasoning steps, offering insight into the decision process.\n> * Error Source Differentiation: By combining detailed logs and ReAct traces, we can distinguish between high-level reasoning failures (strategy errors) and lower-level execution errors (tool usage mistakes). An automated process categorizes failures, enabling quantitative analysis of failure modes.\n\n\n> We provide 2 case studies as exemplars on SRE agent:\n\n> **Prompt problem**: Trajectory analysis revealed flaws in an agent using Granite-3.1-8B (e.g., tool misuse linked to prompt errors). Fixing the prompts based on this analysis significantly improved success rates (3.3% to 8%), reduced errors (incorrect tool calls 7% to 0.7%), and balanced tool usage.\n\n> **Planning problem** To quantitatively assess reasoning strategy (e.g., SRE diagnosis), we compare the agent's explored path (from 'Thoughts'/tool use) against the ground-truth fault propagation chain (causal sequence). Rationale: Diagnosis often traces this chain in reverse. Metrics developed across trajectories:\n> * Detoured services: Avg services explored off the ground-truth path (lower = better focus).\n> * Relative covered services: Avg ratio of relevant on-path services explored vs. ground-truth length (higher ≈ 1 = better alignment).\n\n> We analyzed these metrics separately for successful and unsuccessful trajectories, focusing on GPT-4o versus Granite-8B:\n* > For successful trajectories: GPT-4o demonstrated significantly better reasoning quality. It achieved much higher alignment with the ground-truth path (avg Relative Covered Services: 0.75 for GPT-4o vs. 0.30 for Granite-8B) and substantially less deviation into irrelevant services (avg Detoured Services: 0.98 for GPT-4o vs. 2.00 for Granite-8B).\n* > For unsuccessful trajectories: Even when failing, GPT-4o maintained better reasoning metrics compared to Granite-8B. GPT-4o still covered more of the relevant path (avg Relative Covered Services: 0.48 vs. 0.27) and explored fewer irrelevant services (avg Detoured Services: 2.1 vs. 3.1) than Granite-8B did during its failures.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "N/A, no theoretical claims", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "1. The agent framework used in this paper follows the widely used ReAct style.\n2. The evaluation is consistent with the previous setting.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The main limitation of this paper is the theoretical depth. As the current LLM agent systems are black-box due to the LLM, we cannot clearly analyze and understand the decision process. This paper shows that existing agent systems cannot solve current IT challenges but fail to give a theoretical analysis.", "other_comments_or_suggestions": "No", "questions_for_authors": "Can you explain more about the unique challenge of IT tasks compared with the broader SWE tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HzpOtcpOt6", "reviewer": "Reviewer_5pPq", "summary": "The paper introduces IT-Bench, a specialized benchmarking framework designed to evaluate AI agents on real-world IT automation tasks across three key domains: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). Built from 94 scenarios derived from actual incidents, CIS benchmarks, and FinOps guidelines, IT-Bench operates in realistic environments—such as Kubernetes clusters—integrated with industry-standard tools like Grafana and Prometheus. The framework assesses agent performance using metrics such as pass@1 and time to resolution. Testing reveals significant challenges for even advanced models like GPT-4o, with success rates of 13.8% in SRE, 25.2% in CISO, and 0% in FinOps. These results highlight the complexity of IT automation and the pressing need for enhanced AI capabilities in this field.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Can IT-Bench incorporate resilience testing (e.g., telemetry noise) to better simulate production unpredictability?**\n\n> Yes! We already incorporate a few resilience testing tools like Chaos Mesh in ITBench, which can be used to evaluate agentic technologies under different resilience testing scenarios. ITBench also supports the evaluations of the effectiveness of agents with different kinds and verbosity levels of telemetry data – one can easily turn on or off any telemetry. \n\n> We currently do not add random noises to telemetry data. The mechanism of adding noises is straightforward. However, one of our key principles is to achieve high-fidelity of real-world IT scenarios; we are working on policies for adding noises that can resemble faulty telemetry in the field.\n\n**We also respond to the following important comments.**\n\n**C1. Many real-world IT operations involve complex, collaborative environments where multiple agents or human-AI interactions are critical. Extending IT-Bench to incorporate multi-agent collaboration or human-AI workflows could significantly enhance its generalizability, addressing scalability and robustness concerns raised in this review.**\n\n> Those are excellent suggestions! Multi-agents and human-AI interactions are important components on the roadmap of ITBench, and we will discuss them in the paper. We use agents inexplicitly to refer to multi-agents – our SRE agents already have a form of multi-agents: the mitigation agents interact with the diagnosis agents to determine the resolution based on the root causes. Similarly, CISO agent is also a multi-agent system including skilled agents targeted for OPA-Ansible, Kyverno, OPA-Kubectl.\n\n**C2. Potential biases in scenario selection (e.g., over-representation of Kubernetes-based systems) and lacks discussion on generalizability across diverse IT ecosystems**\n\n> Thanks for the question. We will discuss the generalizability in the final version. The design of ITBench is not specific to Kubernetes-based stacks; it can be easily extended to other IT infrastructures (e.g., Docker Swarm and Nomad from HashiCorp). Certainly, it would require engineering effort.\n\n> Kubernetes is chosen because it is the de facto open-source IT infrastructure for cloud and datacenter systems today. Its design is in principle similar to proprietary infrastructure systems such as Google’s Borg, Meta’s Twine/Tupperware, AWS’s ECS, and SnowFlake’s ECS; Kubernetes as a main platform services are offered by all major cloud services (e.g., Google, Azure, AWS, IBM). To make ITBench an open platform, we intend to use only open-source systems as the components instead of proprietary ones. So, Kubernetes seems to be the best choice. Note that most cloud system research uses Kubernetes as the representative infrastructure (in a similar vein as how Linux is used in OS research and how x86-64 is used in architecture research).\n\n> In the context of ITBench, Kubernetes only serves as a container orchestration infrastructure. Many IT tasks are beyond the Kubernetes layers, e.g., the applications like Hotel Reservation are not specific to Kubernetes, and thus misconfigurations in applications are orthogonal to Kubernetes (they happen in the same form regardless of the orchestration infrastructure). Similar to node failures and network disconnections.\n\n> Note that the term “Kubernetes” refers to the broader container orchestration based IT infrastructure, which is not limited to the original Kubernetes project (https://github.com/kubernetes/kubernetes). In fact, we use different backends like MiniKube, K3d, and Kind (for laptop-based setups).", "claims_and_evidence": "The authors assert that IT-Bench is a robust, extensible, and practical framework for evaluating AI agents in IT automation. This claim is supported by several key points:\n\n* Real-World Scenarios: The benchmark incorporates 94 tasks rooted in genuine IT incidents and established industry standards, ensuring relevance and applicability.\n\n* Framework Design: By integrating authentic environments and observability tools, IT-Bench mirrors the conditions AI agents would encounter in practice.", "methods_and_evaluation_criteria": "IT-Bench models agent-environment interactions as a Partially Observed Markov Decision Process, capturing the inherent partial observability of IT systems. Agents are evaluated using a suite of metrics, including pass@1 (success on the first attempt), fault localization accuracy, and fault propagation chain analysis, among others. These metrics provide a comprehensive measure of performance across diverse dimensions.", "theoretical_claims": "The use of POMDP to model agent-environment interactions is theoretically sound and aligns with AI research paradigms. The NTAM metric is a novel contribution, accounting for topology-aware fault localization. However, the paper overlooks potential biases in scenario selection (e.g., over-representation of Kubernetes-based systems) and lacks discussion on generalizability across diverse IT ecosystems. NTAM’s parameter tuning is heuristic-based, needing empirical justification.", "experimental_designs_or_analyses": "Experiments\n\n\nThe paper evaluates baseline agents, such as GPT-4o and Llama-3.3-70B, across IT-Bench’s scenarios, yielding the following insights:\n\n* Low Success Rates: Performance is notably weak, with FinOps tasks achieving a 0% success rate.\n\n* Complexity Challenges: Agent effectiveness diminishes as scenario complexity increases.\n\n* Environmental Factors: Non-deterministic elements, such as real-time telemetry data, pose significant hurdles for agents, underscoring the unpredictable nature of real-world IT settings.", "relation_to_broader_scientific_literature": "The introduction of IT-Bench adds a valuable contribution to the literature on AI agent evaluation for IT automation. Its focus on single-agent performance establishes a strong foundation for assessing individual AI capabilities in these contexts. However, many real-world IT operations involve complex, collaborative environments where multiple agents or human-AI interactions are critical. Extending IT-Bench to incorporate multi-agent collaboration or human-AI workflows could significantly enhance its generalizability, addressing scalability and robustness concerns raised in this review. Such an advancement would align the framework with emerging research on collaborative AI systems, positioning IT-Bench as a versatile tool for evaluating AI-driven solutions in dynamic, multi-actor settings.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths\n\n* Comprehensive Metrics: IT-Bench employs a diverse set of evaluation criteria, including pass@1 for accuracy and time-based metrics like Mean Time to Diagnosis (MTTD) and Mean Time to Repair (MTTR). This multifaceted approach assesses both precision and efficiency, offering a holistic view of agent performance.\n\n* Openness: The framework and baseline agents are open-sourced, encouraging collaboration and further development by the research community.\n\nWeaknesses\n\n* Narrow Agent Focus: The evaluation centers on large language model-based agents (e.g., GPT-4o, Llama), neglecting alternative AI approaches, such as rule-based systems or reinforcement learning, that might outperform in specific IT automation contexts.", "other_comments_or_suggestions": "Refer to the former comment.", "questions_for_authors": "Can IT-Bench incorporate resilience testing (e.g., telemetry noise) to better simulate production unpredictability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers are universally positive about this paper, including that it 1) makes a significant contribution to the community in the form of a benchmark and evaluation infrastructure for using AI agents to automate IT tasks, 2) is comprehensive, includes a wide variety of real world data, and may be extended easily in the future 3) includes automatic evaluation with a leaderboard, which makes it accessible and will further motivate work in the community, and 4) is a clear and detailed examination of the benchmark, including assessment of related methods. There are some minor issues, such as imbalances among the categories of IT automation tasks, but the strengths of this work and its likely impact far outweigh any minor negatives.
Decision: Accept (oral)