STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds
Authors: Yinfang Chen
Jiaqi Pan
Jackson Clark
Yiming Su
Noah Zheutlin
Bhavya Bhavya
Rohan R. Arora
Yu Deng
Saurabh Jha
Tianyin Xu
Venue: NeurIPS 2025 poster ()
Keywords: Agentic AI
Large Language Model
System Reliability
Primary area: applications

Abstract: In cloud-scale systems, failures are the norm. A distributed computing cluster exhibits hundreds of machine failures and thousands of disk failures; software bugs and misconfigurations are reported to be more frequent. The demand for autonomous, AI-driven reliability engineering continues to grow, as existing human-in-the-loop practices can hardly keep up with the scale of modern clouds. This paper presents STRATUS, an LLM-based multi-agent system for realizing autonomous Site Reliability Engineering (SRE) of cloud services. STRATUS consists of multiple specialized agents (e.g., for failure detection, diagnosis, mitigation), organized in a state machine to assist system-level safety reasoning and enforcement. We formalize a key safety specification of agentic SRE systems like STRATUS, termed Transactional No-Regression (TNR), which enables safe exploration and iteration. We show that TNR can effectively improve autonomous failure mitigation. STRATUS significantly outperforms state-of-the-art SRE agents in terms of success rate of failure mitigation problems in AIOpsLab and ITBench (two SRE benchmark suites), by at least 1.5 times across various models. STRATUS shows a promising path toward practical deployment of agentic systems for cloud reliability.

STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds
Yinfang Chen
Jiaqi Pan
Jackson Clark
Yiming Su Yu Deng!
Noah Zheutlin
Bhavya Bhavya!
Rohan Arora
Saurabh Jhat
Tianyin Xu
University of Illinois
Urbana-Charl "honna-Champaig"
IBM Researcht III, Tsinghua University
Abstract
In cloud-scale systems, failures are the norm. A distributed computing cluster exhibits hundreds of machine failures and thousands of disk failures; software bugs and misconfigurations autonomous, Al-driven reliability engineering continues to grow, as existing human- in-the-loop practices can hardly keep up with the scale of modern clouds. This paper presents STRATUS, an LLM- based multi-agent system for realizing autonomous Site Reliability Engineering (SRE) of cloud services. STRATUS consists of multiple specialized agents (c.g., for failure detection, diagnosis, mitigation). organized in state machine to assist system-level safety reasoning and enforcement. We formalize a key safety specification of agentic SRE systems like STRAY ATUS, termed Transactional No- Ro-RRegession (?'NR), which enables safe exploration and iteration. We show that TNR can effectively improve autonomous failure mitigation. STRA significantly outperforms state-of-the-ae SRE agents in terms success rate of failure mitigation problems in AlOpsLab and ITBench (two SRE benchmark suites), by least 1.5 times across various models. STRATUS shows a promising path toward practical deployment of agentic systems for cloud reliability.
reported to be more frequent. The demand for
Introduction
Cloud systems are the backbone of today's large-scale computing and end-user applications. Their reliability is crucial, with few minutes of outages leading to significant user impacts and financial losses [T. Given the ever-growing scale, complexity, and dynamics of cloud systems, cloud
reliability continues to be grand challenge. Despite extensive efforts [32 39 58 82). modern clouds
still struggle to detect, localize, and mitigate system failures effectively and timely. One essential reason is the existing human-centric reliability engineering practice
12 I3 human engineers are
the loop, responsible for decision making, e.g., which alerts to act upon, which tools to use, and
Stratus
Detection
Detection Agent
Toolset
Observability
Cloud
Control Plane
TelemetryCollectin
Localization
Dingnosis Agent
NL2Kubect1
RCA
Mitigation Agent
Linting: Notification
Logs
Services
Oracles
Metrics
Mitigation
Aleri clearance
Undu Agent
System health
System states
Failure
SRE Tasks
Workleads
Figure I: Overview of STRATUS, LLM-based multi-agent system for autonomous Site Reliability Engineering (SRE) of modern cloud services.
"Co-primary authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

which mitigation paths to choose. However, as failures are the norm in cloud-scale systems, it is increasingly costly for human suman-in-the-loop approaches to keep up with the scale of modern clouds. Autonomous Site Reliability Engineering (SRE), empowered by generative AI, is desired [19 35 62]
This paper explores agentic AI approach for autonomous Site Reliability Engineering (SRE) of modern clouds. We present STRATUS, an LLM-based agentic system that realizes autonomous SRE for cloud services through failure detection, localization, root-cause analysis, and mitigation. STRATUS differs from prior work on developing Al or agentic tools that assist human engineers [20]
36 37
81 (for data collection, summarization, and root-cause prediction). STRATUS aims to govern live production systems and mitigate failures, without human intervention (see $2).
/
STRATUS is multi-agent system that orchestrates specialized agents of detection, diagnosis, miti- gation, etc., each responsible for one or multiple tasks (Figure [ We chose a multi-agent design because (1) it specializes agents for SRE tasks with well-defined roles, (2) it offers customizability of individual agents and extensibility of the system, and (3) it makes it easy to reason about agent safety. STRATUS uses deterministic control-flow logic that orchestrates agents in state machine, while leveraging LLMs in data flows that make agents intelligent and creative.
Safety is key challenge of SRE agents (or any agentic systems that operate critical live systems). STRATUS must not worsen the state of the target system, when mitigating its failures; however, is hard to ensure that mitigation plan generated by the agent (or even by a human) is always successful, considering the dynamics and complexity of the system and imperfect AI models. In practice, the risk of "making things worse" is fundamental barrier deploying autonomous agentic techniques in high-stake systems. We formalize a safety specification of agentic SRE, termed Transactional Ro-Resgessoonn TNR. TNR constructs transaction semantics to ensure: (1) the agent's mitigation actions, if unsuccessful, can always be "undone," and (2) the agent keeps improving the system in terms of its health state, by undoing actions that worsen it (i.e., no regression). We show that TNR enables safe agent exploration and iteration, which effectively improves failure mitigation.
We describe implementation that realizes TNR using (1) state- machine based scheduling (for write exclusivity), ) (2) sandboxing (for confinement), and (3) an Undo Agent that realizes a system-wide undo operator [15]. We implement stack-baset based undo mechanism that tracks the agent actions with regards to specific system state and reverts them in the correct order needed. We also describe other notable implementations, e.g., agent tooling, bootstrapping, and termination.
We evaluate STRATUS on two SRE benchmark suites, AlOpsLab [19] and IT'Bench (35). STRATUS significantly outperforms state-of-the-art agentic solutions in terms of success rate of failure mitigation problems in AIOpsLab and ITBench, by at least 1.5 times across various models (GPT-40, GPT- 4o-mini, and Llama3). We show that TNR, which enables the undo-and-retry practice, is key to the ability of STRATUS in solving complex mitigation problems; it allows STRATUS to safely and iteratively explore new mitigation paths, while avoiding unrecoverable error states.
Summary. This paper makes the following contributions:
An attempt of an agentic Al system that autonomously mitigates system failures, towards fully autonomous Site Reliability Engineering of modern clouds.
STRATUS, multi-agent system that implements an end-to-end agentic SRE pipeline, from failure detection, localization, to mitigation, with system-wide safety properties.
Transactional Ron-Regresnion (TNR), a formalization of agentic SRE safety specification, and its implementation in STRATUS that enables safe agent exploration and iteration.
STRATUS significantly outperforms nate-of-thh--rt SRE agents in terms of success rate of various SRE tasks on two SRE benchmark suites (AIOpsLab and l'TBench).
The artifacts of STRATUS can be found at httpt:://giiiit thubb cww /xbbbblaabbbaaaaa uiuc/stratus
2 Background and Problem Definition
STRATUS aims to realize autonomous SRE which we define as an automated system on detecting failures, localizing failing components, analyzing the root causes, and mitigating the failures, with minimal human intervention. In other words, autonomous SRE systems like STRATUS imitate human engineers with agentic AI, and hence is fundamentally different from Al or agentic tools designed for assisting human engineers specific SRE tasks [20 73 81). A key differential feature of STRATUS

---

is to automatically mitigate failures, beyond summarizing failure information and predicting root cause categories as in prior work. Mitigation poses unique challenges for autonomous SRE: unlike detection and diagnosis tasks that only need to observe the system, mitigation needs to change the mmiiiattion actions must not cause new (nested) failures and must not drive the system to worse state (i.e., no regression).
system state. How to ensure safety of failure mitigation is a key challenge-mitigationn
We model a target cloud system an environment & characterized by:
A set of states S {s}. including a crash state 1 where the system is completely unavailable;
A severity metric f=(*): U { - NU {oo} that represents the severity of an error state s".
An error state can be caused by different types of faults (root causes) [10], including software
bugs [2 misconfigurations (75). and hardware issues (29 44 46 47]. The severity of the error for a given state is measured as a weighted sum of the sets of alerts (A), violations of service-level agreement (SLA) (V), and system capacity loss (unhealthy nodes) (L); it is formally defined as:
(ss) W1 |A| + W22 [V] + W/ L| where > 0, ((L) oo. If the entire system is unavailable,
then is in the state 1, where (L))= co. Just like human SRE engineers
[12]. STRATUS addresses following SRE tasks in order to resolve the error in the system state:
Detection. SRE must promptly detect production failures via logs, traces, and other telemetry data; detecting failures is the first step to prevent incidents.
Localization. SRE must localize the faulty components so that they can be isolated to minimize the blast radius and impacts.
Root Cause Analysis (RCA). SRE shall identify the root causes of the failures in order to repair the faulty components (e.g., fixing bugs and misconfigurations).
Mitigation. SRE must mitigate the failures to prevent their propagations that lead to production
incidents and service outages.
Note that RCA is not on the critical path of mitigation. In practice, RCA is typically done offline, while mitigation directly determines service availability [69]. A successful mitigation may not need to know the failure root causes. For example, mitigating faulty machine can be done by migrating deployed jobs to a healthy machine (70), without knowing the faulty component; mitigating a faulty software component can be done by rebooting it [17 56 reverting recent changes [14).
3 STRATUS Design
STRATUS uses a multi-agent system design that orchestrates multiple specialized agents A = {a;} (a, refers to an agent) to interact with & and mitigate any system failure represented by an error state s". Our implementation currently uses four agents A {ap, ag, aM, au))
ap (Detection agent): Observes & to identify failures and establishes the initial error state sh-
ag (Diagnosis agent): Observes & (sf and telemetry) to determine the root cause(s) of the detected failure; its output is primarily analytical. ag is responsible for both localization and RCA.
aM (Mitigation agent): Takes the diagnostic output from aD and ag. It is responsible for (1) devising a high-level mitigation plan, (2) decomposing the plan into sequence of concrete mitigation actions, and (3) executing each action by issuing commands through agent tools with Agent-Computer Interfaces (ACI) (77) (see $
ay (Undo Agent): Executes an undo sequence on if a transaction executed by aas is aborted.
Agents interact with the environment using actions from a defined Action Space through ACI:
Arend: Read-only commands that do not change the state of E (e.g., kubectl get
Awiit: Write commands that can mutate the state of & (e.g., kubectl apply).
Aundo: A specialized sequence of commands executed by ay to undo the effects of write commands.
Agents ap, aG are restricted Aread Agent a,s executes actions from Awiite as well as Aread for reading system states and checking pre/post conditions. Agent au executes actions effectively in Aundo (which internally uses Awrite commands restore a prior state).
kubectl is the command line tool of Kuberneees, facto cloud platform that manages cloud systems 16
3

---

The system state of & changes based on State-Transition Relations (see (26)). R S x Awrite U {1}. The undo is performed by an operator U + realized by av.
3.1 Safety Specification
STRATUS must guarantee safety. However, an agent's action, whether due to an imperfect under- standing, a planning flaw, or hallucinations of the generative model, could inadvertently worsen an already degraded system state (μ($E) > 0). In our experience, this risk of "making things worse" is main barrier to deploying autonomous agentic techniques in high-stake systems. How can we grant autonomous agents the authority to execute sequences of potentially state-changing commands (e.g.. via kubectl) for critical tasks like failure mitigation, without risking worse outcomes?
We formalize safety specialization of agentic SRE systems like STRATUS, termed Transactional Non- Non-Regression (TNR), atop the classic transition abstraction [28]. TNR ensures that (1) the agent's mitigation actions, if unsuccessful, can always be "undone", and (2) the agent keeps improving the system state (i.e., the severity metric decreases monotomically, with no regression).
3.1.1 Assumptions
We state the following assumptions, which are enforced by our implementation
A1 Writer Exclusivity (Agent-Lock or A-Lock). At most one writer agent (a:M or ouu) is scheduled execute commands that mutate system states time and have exclusive access; reader agents (OED, OG) do not mutate state relevant to μ.. A-Lock is a readers-writer lock.
A2 Faithful Undo. The undo operator U, when invoked by ag on 8post after an abort decision, restores the checkpointed Spre exactly, i.e., U(ppou) = Spre-
A3 Bounded Risk Window. The length ls (the number of commands) of any transaction executed by am is bounded by system-wide threshold K (1 k < K). This limits the duration the A-Lock is held and the complexity of any single transaction.
3.1.2 Transaction Semanties
We construct transaction semantics atop the above assumptions, where transaction of length
(1 < k < K) is sequence of read or write commands 7 = (a1,..., "oi,......))) ,.)) E (Awriee Areza) A
transaction encodes mitigation plan executed by a&M while holding the A-Lock (see A1):
RI Checkpoint: a& records the entry state Spre before aj is executed.
R2 Execute: aM a1 ... dg sequentially. Let Spost be the state after as completes (or 1 if any aj (1Sj<k) causes a crash).
R3 Commit/Abort rule: ag evaluates: Commit if Spost # 1 and p(spon) I ((spec;; otherwise, abort by instructing ay to invoke U once. Per A2, this restores Spre exactly.
An aborted transaction leaves no trace on externally visible state transitions. We call the internal sequence of states visited during transaction execution, Spr "i, s()) _2, 22, Spost. and their corresponding severity metrics the hidden p-path. Only ((pprr) and, if commits, (sposs) are part of the externally visible state transitions that define the system's 's trajectory viewed by other agents.
3.1.3 Transactional Non-Regression (TNR)
Let s6 be the error state observed by opp when an failure occurs, with an initial severity = (a(5))- Lemma 3.1. Every state externally visible state transitions (i.e., 8 = sy or s is a state immediately following read-only action by aD ag. is state following the completion (commit or abort) of a transaction by ( satisfies p(s) S b.
Proof Sketch. The proof is by induction on the sequence of externally visible states, where an externally visible state is one observed outside of any active transaction T'.
Base Case. For the initial state st (established by op when the failure is detected), ((si) by definition. Thus, the invariant [(s) = pa() < b holds for sñ-

---

Inductive Step. Assume that the invariant [(s: (uss)) S b holds for an externally visible state We show holds for the next externlly visible state 8:+1-
If 8,41 results from read action (by op or (x)) on 8;: These agents execute actions from Arad which do not mutate state. Thus, 81+1 8,, and ((${+1)) (u(s;) S b. The invariant holds.
If 94+1 results from the completion of T executed by a starting from 8, (so Spre = st)::
The T's execution (a)....,00) (22) and its commit/abort decision occur atomically with respect to other writer agents due A-Lock (AI).
If the commits, the new externally visible state is 8441 = spost- The commit condition requires p(sposs) I ju($pre). Since Spre 5; and p(s;) S 6 by the inductive hypothesis, it follows that ((8pos)) ) Σ b. The invariant holds for 8i+1-
If the T aborts, ay restores Spre with Faithful Undo (A2). The externally visible state is 8i+1 = Spre (which 8;). Since (((s;) S b by the inductive hypothesis, the invariant holds for 8441- Note that the effects of 01,-... 0g by am are not part of the externally visible sequence.
Thus, any state in the externally visible sequence satisfies p(s) S b.
Table T gives an example that contrasts the hidden and visible severity trajectories for four common mitigation plans, orchestrated by STRATUS.
Table 1: The TNR safety guarantee under four failure mitigation plans.
Mitigation
TNR actions (by arr)
Hidden p-path
Commit?
Visible μ
Node drain/rebalance
cordon, evict, scale
18-9
12 12-+9
Rolling upgrade
scale 0, patch, scale
15-+22+11
15-+11
Bad image attempted
scale 0, patch(bad), scale
15-24-300
x
15-+15
/ 1 otherwise
Single hot-fix (K=1)
apply hotfix
15-+x
S 15
3.2 Implications
TNR, which states that "severity never increases over the initial baseline b in the observable trace of states," is instance of the Alpern-Schneider safety property [6.8). This safety property is not merely a theoretical construct; it forms foundation for building trustworthy and effective Al agents that interact with critical live systems. For any agentic system employing an ACI to enact changes, especially those using LLMs for complex reasoning and planning (like STRATUS's a the risk of generating sub-optimal flawed action sequences non-trivial.
By ensuring that the potential "daaage" of an agent is capped (it cannot make the system observably worse than So), TNR provides powerful tool for safe exploration and iteration. This allows the agent to attempt ambitious, complex repairs, learn from outcomes (via observing 1 post- cost-transaction), and adapt its strategy without the risk of digging deeper hole.
Moreover, preventing agent-driven escalations beyond the baseline b makes the liveness property of eventually reaching healthy state tenable 87 It empowers robust, reliable Al agents capable of sophisticated, autonomous interventions the real world, fostering the development of AI systems that can not only act, but also act safely through recoverability.
3.3 Extension
TNR currently assumes concurrent writer agents and has concurrency control. Writer agents are strictly serialized with the readers-writer lock (A1). Our current deployment model uses one STRATUS instance per cloud system (each system managed in separate namespace).
STRATUS can be extended to support multiple concurrent writers by evolving A-Lock into a fine- grained, resource-aware concurrency control system. The idea is to allow multiple mitigation agents to execute transactions concurrently, provided their intended actions do not conflict over shared resources. This can be achieved through Coordination Controller that functions distributed lock manager. Each mitigation agent first comes up with plan P., which associated with a set
'Hidden values arise inside the transaction during execution by aus and are not visible to other agents, nor do they form part of the externally visible state sequence against which the lemma proven.
5

---

of resources, R(Pi), containing the unique identifiers of all system components its write actions will mutate. The controller maintains a global set of locked resources, Riccked, which is the union of resource sets for all currently executing plans, Rlacked = Jeexxcuuing R(P;). A newly submitted plan P: is admitted for execution if and only if its resource set is disjoint from the global lock set: Admit(P.) = R(P.) Rlocked = Ø.
Upon admission, the plan's resources are atomically added to the global lock set (Riockedd &- Riacked U R(P.)), effectively acquiring distributed lock them. The agent then executes its transaction under the original TNR semanties, guaranteeing that the severity metric does not increase: ( S a u(Spre). Once the transaction completes (via commit or abort). its resources are released: Rlocked f Rlocked \\ R(P.). This mechanism upholds the "Writer Exclusivity" (A1) at the resource level, ensuring that the safety properties of TNR hold for each concurrent transaction. Furthermore, this strict resource isolation can prevent read-after- write hazards, as an agent is guaranteed a consistent view of any resouree not explicitly locked by another transaction.
4 Implementation
Detection
Diagnosis
STRATUS is implemented using the Crew multi-agent framework ( The agents are orchestrated via state-machine based control-flow logic (Figure 2 STRATUS is an extensible framework. The agents are loosely coupled and can be independently developed and customized. We currently use off-the-shelf LLMs such as GPT and Llama models without fune-tuning. One can also add new agents or replace existing agents. The control plane can also be customized (e.g., using a different state machine) as is decoupled from the data plane of the agents (each agent's data-flow logic).
Agent
Agent
Undo Agent
Mitigation
Agent
Figure 2: The state machine
based control- logic.
4.1 Realizing TNR in STRATUS
In STRATUS, the assumptions AI-A3 are enforced as safety invariants with the following mechanisms. Writer Exclusivity. First, sandboxing- based confinement is implemented for each agent. The detection and diagnosis agents are not allowed to issue any commands that may mutate system states (the confinement rules detailed in the appendix). Second, the state machine ensures mutual exclusion when a writer agent (the mitigation agent or the undo agent) is scheduled to run, i.c., the sequences of the actions of the writer agents strictly serialized. We also instruct the agent to dry-run the commands (kubectl --dry-rum whenever applicable to simulate command execution and identify errors without altering the system state.
Faithful Undo. STRATUS ensures that every agent action has a corresponding undo operator; otherwise, the action is not allowed. STRATUS rejects destructive actions which cannot be recovered, or turns them into recoverable actions through the agent tools (e.g.. a file deletion is replaced by
moving the file to a backup volume, which is recoverable). A common pattern of the undo operator is system-star state rollback. Modern cloud platforms such as Kubernetes [6]] Borg (7 Twine (68), and ECS [45) are implemented based on the staterrcconciiiation principle 655 66 with declarative interfaces 26 67 Rollback in state-reconciliation systems can be implemented by recording the
state changes and later reconciling to the recorded states.
Push
Nacetapp (4:12:45
Tonestamp
STRATUS implemented stack-based rollback mech- anism, as exemplified by Figure 3 The stack-based mechanism allows fine-grained rollback the related system resources (represented as state objects in Ku- bernetes) instead of the system-wide state. Our current undo agent ay mechanically walks over the stack and performs undo, and thus is arguably not intelligent. The rationale of keeping ay an independent agent is to integrate more advanced undo policies and mechanisms. For example,
14:1541
(delete pod, Pod, State)
14:12:24
(potcch deployment geo, Deployment, State 2)
14:13:5
14:10:19
(patch clep eyoloyment L geo, Deployment, State
14:16:1
Figure 3: An example of the action stack used for reconciliation-basee undo.
exploring learning- based rollback policy where the LLMs are given the choice of undoing a subset of the actions in the stack (instead of always rolling back to the start state).
Bounded Risk Window. The implementation is trivial. However, the window size has an impact on mitigation effectiveness. We set K to 20 based on an empirical analysis (in appendix).

---

Limitations. From an implementation perspective, realizing perfect undo for all conceivable state changes in complex environments like cloud systems remains a practical challenge (e.g.. involving application-speciie states and external interactions). Certain operations may require more sophisti- cated compensation logic not covered by a simple (Spus) 8pre- Fortunately, modern cloud-native systems are typically equipped with an operator program that follows the iate-reconniiiationn princi- ple [21 57], which can be leveraged by STRATUS. Moreover, our rule-based confinement may not be perfect to capture all destructive actions. Recent work advanced guardrails can help improve the confinement (see Related Work in (6).
The principle of segmenting agent interventions into transactionally bounded stages, each adhering to non-regression with respect to a measurable system health metric (μ), significantly de-risks autonomous operation. It transforms the problem from hoping an entire complex plan is perfect to verifying the safety of smaller, manageable steps.
4.2 Other Notable Implementations
Agent tools. We develop tools that enable agents to interact with the environment using natural languages, following the Agent-Computer Interface (ACI) principles (77). Specifically, we developed two sets of tools: (1) observability tools for agents to query different kinds of observability data such as state objects (maintained by the cloud platform like Kubernetes), system logs, distributed traces, performance metrics, etc; We use LLMs to summarize the observability data instead of directly feeding a large volume of data the agents, as suggested by recent work (20). (2) command-ling -ine tools that allow agents to execute commands and change system states using natural language, e.g.. NL2Kubect1 takes descriptions natural language and generates concrete kubectl commands. The confinemen described in 44.1 is implemented as the command-ling line tool wrappers. STRATUS's design supports hybrid operations where writer agents can request human approval before execution, with interceptable action stacks and tool interfaces.
Bootstrapping (Where shall the diagnosis agent start?) The multi-agent design makes it easy to specialize agents with different roles. We boost the diagnosis agent ag with bootstrapping technique that helps ag localize the failure region. In cloud systems, failures are propagated along request-response paths [40 69). which is reflected by distributed traces [88 64). STRATUS localizes the faults by constructing the call graph and establishes an initial fault localization hypothesis to guide the diagnostic process. Despite the simple idea, the Bault-ocaalization based bootstrapping is important for large-scal scale systems and massive volumes of observability data.
Termination (When shall the agents stop?) Agents need to correctly determine whether the target failure is successfully resolved stop its actions; a perpetual agent could cause unintended effects, destabilize the system, or waste computing resources. STRATUS integrates a structured validation-an ando- termination approach. After each diagnosis or mitigation procedure (a series of actions), STRATUS assesses system health based on three kinds of oracles: (1) alerts: whether the alert that reports the target failure is cleared, (2) user requests: whether user requests can be successfully returned (c.g.. no SXX or 4XX HTTP responses), and (3) system health: whether system components (e.g., pods and volumes) are running in healthy states. These oracles act as weak oracles individually, each capturing a partial view of system health. STRATUS combines multiple oracles to form a stronger oracle. If all oracles pass, ous concludes a successful mitigation and terminates.
5 Evaluation
We evaluate STRATUS on two state-of-the-ar benchmarks, A1Op- sLab [19] and ITBench (35). Both benchmarks provide a live, arena-like environment, where AI agents are asked to resolve problems; each problem encodes a failure in emulated cloud sys- tems 23 as exemplified by Figure 4 We focus on STRA- TUS's s ability mitigate the failures (which has dependencies with detection and localization); we also report results of the other tasks, including detection, localization, and RCA. Each benchmark takes STRATUS 15-20 machine hours to finish.
Client
Memcached
Load Baluncer
MongoDB
NGINX
Post Storape
Compose
MongoDB
Timeline
Redis
Text "Froe Failed
fcstioaa ssc- exs--nanion servie!!
User Mintion
KRS TuretPon pureriiii
Dunhase
prc.
Dirss
MongaDii
Memeached
Baseline Agents. We compare STRATUS with the reference agents offered by AlOpsLab and ITBench, referred to as AOL-
Figure 4: An example problem.
7

---

agent and ITB-agent respectively, For AlOpsLab, we also compare STRATUS with two other agents, ReAct and Flash, included in the benchmark (19). We fuel these agents with different LLMs, includ- ing GPT-4o, GPPT-4o-mni, and Llama 3.3. The L.L.M temperature is set to 0 to make the results more deterministie.
We measure agentic systems using metrics: Success Rate (the percentage of problems being success- fully solved), average time (in seconds), the number of steps solve a problem, and monetary cost in US dollars (calculated based on token consumptions). We also profile the memory footprint for one heavy problem across different agents in STRATUS.
Failure Mitigation Effectiveness. As shown in Table 2 STRATUS (GPT-40) significantly outper- forms state-of- of- the-art SRE agents mitigation problems in both AIOpsLab and ITBench. Concretely, it successfully solves 69.2% (9/13) and 50.0% (9/18) mitigation problems in A and ITBench, respectively. This leads improvements the success rate by 1.5X and 5.4X over the second best performing solution (AOL-agent and aTB-agent), respectively. The advantage of STRATUS over the other SRE agents is consistent with different models, including both GPT-4o-mini and Llama 3.3.
We carefully inspect the mooleem-solving trajectories of the evaluated agents. For the mitigation problems solved by STRATUS in AIOpsLab, STRATUS correctly mitigated the triggering conditions or the root causes of the failures. In eight ITBench problems, STRATUS exploits an observation that the injected faults do not persist, after the failing pods restarted (the fault injector cannot recognize the original pod); thus, STRATUS tends to restart the failure pods one by one and thus solves the problem. Such mitigation strategy may not work in reality, especially for failures caused by persistent faults (e.g., misconfigurations and hardware defects).
Table 2: Effectiveness of STRATUS in solving mitigation problems of AIOpsLab [19] and ITBench [35]. ITB-agen numbers are taken from (35) (*-": unknown). The "S" column reports the average cost (in USD) of running the agent over all the mitigation tasks in each benchmark.
(a) ATOpsTab (13 Mitigation Problems)
(b) ITBench (18 Mitigation Problems)
Agent
Suce.
Time (s)
Steps
$
Agent
Succ.
Time (s)
Steps $
ReAct (40)
23.1 1%
46.0
23.0
0.112
ITB-ttent (4o)
9.2%
251.7
Flash (4o)
38.5%
154.0
23.1
0.150
ITB-agent (Ilama)
5.7%
440.8
AOL-agent (4o)
46.2%
223.3
21.7
0.206
STRATUS (4o)
50.0%
1720.8
115.7
6.11
AOL-agent (mini)
7.7%
58.9
22.7
0.003
STRATUS (mini)
19.4%
3874.9
468.9
9.38
AOL-agent (Ilama)
15.4%
98.2
13.0
0.037
STRATUS (Ilama) 28.0%
2566.6
160.3
0.76
STRATUS (4o)
69.2%
811.9
46.3
0.877
"46" refers GPT-4o (got-4o-2024-08-06)
STRATUS (mini)
23.1%
3557.9
125.7
0.036
"mini" refers to GPT.4o mini (gpt 4n mini 2024 07. 18)
STRATUS (Ilama) 23.1%
1486.9
71.8
0.360
"Ilama" refers to Llama 3.3 (llama 3.3. 706 instruct)
STRATUS overall takes longer time to solve problems; for the hard problems, STRATUS retrics multi- ple times to solve them-the retry is enabled by the Transactional No-Regression safety guarantee. In comparison, the other agents do not have the safe retry capability. In fact, they provide no safety guarantee over their mitigation actions- for the failures they fail to mitigate, these agents can leave the system in worse states than the original error state. For the same reason, STRATUS spends more in dollar amount, but the amount is significantly cheaper than human cost.
Biddo-and-Retryy The TNR safety is crucial to STRATUS's effec- tiveness in solving complex mitigation problems. Table 3 shows this point through an ablation study of STRATUS (GPT-4o) (more detailed ablation analysis can be found in the appendix). We compare STRATUS (GPT-4o) with two variants: (1) No retry: Only attempting one mitigation path, and (2) Naive retry w/o undo: exploring a new mitigation path if the previous mitigation path was not successful; however, the new exploration is conducted directly from the ending state of the last attempt, instead of rolling back to the original state as in STRATUS.
Effectiveness of based
We see that (1) the ability of retrying a different mitigation plan is important-oni in two problems, the first mitigation plan generated by STRATUS successfully mitigates the failure, and (2) the ability to undo and rollback the system state is critical- new mitigation from error state left by failed attempt can hardly succeed (only one new problem solved in this way). Intuitively, failed mitigations further worsen the system states, making the system harder and harder to save. TNR prevents such
8

---

failure patterns and thus is essential to mitigation. Figure shows the probability density function of the number of retries (per problem) done by the STRATUS (GPT-40) in AlOpsLab mitigation problems. STRATUS sets limit of nine retries. In 80+% of the problems, STR retries at least once; 30+% of the problems, STRATUS retries no less than five times.
Table 3: Ablation analysis on TNR-enabled undo-and-retry of STRATUS (GPT-4o) using the mitigation problems in AIOpsLab, which shows the importance of TNR safety.
25
20
Probability
15
on 10
Ablation
Succ. Rate
Time (s)
Cost (S)
Number of Retries
STRATUS (4o)
69.2%
811.9
0.877
No retry
15.4%
72.6
0.163
Figure 5: Probability density of the retry times per problem.
Naive retry w/o undo
23.1%
1221.5
0.929
Effectiveness on Detection, Localization, and Root Cause Analysis (RCA). We also evaluated the effectiveness of STRATUS on other SRE problems, in comparison to other SRE agents. Table 4 shows the results for problems in AIOpsLab. Note that the number of problems in the categories different. STRATUS outperforms the reference agents both detection and localization problems. Specifically, STRATUS with Llama and GPT-4o achieves over 90% of success rate for detection. Localization and RCA problems, which require the agents to pinpoint the roof-cause components (e.g., the pod) and the fault type (e.g., misconfiguration solutions; STRATUS (GPT-4o) has the highest success rate for localization problems and is second highest for RCA problems. Note that RCA not a prerequisite of failure mitigation (see 2
bug), present greater challenges to agentic
Table 4: Effectiveness of STRATUS in solving other SRE problems in AIOpsL .ab. The problems in AlOpsLab are independent; there are total 86 problems of four SRE types ($2).
Detection (32 Problems)
Localization (28 Problems)
RCA (26 Problems)
Agent
Suec. Time (s)
$
Succ.
Time (s)
Suce.
Time (s)
ReAct (4o)
87.5%
33.2
0.086
26.8%
$9.6
0.328
23.1%
28.5
0.065
Flash (4o)
59.4%
30.7
0.013
39.3%
165.0
0.190
26.9%
30.5
0.019
AOL-agent (4o)
62.5%
14.4
0.061
46.9%
34.8
0.083
38.5%
12.3
0.061
AOL-agent (mini)
25.0%
43.0
0.002
9.5%
34.1
0.001
7.7%
57.7
0.003
AOL-agent (Ilama)
84.4%
19.8
0.019
32.1% 1
40.5
0,018
30.8%
13.8
0.014
STRATUS (4o)
90.6%
48.4
0.118
51.2%
65.3
0.126
34.6%
39.6
0.068
STRATUS (mini)
78.1%
34.4
0.010
25.0%
37.2
0,013
30.8%
279.0
0.007
STRATUS (Ilama) 93.8%
50.0
0.111
36.3%
90.5
0.112
26.9%
60.2
0.095
Note that the relatively low RCA success rate across other tasks for STRATUS (and all the other agents) is largely attributed to the ambiguous evaluation metrics used by AIOpsLab. AIOpsLab measures RCA success by matching the agent's s output against pre-defined category labels; however, some labels are ambiguous and not mutually exclusive. For example. for port misconfiguration problem, AIOpsLab defines the root cause to be "Virtualization" (as the layer) and "Misconfiguration" (as the fault type). STR ATUS answers the root cause to be "Application" (as the layer) and "Network/Storage Issue" (as the fault type). Arguably, STRATUS's answer is more precise in describing the root cause-a port is more of an application-level issue than "Virtualization" since there is no traditional hypervisor-like virtualization layer in AlOpsLab. However, STRATUS is considered incorrect for this problem in AlOpsLab due to such label mismatch. As ongoing work, we are actively investigating new metrics designs to improve the benchmarks for SRE agents.
Memory usage. To understand the memory usage of STRATUS, we profile STRATUS running heavy mitigation problem (misconfig_app hww.riigaapphhtteeeeee hotel_res mi tigationn-11 in AIOpsLab), which takes 13 steps to resolve. The dominant memory cost (over 530 MB) comes from loading the agent runtime and its dependencies (e.g., CrewAl and benchmark tooling). The Detection agent uses the most memory as it collects and analyzes raw telemetry data (e.g., using pandas for trace analysis). The Mitigation agent consumes the largest context memory (31.02 1.003 KB) as accumulates trajectories across steps, whereas the Undo agent uses the least (5.79 KB) due its simple rollback operations.

---

6
Related Work
AI and Agentic Research for SRE. Applying AI/ML techniques to solve SRE problems has long history of research, as AI/ML has the potential to accommodate the large scale of modern systems and infrastructures (e-g., .. the clouds) 48 84] well massive volumes of observability
data (38] Specialized AI/ML techniques are designed for concrete SRE tasks, such as failure 54
79,, triage (59 60). and diagnosis 24 33 42 49 Recently, generative AI models. as well as agentic approaches, are increasingly used to further generalize and automate these SRE However, the aforementioned efforts focus on building tools to assist human Differently, we aim for autonomous SRE. A differential feature is failure mitigation. Prior work mostly focuses on helping human engineers to detect, understand the failure or recommend potential mitigation documents, where mitigation is the main goal of STRATUS
detection 52
tasks 20 36 78
engineering (see 2
Safety of Agentic AI Systems. Safety is an emerging concern of agentic AT systems 22 50 63
Prior work focuses on implementing safety guardrails [2] 21345111 72), which aim to prevent agents from producing or acting upon harmful inputs and outputs. often through prompt-level or static analysis. However, such preventative approaches are limited in environments like cloud systems, where the system state evolves over time and side effects emerge dynamically only after actions are executed. For example, AutoSafeCoder [5T] mitigates code vulnerabilities via static checks, but such methods cannot anticipate runtime failures or cascading effects that depend on temporal system dynamics. In addition, few works provide guarantees on recovery. For example, TrustAgent (31) proposes the Agent- Constitution framework that address agent safety. However, it does not provide guarantees on the safety of the agent action or any recovery from unsafe actions. For SRE, this could allow failures
to regress into a worse situation due to unsafe agent actions.
STRATUS addresses this challenge by enabling safe exploration with Transactional Non-Regression (TNR), TNR allows agents to explore multiple action sequences safely and iteratively, giving the agent more opportunities to mitigate failures. Concurrent work has also started integrating transactional primitives into LLM-based frameworks [18]. STRATUS extends this direction by enabling rollback of both the agent and the system state, combined with a continuous severity metric to assess action outcomes. This dynamic and fine- -rainegaiddd safety mechanism critical for real-world deployment of autonomous agents in evolving operational contexts like SRE.
Multi-agent Systems. Multi-agent has become common design pattern, with the development of many LLM--bsed multi-agent frameworks (1 30 41 74) and use cases [55] 76 83 85). Recent
work proposes multi-agent conversations (74] and debates 23 4
that encourage diverse thinking among agents. However, our experience shows that conversations and debates are not suitable for SRE tasks which require safety reasoning and timeliness of problem solving. Hence, STRATUS uses a deterministic state machine to coordinate the agents. Recent work also reports common failure modes of multi-agent systems (5380) We detail how STRATUS addresses them in the appendix.
Concluding Remarks
With generative Al and agentic technologies, autonomous SRE for modern clouds may no longer be pipedream. While we can expect new Al models and fine-tuning further improve mooblem-solving abilities of our agents, we assert that safety is the key challenge for agentic SRE systems to operate live production clouds. We also assert that agent safety has to rigorously defined and enforced as a first-class system design principle. This paper makes a first attempt to formalize TNR as useful safety specification of agentic SRE and realize it in our multi-agent SRE system. We envision and hope the work to inspire stronger safety properties and system-level support for agentic SRE systems.
Acknowledgment
We thank the anonymous reviewers for their insightful comments. The work is supported in part by an EilInolinois Discovery Accelerator Institute (IIDAI) grant and NSF CNS-2145295.
10

---

References
[1] Crew AI. hhttps://ww.crreall
[2] Guardrails AL. https://www, www.w.guardrailaa..coom
[3]
OpenTelemetry Astronomy Shop Demo. https///ppeeteekkkett...o//oo//p
[4] Swarm: Scalable infrastructure for multi-agent coordination. httpe/oooooooooooppp//gg/gttoiio
swarm 2024.
[5] AIMME,, T., GIOOSH, S., BANSAL, C., ZIMMERMANN, T., ZIJANG, X., AND RAJMOHAN, Recommending Cause and Mitigation Steps for Cloud Incidents using Large Language Models. In Proceedings of the 45th International Conference on Software Engineering (ICSE'23)
(Jul 2023).
[6] ALPERN, B., DEMERS, A. J., AND SCHNEIDER, F. B. Safety without stuttering. Information Processing Letters 23, (1986), 177-180.
[7] ALPERN, B., AND SCHNEIDER, F. B. Defining Liveness. Information Processing Letters 21 (1985), 181-185.
[8] ALPERN, B., AND SCHNEIDER, F. B. Recognizing safety and liveness. Distributed Computing
2, 3 (Sept. 1987), 117-126.
[9] ARMBRUST, M., Fox, A., GRIFFITHI, R., JOSEPH, A. D., KATZ, R. H., KONWINSKI, A., LEE, G., PATTERSON, D. A., RABKIN, A., STOICA, I,, AND ZAHARIA, M. Above the Clouds: A Berkeley View of Cloud Computing. Tech. rep., University of California at Berkeley, Feb. 2009.
[10] AVIZIENIS, A., LAPRIE, J.-C., AND RANDELL. B. Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Transactions on Dependable and Secure Computing (TDSC) 1, (Jan 2004), 1-23.
[11] BARROSO, A., HÖLZZE,, U., AND RANGANATHAN, P. The Datacenter as a Computer: Designing Warehouse-Scale Machines. 2018.
[12] BEYER, B., JONES, C., PETOFF, J., AND MURPHY, N. R. Site Reliability Engineering: How Google Runs Production Systems. O' O'Reilly Media, Inc., Mar. 2016.
[13] BEYER, B., MURPHY, N. R., RENSIN, D. K., KAWAHARA, K., AND THORNE, S. Site Reliability Workbook: Practical Ways to Implement SRE. O'Reilly Media Inc., Aug. 2018.
[14] BHAGWAN, R., KUMAR, R., MADDILA, C. S., AND PHILIP, A. A. Orca: Differential Bug Localization in Large-Scale Services. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI'18 18) (Oct. 2018).
[15] BROWN, A. B., AND PATTERSON, D. A. Undo for Operators: Building an Undoable E-mail Store. In Proceedings of the 2003 USENIX Annual Technical Conference (ATC'03) (June 2003).
[16] BURNS, B., GRANT, B., OPPENHEIMER, D., BREWER, E., AND WILKES, J. Borg, Omega, and Kubernetes. Communications of the ACM 59, 5 (May 2016), 50-57.
[17] CANDEA, G., KAWAMOTO, S., FUJIKI, Y., FRIEDMAN, G., AND Fox, A. Microreboot A Technique for Cheap Recovery. In Proceedings of the 6th USENIX Conference Operating Systems Design and Implementation (OSDI'04 04) (Dec. 2004).
(18) CHANG, E. Y., AND GENG, I.. Sagal.LM: Context Management, Validation, and Transaction Guarantees for Multi-Agent L.LM Planning. arXiv:2503.11951 (Mar 2025).
[19] CHEN, Y., SHETTY, M., SOMASHEKAR, G., MA, M.. SIMMHAN, Y., MACE, BANSAL, C., WANG, R., AND RAJMOHAN, S. AlOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds. Proceedings of the Conference on Machine Learning and Systems (MLSys 25) (May 2025).
[20] CHEN, Y., XIE, H., MA, M., KANG, Y., GAO, X., SHI, L., CAO, Y., GAO, X., FAN, H., WEN, M., ZENG, J., GHOSH, S., ZHANG, X., ZHANG, C., LIN, Q., RAIMOHAN, S., ZHANG, D., AND XU, T. Automatic Root Cause Analysis via Large Language Models for Cloud Incidents. In Proceedings of the 19th European Conference on Computer Systems (FuroSys'24 24) (Apr. 2024).
11

---

[21] DOBIES, J., AND WOOD, J. Kubernetes Operators: Automating the Container Orchestration Platform. O' RRellly Media, Inc., Feb. 2020.
[22] DONG, Y., MU, R., ZHANG, Y., SUN, S., ZHANG, T., WU, C., JIN, G., QI, Y.. HU, J.,
MENG, J., BENSALEM, S., AND HUANG, X. Safeguarding Large Language Models: A Survey. 2:Xiv:4406.02622 (Jun 2024).
[23] DU. Y., LI, S., TORRALBA, A., TENENBAUM, J. B., AND MORDATCH, 1. Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305. 14325 (May 2023).
[24] GAN, Y., LIANG, M., DEV, S., Lo, D., AND DELIMITROU, C. Sage: Practical & Scalable MI.- Driven Performance Debugging in Microservices. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS' 21) (Apr 2021).
[25] GAN, Y., ZHANG, Y., CHENG, D., SHETTY, A., RATHI, P., KATARKI, N., BRUNO, A., HU,
J. RITCHKEN, B., JACKSON, B., HU, K., PANCHOLI, M., HE, Y., CLANCY, B., COLEN, C., WEN, F., LEUNG, C., WANG, S., ZARUVINSKY, L.., ESPINOSA, M., LIN, R., LIU, Z., PADILLA, J., AND DELIMITROU, C. An Open-Source Benchmark Suite for Microservices and Their Harddware-Sofwarc Implications for Cloud & Edge Systems. In Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS' 19) (Apr 2019).
[26] GU, J. T., SUN, X., ZHANG, W., JIANG, Y., WANG, C., VAZIRI, M., LEGUNSEN, O., AND Xu, T. Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP'23) (Oct 2023).
[27] GUNAWI, H. S., HAO, M., LEESATAPORNWONGSA. T., PATANA-ANAKE, T., Do, T., ADITY- ATAMA, J., ELIAZAR, K. J.. LAKSONO, A., LUKMAN, J. F., MARTIN, V., AND SATRIA, A. D. What Bugs Live in the Cloud? Study of 3000+ Issues in Cloud Systems, In Proceedings of the 5th ACM Symposium on Cloud Computing (SoCC' 14) (Nov. 2014).
[28] HAERDER, T., AND REUTER, A. Principles of transaction-oriented database recovery. ACM Computing Surveys 15, 4 (Dec. 1983), 287-317.
[29] HOCHSCHILD, P. H., TURNER, P., MOGUL, J. C.,, GOVINDARAJU, R., RANGANATHAN, P., CULLER, D. E., AND VAHDAT, A. Cores that don't 't count. In Proceedings of the 18th Workshop on Hot Topics in Operating Systems (HotOS'2) 2 (June 2021).
2
[30] HONG, S., ZHUGE, M., CHEN, J., ZHENG, X., CHENG, Y., WANG, J., ZHANG, C., WANG, Z., YAU, S. K. S., LIN, Z., ZHOU, L., RAN, C., XIAO, L., Wu, C., AND SCHMIDHUBER, J. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In Proceedings of the International Conference on Machine Learning (ICMI. 244 (Jan 2024).
[31] HUA, W., YANG, X., LI, Z., WEI, C., AND ZHANG, Y. TrustAgent: Towards Safe and Trustworthy LLM-based Agents through Agent Constitution. arXiv:2402.00 01.400..1550 586 (Feb. 2024).
[32] HUANG, P., GUo, C.,, LORCH, J. R., ZHOU, L.,, AND DANG, Y. Capturing and Enhancing In Situ System Observability for Failure Detection. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI'18 18) (Oct. 2018).
[33] IKRAM, A., CHAKRABORTY S., MITRA, S., SAINI, S., BAGCHI, S., AND KOCAOGLU, M. Root Cause Analysis of Failures in Microservices through Causal Discovery. In Proceedings of The Thirty-Fifth Annual Conference Neural Information Processing Systems (NeurIPS'22) (Oct. 2022).
[34] INAN, H., UPASANI, K., CHI, J., RUNGTA, R., IYER, K., MAO, Y., TONTCHEV, M., HU, Q., FULLER, B., TESTUGGINE, D., AND KIIABSA, M. Llama Guard: LLM-based Input-Output Safeguard for Auman-Al Conversations. vxXiv:2312.0667 (Dec 2023).
[35] JHA, S., ARORA, R., WATANABE, Y., YANAGAWA, T., CHEN, Y., CLARK, J., BHAVYA,
B., VERMA, M., KUMAR, H., KITAHARA, H., ZHEUTLIN, N., TAKANO, S., PATHAK, D., GEORGE, F., WU, X., TURKKAN, B. O., VANLOO, G., NIDD, M., DAI, T., CHATTERJEE,
O., GUPTA, P., SAMANTA, S., AGGARWAL, P., LEE, R., MURALI, P., WOOK AHN, J., KAR,
D., RAHANE, A., FONSECA, C., PARADKAR, A., DENG, Y., MOOGI, P., MOHAPATRA,
P., ABE, N., NARAYANASWAMI, C., XU, T., VARSHNEY, L. R., MAHINDRU, R., SAILER,
12

---

A., SHWARTZ, L., Sow, D., FULLER, N. C. M., AND PURI, R. ITBench: Evaluating Al Agents across Diverse Real-World IT Automation Tasks. In Proceedings of the International Conference on Machine Learning (ICMI'25) (July 2025).
[36] JIANG, Y., ZHANG, C., HE, S., YANG. Z., MA, M., QIN, S., KANG, Y., DANG, Y., RAJMOHAN, S., LIN, Q., AND ZHANG, D. Xpert: Empowering Incident Management with Query Recommendations via Large Language Models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (CCE''24) (Apr. 2024).
[37] JIN, P., ZHANG, S., MA. M., LI, H., KANG, Y., LI, L., LIU, Y., QIAO, B., ZHANG, C.,, ZHAO, P., HE, S., SARRO, F., DANG, Y., RAIMOHAN, S., LIN, Q., AND ZHANG, D. Assess and Summarize: Improve Outage Understanding with Large Language Models. In Proceedings the Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (2023).
[38] KALDOR. J., MACE, J., BEJDA, M., GAO, E., KUROPATWA, W., O'NEILL, J., ONG, K. W., SCHILLER,, B., SIIAN, P., VISCOMI, B., VENKATARAMAN, V., VEERARAGHAVAN, K., AND SONG, Y. J. Canopy: An End-to-End Performance Tracing And Analysis System. In Proceedings of the 26th Symposimm on Operating Systems Principles (SOSP' 17) (Oct. 2017).
[39] LENERS, B., WU, H., HUNG, W.-L., AGUILERA, M. K., AND WALFISH, M. Detecting Failures in Distributed Systems with the Falcon Spy Network. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOP'11) (Oct. 2011).
[40] LI, A., LU, S., NATII, S., PADIIY,, R., AND SEKAR, V. ExChain: Exception Dependency Analysis for Root Cause Diagnosis. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI' 24) (Apr. 2024).
[41] LI, G., HAMMOUD, H., ITANI, H., KHIZBULLIN, D., AND GHANEM, B. CAMEL: Commu- nicative Agents for "Mind" Exploration of Large I Language Model Society. In Proceedings of The Thirty-Sixth Annual Conference on Neural Information Processing Systems (NeurlPS (Sept. 2023).
[42] LI, M., L1, Z., YIN, K., NIE, X., ZHANG, W., SUI. K., AND PEI, D. Causal Inference- -ased Root Cause Analysis for Online Service Systems with Intervention Recognition. In Proceedings of the 28th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD (2) (Aug. 2022).
[43] LIANG, T., HE, Z., JIAO, W., WANG, X., WANG, Y., WANG, R., YANG, Y., SHI, S., AND TU, Z. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing (EMNLP'S 24 (Nov. 2024).
[44] MANEAS, S., MAIDAVIANI, K., EMAMI, T., AND SCHROEDER, B. A Study of SSD Reliability in Large Scale Enterprise Storage Deployments. In Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST'20) (Feb. 2020).
[45] MELISSARIS, T., NABAR, K., RADUT, R., REHMTULLA, S., SHI, A., CHANDRASHEKAR,
S., AND PAPAPANAGIOTOU, I. Elastic Cloud Services: Scaling Snowflake's Control Plane. In Proceedings of the 13th ACM Symposium on Cloud Computing (SOCC'22) (Nov. 2022).
[46] MEZA, J., WU, Q., KUMAR, S., AND MUTLU, O. A Large-Scale Study of Flash Memory Failures in the Field. In Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS' 15) (June 2015).
[47] MEZA, J., Xu, T., VEERARAGHAVAN, K., AND SONG, J. A Large-Scale Study of Data Center Network Reliability. In Proceedings of the 18th ACM Internet Measurement Conference (IMC"18) 18) (Oct. 2018). 8).
[48] MIAO, G., MOSER, E., YAN, X., TAO, S., CHEN, Y., AND ANEROUSIS, N. Generative Models for Ticket Resolution Expert Networks. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD' 10) (July 2010).
[49] MICKENS, J., SZUMMER, M., AND NARAYANAN, D. Snitch: Interactive Decision Trees for Troubleshooting Misconfigurations. In Proceedings of the 2nd USENIX Workshop on Tackling Computer Systems Problems with Machine Learning Techniques (SYSML'077 (Apr. 2007).
[50] MIEHLING, E., RAMAMURTHY, K. N., VARSHNEY, K. R., RIEMER, M., BOUNEFFOUF,
D., RICHARDS, J. T., DHURANDHAR, A., DALY, E. M., HIND, M., SATTIGERI, P...
13

---

WEI, D., RAWAT, A., GAICIN, J., AND GEYER, W. Agentic AI Needs a Systems The- ory. arXiv:2503.00237 (Feb. 2025).
[51] NUNEZ, A., ISLAM, N. T., JHA, S. K., AND NAJAFIRAD, P. AutoSafeCoder: Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testing. arXiv:2409. 10737 (Sept. 2024).
[52] PALATIN, N., LEIZAROWITZ, A., SCHUSTER, A., AND WOLFF, R. Mining for Misconfigured Machines in Grid Systems. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '06) (Aug. 2006).
[53] PAN, M. Z., CEMRI, M., AGRAWAL, L. A., YANG, S., CHOPRA, B., TIWARI, R., KEUTZER, K., PARAMESWARAN, A., RAMCHANDRAN, K., KLEIN, D., GONZALEZ, J. E., ZAHARIA, M., AND STOICA, I. Why Do Multiagent Systems Fail? In Proceedings of The Thirteenth International Conference on Learning Representations (ICLR'25) (Mar. 2025).
[54] POTHARAJU, R., CHANN, J., HU, L., NITA-ROTARU, C.,, WANG, M., ZHANG, L., AND JAIN, N. ConfSeer: Leveraging Customer Support Knowledge Bases for Automated Misconfiguration Detection. In Proceedings of the 35th International Conference on Very Large Data Bases (VLDB' 15) (Aug. 2015).
[55] QIAN, C., LIU, W., LIU, H., CHEN, N., DANG, Y., LI, YANG, C.,, CHEN, W., SU, Y., CONG, X., Xu, J., LI, D., LIU, Z.,, AND SUN, M. ChatDev: Communicative Agents for Software Development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL'24) (Aug. 2024).
[56] QIN, F., TUCEK, J., SUNDARESAN, J., AND ZIIOU, Y. Rx: Treating Bugs As Allergies Safe Method to Survive Software Failure. In Proceedings of the 20th Symposium on Operating System Principles (SOSP 055) (Oct. 2005).
[57] RATIS, P. Lessons Learned Using the Operator Pattern to Build a Kubernetes Platform. In USENIX SREcon (Oct. 2021).
[58) REN, X. J., WANG, S., JIN, Z., LION, D., CHIU, A., XU, T., AND YUAN, D. Relational
Debugging - Pinpointing Root Causes of Performance Problems. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI'23) (July 2023).
[59] SHAO, Q.. CHEN, Y.. TAO. S., YAN, X., AND ANEROUSIS, N. EasyTicket: a Ticket Routing Recommendation Engine for Enterprise Problem Resolution. In Proceedings of the 34th International Conference on Very Large Data Bases (VLDB '88) (Aug. 2008).
[60] SHAO, Q., CHEN, Y., TAO, S., YAN, X., AND ANEROUSIS, N. Efficient Ticket Routing by Resolution Sequence Mining. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'08) 08) (Aug. 2008).
[61] SHETTY, M., BANSAL, C., UPADIYYYULA, S. P., RADIIKRISSINA, A., AND GUPTA, A. AutoTSG: Learning and Synthesis for Incident Troubleshooting. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE'22 22) (Nov. 2022).
[62] SHETTY, M., CHEN, Y., SOMASHEKAR, G., MA, M., SIMMHAN, Y., ZHANG, X.. MACE,
J., VANDEVOORDE, D., LAS-CASAS. P., GUPTA, S. M., NATH, S., BANSAL, C., AND RAMMOHAN, S. Building AI Agents for Autonomous Clouds: Challenges and Design Principles. In Proceedings of 15th ACM Symposium on Cloud Computing (SoCC'24) (Nov. 2024).
[63] SHINN, N., CASSANO, F., BERMAN, E., GOPINATH, A., NARASIMHAN, K., AND YAO, S. Reflexion: Language Agents with Verbal Reinforcement Learning, XXXv:2303.11366 (Mar. 2023).
[64] SIGELMAN, B. H., BARROSO, L. A., BURROWS, M., STEPHENSON, P., PLAKAL, M., BEAVER, D., JASPAN, S., AND SHANBHAG, C. Dapper, a Large-Scale Dis- tributed Systems Tracing Infrastructure.
http://research. wwweeseerchhggogle...om com/ http/rrchvee/
Sapers/dapper-2000 2onr-2210-11 1.0-1.110 pdf. Apr. 2010.
[65] SUN, X., LUO, W., GU, J. T., GANESAN, A., ALAGAPPAN, R., GASCH, M., SURESH, L., AND XU, T. Automatic Reliability Testing for Cluster Management Controllers. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI222) (July 2022).
14

---

[66] SUN, X., SURESH, L., GANESAN, A., ALAGAPPAN, R., GASCH, M., TANG, L., AND XU, T. Reasoning About Modern Datacenter Infrastructures Using Partial Histories. In Proceedings of the 18th Workshop on Hot Topics in Operating Systems (HotOS'21) (May 2021).
[67] SURESH, L., AO LOFF, J., KALIM, F., JYOTHI, S. A., NARODYTSKA, N., RYZHYK, L., GAMAGE, S.. OKI, B., JAIN, P., AND GASCH, M. Building Scalable and Flexible Cluster Managers Using Declarative Programming. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) (Nov. 2020).
[68] TANG, C., YU, K., VEERARAGHAVAN, K., KALDOR, J.. MICHELSON, S., KOOBURAT, T.,
ANBUDURAL, A., CLARK, M., GOGIA, K., CHENG, L., CHRISTENSEN, B., GARTRELL, A.,
KHUTORNENKO, M., KULKARNI, S., PAWLOWSKI, M., PELKONEN, T., RODRIGUES, A., TIBREWAL, R., VENKATESAN, V., AND ZHANG, P. Twine: A Unified Cluster Management System for Shared Infrastructure. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI'20) (Nov. 2020).
[69] TREYNOR, B., DAHLIN, M., RAU, V., AND BEYER, B. The Calculus of Service Availability. Communications of the ACM (Aug. 2017).
[70] VEERARAGHAVAN, K., MEZA, J., MICHELSON, S., PANNEERSELVAM, S., GYORI, A., CHOU, D., MARGULIS, S.. OBENSHAIN, D., PADMANABHA, S., SHAH, A., SONG, Y. J., AND Xu, T. Maelstrom: Mitigating Datacenter-level Disasters by Draining Interdependent Traffic Safely and Efficiently. In Proceedings of the 13th USENIX Sympositm on Operating Systems Design and Implementation (OSDI' 18) (Oct. 2018).
[71] VERMA, A., PEDROSA, L., KORUPOLU, M., OPPENHEIMER, D.., TUNE, E., AND WILKES,
J. Large-Scale Cluster Management at Google with Borg. In Proceedings of the 10th European Conference on Computer Systems (EuroSys' 15) (Apr. 2015).
[72] WANG, H., POSKITT, C. M., AND SUN, J. AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents. arXiv:2503. 18666 (Mar. 2025).
[73] WANG, Z., LIU, Z., ZHANG, Y., ZHONG, A., FAN, L.,, WU, L., AND WEN, Q. RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models. arXiv:2310. 16340 (Oct. 2023).
[74] WU, Q., BANSAL, G., ZHANG, J., WU, Y., LI, B., ZHU, E., JIANG, L., ZHANG, X., ZHANG, S., LIU, J., AWADALLAH, A. H., WHITE, R. W., BURGER, D., AND WANG, C. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. In Proceedings of the 1st Conference on Language Modeling (COLM (Oet. 2024).
[75] XU, T., ZHANG, J., HUANG, P., ZHENG, J., SHENG, T., YUAN, D... ZHOU, Y., AND PASUPATHY, S. Do Not Blame Users for Misconfigurations. In Proceedings of the 24th Symposium Operating System Principles (SOSP' 13) (Nov. 2013).
[76] XU, T., ZHANG, Y.-F., CHU, Z., WANG, S., AND WEN, Q. Al-Driven Virtual Teacher for Enhanced Educational Efficiency: Leveraging Large Pretrain Models for Autonomous Error Analysis and Correction. arXiv:2409.094 (Sept. 2024).
[77] YANG, J., JIMENEZ, C. R., WETTIG, A., LIERET, K., YAO, S., NARASIMHAN, K., AND PRESS. O. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Proceedings of the Thirty-eighth Annual Conference Neural Information Processing Systems (NeurIPS (Sept. 2024).
[78] Yu, Z., MA, M., ZHANG, C., QIN, S., KANG, Y., BANSAL,, C., RAIMOHAN, S., DANG, Y., PEI. C., PEI, D., LIN, Q., AND ZHANG, D. MonitorAssistant: Simplifying Cloud Service Monitoring via Large Language Models. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE'24) (July 2024).
[79] ZIIANG, J., RENGANARAYANA, L., ZIIANG, X., GE, N., BALA, V., XU, T., AND ZIOO, Y. EnCore: Exploiting System Environment and Correlation Information for Misconfiguration Detection. In Proceedings of the 19th International Conference on Architecture Support for Programming Languages and Operating Systems (ASPLOS') 14) (Mar. 2014).
[80] ZHANG, S., YIN, M., ZHANG, J., LIU, J., HAN, Z., ZHANG, J., LI, B., WANG, C., WANG, H., CHEN, Y., AND WU, Q. Which Agent Causes Task Failures and When? On Automated Fail- ure Attribution of L.LM Multi-Agent Systems. In Proceedings of the International Conference on Machine Learning (ICML 25) (2025).
15

---

[81] ZHANG, X., MITTAL, T., BANSAL, C., WANG, R., MA, M., REN, Z., HUANG,
H., AND RAJMOHAN, S. FLASH: A Workflow Automation Agent for Diagnosing
en-m/en-ussa wtwp://e//eeeeeseeeeeeeeeeeeeeee Recurring Incidents. https://wwas mi microsoft. http://www.iiiiooooooooooaaaeaea
http://muuuurreseeeaaa///iaaaaoo alash-a-workfllow a agentttttnnnn-tt-----ee-eannenee for diggooosiiig-rrr..nnnnniniiieeee
[82] ZHANG, Y., MAKAROV, S., REN, X., LION, D., AND YUAN, D. Pensieve: Non-Intrusive Fail- ure Reproduction for Distributed Systems using the Event Chaining Approach. In Proceedings of the 26th ACM Symposium on Operating System Principles (SOSP'17) (Oct. 2017).
[83] ZHAO. J., Zu, C.,, HAO, X., LU, Y., HE, W., DING, Y., GUI, T., ZHANG, Q.. AND HUANG, X. LONGAGENT: Achieving Question Answering for 128k-Tooken-Lorg Documents through Multi-Agent Collaboration. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing (EMNLP'28 224) (Nov. 2024).
[84] ZHOU, X., PENG, X., XIE, T., SUN, J., J1, C.,, LI, W., AND DING, D. Fault Analysis and Debugging of Microservice Systems: Industrial Survey, Benchmark System, and Empirical Study. IEEE Transactions on Software Engineering 47 (Dec. 2021), 243-260.
[85] ZHOU, Z., LIN, Y., JIN, D., AND LI, Y. Large Language Model for Participatory Urban Planning. / (Feb. 2024).
16

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We list our main contributions in the end of the Introduction, and the contributions are analyzed in Section and evaluated Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in context (see Section 3.2 and Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assump- tions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting- Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
17

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We state our assumptions in Section 3.1.1 and write down our proof's in Section 3.1.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed reproduce the main experimen- tal results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Section 4 discusses our implementation. The artifacts and instructions can be found at https: thub. com wwl./blbabuuutratttt
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
1. If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
2. If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
3. If the contribution is new model (c.g., large language model), then there should either be way access this model for reproducing the results way to reproduce the model (c.g.. with an open-source dataset or instructions for how to construct the dataset).
4. We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. source models, it may be that access to the model is limited in some way (e.g.. registered users), but should be possible for other researchers to have some path to reproducing or verifying the results.
case of closed-
18

---

Open access to data and code
Question:
Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Section discusses implementation. The artifacts and instructions can be found at https://github... wwtp:/oo///auuuusstttaeaaa
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (attps://nips.cc/pubbll http://oddSuummmsssoooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips. https////ipscccppuliccc ssi oisioninlcyy for more details.
guides
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yss]
Justification:
We explain our experimental setting and details in Section 5 and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The reported results are statistically significant. The experiments are run multiple times and the median numbers reported in the tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

---

The factors of variability that the error bars are capturing should be clearly stated (for exam- ple, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the bar is the standard deviation the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetrie error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer re- sources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: YYs]
Justification: We document the compute resources used in the appendix. We list the cost of using LLMs in our evaluation section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
Code of ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https///heerrpps.cccpppi//ttheee
Answer: [Yes]
Justification: All experiments are conducted on publicly available benchmarks. We do not anticipate any negative social impacts. Our agent is designed specifically for Site Reliability Engineering.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics. If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss broader socictal impacts the appendix.
Guidelines:
20

---

The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., dis- information, generating fake profiles, surveillance), fairness considerations (e-g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications. the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any data models, there is potential for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The foundation models used in our experiments are cited properly. Only open-source benchmark suites are used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our only new asset is our code, can be found at bttps::/giittub.. com cot/:lab-uuuucc stratus] We also provide scripts and a README file containing documentation repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: STRATUS is fundamentally an LLM-based multi-agent system for autonomous Site Reliability Engineering (SRE) of cloud services. The LLMs are central to the system's planning. reasoning, and interaction with the environment through natural-languee interfaces, making their usage both original and essential the system's 's methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httpe://aearrissccc//offrreeeese should or should not be described.
for what
23

---

Appendix
In the appendix, we provide additional analysis, detailed results, and more discussions.
Table of Contents
Confinement Rules
25
B Bounded Risk Window
26
Ablation Study on l'Beecch
27
Tool Usage
27
D.1 List Tools
27
D.2 Distribution of Tool Usage
28
E Oracle Usage
28
F Hyperparameter Sweep
30
Result Details
30
Heuristic Evaluation of STRATUS'S Multi-agent Design
31
An Example of STRATUS'S Trajectory
32
24

---

A Confinement Rules
To ensure safety and predictabilitty, each STRATUS agent operates under sandboxed confinement rules. As detailed in Table detection and diagnosis agents restricted to Read-Only role and are only permitted to observe the system using commands such as Tel eeleemtryCollee explicitly prohibited from executing any operation that alters the state of the system.
tools. These agents
To perform mitigation, STRATUS will need use the Writer role, as changing the system state essential for mitigation. For state-changing tasks, STRATUS performs command-level validation to verify that generated actions are syntactically correct and safe to run in production. Risky operations such deleting namespaces are prohibited.
Certain kubectl commands, such kubectl exec -it and kubectl edit, launch interactive shells or editors. These commands are prevented because STRATUS cannot participate in interactive sessions, which would cause execution to hang. Additionally, granting arbitrary shell access significantly increases the potential for unintended and unsafe behaviors. Therefore, we explicitly disallow all interactive commands.
Table 5: Agent confinement rules in STRATUS. shows the allowed sandboxing rules according to the agent's role, while shows the blocked actions enforced by STRATUS.
Category
Subcategory
Confinement
Example(s)
Noo-mutating commands only
kubectl get. kubectl describe
Read-Only Agents
kubectl logs,
Agent Role-Based
TelenetryCollection
Sequential execution
Only serialized state moditications allowed
Writer Agents
unly
Command-Leve Confinement (Blocked for all agents):
Destructive Operations
Namespace deletion
kubect1 delete namespace my-ns kubectl debug pod/fon kubect] edit deployment//bar kubectl apply -f
debug.
Interactive
Kubectl Subcommands
edit
Input from sidia
pattern
Kubectl Flags
Interactive Terminal
-stdin, -tt.. -it
kubectl attach -tty pod/foo kubect. -it pad/bash
Pipelines
Pipe operator 68. Il.;
kubect1 get pods grep Running kubect] get pods & echo success kubectl delete pod 50ubbett get pods name)
Compound Commands
Command Substituionn
$C SC...), backticks
Shell Syntax
if, for, while, until, Function definitions
if [ ... 1; then ...: for ...: do ...: done nyfunc() {
Flow Control
Shell Functions
Confinement Examples
Example (Input from stdin):
T. STRATUS:
kubect1 apply -f
2. Confinement:
"Stdin redirection is not allowed."
Example (Interactive Terminal):
1. AOL-agent
kubectl exec it www.-ratebbbff577mmasa -n test-hotel-reeervee Hang there until timeout.
-- mongo
2. AOL-agent:
I. STRATUS:
kubectl exec -it mongodb wtw.cc-rrat--bff477--7111-. -n best-hotel-reservare "Interactive flag detected: -it. Such commands are not supported."
mongo
2. Confinement:
To further enforce safety, STRATUS uses static linting to validate command syntax before execution. This validation includes rejecting shell pipes, compound commands (e.g., using & I. or semicolons), and other complex shell constructs. These disallowed two reasons: (1) there are exponentially many arbitrary shell expressions that T STRATUS may generate; comprehensively verifying the safety
25

---

of all possible shell expressions is not realistic. On the contrary. individual kubect1 invocations can be easily checked. (2) we require the agent to operate incrementally, issuing and validating one command at time, to work effectively with our safety infrastructure and confinement layers.
When supported, STRATUS employs Kubernetes' dry-run mechanism produce the effect of command without persisting changes. In this mode, the Kubernetes API server processes the request and returns the result without modifying the system. This simulated output is presented the agent, which can then decide whether to proceed, adjust the command, or cancel the operation entirely. This approach allows the mitigation agent to preview the impact of state-changing actions and apply corrections before execution.
The examples above show the effectiveness of command-leved confinement in the mitigation agent of ATUS,US, particularly for preventing execution failures or system hangs due to unsafe operations. In Example 1, the agent attempts execute the command kubectl apply -f which will make the agent hang due to missing input via stdin. However, it is prevented by STRATUS with feedback. In Example the agent issues an interactive command using the -it flag, attempting to launch terminal session within a running pod. Without confinement, this results in the execution hanging until timeout occurs. STRATUS's confinement mechanism blocks such interactive terminal invocations, returning a clear message to the agent: "Interactive flag detected: -it. Such commands are not supported."
In our evaluation, 10.5% of commands executed by STRATUS (GPT-4o) and 9.7% by STRATUS (LLaMA3) would result in system hangs, in the absence of our confinement mechanisms.
B Bounded Risk Window
key safety mechanism in STRATUS the bounded risk window, which specifies the number of actions that the mitigation agent (ars) can execute in a single transaction. In this section, we empirically investigate how the total step limit impacts STRATUS's success rate on Lab problems (detection, localization, root-cause analysis, and mitigation). For each problem, we run the evaluated agents with total step limits setting to different values (3, 5, 10, 15, 20, and 30). Results are presented in 6
STRATUS (GPT-4o) improves from 0% 53.53%, after switching the step limit from 3 10. This shows that allowing STRATUS to take more steps can significantly boost its effectiveness. We also find that among all evaluated agents, agent performance plateaus after 15 steps, indicating that taking more steps after certain threshold does not yield better performance significantly.
The effectiveness gain be observed STRATUS with other LLM backends, but with lower overall accuracy, relatively scaling with the LLM'ss capability. Based on these observations, we set the default bounded risk window to K 20.
1.0
GPTio-ACe
Stratut (GPT)a)
ReAet Rash
Uana2.2-AC2 2-Acls
Stratus (Llama3.3]
do-mini-ACle
Success Rate
Stratus (oo-mnn)
0.6
0.2
0.0
30
Steps Taken
Figure 6: Impact of step limit on success rate of different agents A1OpsLab.
26

---

C Ablation Study on ITBench
In $ we presented an ablation study of STRATUS (GPT-40) on AlOpsLab, demonstrating the effectiveness of its retry and undo mechanisms. We now examine STRATUS's behavior under ablation on the ITBeench benchmark, as shown in T 6 drops the success rate to 11.1%. %. Note that this configuration implicitly disables the undo agent, as there are no prior actions to undo. Without retries, the agent must determine the root cause and devise mitigation plan correctly with attempt, which is significantly challenging. (2) STRATUR achieves the same task-level success rate (solving 9/18 problems) when the undo agent is disabled. As mentioned in $5 STRATUS exploits the observation that in out of 18 problems, restarting the target pods clears the incident alerts. Since ITTeench evaluates correctness primarily based on the absence of immediate alerts, this allows the agent resolve the incident without trying more destructive, state-changing actions, where our undo mechanism will be most effective.
We observe that (1) disabling retry substantially
To better understand this, analyzed these 8 problems in detail. Among all mitigation actions issued by the agent, approximately 15.2% are pod deletion operations managed by Kubernetes Deployment, which can automatically recreate the pods. Furthermore, significant portion (44.2%) of state-changing commands, e.g., changing a container image to an image with a hallucinated name, easily detected and fixed by STRATUS in the subsequent run, since STRATUS has reflection mechanism, which can help the agent to realize what went wrong in the previous run. The rest of the commands are a mix of direct restart or redeployment of the pods, which again lifts the underlying fault directly. In this regard, the undo mechanism does not improve the success rate in ITBench. However, the absence of undo capabilities can still compromise the cloud over time. Latent or non-critical side effects from agent actions, such as misconfigurations or performance degradations that do not immediately trigger alerts. may go undetected. Without TNR, there is no guarantee ensuring that such side effects are identified and reverted.
Table Ablation analysis of STRATUS (GPT-40) on mitigation problems in ITBench.
Ablation
Suee. Rate
Time (s) Cost ($)
STRATUS (40)
50%
1720.8
6.11
No Retry
11.1%
267.2
0.72
Naive retry w/o undo
50%
1711.9
5.13
D
Tool Usage
We develop of tools that enable agents to interact with the cloud environment through natural language. We describe the implemented tools in $D.1 and analyze their usage distribution across execution steps in $D.]
D.1 List of Tools
In order do complex reasoning and safe system manipulation. STRATUS has comprchensive suite of tools, including observability tool and command-lin tools. We describe some of the tools here.
Observability Tool. (Tel steeemetrycoll)cionn) or retrieving system states.
STRATUS has tools to observe the clouds, through collecting telemetry data
GetLogsTool: Retrieves logs from specific component, e.g., application logs from given pod.
GetTracesTool and ReadTracesTool: Retrieve traces if the system is traced, e.g., traces from Jaeger. ReadTracesTool handles large trace files stored disk.
Support trace-based boot- strapping by identifying likely fault locations. Instead of digesting the entire trace data, the agent will only read the traces with errors (e.g., 500 HTTP responses) as initial analysis results.
GetBootstraknnalyyssstoolll and ReaBBotstrrppnallyssstod
27

---

OracleTools: After each iteration of diagnosis or mitigation, oracles can be used to validate system health (e.g.. alert clearance, workload throughput, cluster health). These oracles determine whether the agent should continue, retry, or terminate.
Note that STRATUS disables direct metrics consumption, raw numerical data can be difficult for the agent to interpret effectively. Instead, such information is surfaced in logs a more proppeeeseed form, e.g., filtered, aggregated, or human-readabn summaries. These tools abstract away raw observability data, enabling the agent to reason over high-level, interpretable signals, and to determine its termination.
Command-line Tools. On the action side, STRATUS has specialized tools to construct the command to change the system states and perform the failthful undo.
NL2Kubect1: Translates natural language instructions generated by the agent into Kubernetes commands. This decouples high-level reasoning from low-level execution syntax.
LintingTool:
Acts as an internal tool to check the correctness of the command, e.g., the commands generated by NL2Rubect1; if the command is not correct allowed, the tool will return a message to ask the agent to improve the command.
RollbackTool: Supports the undo agent to perform failthful undo actions.
Noti NotificationTToll. Whenever the agents feel confident that the task is finished, it can use the NotificationTool to notify human of task completion, or escalation. Notification can be made in different troubleshooting phases e.g., detection phase should notify human whether there a nomaly in the system; while the agent should notify human whether the mitigation is successful or complete.
D.2 Distribution of Tool Usage
To analyze the tool invocation patterns in STRATUS, we log the tool usage of the STRATUS (GPT-40) across all problems in the AlOpsLab. For each execution step (i.e., each iteration within the retry loop), we record which tools that interact with the clouds are invoked. In Figure 7 we show the distribution of external tool usage (including the Selemeerryoollee RollbackTood by the STRATUS (GPT-40)) at different steps in our AIOpsLab experiments.
N12KubectI, Notificationnoon and
STRATUS exhibits distinct phases of tool usage across execution steps. In the initial stage (Iterations 1 1-2), the RollbackTool is predominantly used by the undo agent, and bootstrapping tools are used to perform initial analysis, which specified in the control flow. In early steps (Iterations 3-8), the agent primarily engages TeleeetryCollectiie
tools to gather observability data from logs and traces. As execution progresses (Iterations 9-23), NL2KubectlToo dominates, indicating the agent's transition from information gathering using provided tools to more customized command generation for diagnosis and mitigation. Additionally, we observe a shift in preference from trace retrieval log retrieval over time.
E
Oracle Usage
In $4.2 we describe how STRATUS determines termination based on the oracles. Here, we use an example (Problem ID: brong_bin_usage-mittbe
in A1Opslab) to illustrate the usage of our oracles in STRATUS. In this problem, a microservice is misconfigured to use an incorrect binary during container initialization. As a result, while the container is able to start and all pods appear healthy at the orchestration level. the application has functional failures. Specifically, the service responds with a high rate of HTTP errors -xxn-xxx33xx responses), indicating that the application is not working expected.
As illustrated in Figure during the initial validation phase, the system health oracle confirms that all pods are running normally. However, the workload oracle detects number of erroneous responses (115), revealing that the issue has not been fully resolved. Based on this feedback, STRATUS further troubleshoots the issue, and later correctly identifies the root cause an incorrect binary and proceeds to patch the deployment with the appropriate version. second round of oracle validation shows that all checks pass: the cluster remains healthy and the workload oracle confirms the climination
28

---

AaudBlosstrippAmllsssttt Tool
GororacesTol
ReadIraces
GogToo
200
No22
NotificationIool
Frequency
Rollbuck Tool
150
100
50
10
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Turn
Figure 7: Distribution of tool usage by STRATUS (GPT-40) across steps. The figure shows histogram of tool invocations grouped by type and step index (Iteration).
of HTTP errors (zero non -xx/xxx responses). With successful validation, STRATUS notifies the benchmark, marking the resolution of the issue.
Oracle
Pod-1 Pod-2 Pod-3
Cluster health oracle: checks status of the pods
117 requests in 10 seconds;
Workload throughput
oracle: simulate requests
Non-2xx 3xx
responses: 115
h 0O
Mitigation Agent
Find the wrong binary
Oracle feedback
Aprofile being used and fix
Pod-1 Pod-2 Pod-3
Cluster health oracle: checks status of the pods
Workload throughput
128 requests in seconds; Non-2xx 3xx responses:
oracle: simulate requests
Second Oracle check
Mitigation Agent
Submit and pass the problem
Figure 8: Oracle usage when STRATUS solving the wrong beu _sage mi tigatioonnnin task. In the first validation round, the workload oracle detects 115 failed responses, indicating the issue is unresolved. STRATUS continues troubleshooting and patches the deployment with the correct binary. In the second round, both oracles confirm success.
29

---

F Hyperparameter Sweep
We perform hyperparameter sweep over the temperature parameter in STRATUS (GPT-40) using all 13 mitigation tasks from AlOpsLab. Table 7 the results. We can that the temperature of 0.0 yields the best results. Setting temperature to 0 allows the LLM to output deterministically with given input. Setting non-zero temperature allows "creativity" in LLM's output. However, in the SRE context, this may not be desirable. A non-zero temperature permits the L.M to output non-deterministic results given the same input. For example, given a microservice application trace that clearly indicates the root cause of a problem, we would like STRATUS to confidently take one correct mitigation action. In addition, setting overly high temperature can directly affect the tool's ability in generating the correct commands, which harms STRATUS's performance critically.
Table 7: Effect of Temperature on Mitigation Success Rate.
Temperature
Suce.
Time (s)
Step
$
0.0
69.2%
811.9
46.3
0.877
51.3%
961.9
71.6
0.647
0.4
46.2%
950.58
67.9
0.493
0.6
56.4%
836.8
55.1
0.388
0.8
33.3%
1209.51
77.2
0.533
G Result Details
We present detailed evaluation results for STRATUS (GPT-4o) in A1OpsLab. Table SeieeSshone shows the results of detection, localization, and RCA tasks. Table 9 shows the mitigation results.
Table 8: Detailed results of Detection, Localization, and RCA for STRATUS (GPT-40) in AlOpsLab. We use the Pass @3 metric to calculate localization accuracy. Non-existent tasks are denoted as
Detection
Localization
RCA
Problem ID
Time (5)
Pass
Time (6) Pass@3 Suce.
Time (S)
Suce.
has__trrgt___pee purget__ort-miieee port-misconfgg-1
39.04
T
0.100
24.90
T
100
0.093
49.33
0%
0.073
target. target.port-mmsoone mot-mof
27.97
18s target port-misemfig-3
42.46
T
0.141
41.19
T
100
0.147
41.00
50%
0.075
auth mise, mongodls
42.26
0.117
50
0.115
43,30
0%
0.083
revoke_auie auth mongulb-1
42.98
T 0.066
58.26
100
0.050
37.05
50%
0.042
revoke auth mongodb-2
45.04
T 0.089
33.27
100
0.041
45.45
50%
0.056
maregistered mongoddh-1
35.63
T
0.043
34,47
100
0.041
34.10
50%
0.042
user unregistered mongodh-2
38.21
T 0.044
34.26
T
100
0.042
35.72
50%
0.042
miscontig app hotel
scale pod sero social net
72.04
T 0.113
35.
T
100
0.110
42.39
0%
0.076
assign lo nonnoooeii eon__exitent_ mode social mode_sociallnnnneomooooo.. nel
28.62
0.045
28.91
100
0.045
28.54
0%
0.046
container Jill
43,77
T 0.042
39.72
100
0.063
pod. Pallure, hesel_res
68.94
T 0.050
42.97
100
0.064
hotel res
55.75
0.072
275.83
0.120
network, Joss, hotel_ hose__otee__re res
43.17
T 0.061
$8.97
F
D
0.070
network delay
35.87
noup detection hotel reservation
38.62
0.101
noup deteectioonssccall social network
38.10
0.052
astronomy_shmie _d___aa______d service_ fullure
87.84
T 0.094
67.51
F
0.067
astronomy. shop. ad service. high. epn
40.44
T 0.060
67.16
T
SO
0.071
astronomy shop service sstronomy Shop. cart service servic, falbure astronomy shop image lond astronumy_ shop. Jafka, queue problems
42.17
0.886
64,73
0.055
38.37
T 0.092
60.96
T
100
0.111
33.33
66.00
T 0.126
65.48
F
0.059
astronomy sshopyy___p___eee wtt___hhhop__eeee _erteeee__errrceee__ servicefaaluue
65.14
T 0.054
67.U3
0.085
astronomy_ shop_ homppppyuentt...m service_umeeeeeee umreachable
52.18
T
0.065
66.44
0.045
sstronomy shop product catalog serviee failure
36.90
T 0.084
62.56
F
0.046
aetrenomy shop recommendation service. cache failure
78.68
0.090
sstrenomy shop. londgenerator foood homepage
29.59
T 1.183
35.45
F
D
1.345
redeplay without
40.72
0.044
42,74
0%
0.070
wrong bin,
40.32
T
0.170
298.09
F
0.149
40.86
50%
0.137
noop cetp__deeccionnnttee astronomy_sshme
68.54
0.056
Average
48.35
29/32
0.118
65.34
16288
51.19 0.126
39.68
34.62%
0.068
30

---

Table Detailed results of Mitigation tasks for STRATUS (GPT-40) in AlOpsLab.
Problem ID
Time (8)
Step
Succ.
$
k8s
Larssttree__pee poop-poooooooooo mort-misconfig misconfig-miiigatonn 1
124.59
13
T
0.116
k8s porget_port-niin mortimioo
210.08
23
T
0.338
k8s
148.49
T
0.084
assign_to__s
__td___o______ social soccill_net-miteeee net--mitiattion-
115.42
12
T
0.051
scale_pod_zero, zero__rro___ zerooeoccilllooeeeeeeeeeee Doocialnet-miriiationn-
139.03
88
T
0.232
user unregistered user unregistered soongod--mitigationn
2,007.58
123
T 0.330
2,090.04
46
F 0.955
revoke auth coongodb-mitigatinn
1,405.48
F 0.436
revoke_aue corrrhhmooobbboo mongodd--iitgatioon mittiaation-2 moniissmonoodbbboooo
1,321.47
108
F 0.644
136.60
20
T
0.148
misconfig_ayy _e____pppphht_llll hotee__reeee reree--mm 1
1,367.99 .99
10
F 0.805
redeploy rrdeploy_without, withoutt PV-mitigation--
839.00
152
T
6.704
wrong bin usage-mitigation11
649.44
T
0.562
Average
811.94
59.85
9/13
0.877
H Heuristic Evaluation of STRATUS's Multi-agent Design
STRATUS adopts a multi-agent design. Recent work [53 categorizes common failure modes of multi-agent systems, including Specification Issues, Inter-Agent Misaligument, and Task Verification. We present heuristic evaluation of the STRATUS design and how the design addresses common failure modes. Table 10 provides summary.
Table 10: Failure modes from 53 and STRATUS techniques that mitigate them. Abbreviations: DCF=Deterministic Control Flow, TRBBConnfinmmen (role-based), CCL=Confinement (command- level), ORC=Oracles STS=Specialized Toolset, UDO=Undo, LTR=Linter, BTS-Bootstrapping. ATD=Agent Thought Dropout, RWR=Reflection with Retries. Each row shows how the technique conceptually addresses the failure.
Failure Mode
StRAtuS Technique(s)
Deseription and Examples
Specification Issee (System Design)
Disobey Task Specification
CRB, CCL
Role-hased confinement assigns specific tasks for different trou- bleshooting stages; command-level confinement will validate cach action:
agent has clear (diagnosis, mirigation, etc) is given tools corresponding to the agent type. This allows for agent-spocifi confinement where each agent is only given the necessary tools complete task it's been assigned.
Disobey Role Specification
CRB, STS
Agent thought dropout prevents agents from heing trapped in previous mislcading thoughts; bootstrapping gives focused start.
Step Repetition
ATD, BTS
of Conversation History
RWR
Persistent agent thoughts retain context across retries.
Unaware ol Termination Conditions
ORC
Oracles identify task completion and provide exira context.
Inter-Agent Misalignment (Agent Coordination)
Datatlow direction between agents is the same as control flow; undo
Conversation Reset
DCF, UDO
restores state incorrect or destructive actions.
Fail Ask for Clarification
RWR
retry round can be initiated the agent fails the task.
Booistrapping orients intiaal step lic. where the agent starts): contam-
Derailment
BTS, ATD, UDO
to restore Agents will be forced give linal answer part of CrewAl's design) to deternine its success or notify of failure.
inated thought
dropped; undo
Information Withholding
Ignored Agent's Impat
DCF,
Agents consume predecessor's output (including output the previous round in rerrying) in the dataflow until the task completed.
Reasoning Action Mismalch
Prohubly related more to the model's is own capahilities.
Task Verification (Quality Control)
Premitture Termination
ORC, DCF
Task ends only after all agents linish and orucles confirm completion. completion single action are both verified.
incomplete Verilication
ORC, CRB, CCL
Incorrect Verification
ORC
Multiple oracles peovide validation to reduce false positives.
Specification Issues. common failure pattern arises from incomplete or ambiguous role/task definitions, or insufficient system design encoding in agent prompts (53) STRATUS avoids this failure pattern in multiple aspects (see Table TOP. Role-based confinement (CRB) is used to confine
the agent's 's behavior according to the role and task specifications; command-level confinement
31

---

(CCL) will validate each action. Also, STRATUS has specialized toolset (STS) for different agents, to retrieve the relevant information from the system, which can facilitate their task completion. Inter-Agent Misalignment. The Inter-Agent Misalignment category includes coordination failures among agents, such as one agent's incorrect output misleading downstream agents [53]. Take our context the example. If the detection agent identifies the wrong error location, the mitigation agent may inadvertently repair the wrong service. STRATUS addresses such issues by employing deterministic control flow (DCF), which coordinates the agents work together and communicate clearly with each other. STRATUS also uses reflection with retries (RWR), which enables the system to reassess and relaunch task (within the bounded risk window K). In addition, the undo capability (UDO) allows the system to revert to a safe state after erroncous executions. Furthermore, agent thought dropout (ATD) prevents potentially corrupted misleading context propagated from previous agents. Task Verification. Inadequate verification that fails to detect or correct errors or premature termination of tasks can make multi-agent systems fail (53) STRATUS deploys three oracles (ORC) including alert oracle, user request oracle, and system health oracle, which periodically assess system states and validate whether termination conditions have been met, to ensure the agent is terminating appropriately. Complementing these, STRATUS has confinement (CRB, CCL, LTR), c.g., command dry-run and syntax check, ensure that commands are semantically and syntactically checked prior to execution.
I An Example of STRATUS's Trajectory
We show the trajectory of STRATUS when solving a mitigation task (Problem ID: redeploy_wi
in AIOpsLab. In this problem, the storageclass of the persis- tent volumes (PVs) is mistakenly removed during redeployment. As result, the persistent volumes claims (PVCs) will be able to find the corresponding storageclass in the new deployment, and most of the pods will be in an erroneous state.
We show the key actions taken by STRATUS in "Gold Action." Statements marked "Note" are not part of the trajectory but are added for clarity.
Phase: Initialization
Run #1
[21:34:59] 59] All pods in nanospace "test horel- reservation are ready.
Pault Injection
[21:34:51] Waiting for
best-hotel-resete [21:35:18] Nanespace sest-bote--resevvttion
deleted...
been deleted.
Kanespace
deleted successfully.
Deleting nanespace test-hotel ret-booel--esevatiee vithout deleting the PV.
Redeploying sest-hoeel-rssevataee
Kanespace ret-hutel-rseeraation
found. Creating saccessiully: nanespace/tess hoter-hotel-eeee reservation created
Kanespace test-hoted
Task: Bootstrapping
Diagnosis Agent
Analyze the traces to localize shich part of the syster, e.g.. a service, a component, a pod, mode, etc., has + anonalien ies. Always use the get_traces tool first: then use a read_traces tool to help you with the Iocaiization. the tools times. You working the nanespace
that may not be helpful "tett-hntel-reeeraatioi"" please
nanespace using
Thought & Action
Diagnosis Agent
analyze the and localize anoralies in horel reservarion nanespace, first traces tool to fetch the traces for specified duration. Then, I kill use the read_traces tool to read analyse the traces.
get ("camespace":
"'est-botel-reservatior
"duration": 36)
Tool Usage: get traces
Diagnosis Agent
exported to: /eval/xagent/urac output /trree/rress... 1747186692.cov
Please the next action
32

---

Thought & Action
Diagnosis Agent
call teades with p'Ethe_pathh:
Tool Usage: read traces
Diagnosis Agent
The following of services are last service reqaest reached before returning an error, these services
nay not he the exact place of the voot cause, so you may need to dig deeper. These services correspond to
list of operations happened there, which is the service relying and could be shere the root cause cares
the list:
(service: "search", operation: "gee")
(service: "frontend", operation: "recommonattinn")
Task Bootstrapping Finished
Diagnosis Agent
entities with anmalless:
Service: Operation: "gee"
"frontend", Operation: "recomendatioo"
Possible root cause ittities for further analysis:
Service: "search", Operation: "geo"
Service: "fruntend", Operation: "recomendation"
Task: Mitigation
Mitigation Agent
The probles description is "You are an export Devōps engineer assigned to nitigate anomalies in deployed service...
Thought & Action
Mitigation Agent
begin the nitigation
need gather information the "Frontend" service "recomendatton
anoralies. anoralies
are present
"search"" service with "gea" operation operation. will start by fetching the logs for these services to understand the issues better.
First, I will ferch the logs for the "searca" service in the "test-hotel testt-hote--reeeraaaee reservation" nanespace.
Call get logs with ("nanespace": "test-botel-ressevvaie "service": "search")
Tool Usage: get logs
Mitigation Agent
("1evel" ": "info" "time"="2022 -2025-05-10T00:366000
"message":"TL "TLS disabled. "}
("levei":" " "indo" "tine":"000 12025-0925555--
"messae":"""e global log Level: info")
("level" "info", "112"==222-0--110100:
"message": setoCPercent 199")
2825-09-14701:3:0022
cml/search/main.gg
config...
9825-05-14701:3:-00
Read target port:
9025-03-44991:0:022
137
aat/search//aan..o:: go:44 10:41 Read consul address: consul:8500
2825-05-14191:3::022 IN7
m//ssaarch/main.oo01
Read jaeger address: jaeger:6811
2825-05-14101:3611
IN7
cnt/search/main.gr
Initializing
[service search bost:
2625-05:
traccin/ttracer.co g6:
Jaeger
adjusted sarple ratio
0225-05-14911:36:00
IN7
cml/search/main.ge
5o:6
Jaeger agent initialized
1155-05-14741:330
TNP aml/search//ain.go 58:58
Initializing consul agent Thost: consul :85861...
2625 0 :36:01
TNP
mm//saarch/aain.go go:66
Consul agent tnitialized
2825-05-14191:360 36:
187
cml//search/miin.gor
Starting
registry/regiatryy.
Trying
name: sev-search,
254379a6-hef4-441
2828-26664434488.
address: 18.244.2.21:8033
2625.
16:01:36:032
IMF sww.ceee//aacch/ereeeee
go:188 Successuully registered in consul
Please take the action
Thought & Action
Mitigation Agent
logs "search" service indicate any arrors anonalies. The service seens indtializing registering successiully. Next, will fetch the logs for the "Irontend" service in the "test-hotel-ee" restevlresevaaiic nanespace investigate the "recozmendation" operation anonaly.
Call lops with ("nanespace"; "test-botel-reservatinr"" "service": "frontena")
33

---

Tool Usage: get_ get_logs
Mitigation Agent
("level": "iinel""""nio"" ""iie":""023" 35-113333544 ""esssae";"""" disabled.")
("level" "inno""" 12205-00--1111 :35:4235: ""essage":""e global log level: info") ("Ievel":" "i"na"",, "tine" "2225 35-14481:35:-44
"message", "Tuune: setcCPercont to 199")
2825 05 14701:35:467
INF cnd/Irontend/en weaeeenddddddd.neeee go:23
Reading config...
2825/05/14 41:35:46
retrieving
"bttp://consul: httpa//////////aaaa//////ooooo search7passingg=" refused
tep 18.96.73.138:8888 connect: comeection
Please take the action
Thought & Action
Mitigation Agent
The logs for the "frontend" service indicate connection issue with the Consul service. The ezzor pessage "dial 10. 73. 13:88:80:: likely causing anonaly service,
connection refused" that the frontend service unable connect
ninespace to "recomendation" operation. To address this issue, 1 will check the status of the Consul service in the "test-botel-resee ensure is running and accessible.
NL2Kabect] Tool with ("nl_query": "get pods -n test-botel lil-botel-reserattion""
Tool Usage: NL2Kubeetl Tool
Mitigation Agent
RESTARTS
consul-bi54577 (es2lbbb667776--9999g
Running
frontend- 69d95967 6.895967hh----rr--e
Running ng
73%
gee- 900-998c18c--20 Swcbn
0/1
Ecror
(32s ago)
735
jaeger- (ii4455b66667/bbp
1/1
Running
715
rencached few.lre-ffffleeeeeeee 6--6686966598----
Running
78s
zencached rate- 68588796cb
Running
sencsched-reserve--e
-xeett
Running
bggeer
Pending
715
wangodb-profilee profile-758cb77330 thip1
0/1
Pending
785
0/1
Pending
665
(169966577--99mm
Pending
64s
zangodb 1994899999
Pending
Pending
Profile--76fccBdd---1
Running
(26s ago)
695
rate cloc38c55 kica?
1/1
Running
(245 ago)
65s
recommedattion- -et-8-666c6c-bbbbb0
1/1
Running
2 (22s ago)
64s
reservation- F651--n---t
Error
68s
65d19648d1 -1mm2
Running
59s
ago)
Thought & Action
Mitigation Agent
suggests zessage comectivity configuration. is not reachable at the specified address or port. Given that the Consul pod is rumning, will now check the service configuration for Consul to easure it is correctly
Consul pod
with Consal service itself but rather with typically indicates that
Call 3L2Bbbet1 Tool with quel":" "describe service consul nest-hotel-reservatie
Tool Usage: N1.2Kueett Tool
Mitigation Agent
Port:
5399 8380/TCP
TargetPort:
5399/TCP
Endpoints:
19.264. 17:8800
5429
TargetPort:
Endpoints:
19.246.2.17.840
Port:
5529
8508/TCP
TargetPort:
5599/TCP
Endpoints:
29.244.2. 2.1.8000
TargetPort:
Endpoints:
19.244.2.17:53
34

---

Thought & Action
Mitigation Agent
Consul service correctly configured expose part uhich the the frontend service trying connect to. The the iasue night he related to network policies or firesall rales that are prowent ing the Frontend service from connecting the Consul service. further investigate, betseen the
port correctly set, endpoints are available. This suggests that
check if services.
network policies place night affecting comectivity
Call 3L28abect1 Tool with quer":: "get netwockpolicy -a test-botel-reservattinn"
Tool Usage: NI.2Kubectl Tool
Mitigation Agent
Thought & Action
Mitigation Agent
There are no network policies listed in the "test- hotel reservatoon nasespace, visih neans there are no restrictions network traffic between the services. This supgests that the issue night be related to the frsel
configurat
the frontend
misconfigerat
service.
civen the inforsation gathered, appears to be a comeetivity issue with the Consul service. The error zessage "comeectin refused" indicates that the frontend service is unable to reach the Consul service at the speci address and port.
root
anonaly
"Frontend" service's "recomeedattonn" operation
Kitigation
service part 8566. 2. Check the firewall settings and ensure that there are no roles blacking traffic hetseon the frontend and Consul services.
Verity the network configuration ensure that
service
configuration
frontend service
correctly
4. necessary, restart the Consul service to ensure it is running correctly and accepting connections.
Thought: I have fied the root cause of the anmmaly and outlined mitigation plan. T will now submit the solution.
Call subnit with
Phase: Submission Validation
Run #1
Subnission triggered. Validating
Kaiting 15 seconds for changes to take effect....
Testing workload
117 requests 16.00s,
Bon-2ax or 3ax respoases: 117
Requests/sec:
11.70
Transfer/Sac: 3.
Tool Usage: submit
Mitigation Agent
The
valid Please resolve issues before subritring.
Thought & Action
Mitigation Agent
is in valid indicating unresolved issues. need further investigate resolve any resaining issues before tting The "geo" operacion in the "search" service also has an anonaly. I will check the logs far the "geo" pod to gather foroat fon about the error status.
subnission failed because
call get_logs ("nanespace" "test horel hwwehh-osseeeeeeee reservation", "service":
Tool Usage: get_logs
Mitigation Agent
2825-05- 141-1:111117100 cnd/gen//ain..or Initializing connection.
14791:37 cnd//ee//th.c de.2////hh....6666 reachable servers
panie: no reachable servers
inrourine 1 frunning]:
github.comm// www.:///ereooog ('Logger). Panic.
the next action
35

---

Thought & Action
Mitigation Agent
for the "gea" service indicate the RongsDB database.
reachable which suggests comsettiitt issue connection, indicating that
error occurs during initialization the "geo" service is unable to connect to the NongoliR instance at "wongedh geo:27017."
To address this issue, will check the status of the NongeE8 pod for the "gea" service ensure it is rumning and accessible.
Wl.Kaheet]
que__y" "describe pod vongodb-geo Set.od--geo-SFSatbbbbe www.--fff6-55bgoeeeeee bbgeer -n test reservation"]
Tool Usage: NL2Kubectl Tool
Mitigation Agent
Mame:
nongodb geo Sf1578bcb8 bgäcrr peat-botel-resevveee
Kamespace:
Prfority:
Service Account:
Mode:
<none>
Labels:
1o. korgose service servicemmngooll-gee pod- template 5F5h-55f5788chh
Type
Age
Message
default scheduler 8/3 nodes are available: pod has unbound immediate - WersistentVoluree Voluuooolaaaaann inss procuption: 0/3 modes available: Proseption not helpful for scheduling.
Narning FailedSchedul ing 2a
Thought & Action
Mitigation Agent
The "mongodb-geo" pod is in "Pending" state due to failed scheduling event. The message "pod has unbound irmediate PersistentVolumecl PersistentVolumellaii (PVC) naxed "gea-pre." status "geo-pve"
indicates that the pod is unable start because it camnot biad to the required
resolve issue, available.
correctly configured
call 312Babect1 Tool with ('nl query": "describe geo-pvc -a
Tool Usage: NL2Kubectl Tool
Mitigation Agent
geo pvc
Kanespace: test-hotel rreservation
StorsgeClass:
geo-storage
Status:
Pending
Volume:
Labels:
Annotations:
Final iiers:zers:
[kuberntees, proppppoottttt
Capacity: ty:
Access Bodes:
Volunefode:
Uilesysten
xangedb 511578bch8 bçter
Type Reason
Age
Prun
Message
Warning
ProvisioningFaille is (x9 over 2nts)
persistentwolme controller
starageclass storrgg......ss................. sto. .8
not found
Thought & Action
Mitigation Agent
The "geo-pve" is in "Pending" state because the storage class 'geo-storage" not found. This is preventing the Ponsisttmtooluneelaaan from heing bound, which in turn is causing the "songodh-gen pod to renain in "Ponding"
resolve issue, the pod can start.
create
storage "geo-storage"
Mitigation Plan:
PersiatenttoouueClaaa class named "peo-sterage" with appropriate configuration
"ceo-pve"
2. Ensure that the storage class is avatlable and carrectly configured allow the PVC to bind.
3. Once the storage class is created, the "geo-pue" should bind successfully, allowing the "wongodh-geo" pod
now proceed create storage "geo-ste storage."
KLaebtl Tool with ('nl query': "create storageclass -parsneters type:gp2")
36

---

Tool Usage: N122Kubecl Tool
Mitigation Agent
Dry ruy failed. Potentially it's comsand. stders: error: unknown flag: provisioner See "subect] create -help' usage.
Thought & Action
Mitigation Agent
seens the comand to create storage class was incorrect. will correct the comand and attempt to create the storage class "geo-storage" using valid comfiguration.
ML.8bbect ("nl_query": "apply #EFayaapTerrsion storage Ja8s. ltt...../....kaa......
StorageClass StorageCiassmmetaatttaa Ve name:
kubernetes 1o/aws aas unprraeters. AR type: AP2NNEEF "
Tool Usage: NL2Kubectl Tool
Mitigation Agent
storageclass. storage. kás. itgggoeee created
Thought & Action
Mitigation Agent
storage class ppo-sstorage" successiully ensure St is hound and the "rongodh-ggo" pod can start.
nox check status "geo pvc"
XL2abbect queryy: pve geo-pvc ----httel-reeerattion)
Tool Usage: NL2Kubectl Tool
Mitigation Agent
peo-pve
Kamespace:
test-botel-reseee reservreesevvtionn
StorageClass:
geo-storage
Status: Pending
Volume:
Labelsi
Annotations:
Pinalizes
Dubeeretes.ibbppeeppoeee pppopppoottoppo
Capacity::
Access Modes:
Velunefode:
Pilesysten
Used By:
mungodb-gee geo- peo--$9578bcbb-bbbee
Type
Nessage
Narning peniisioiingPallel 1és (e9 over 2m14s) persistenttolare-cotrtllee storageclass. sttrrggglsssssssrag....... iis.io
b storagtrage" not found
Thought & Action
Mitigation Agent
The "geo-pve" is stfll in "Pending" state, and the arror ressage indicates that the storage class "geo-storage" is there be with the creation the storage class recognition by persistent controller.
found.
To resolve this, T will fyrlify the existence of the storage class "gea-starage" and ensure it is correctly configured.
Call SuZubbectl Tool with ('nl query': "get storapeclase gco-storage")
Tool Usage: N122Kueetl Tool
Mitigation Agent
PROVISIONER
RECLATMPOLICY
VOLUMEESTDDINNGIEE Immediate
ALLOMPOLIMEEXPASSTOCO
AGB 8s
geo-storage
kubernetes. 10/ aws-ebs
Delete
false
Thought & Action
Mitigation Agent
storage class "geo-storage" exists is correctly configured with the provisioner "kubernetes. wtt.ees../////eeeeeeeee Hovever, the persistent volume controller is still unable to find it, which suggests potential issue with the cluster configuration
the controller itself. Given inforsation gathered, the inability to bind the "geo-pve" to the persistent volune controller not recognizing the storage class "peosstorage."
anonaly "gea" operation the "search" service is
kitigation Plan:
configurstion
persistent
correctly up recognize
storage classes.
2. Cbeck the logs of the parsistent volune controler for ay arrors misconfiuuratins.
37

---

port becessary, restart persistent volume
running correctly recognize
storage class.
Thought: I have identified the root cause of the anozaly and outlined mitigation plan. I kill now submit the solution.
Phase: Submission Validation
Run #1
Subnission triggered. Yalidacing.... Waiting seconds for changes to take effect... Testing workload
117 requests 18. 18.00s, 39.62KB read
Son-22x or 3xx respoeses: 117
Requests/sec:
11.70
transfer/sec: 3.06K3
Tool Usage: submit
Mitigation Agent
system not state.
subnitting.
series of observation and submission.
Task Mitigation Finished
Mitigation Agent
Boot Cause Analysis:
The "frontend" service's is "recomsondatioe" operation anonaly is due consectivity issue with the Consul "comecttion refused" indicates port.
The
frontend service unable
specified The "geo" operation in the "search" service is experiencing a comectivity issue with the MongoD8 database. The
service
+ error "no reachable servers" sugpests that the "geo" service is unoble to comnect to the MongoDB instance due the "peo-pve" being in "Pending" unable hind "goo-storage," despite it being correctly configured.
The "geo-pus
porsistent volume controller recognizing storage
Kitigation Plan:
For the "fruntend" service: Verify the
configuration
service reach
8599.
Chack firemal settings to ensure there are no rules blocking traffic betueen the Frontend and Consul services, Review the configuration of the frontend service to ensure it is correctly set to connect to the Consul
the Consul service if
rumning correctly and accepting consections.
2. For the "geo" operation:
Vorify the cluster configuration to ensure the persistent volure controller is correctly set up to recogrize
classes.
Check the cluster
installed and configured running correctly recognize
Bestart persistent
necessary
storage class "geo-storage."
These stops should help resolve the anoralies in the "Frontend" and "search" services.
Phase: Validation & Retry
Run #1
117 requests 16.06s, 57XB
Non-2ax or 3ax respoases: 63
Requests/sec:
11.70
Transfer/sec: 4.46K3
issues
responses:
Issues found with pod geo-9918c58c- Swrbn:
Container hatel reserv-geo is in CrashhoopBack01
Issues found with pod (e17-88777778--daatt
Container hatel
CrashloopBack011
cluster.
Adding issues ('Container botel-reserv-geo is in CrashLoopSackoff" ...]
Validation result: ('success' False, 'issuas': [' Non 2xx or 3xx responses: 117', 'Container botel reserv-ges geo is
= in Crashl.mpRackOFF ....]
written to: eval/05-13. 111-13221------1 21-28-46-redepoo xithout Pt-niiigstionn--L iimpattiiionnnnunnee Analyzing
0sjson
improvement...
Reflection generated sulcessfully.
38

---

Note (not part of the trajectory)
A stroggeclass has been created but not deleted in the lirst run. However it's created using the wrong provisioner. The Persistent
Volume Controller report following error:
Waiting for volume be created either the external provisioner "ebs.csi csi. com" manually by the systen adninistrator. If volume creation is delayed, please verify that the provisioner is running correctly registered.
overwrite such existing stroegoolass will result enors like:
The StorageClass essorage-class-name is invalid:
provisioner: Forbidden: updates
forbidden.
volumeBindinggMod::
Invalid value: "BaitForFirstConsumee field is immutable
Rollback prevents such overwrites and errors the next run.
Task: Rollback
Rollback Agent
You are given systen and rollback tool. The system is either in initial faulty state, or in the state you nitigation. You previous round may negatively impact your
left incorpletely repaired
rollback
initial state
operations you perforsed
nitigation.
You should run the rollback tool as rany times as you can, since this rollback tool undos the last operation perforsed the systen. You should the rallback tool until returns that there are nore operations to undo.
Thought & Action
Rollback Agent
rollback the initial need rollback repeatedly until indicates that operatioon undo.
Gall rollback_tul with ("argument":
"rollback the last operation")
Tool Usage: rollback tool
Rollback Agent
Rolled back the previous command: kubectl apply +f <xE0F
apiversion: storage. lid.
kind:
name: geo-sterage
provisionar: kubernates. to//ns ebs
paraneters:
gp2
using tobbet delete storageclass. storage. geo storage hotel reservation
Thought & Action
Rollback Agent
Thought: need provide different
rollback tool continue the rollback
call tool with ("argiment": "undo the previous operation"))
Tool Usage: rollback tool
Rollback Agent
Ke nore actions to rollback.
Task Rollback Finished
Rollback Agent
Hone.
Task: Bootstrapping
Diagnosis Agent
ynalyze the traces to Tocalize which of the syster, service, component, mode, etc... has tool first; then reacer tool nazy times. working nanespace "rest hotel reservationn. please use this navespace when using the tools.
asonslies. may helpiul
lncalization.
39

---

on part Thought & Action
Diagnosis Agent
fetch from the hotel reservation' start by using get. traces the necessary data.
and identify any anomalies in
systen.
Call get with ("camespace": "test-botel-resevvviie"""""""
"duration":
(6)
Tool Usage: get traces
Diagnosis Agent
exported /ewal/xagent//eeeeeeeeeeeeee output/ 174718896.ccM Please the
Thought & Action
Diagnosis Agent
successfully fetched identify any anmalies in system.
need the fros the file
call read with ("file path": "/eval/aagentttrae output/traces 1747186861.c5'))
Tool Usage: read traces
Diagnosis Agent
The following list of services
reached before returning place zeot cause, so you may diy deepex. services correspond list operations bappened there, which is the service relying on and could where the root cause cames
last
services
may not
from. Here is the list:
(service: "Frontend", operation: "an//etBaccmmmmaattoo""""
(service: "search",
"gee")
(service: "frontend", operation:
Task Bootstrapping Finished
Diagnosis Agent
List entities with axcaalies: Service:
"Frontend" Operation: "at//eetBacoemmmddtiissss"
2. Service:
"search", Operation: "geo"
Service:
"frontend", Operation: "user"
Possible cause entities:
The anomalies may originate from "frontend" service, particularly the "on/Getleecmmeedattttt ioms" "user" operations, as well as the "search" service in the "geo" operation. Purther analysis can start froz these entities.
Task: Mitigation
Mitigation Agent
probles description service...
an expert DevOps engineer assigned tigaee
deployed
Roflection the previous ru:: Root Cause:
The anoualy the "frontend" service's "recomendaticn" operation due comectivity issme with service, indicated by error nessage "conectioon refused." 2. The anomaly in the 'geo" operation of "search" service is due the inability to bind the "geo-pue" because
+ the persistent volune contraller is not recognizing the storage class "geo-storage."
Kitigation Plan:
For the "frontend" service:
service can reach the Check the firevall settings and ensure that there no rules blocking traffic between the frontend and Consul services.
Verify network configuration ensure that
+ 3599.
ensure correctly set connect If necessary, restart the Consul service to ensure it is running correctly and accepting connect
configuration
frentend
service.
2. Por the "geo" uperation in the "search" service:
Verify the cluster configuration
persistent voluse controller correctly up recognize
storage classes.
Chock the cluster setup to ensure all necessary corporents are installed and configured.
If necessary, restart the persistent volime controller to ensure it running correctly and can recognize the class..
The following issues vere found in the cluster:
Boo--2a ur box responses: 117
Container hotel: reserv- geo is in CrastLoogBack01
Pod wet-----eo------eee--ee-- bygcr is in Pending
mopodi-profile--155t///7 thlpl Pending
40

---

Thought & Action
Mitigation Agent
resolve the issues. plan involves nain tasks: provided airigation
follow comnectivity issue Consul service the end" service and addressing the persistent volume contraller issue for the "geo" operation fon in the "search" service.
previous has already idmniified
frontend service reach Consul check firewall settings review configuration the frantend service. necessary.
configuration
port 8500.
I will restart the Consul service.
Next, will verify the cluster configuration to ensure the persistent volure controller is correctly controller if
storage
will check
setup and the persistent
nacessary. Let's start by checking the network configuration for the "Frontend" service to ensure it can reach the Consul service.
XL2Kabect1 quelyy" "describe
test-botel-resevvttinn")
Tool Usage: NI.2Kubectl Tool
Mitigation Agent
Nane:
frontend
Kanespace::
test sostt-hote--reee rrsservat Fon
Labels:
ko.xorpose. cervicee-rroteed
Amnotations:
kompose. korpose convert
korgose versioon 1.22.8 (955b78124)
Selector:
So. Korpoopsee..co cervice-frontent
Type:
ClusterIP
Panily ly Policy:
SingleStack
Fies:
TPu4
19.96.54.222
IPs:
10.96.54.22 225
Port:
5009 5888/TCP
TangetPort:
5000/TCP
Thought & Action
Mitigation Agent
The "frontend" service is configured with ClusterIP and is Listening port 5800. Bowever, the issue is related connectivity with the Consul service accessible expected port, Next, I will describe the Consul service to verify its configuration and ensure is running correctly.
8588. need check the Consul service rumning and
call Mu.aubectl Tool ("al_query": "describe consul hotel reservation")
Tool Usage: NL2Kubectl Tool
Mitigation Agent
Name:
consul
Port:
5399 8386/TCP
TargetPort:
Endpoints:
18.244.2.. 2.:88360
Fort:
5499 B088/TCP
TarpetPort:
$199/TCP
Endpoints::
19.244.2. 2. 17:8440
Port:
5599 8586/ICF
TargetPort:
Endpoints:
18.244.2.17:880 17:8500
Fort:
5699 B688/UDP
TargetPort:
$3/UDP
Endpoints:
19. 10.246.2.17:5
Thought & Action
Mitigation Agent
The Consul service is correctly configured with ClusterTP and is listening port 8599, which natches the configuration. The endpoints carrectly setcings I will now check the firesall sectings to ensure there are to rules blacking traffic betwoon the frontend and Coesul services.
that the related
service configuration.
M..Keeeet1 with ("el_yu "get netsorkpolicy bot-hotel--o hww.hhl--eeeeatioonn
Tool Usage: Nu2Kubeetl Tool
Mitigation Agent
41

---

Thought & Action
Mitigation Agent
network polictes defined in "test hotel reservation" naresace, which there firewall rules blocking traffic between Irontend services. issue night related configurat ton,
I will now check the configuracion of the frontend service to ensure it correctly set to connect to the Consul service.
Ni.Kaheet]]
quee_yu "describe deployment Frontend test toot-hote ressevvation")
Tool Usage: NL2Kubectl Tool
Mitigation Agent
Frontend
Nanespace:
test- test-hotel-no -oteee-esseretttoon
Tae, 13 May 2825 21:35:37 -8188 To. koroose. service- frontend dep oymeet Juhemmtess ctttemmtte..t//viiiioooo kompose:
Labelst
Annorat fons:
korpose convert
korpose version: 1.22.9 (955h78124) to, kerpose, seeposs.eerviceee farvice--Frontom desired 1.1 updated total available unavailable Roll inglldngee
Selector:
Replicas:
Strateg//ype:
Thought & Action
Mitigation Agent
frentend deployment configured correctly, or command setup. The deployment is running and available, which suggests that the comnectvvity issue might be transient related to the Consul service itself. will proceed
apparent issues environment variables
Consul service
it raming correctly and accepting consections.
call M..ubbcct Tool with ("nl_query":
"rollout restart daployment consul -B test hotel reservation")
Tool Usage: NL2Kubectl Tool
Mitigation Agent
deployment, apps/consul restarted
Thought & Action
Mitigation Agent
The Consul deploysent has been restarted. This should help resolve any transient comectivity issues between the frontend service and the Consul service. Next,
operation "search" service, volune controller not recognizing the sterage class "pee-storage. will start checking the persistent volume clains to verify their status.
issue
persistest
("ul_query":
"get pvc
Tool Usage: NL2Kubectl Tool
Mitigation Agent
VOLUME
CAPACITY
-
AGE
geo-pes peo-PPCc
Pending
peo-storage
<unset>
7m2
paofile-pvc
Pending
profile-storage
7m26s
rate-puc
Pending
rate-storage
cunset>
<. 7n16s
recomnendation pvc
Pending
recomsendation
7m14s
reservation-pve
Pending
reservtion-storape
<unset>
-
7ml is
user-puc
Pending
utor-storage
<Unset>
Thought & Action
Mitigation Agent
"Pending" state, indicating not any persisten: volumes. This is likely due to the storage classes not being recognized by the persistent valume controller,
persistent
(PVCs)
I will check the classes they correctly configured and recognized the cluster.
Call M.KKhhee1 Tool with ("nl_query": "get storageclase";
42

---

Tool Usage: N122Kubecl Tool
Mitigation Agent
RECLATIFOLICY
VOLUNEBBINDNNNNDE
AGE
upenebs-device
openebs. io/local
Delete
Woitfor7irstCnnsuuee
false
-
13n
openebs-hosttpeee bosttshhh (default)
openebs.ie/lock
Delete
Concussrer
falise
standard (default)
smncher.io/locall
Delete
Xad SaltFoorrsttnnsaee
false
14m
Thought & Action
Mitigation Agent
storage classes "geo-storage". "profile-storaed stooagge, "sate-storage" "recomentiation starraag""""""" "reservation-strrae""" and "user-sterage" are not listed ancog the available storage classes. This indicates that these storage classes are not defined in the cluster, which is shy the PVCs are in "Pending" state. creating the seo-sttragee
resolve this
need
classes. will
Call ML.ubbbctl Tool with ("ml "create storageclass geo-storage prooris wwwaraannnaaeeeeoeroareeeee 1o/local path --reclatn-police Doley--eeeee -vollme-binding mode=hhttfforrrssteeeeeee"
Tool Usage: NL2Kubeetl Tool
Mitigation Agent
Bry failed. Potentially it's invalid corsand. 'lubect1 bbelp' usage.
flag: p-rrovisioner
Gold Action
Mitigation Agent
seens the command to create storage class was incorrect. I will use the correct syntax to create the "geo-storage" storage class using YAMI. file.
XL.obbect Tool with ('nl. query": "apply -f vBDFiaapiVersion storage k85. io/ lo/vi'nkind:: StorageClass umetadata.Va name: geo- wtw.ssttrgeeeeesssttttote spprovisione:: rancher. io/local path aatiiarreclassllicyy: Deletelpvo selete/owolumeiiiddiiddde:::: MaieforFirsstoossmeeet
Tool Usage: NL22Kbeetl Tool
Mitigation Agent
storageclasss. storage. www.://///oootreeee
Gold Action 2
Mitigation Agent
The "geo-storage" storage class has been successfully created. will proceed to create the other nissing storage classes: "prufile-storage" "rate-storage" "recosendation tion-storaee" "reservation-storee and "user-storage"
Call SL.ubbett Tool with ('nl query": "spply -f 4EC <EDF/papiVersson storage its. io/iiinidd: tmedadtaa S name: profile iaposssssppo rancher. rancher.to//ocal pathyprellateeolliey: beletee/avolmmeBBidinggoeeoe Maiffarfirssccsssurrrerr""
Tool Usage: N1.2Kueetl Tool
Mitigation Agent
storageclass, storage kös. wot.rroiolleeestorage created
Gold Action 3
Mitigation Agent
The "profile-starage" storage class has been saccessfully created. I will contimue create the remsining storage classes.
st.aabectl
query": "apply aappaaaperrssonn storage .r5..
StorageClass umetadata: in name: rate-storsaesmroviiii runcher. Delete ialltee/wollaeiiiidnngggdeedd Waitffrriirsstoossseeeetee rancher.io/leca ww/.ocal--peeeeeeeecceeeeeeee--e pathyarecacrelllllollorro
Tool Usage: NL22Kueetl Tool
Mitigation Agent
storageclass, starage. io/rate created
43

---

Gold Action 4
Mitigation Agent
"vate-storage" storage
been created.
"recommendation storage" storage
Call SLKabbect Tool with ('nl query": "spply -f veD.aaapierrsion
k85, in/vi'nkind:
StorageClass name: recommoiatioon-ssraap wtr.rioonnnoorreeerrrrrssieeeee rancher: http//oallpathhreeeiieeeeeeee patthaareelaiiiooleey..
Tool Usage: NI.2Kubectl Tool
Mitigation Agent
storageclass, storage. fo//recomsedaaton storage
Gold Action 5
Mitigation Agent
The "recemendation sterage" storage class bas been successtully created. I xill proceed to create the class.
"reservation
Call 3L2Kbbect] Tool with qunl_query":: "apply -f <E00 vDDVaaapiVerssin sterage à8s. in/vi'nkind:
StorageClass lmttlsssmmttaataaaa nine: inseevatioonstoragee enprrviiinooviiooneer. rancher. io/local hwweToocll-pphhhheeee pethiareelainnolliey:: oelete/bwwlamediiddnngooo::s MeitforFirrttoossmereeeeee
Tool Usage: NL22Kbeeel Tool
Mitigation Agent
storagecTass, storage. kds. www.essrvaatinnntteeeee created
Gold Action
Mitigation Agent
The Pessevationnsteraeee storage class has been: successfully created. I xill proceed to create the "user-storagee" storage class.
N.2Kahect] with ("nl_query": "apply -f #FFFinapiVersion storage k8s. in/viinkind:
Mirageclasssmmetddatte nams:: wor.--stooieeeprravi.. sitners rancher, in/Tocal htt.aallppptppreaaaattolooooo Bolltre/pwollmmeeEndinnnooooo RaitForFFisstcoossaarrreeeee
Tool Usage: NL2Kubeetl Tool
Mitigation Agent
storageclass, staragg kds. to/user strrrse--- created
Thought & Action
Mitigation Agent
necessary storage This should resolve the issue with persistent volume clains being "Pending" state. I will now check the status of the persistent volume clains to ensure they are bound correctly.
iser-storage" storage
successfully
qunl'query":
"get pvc reservarion")
Tool Usage: Nu2Kubettl Tool
Mitigation Agent
MAMB
STATUS
VOLUME
CAPACITY
ACCESS MODES
STORAGECLASS
VOLUVEATTRRBBUESCCAAS
geo-pwe
pvr- Nwc-2297753a-c660
Becd
geo storage
cunsets
7a52s
profile-pme -VC
Bound
(el-dd99--9cca4cde (.20-- 095894222243
IG1
FNO
profile proiile-storage
7a58s
zate-pvc
Pending
rate-storage
cunsets
7a66s
recoerendatinn ion- ion-poc pve
Pending
<a recormsndation storage cunset>
7i446
pvc Pending
reservation- storage
user-pvc
Pending
user-storage
<unset>
7x38s
44

---

Thought & Action
Mitigation Agent
"geo-peet and "profile pvc". bound, indicaring storage classes being recogaized. the other still "Fending" state. This night resource constraintss scheduling issues. I will check the events for one of the panding PVCs to understand why it is nor being hound.
XL2Kabect1 Tool quelyyu
"describe pve rate-pve rest-hote--reserveiime
Tool Usage: NL2Kubectl Tool
Mitigation Agent
Kame:
rate- pee
Nonespace:
test-botel- peet-botel-reservvee ressevvaion
StorageClass:
rate-storage
Status:
Pending
Volune:
Labels:
cnone>
Annorat tons:
valure.beta bww..beta.kke.. kuhhenetes.. to/storage-pravte provisivvvis:::: rancher. rancher.in/loce wwww/cal-llaa.... -athh
volune, kubernetes, tt/selected-nod:: kind-worker2
volure. kubernetes, in/storage weterage-proiisiiee provissons:::: rancher, ii/loccll-ppteee
Final
[kubemetes, www.:c/pppppeeeeeccccc
Capacity:
Access Modees:
VoluneFode:
Filesysten
Used By:
wongodb- rare- 56cc8659c9-7dv4
Events:
Kessage
Narning ProvisieningPsilel 186s (x26 7m4ls)
persistentvolue-caatroi
storageclass, storage kés.io sate-storage" persistentvolume-comd 18s
Normal Wait ParPPSosoSseeallll lo waiting for pod vongodh-rate wete--rattt----eeeee 56ee865999- to be schedul ed Norral ExternaProvtsiiont
5s either by paralatentvolure controller
Waiting for the systes adeinfatrator. velune creation delayed, please verf
'rancher, .io/lccal path' the provistoner running
correctly registered.
Nomall Provisioning
Ss
rancher, io/local-patt. local-path- path-grooissooer- proristone--9ecc755500 58cc785666-229r 4374 -a55---------5-5 aSIf-Bac1177steel "test reservarion/rate gvc"
External
provisianer provisioning
Thought & Action
Mitigation Agent
The "rate pve" is pending due to the sterage class "rate-sie storage" not being found initially, but it seees the provisionar atterpting provision voluze. This indicates the issue night resolve itself provisioner completes will check the status of the persistent volume claiss agatn to see they have been hound.
XL2Kabect1
("nl_query": "get pve test-hotel- peee-hotllreeservatio""
Tool Usage: N1.2Kubectl Tool
Mitigation Agent
NAMB
STATUS
VOLUME
CAPACITY
ACCESS KODES
STORAGECLASS
-
TOLUPATTRIBITTSSTASS
AGE
Bound
1GL
RMO
profile-gve
Bound
IG1
KNO
profile-storanl
cunsets
7a54s
rate pvc
Pending
rate storage
7a58s
recommendation- pvc
Pending
recormendation- ion.eeattiinnssrraaee
<unset>
7rd88s
reservation-pue Bound
161
RMO
L reservati imn-storage
cunset>
7e45s
Pending
cunsets
7m42s
Thought & Action
Mitigation Agent
The "reservation pve" has been successfully bound, but "rate prc", rrccomsendation pvc", and "user prc" are still pending. The provisiin provisioner working now check logs "hotel- heall-reerr-aaee gea" container see there are any arrors related CrashlooplaackOff state.
may resolve shortly.
get ("nanespace": "test-botel-reseetee resevvireservation"" "service": reserv-geo")
45

---

Tool Usage: get get_logs
Mitigation Agent
service/eaneppce exist.
Please take the next action
Thought & Action
Mitigation Agent
seens there was an exror retrieving logs the "hotel-reserv-ge"" container, possibly due to aesonfiguration the service not existing. will check the pods in the teer-hoot--resevatiion nanespace to verily their status and ideatify issues.
Call ML.Rbbectl Tool with qualqqury":
"get pods -n test hotel reservation")
Tool Usage: NL.2Kubeel Tool
Mitigation Agent
HAMR
READY
STATUS
RESTARTS
AGF
consul-6$8 -55--555-5555-5
1/1
Running
48s
Irontond 634959671b carjj
1/1
Running
8x36
380-998c58c--35 Sw-nh
CrashLoosaac001
jasger-67445bedd-7755
Running
nencached-grofil profile--0bbckkkol----
Running ng
sancached-roe rate 68585796cb gtpas
1/1
Running
7a57s
1/1
Running
7x52s
xongodb-eeo-1555/55 Sateoo--5558abc--booe -plcr
Running
8x1s
zongodo wrt.ppoooolelceeeeeeeeee
Pending
f9hnng
0/1
Pending
7254s
songodb- reservation 5994859869 zVVEE
1/1
Running
7x58s
xongodb-usez: Beroodb-ussr--9-38- 1155a-9933666449-38888
-6th
0/1
Pending
7xd8s
Error
(3nlés
7259s
cloc5cc55 klcq7
CrashLoop@aa0011
7x556
recommesdation
bobl8
CrashLoopBack011
ago)
reservatinn769900 76690---67699
0/1
Crashi copSack0ff
(2n58s ago)
7a58s
search 65d196dRdE 61ma2
1/1
Running
7249s
user 5477d757dB 48645
0/1
Error
(3nls ago)
7x47s
Thought & Action
Mitigation Agent
The "geo" pod is in crashlloogBackfr state, and several other pods are pending experiencing errors. This constraints.
there underlying issues check the Togs "gea" pod to identi fyy any specl Fil errors causting the Crashl.ooRaacooF seate.
conliguration
call get logs with
("banespace"; "test-bootsl-rsseevvtiii"""""" "service": "geo")
Tool Usage: get logs
Mitigation Agent
("level"="info"" "20p::::222255111110"
'message":""E disabled.")
["Ievel":" "info"" "nfo"""tie""" "tinm""""2255 14:40-00--1101:2:--111111
"message" global lng info"]
("Ievel": "time" ""im==="002000-111000:::
'aeessage " "Tune:
199")
2025-05-14701:42:--5 INP cné/geo/sain.go.
Reading config...
2825 1225-05-14791:66 :4:47 TNP cwd/gen/sain.go:
Read database URI.: mongodb- umngodt-geo:2700
113-00--1091144:-8222 I87 cms//ee/sain..ge 37
Initializing DB comection...
2825-0--41001-3::66 PNC end/gen/ go:26
reachable
poroutine [running]:
github.con/rs//// wwt.c///////ooogg./ooo/ ("Legger) Panic. Punic. Flnc1C109966211 Beb))
/gn/src/gi whw..hhhhhaaaaaaeee wwtpcc//o//////eeeeooooooo pig.--lccooo..... www.ioss/vwdor/gtlttt thtbor httpu//.o//////eeeeeeee http://s/////e//oo//////////// og/looggg:16555555 +6x2d
github con/rs/////e////ee (*Event) 2sg(0xc661191199 (8x96e211, 0x14))
/go/src/githor com/harlow/poRiicnn wttpoeseeviceeeeenrrrr/eoeo wttp:/cwww///eeeoleeeeoooo go:118 -6x2b6
gi withub.cco/// waabbb/co/////eeeeeeeeee
(*Fvent) .)
/go/ poo/sre/gii thub. con cow/harlow/go-zitt
/ii thub http://erroooo/ee..eeeeoeoo////e
rain. initi ali tellatabassee((eeee 9999289222888 #xcl99Faddd)
/gn/sre/ /ii thub. con/harli wow/haloo/ghh tow/gn-niereo services/cedl cend/ggo//tt g6: +8x167
sain.ming
hwt/sr//githbb... www.bl.ccc/chaaaaee wwwp.ocloww.oooooeo.oooooooooooo hhw.oo/go-mmccro- servicess/ca///eo//iiii
Please take the next action
Task Mitigation Finished
Mitigation Agent
Boot Cause:
The frentend" service's "recaseenaaton"" operation experiencing connectivity with the Consul service, indicated nessage refused. Consul was restarted any connectivity issues.
46

---

2. The "geo" operation in the "search" service is unable to bind the "gec-pve" due to the persistent volure - controller not recomiizin the storage class "geo-storage." The missing storage classes were created to resolve
error, specifically - servers." This indicates probles. with the Nongote connection.
"geo" pod
CrashloophackOff
Kitigation Plan:
For the "fruntend" service:
correctly configured connect blocking between
service port
Varlfy
policies
- services.
Monitor the Consul service to ensure it remains operational and accepting comnection..
For the "gea" operation in "search" service:
correctly aow storage classes Investigate the Kongobü connection issue causing the "geo" pod to crash. Verify the KongeD8 service is running + and accessible at the expected endpoint.
persistent
For the pending PVCs:
binding remsining Check for any resource comstraints or schedul ing issues that may preventing the from being bound.
provisioner
it completes
Conclusion:
The primary issues have been addressed by restarting the Consul service and creating the missing storage classes. system Kongobi connection issue for "geo" pod.
The iwininggg involve
changes effect investigating
Note (not part of the trajectory)
Although nor all the pods ready in the last observation, this docsa't means the agent failed this attempt. The operations it performed are correct, the system still the process stabilizing. Kubernetes will attempt restart faulty pods periodically. Aller few minutes of wailing, all pods will be ready and the workloads will be able without issues.
Phase: Validation & Retry
Run #2
Waiting 399.8 seconds for changes
take effect....
117 requests 18.60s, 48.95KB read Anguests/sec" 11.70
fersec: 4.39KB
Phase: AlOpsLab Validation
Run #2
Evaluation
Results:
('success's True)
47


Reviews:
[{"review_id": "Mjemi84T3B", "reviewer": "Reviewer_Q1w9", "summary": "The authors present STRATUS, a method for Site Reliability Engineering of cloud services. The multi-agent approach presents TNR which allows for failure mitigation, a key advantage over existing approaches. It shows improved mitigation of failures, though at a longer cost and computational overhead.", "questions": "•\tWhy does Stratus underperform on RCA tasks?\n•\tWere any other multi-agent decompositions considered and were any experiments run on these?\n•\tCan the system be adapted for hybrid settings i.e. how would you balance agent autonomy with human oversight in safety-critical environments?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths\n  - Clarity: The algorithm is explained in detail and TNR is proven by induction\n  - Quality: The experiments show strong results across different takss.\n  - Significance: The work addresses a key problem in the real world and shows string promise\t\n  - Originality: TNR is an important aspect for SRE \n\n- Weaknesses\n  - Clarity: The authors must formally state exactly what ε is. They must also \n add a diagram or clear example of how the LLM is used by the agents.\n  - Please move around figures and rename captions to have the full information and not really on the text e.g. Figure 5.\n  - Quality: The reasons for Stratus struggling on RCA problems needs to be expanded upon: for the rest of the paper, the user has expanded on limitations but should be here as well.\n  - Would like to see an ablation of the multi-aent system i.e. with different methods of separating the agents\n  - While authonomy rovides obvious benefits, the authors must be careful of downplaying the importance of human intervention.", "was_revised": true, "final_justification": "I commend the authors for thoroughly engaging with all reviewer feedback.\n\nTaking into account the rebuttal and the other reviews, I am content to keep my score the same, contingent upon the authors incorporating the changes outlined in their rebuttal.", "rebuttal": "Thank you for the insightful review! We will carefully address the review comments, including: improving the presentation (e.g., formally define ε, adding clear examples, and improving figure captions). \n\nWe answer the questions as follows.\n\n------------------------------------------------------------------\n\n**> Q1: Why does Stratus underperform on RCA tasks?**\n\nWe looked into the results and trajectories; the low RCA success rate is largely due to the ambiguous metrics used by AIOpsLab. The way AIOpsLab measures RCA success is to match the agent’s output with pre-defined category labels; however, the category labels are often ambiguous and not mutually exclusive. \n\nFor example, for a port misconfiguration, AIOpsLab defines the root cause to be “Virtualization” (as the layer) and Misconfiguration (as the fault). Stratus answers the root cause to be “Application” (as the layer) and “Network/Storage Issue” (as the fault). Arguably, Stratus is more precise in describing the root cause, e.g., a port is more of an application-level issue than “Virtualization” (AIOpsLab doesn’t have a traditional hypervisor-like virtualization layer). However, Stratus is considered wrong due to mismatch of the labels. We reached out to the AIOpsLab authors and they acknowledged this problem.\n\nWe manually go through all the Stratus RCA answers; the true success rate of Stratus (GPT-4o) is 50% (13/26), which is confirmed by the AIOpsLab authors.\n\n------------------------------------------------------------------\n\n**> Q2: Were any other multi-agent decompositions considered and were any experiments run on these?**\n\nYes. We considered and experimented with a conversation-based multi-agent framework, AutoGen, to construct a team of conversable agents with the roles of Application Developer, Platform Engineer, System Administrator, and a Team Manager to aggregate opinions and issue actions. However, we found that such a decomposition was neither effective nor efficient.\n\nWe sampled 48 problems from AIOpsLab, where the incidents happen on the application and platform layers. However, the role-based decomposition only solves 10 out of 48 problems. The multi-agent conversations tend to over-communicate and each agent only has a partial view of the entire system stack. For example, when round robin is used as the speaker selection method, every agent speaks in each step, slowing down decision making and causing agents to repeatedly collect telemetry data to investigate/confirm other agents’ observations and hypotheses. For example, solving a simple detection problem `user_unregistered_mongodb-1-detection` took 893 seconds, which is 25X longer than Stratus (Table 8 in Appendix). We believe that multi-agent conversation is not the right design for SRE agents, despite their promises in use cases like coding, Q&A, and entertainment as shown in the AutoGen paper.\n\nThe design rationales of our decomposition are:\n* Each agent has a global view of the system stack (which is a key advantage of AI agents over human engineers).\n* Each agent is corresponding to a concrete SRE task.\n* The design enables timely decision making as SRE tasks are time sensitive.\n* The state-machine based control logic facilitates safety reasoning (unlike many use cases like coding and writing, safety is the first principle of SRE agents). \n\n------------------------------------------------------------------\n\n**> Q3: Can the system be adapted for hybrid settings i.e. how would you balance agent autonomy with human oversight in safety-critical environments?**\n\nThanks for the question! Full autonomy is our moonshot; in practice, human-agent interaction is very important. We will make this point clear.\n\nOur design supports hybrid operations in the following ways:\n* Agents with the writer role (which can change system states) can be enforced to always request human approval before execution. Agents’ action stack and tool interface are interceptable before every tool call or action pushed/popped. Before executing a write operation, the system can present a concise summary of the proposed command, its intent, and relevant telemetry, enabling the human to verify it.\n* Policy-based control for human-in-the-loop preferences such as: \n    * Always ask before executing write operations.\n    * Approve specific commands this time, require confirmation next time.\n    * Remember and always allow specific commands after approval.\n\nCertainly, how to design an effective human-agent interface is still an open question and we plan to explore hybrid designs as future work."}, {"review_id": "6IkHSsEvv5", "reviewer": "Reviewer_fQgT", "summary": "STRATUS is a multi-agent system for autonomous Site Reliability Engineering (SRE) in cloud environments. It automates failure detection, diagnosis, and mitigation. Its main contribution is \"Transactional No-Regression (TNR),\" a safety specification ensuring that mitigation actions can be undone and system health never worsens. STRATUS uses specialized agents (detection, diagnosis, mitigation, undo) and tools for cloud interaction, with sandboxing and a stack-based undo for TNR. Evaluations on AIOpsLab and ITBench show STRATUS significantly outperforms existing SRE agents in mitigation success rates, primarily due to TNR's undo-and-retry mechanism, validating its potential for practical autonomous cloud reliability.", "questions": "Section 3.1.1 lists three assumptions for TNR: Writer Exclusivity (A1), Faithful Undo (A2), and Bounded Risk Window (A3). Could you discuss a scenario where one of these assumptions might be violated in a real-world cloud system and the potential consequences for STRATUS's safety guarantees?\n\nThe paper mentions that \"realizing perfect undo for all conceivable state changes in complex environments like cloud systems remains a practical challenge.\" What are some specific types of state changes or operations in a cloud environment that would be particularly difficult to perfectly undo, and how might STRATUS's current implementation address or be limited by these challenges?\n\nPersistent vs. Non-Persistent Faults in ITBench: Your analysis of ITBench (Table 6) points out that \"in 8 out of 18 problems, restarting the target pods clears the incident alerts\" because \"the injected faults do not persist.\" This suggests a specific characteristic of the benchmark. How would STRATUS's performance and the effectiveness of TNR differ when confronted with persistent faults (e.g., a hardware defect or a deeply embedded misconfiguration) that cannot be resolved by simple restarts?", "limitations": "The authors acknowledge some technical limitations (perfect undo challenges, no concurrency control, incomplete confinement rules, ITBench's non-persistent faults). However, they don't adequately address potential negative societal impacts.\n\n### Suggestions:\n\"Perfect Undo\": Detail why certain changes are hard to undo and propose handling for partial/irreversible actions.\nLLM Errors: Beyond TNR, explain strategies to minimize initial LLM errors caused by non-determinism and hallucination (e.g., structured generation, fine-tuning, prompt engineering, ensemble methods, human-in-the-loop for high-risk ops)\n\n### Address other concerns: \nSecurity Implications: Detail safeguards against compromise and the risk of amplified attacks.\nAccountability/Liability: Who is responsible for failures caused by STRATUS?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\nI appreciate the author tackling a very important problem in AIOps and system reliability. \n*   **Significance:** Addresses critical, real-world SRE challenges by pursuing fully autonomous cloud reliability, moving beyond human-assisted approaches. This could transform incident response.\n*   **Originality:** Introduces and formalizes \"Transactional No-Regression (TNR),\" a novel safety mechanism enabling safe exploration and iterative mitigation through guaranteed undo capabilities. Its specialized multi-agent architecture for SRE is also a unique design choice.\n*   **Quality:** Backs TNR with rigorous formalization and implementation details. Demonstrates strong empirical validation, significantly outperforming baselines in mitigation success rates across multiple LLMs and benchmarks (AIOpsLab, ITBench).\n*   **Clarity:** The paper is well-structured, clearly written, and enhanced by illustrative examples and a detailed appendix, making complex concepts accessible.\n\n**Weaknesses:**\nThe overall evaluation is constrained. \n*   **Quality - Practical Limitations:**\n    *   **Imperfect Undo:** Acknowledged challenges in achieving \"perfect undo\" for all complex, stateful, or external interactions mean some subtle degradations might persist post-rollback.\n    *   **Performance/Cost Overhead:** STRATUS is significantly slower and more expensive than other agents, posing a concern for urgent incidents where rapid resolution is critical. It would be helpful to state solutions that accelerate the process. \n    *   **Benchmark Generalizability:** TNR's benefits were less evident in ITBench due to a prevalence of non-persistent faults, raising questions about its effectiveness for persistent, harder-to-undo real-world issues.\n*   **Significance - Applicability Constraints:**\n    *   **No Concurrent Writers:** The assumption of no concurrent writer agents significantly limits its applicability in highly dynamic, multi-actor cloud environments where simultaneous changes are common.\n    *   **Oracle/Success Definition:** The success criteria, based on immediate alerts and basic health checks, might not fully capture subtle, latent, or long-term system degradations, potentially leading to a false sense of full recovery.", "was_revised": true, "final_justification": "The aurhors addressed most of my concerns, therefore I enhanced the score.", "rebuttal": "Thank you for the insightful review! We'll carefully address the review comments.\n\nWe first answer the explicitly listed questions and then respond to other concerns.\n\n--------\n\n**> Q1: Could you discuss a scenario where one of these assumptions might be violated in a real-world cloud system and the potential consequences for STRATUS's safety guarantees?**\n\nThe simplest example is A2 (Faithful Undo). AI agents are imperfect. If the mitigation agent had a wrong mitigation attempt, say deleting a large file (e.g., to clean up disk space) but the root cause is not about the disk being full, it needs to recover the file in order to restore the system state as if the wrong mitigation procedure was not applied. However, without special handling as in Stratus, file deletion cannot be undone, and thus violates TNR. Basically, without TNR, the cure could be worse than the disease. TNR ensures that unsuccessful mitigation actions can be rolled back, which enables Stratus to try different mitigation strategies.\n\n--------\n\n**> Q2: What are some specific types of state changes or operations that would be particularly difficult to perfectly undo; how might STRATUS address or be limited by these challenges?**\n\nModern cloud platforms (e.g., Google's Borg, Meta's Twine, Kubernetes, DBOS) all follow _state-centric_ designs where the system states are maintained in fault-tolerant, strongly consistent data stores (e.g., etcd in Kubernetes). Any operations are realized by _state reconciliation_, which can be reliably done through declarative, state-centric interfaces. In principle, undo can be cleanly done by reconciling the system to a recorded state. Stratus leverages such state-reconciliation principles to realize systematic undo using a stack-based mechanism.\n\nThe state-reconciliation principle requires modern cloud systems to expose a state-centric management interface (e.g., CRD in Kubernetes). Modern systems support such interfaces and thus are referred to as \"cloud-native\" systems. Therefore, we believe that the Stratus style of system undo is generic and practical.\n\nHowever, recent research [Acto-SOSP'23, Sieve-OSDI'22] shows that implementing a declarative, state-centric interface is nontrivial because (1) it’s error-prone to implement a declarative state-centric interface using common programming languages (e.g., Go, Rust, Python, etc) which are imperative, and (2) some cloud-native systems are transformed from legacy codebase and thus are inherently difficult to support arbitrary state reconciliation; this is particularly true for database systems, where on-disk data management is known to be challenging and expensive.\n\nThe aforementioned limitations are arguably orthogonal to Stratus agent designs. Currently, such problems are mostly worked around by disallowing certain state reconciliation that cannot be easily undone (without system redesigns). The same practice can be applied to Stratus by only allowing it to explore operations that can be effectively undone. Going beyond, we shall explore ways to automatically analyze the side effect of a given operation and determine whether it can be undone and how costly would the undo be.\n\n--------\n\n**> Q3: How would STRATUS's performance and the effectiveness of TNR differ when confronted with persistent faults that cannot be resolved by simple restarts?**\n\nWe shall clarify that _only 8 out of 18_ ITBench problems have non-persistent faults. The remaining 10 ITBench problems _all_ have persistent faults and _all_ the 13 AIOpsLab mitigation problems have persistent faults (e.g., misconfigurations and persistently failing nodes) – simple restarts cannot resolve these failures. \n\nStratus successfully mitigates 9 out of 13 mitigation problems in AIOpsLab (all persistent faults), showing its ability to resolve persistent faults. In terms of TNR, Stratus retried at least once in over 80% of the problems and at least five times in over 30% of the problems. In the 10 ITBench problems with persistent faults, Stratus mitigated 2 of them. Stratus retried at least 8 times per problem.\n\n--------\n\n**> Q4: STRATUS is significantly slower and more expensive than other agents; It would be helpful to state solutions that accelerate the process.**\n\nStratus is not slower or more expensive than the other evaluated agents; the reason Stratus took longer on harder problems is that TNR enables Stratus to make multiple attempts to solve the problem, while the other agents can only make fewer attempts. It is a key advantage of Stratus, not a limitation.\n\nWe have a few ideas on accelerating Stratus: (1) caching solutions for learning so future resolutions of similar type can be much faster (which becomes domain knowledge in a similar vein as Q4 to Reviewer sLKR); (2) using smaller models (Q3 to Reviewer sLKR), and (3) asking the AI model reason about efficiency of mitigation strategies (e.g., cheapest strategy first).\n\n--------\n\n**> Q5: No Concurrent Writers: The assumption of no concurrent writer agents significantly limits its applicability in highly dynamic, multi-actor cloud environments where simultaneous changes are common.**\n\nOur current operation model is to deploy one Stratus per cloud system (each cloud system is managed in a separate namespace).\n\nWe plan to extend Stratus for multiple writers by evolving `A-Lock` into a fine-grained, resource-aware concurrency control system. This can be achieved by designing a Coordination Controller, which effectively functions as a distributed lock manager. The basic idea is to allow multiple mitigation agents to execute their transactions concurrently, with intended actions not conflicting over shared resources. This controller dynamically inspects the \"mitigation plan\" from each agent, identifies the specific resources the plan needs to modify, and only grants execution rights if those resources are not already locked by another active agent; otherwise, the plan aborts and the locked resources are released This preserves TNR's safety guarantee for each concurrent transaction by ensuring that concurrent operations are properly isolated.\n\nThis model can be formalized by extending TNR. \n\nEach mitigation agent first comes up with a plan $P_i$, which is associated with a set of resources, ${R}(P_i)$ its write actions will mutate. The controller maintains a global set of locked resources, $R_{\\text{locked}}$, which is the union of resource sets for all currently executing plans, $R_{\\text{locked}} = \\bigcup_{j \\in \\text{executing}} R(P_j)$. A newly submitted plan $P_i$ is admitted for execution if and only if its resource set is disjoint from the global lock set: $\\text{Admit}(P_i) \\iff R(P_i) \\cap R_{\\text{locked}} = \\emptyset$.\n\nUpon admission, the plan's resources are atomically added to the global lock set ($R_{\\text{locked}} \\leftarrow R_{\\text{locked}} \\cup R(P_i)$), effectively acquiring a distributed lock on them. The agent then proceeds to execute its transaction under the original TNR semantics, guaranteeing that the severity metric does not increase: $\\mu(s_{\\text{post}}) \\le \\mu(s_{\\text{pre}})$. Once the transaction completes (via commit or abort), its resources are released: $R_{\\text{locked}} \\leftarrow R_{\\text{locked}} \\setminus R(P_i)$. This mechanism upholds the \"Writer Exclusivity\" (A1) at the resource level rather than the system level, ensuring that the safety properties of TNR hold for each transaction. Note that if the transaction aborts, the undo agent is responsible for undoing all mutation actions of the plan. This strict resource isolation inherently prevents read-after-write hazards, as an agent is guaranteed a consistent view of any resource not explicitly locked by its own transaction.\n\n--------\n\n**> Q6: Oracle/Success definition; subtle, latent degradations**\n\nWe use standard observability data, which defines Service Level Agreements, as the oracles (SRE agents like Stratus target observable symptoms and user-perceived incidents.) Addressing subtle, latent degradation (e.g., small performance regression) may need to enhance system observability. One intriguing direction is agentic techniques to autonomously enhance observability for a given problem (e.g., exposing more low-level knobs and metrics).\n\n--------\n\n**> Q7: Detail safeguards against compromise and the risk of amplified attacks.**\n\nWe’ll expand on the discussion of safeguards against LLM errors, including sandboxed execution, access control, human-in-the-loop approval, and tamper detection.\n\n--------\n\n**> Q8: Accountability/Liability: Who is responsible for failures caused by Stratus?**\n\nThis is a deep question! The accountability requires deep thoughts – it doesn’t resemble the current human-based DevOps/SRE accountability system. The system needs to account for the LLM developers, agent developers, and the agent operators.\n\nAs of the current industry practices:\n* Agents with writer roles (which can change system states) are enforced to request human approval before execution. \n* Agents’ action stack and tool interface are intercepted for every tool call or action pushed/popped. \n* Before executing a write operation, the agent presents a concise summary of the proposed command, its intent, and relevant telemetry, enabling the human to verify it.\n* The success rates of agents are monitored to decide their autonomy.\n\n--------\n\n**> Q9: Potential negative societal impact**\n\nWe don't expect negative societal impacts. Our goal is to emphasize safety as the first-class principle of SRE agents and demonstrate one way of doing it. With the safety guarantee, system states do not degrade beyond the initial error state – Stratus/TNR enables more effective failure mitigation, which increases the availability of cloud systems (the computing infrastructures of many end-user applications) and leads to positive societal impacts. Note that all experiments are conducted in emulated environments (AIOpsLab and ITBench); the research does not reduce availability of real-world systems."}, {"review_id": "M4Y2rKc45g", "reviewer": "Reviewer_sLKR", "summary": "This paper proposes Stratus, an LLM-based multi-agent system for fully autonomous Site Reliability Engineering (SRE) in cloud environments. Specifically, it presents a multi-agent architecture with specialized agents (detection, diagnosis, mitigation, undo) coordinated via a state machine and a formal safety framework, i.e., Transactional Non-Regression (TNR), to ensure mitigation actions not degrade system beyond initial fault severity. Extensive experimental evaluation on SRE benchmarks (AIOpsLab/ITBench) shows 1.5–5.4× higher mitigation success rates compared to state-of-the-art baselines.", "questions": "1. Can Stratus handle correlated faults (e.g., simultaneous node/memory failures)? If not, what TNR extensions are needed? Real-world validation?\n2. How would you modify TNR for environments with parallel agents? Would distributed locking suffice?\n3. Is it possible to replace GPT-4o with smaller LLMs for undo agent? In this way, it may provide latency/accuracy trade-offs.\n4. RCA failures often misattribute hardware issues to config errors. Would embedding domain knowledge improve it?\n5. Could agents be spawned on-demand? How would state-machine consistency be maintained?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This work claims to make the first effort to enable fully autonomous failure mitigation (beyond diagnosis/recommendation).\n2. The proposed TNR formalizes safety for agentic systems operating critical infrastructure.\n3. It provides rigorous TNR proof and ablation studies to validate design choices. \n4. The code is open-sourced, which makes it easy to reproduce the results. \n5. This paper is well-structured and easy to follow. \n\nWeaknesses:\n1. Benchmarks (AIOpsLab/ITBench) emulate isolated faults. Absent evaluation on cascading failures (e.g., network partitions inducing state corruption) or multi-tenant edge cases.\n2. Diagnosis agent achieves only 34.6% RCA accuracy. Root causes (e.g., hardware vs. misconfiguration) conflated in failure traces.\n3. Mitigation latency and cost hinder real-time deployment. There is no analysis of cold-start delays or agent memory footprint.\n4. Natural language tools may permit prompt injection attacks. Sandboxing lacks adversarial testing.\n5. State-machine orchestration requires manual workflow definitions. Contrasts with dynamic agent frameworks. Limits adaptability to novel failures.", "was_revised": true, "final_justification": "I am grateful for the authors' significant efforts in their response, which have addressed the majority of my concerns. Taking into account the overall quality of this work, I would like to maintain my rating score.", "rebuttal": "Thank you for the insightful review! We'll carefully address the review comments.\n\nWe first answer the explicitly listed questions and then respond to other concerns.\n\n--------\n\n**> Q1: Can Stratus handle correlated faults (e.g., simultaneous node/memory failures)? If not, what TNR extensions are needed? Real-world validation?**\n\nStratus can handle cascading failures. Many problems in the two benchmarks are _cascading failures_. As the applications all use the microservice architecture, the initial fault propagates to dependent services. Mitigating such failures requires the agent to resolve the root cause, instead of the dependent services on the propagation chain. \nStratus can also handle _correlated_ failures. 25 problems in the two benchmarks manifest as correlated failures where the root cause fails multiple dependent services. Similar to cascading failures, mitigating correlated failures requires Stratus to resolve the root cause.\n\nA more challenging case is _concurrent faults_ that are inter-dependent. We crafted such a new problem in ITBench, where we have a quorum system of three nodes (one leader and two followers); we inject data corruptions in the followers’ data volumes and inject a crashing bug in the leader. A correct mitigation requires to first fix the data volumes of the followers and then restart the leader. Unfortunately, Stratus failed to mitigate this problem as it didn’t localize follower data volumes as one of the root causes. It did restart the leader; however, due to the dependencies, restarting the leader without fixing the two followers cannot mitigate the failures (as the leader cannot form the quorum). Note that TNR still applies in this problem, despite Stratus failing to solve it after many attempts.\n\n--------\n\n**> Q2: How would you modify TNR for environments with parallel agents? Would distributed locking suffice?**\n\nTo adapt Transactional No-Regression (TNR) for environments with concurrent agents, we can evolve the single-writer lock (`A-Lock`) into a more fine-grained, resource-aware concurrency control system. This can be achieved by designing a Coordination Controller, which effectively functions as a distributed lock manager. The basic idea is to allow multiple mitigation agents to execute their transactions concurrently, with intended actions not conflicting over shared resources. This controller dynamically inspects the \"mitigation plan\" from each agent, identifies the specific resources the plan needs to modify, and only grants execution rights if those resources are not already locked by another active agent; otherwise, the plan aborts and the locked resources are released This preserves TNR's safety guarantee for each concurrent transaction by ensuring that concurrent operations are properly isolated.\n\nThis model can be formalized by extending TNR. \n\nEach concurrently executing mitigation agent works on the output of the diagnosis agent. The mitigation agent first comes up with a plan $P_i$, which is associated with a set of system resources, ${R}(P_i)$ containing the unique identifiers of all system components its write actions will mutate. The controller maintains a global set of locked resources, $R_{\\text{locked}}$, which is the union of resource sets for all currently executing plans, $R_{\\text{locked}} = \\bigcup_{j \\in \\text{executing}} R(P_j)$. A newly submitted plan $P_i$ is admitted for execution if and only if its resource set is disjoint from the global lock set: $\\text{Admit}(P_i) \\iff R(P_i) \\cap R_{\\text{locked}} = \\emptyset$.\n\nUpon admission, the plan's resources are atomically added to the global lock set ($R_{\\text{locked}} \\leftarrow R_{\\text{locked}} \\cup R(P_i)$), effectively acquiring a distributed lock on them. The agent then proceeds to execute its transaction under the original TNR semantics, guaranteeing that the severity metric does not increase: $\\mu(s_{\\text{post}}) \\le \\mu(s_{\\text{pre}})$. Once the transaction completes (via commit or abort), its resources are released: $R_{\\text{locked}} \\leftarrow R_{\\text{locked}} \\setminus R(P_i)$. This mechanism upholds the \"Writer Exclusivity\" (A1) at the resource level rather than the system level, ensuring that the safety properties of TNR hold for each transaction. Note that if the transaction aborts, the undo agent is responsible for undoing all mutation actions of the plan. This strict resource isolation inherently prevents read-after-write hazards, as an agent is guaranteed a consistent view of any resource not explicitly locked by its own transaction.\n\n--------\n\n**> Q3: Is it possible to replace GPT-4o with smaller LLMs for undo agent? In this way, it may provide latency/accuracy trade-offs.**\n\nYes. A smaller model can effectively replace GPT-4o in the undo agent (the model is used for undo API calls). We tried Llama3.3-70B and it has the same effect as GPT-4o. We also tried Llama3.2-11B (a further smaller model), but the 11B model was ineffective (e.g., incorrect tool usage such as wrong API invocation and wrong parameters).\n\n--------\n\n**> Q4: RCA failures often misattribute hardware issues to config errors. Would embedding domain knowledge improve it?**\n\nIn principle, domain knowledge is definitely helpful for enhancing RCA agents especially for known problems; for example, prior work (e.g., Ref. [20, 61] in the submission manuscript) shows that leveraging Troubleshooting Guides (TSG) can improve the effectiveness of RCA. Such domain knowledge can be encoded either through model fine-tuning or by RAG (Retrieval-Augmented Generation). Unfortunately, domain knowledge such as TSG is largely proprietary. Future work can explore using publicly available data (e.g., user manual and GitHub issues) to enhance RCA agents.\n\n--------\n\n**> Q5: Could agents be spawned on-demand? How would state-machine consistency be maintained?**\n\nYes. Only the Detection agent is long-running. The Diagnosis and Mitigation agents are spawned on demand when a failure is detected. We expect these agents to have lifecycles till the end of the failure; therefore, the state machine is maintained throughout the failure duration. We don’t see benefits of having more fine-grained agent life cycles.\n\n--------\n\n**> Q6: Mitigation latency and cost hinder real-time deployment.**\n\nStratus is not slower or more expensive than the other evaluated agents; the reason Stratus took longer on harder problems is that TNR enables Stratus to make multiple attempts to solve the problem, while the other agents can only make fewer attempts. It is a key advantage of Stratus, not a limitation.\n\nWe have a few ideas on accelerating Stratus (inspired by your questions; thank you!): (1) caching solutions for learning so future resolutions of similar type can be much faster (which becomes domain knowledge in a similar vein as Q4); (2) using smaller models (see Q3), and (3) asking the AI model reason about efficiency of mitigation strategies (e.g., cheapest strategy first).\n\n--------\n\n**> Q7: Analysis of cold-start delays or agent memory footprint.**\n\nOur agents have no cold-start delay as they directly use existing LLMs; it doesn’t maintain long-term memory.\n\nWe measure the runtime memory footprints of the agent when addressing a heavy problem (taking 13 steps to mitigate). The total DRAM usage of the Detection, Diagnosis, Mitigation, and Undo agents are 548.3 MB, 533.3 MB, 545.7 MB, and 534 MB, respectively. The agent runtime (CrewAI) and dependencies (e.g., tools) dominate the memory usage (around 530 MB). The remaining memory usage comes from prompts, and runtime in-memory states.\n\n--------\n\n**> Q8: Natural language tools may permit prompt injection attacks. Sandboxing lacks adversarial testing.**\n\nWe will expand on the discussion of safeguards against LLM errors, including sandboxed execution, access control, human-in-the-loop approval, and tamper detection.\n\n--------\n\n**> Q9: State-machine orchestration requires manual workflow definitions. Contrasts with dynamic agent frameworks. Limits adaptability to novel failures.**\n\nFair point. There’s a tradeoff between creativity and the ability of safety reasoning; we prioritized the latter as safety is the first-class principle. We believe that the current state machine doesn’t impair creativity as Detection, Diagnosis, Mitigation, and Undo are essential to SRE; novel failures are addressed by creativity and autonomy of the corresponding agents."}, {"review_id": "EJQDXVh1eh", "reviewer": "Reviewer_j3f4", "summary": "This paper introduces STRATUS, a novel LLM-based multi-agent system designed to automate Site Reliability Engineering (SRE) in large-scale cloud environments. As cloud systems scale and become increasingly complex, traditional human-in-the-loop reliability practices struggle to keep up with the volume and speed of failures such as software bugs, misconfigurations, and hardware faults.", "questions": "Issue: STRATUS includes several specialized agents (detection, diagnosis, mitigation, undo), but it remains unclear how essential each component is.", "limitations": "as question", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "STRATUS addresses this challenge by coordinating a set of specialized agents—such as detection, diagnosis, mitigation, and undo agents—within a structured state machine to autonomously perform end-to-end failure management. The authors also propose a formal safety property called Transactional No-Regression (TNR) to guide safe and reversible actions during agentic operation, enabling iterative and safe learning and recovery.\n\nEmpirical evaluations on benchmark SRE testbeds (AIOpsLab and ITBench) show that STRATUS significantly outperforms existing SRE systems, achieving at least 1.5× higher success rates in failure mitigation across various modules. The work demonstrates the practical feasibility and advantages of deploying LLM-driven autonomous systems for cloud reliability management.", "was_revised": true, "final_justification": "The rebuttal squarely addresses the core concern about the necessity and design of STRATUS’s specialized agents. The authors (i) justify detection/diagnosis/mitigation/undo as first-class SRE tasks, with undo motivated by production safety. I will keep the score 4.", "rebuttal": "Thank you for the insightful review! We'll carefully address the review comments.\n\nWe answer the listed question.\n\n**> Q1:  STRATUS includes several specialized agents (detection, diagnosis, mitigation, undo), but it remains unclear how essential each component is.**\n\nThanks for the question. The Detection, Diagnosis, and Mitigation agents are corresponding to the common SRE tasks (see Ref. [12] in the submitted manuscript) and are thus essential components of an SRE system. In AIOpsLab and ITBench, those tasks are explicitly required for evaluating any agentic SRE systems, except that ITBench does not evaluate on detection. Undo agents are introduced for safety guarantee which is the focus of the paper and we argue that production safety must be a first-class principle of any SRE agents.\n\nNote that our multi-agent architecture is customizable and extensible – one can add, remove, and replace agents. For example, ITBench does not need the detection agent.\n\nIf the question is about our design of the specialized agents, we did consider and experiment with a conversation-based multi-agent framework, AutoGen, to construct a team of conversable agents with the roles of Application Developer, Platform Engineer, System Administrator, and a Team Manager to aggregate opinions and issue actions. However, we found that such a design was neither effective nor efficient.\n\nWe sampled 48 problems from AIOpsLab, where the incidents happen on the application and platform layers. However, the role-based design only solves 10 out of 48 problems. The multi-agent conversations tend to over-communicate and each agent only has a partial view of the entire system stack. For example, when round robin is used as the speaker selection method, every agent speaks in each step, slowing down decision making and causing agents to repeatedly collect telemetry data to investigate/confirm other agents’ observations and hypotheses. For example, solving a simple detection problem `user_unregistered_mongodb-1-detection` took 893 seconds, which is 25X longer than Stratus (Table 8 in Appendix). We believe that multi-agent conversation is not the right design for SRE agents, despite their promises in use cases like coding, Q&A, and entertainment as shown in the AutoGen paper.\n\nThe rationales of our design are:\n* Each agent has a global view of the system stack (which is a key advantage of AI agents over human engineers).\n* Each agent is corresponding to a concrete SRE task.\n* The design enables timely decision making as SRE tasks are time sensitive.\n* The state-machine based control logic facilitates safety reasoning (unlike many use cases like coding and writing, safety is the first principle of SRE agents)."}]

Author rebuttal:

Meta-review: The paper introduces STRATUS, an LLM-based multi-agent system designed for autonomous Site Reliability Engineering (SRE) in cloud services. The system uses multiple specialized agents and a key safety specification called Transactional No-Regression (TNR), which allows for safe, iterative failure mitigation with an undo-and-retry mechanism. Experimental results on the AIOpsLab and ITBench benchmark suites demonstrate that STRATUS significantly outperforms state-of-the-art SRE agents, improving the failure mitigation success rate by at least 1.5 times. The reviewers’ concerns were adequately addressed to the reviewers’ satisfaction by the authors during the rebuttal period. Both the reviewers and I found the paper to have strong potential impact for applications with solid theoretical foundations in SRE and the broader machine learning community.
Decision: Accept (poster)