Title: STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds
Venue: NeurIPS 2025
Accepted: yes
Keywords: Agentic AI

Large Language Model

System Reliability
Abstract: In cloud-scale systems, failures are the norm. A distributed computing cluster exhibits hundreds of machine failures and thousands of disk failures; software bugs and misconfigurations are reported to be more frequent. The demand for autonomous, AI-driven reliability engineering continues to grow, as existing human-in-the-loop practices can hardly keep up with the scale of modern clouds. This paper presents STRATUS, an LLM-based multi-agent system for realizing autonomous Site Reliability Engineering (SRE) of cloud services. STRATUS consists of multiple specialized agents (e.g., for failure detection, diagnosis, mitigation), organized in a state machine to assist system-level safety reasoning and enforcement. We formalize a key safety specification of agentic SRE systems like STRATUS, termed Transactional No-Regression (TNR), which enables safe exploration and iteration. We show that TNR can effectively improve autonomous failure mitigation. STRATUS significantly outperforms state-of-the-art SRE agents in terms of success rate of failure mitigation problems in AIOpsLab and ITBench (two SRE benchmark suites), by at least 1.5 times across various models. STRATUS shows a promising path toward practical deployment of agentic systems for cloud reliability.


Peer reviews:
comments: SUMMARY:
This paper introduces STRATUS, a novel LLM-based multi-agent system designed to automate Site Reliability Engineering (SRE) in large-scale cloud environments. As cloud systems scale and become increasingly complex, traditional human-in-the-loop reliability practices struggle to keep up with the volume and speed of failures such as software bugs, misconfigurations, and hardware faults.

STRENGTHS AND WEAKNESSES:
STRATUS addresses this challenge by coordinating a set of specialized agents—such as detection, diagnosis, mitigation, and undo agents—within a structured state machine to autonomously perform end-to-end failure management. The authors also propose a formal safety property called Transactional No-Regression (TNR) to guide safe and reversible actions during agentic operation, enabling iterative and safe learning and recovery.

Empirical evaluations on benchmark SRE testbeds (AIOpsLab and ITBench) show that STRATUS significantly outperforms existing SRE systems, achieving at least 1.5× higher success rates in failure mitigation across various modules. The work demonstrates the practical feasibility and advantages of deploying LLM-driven autonomous systems for cloud reliability management.

QUESTIONS:
Issue: STRATUS includes several specialized agents (detection, diagnosis, mitigation, undo), but it remains unclear how essential each component is.

LIMITATIONS:
as question
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Stratus, an LLM-based multi-agent system for fully autonomous Site Reliability Engineering (SRE) in cloud environments. Specifically, it presents a multi-agent architecture with specialized agents (detection, diagnosis, mitigation, undo) coordinated via a state machine and a formal safety framework, i.e., Transactional Non-Regression (TNR), to ensure mitigation actions not degrade system beyond initial fault severity. Extensive experimental evaluation on SRE benchmarks (AIOpsLab/ITBench) shows 1.5–5.4× higher mitigation success rates compared to state-of-the-art baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This work claims to make the first effort to enable fully autonomous failure mitigation (beyond diagnosis/recommendation).\n2. The proposed TNR formalizes safety for agentic systems operating critical infrastructure.\n3. It provides rigorous TNR proof and ablation studies to validate design choices. \n4. The code is open-sourced, which makes it easy to reproduce the results. \n5. This paper is well-structured and easy to follow. \n\nWeaknesses:\n1. Benchmarks (AIOpsLab/ITBench) emulate isolated faults. Absent evaluation on cascading failures (e.g., network partitions inducing state corruption) or multi-tenant edge cases.\n2. Diagnosis agent achieves only 34.6% RCA accuracy. Root causes (e.g., hardware vs. misconfiguration) conflated in failure traces.\n3. Mitigation latency and cost hinder real-time deployment. There is no analysis of cold-start delays or agent memory footprint.\n4. Natural language tools may permit prompt injection attacks. Sandboxing lacks adversarial testing.\n5. State-machine orchestration requires manual workflow definitions. Contrasts with dynamic agent frameworks. Limits adaptability to novel failures.\n\nQUESTIONS:\n1. Can Stratus handle correlated faults (e.g., simultaneous node/memory failures)? If not, what TNR extensions are needed? Real-world validation?\n2. How would you modify TNR for environments with parallel agents? Would distributed locking suffice?\n3. Is it possible to replace GPT-4o with smaller LLMs for undo agent? In this way, it may provide latency/accuracy trade-offs.\n4. RCA failures often misattribute hardware issues to config errors. Would embedding domain knowledge improve it?\n5. Could agents be spawned on-demand? How would state-machine consistency be maintained?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSTRATUS is a multi-agent system for autonomous Site Reliability Engineering (SRE) in cloud environments. It automates failure detection, diagnosis, and mitigation. Its main contribution is \"Transactional No-Regression (TNR),\" a safety specification ensuring that mitigation actions can be undone and system health never worsens. STRATUS uses specialized agents (detection, diagnosis, mitigation, undo) and tools for cloud interaction, with sandboxing and a stack-based undo for TNR. Evaluations on AIOpsLab and ITBench show STRATUS significantly outperforms existing SRE agents in mitigation success rates, primarily due to TNR's undo-and-retry mechanism, validating its potential for practical autonomous cloud reliability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\nI appreciate the author tackling a very important problem in AIOps and system reliability. \n*   **Significance:** Addresses critical, real-world SRE challenges by pursuing fully autonomous cloud reliability, moving beyond human-assisted approaches. This could transform incident response.\n*   **Originality:** Introduces and formalizes \"Transactional No-Regression (TNR),\" a novel safety mechanism enabling safe exploration and iterative mitigation through guaranteed undo capabilities. Its specialized multi-agent architecture for SRE is also a unique design choice.\n*   **Quality:** Backs TNR with rigorous formalization and implementation details. Demonstrates strong empirical validation, significantly outperforming baselines in mitigation success rates across multiple LLMs and benchmarks (AIOpsLab, ITBench).\n*   **Clarity:** The paper is well-structured, clearly written, and enhanced by illustrative examples and a detailed appendix, making complex concepts accessible.\n\n**Weaknesses:**\nThe overall evaluation is constrained. \n*   **Quality - Practical Limitations:**\n    *   **Imperfect Undo:** Acknowledged challenges in achieving \"perfect undo\" for all complex, stateful, or external interactions mean some subtle degradations might persist post-rollback.\n    *   **Performance/Cost Overhead:** STRATUS is significantly slower and more expensive than other agents, posing a concern for urgent incidents where rapid resolution is critical. It would be helpful to state solutions that accelerate the process. \n    *   **Benchmark Generalizability:** TNR's benefits were less evident in ITBench due to a prevalence of non-persistent faults, raising questions about its effectiveness for persistent, harder-to-undo real-world issues.\n*   **Significance - Applicability Constraints:**\n    *   **No Concurrent Writers:** The assumption of no concurrent writer agents significantly limits its applicability in highly dynamic, multi-actor cloud environments where simultaneous changes are common.\n    *   **Oracle/Success Definition:** The success criteria, based on immediate alerts and basic health checks, might not fully capture subtle, latent, or long-term system degradations, potentially leading to a false sense of full recovery.\n\nQUESTIONS:\nSection 3.1.1 lists three assumptions for TNR: Writer Exclusivity (A1), Faithful Undo (A2), and Bounded Risk Window (A3). Could you discuss a scenario where one of these assumptions might be violated in a real-world cloud system and the potential consequences for STRATUS's safety guarantees?\n\nThe paper mentions that \"realizing perfect undo for all conceivable state changes in complex environments like cloud systems remains a practical challenge.\" What are some specific types of state changes or operations in a cloud environment that would be particularly difficult to perfectly undo, and how might STRATUS's current implementation address or be limited by these challenges?\n\nPersistent vs. Non-Persistent Faults in ITBench: Your analysis of ITBench (Table 6) points out that \"in 8 out of 18 problems, restarting the target pods clears the incident alerts\" because \"the injected faults do not persist.\" This suggests a specific characteristic of the benchmark. How would STRATUS's performance and the effectiveness of TNR differ when confronted with persistent faults (e.g., a hardware defect or a deeply embedded misconfiguration) that cannot be resolved by simple restarts?\n\nLIMITATIONS:\nThe authors acknowledge some technical limitations (perfect undo challenges, no concurrency control, incomplete confinement rules, ITBench's non-persistent faults). However, they don't adequately address potential negative societal impacts.\n\n### Suggestions:\n\"Perfect Undo\": Detail why certain changes are hard to undo and propose handling for partial/irreversible actions.\nLLM Errors: Beyond TNR, explain strategies to minimize initial LLM errors caused by non-determinism and hallucination (e.g., structured generation, fine-tuning, prompt engineering, ensemble methods, human-in-the-loop for high-risk ops)\n\n### Address other concerns: \nSecurity Implications: Detail safeguards against compromise and the risk of amplified attacks.\nAccountability/Liability: Who is responsible for failures caused by STRATUS?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
The authors present STRATUS, a method for Site Reliability Engineering of cloud services. The multi-agent approach presents TNR which allows for failure mitigation, a key advantage over existing approaches. It shows improved mitigation of failures, though at a longer cost and computational overhead.

STRENGTHS AND WEAKNESSES:
- Strengths
  - Clarity: The algorithm is explained in detail and TNR is proven by induction
  - Quality: The experiments show strong results across different takss.
  - Significance: The work addresses a key problem in the real world and shows string promise	
  - Originality: TNR is an important aspect for SRE 

- Weaknesses
  - Clarity: The authors must formally state exactly what ε is. They must also 
 add a diagram or clear example of how the LLM is used by the agents.
  - Please move around figures and rename captions to have the full information and not really on the text e.g. Figure 5.
  - Quality: The reasons for Stratus struggling on RCA problems needs to be expanded upon: for the rest of the paper, the user has expanded on limitations but should be here as well.
  - Would like to see an ablation of the multi-aent system i.e. with different methods of separating the agents
  - While authonomy rovides obvious benefits, the authors must be careful of downplaying the importance of human intervention.

QUESTIONS:
•	Why does Stratus underperform on RCA tasks?
•	Were any other multi-agent decompositions considered and were any experiments run on these?
•	Can the system be adapted for hybrid settings i.e. how would you balance agent autonomy with human oversight in safety-critical environments?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-07