ChaosEater: Fully Automating Chaos Engineering with Large Language Models

Authors: Daisuke Kikuta, Hiroki Ikeuchi, Kengo Tajiri, Yuusuke Nakano
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: Chaos Engineering, Software Engineering, Infrastructure as Code, Large Language Models

Abstract:
Chaos Engineering (CE) is an engineering technique aimed at improving the resiliency of distributed systems.
It involves artificially injecting specific failures into a distributed system and observing its behavior in response. 
Based on the observation, the system can be proactively improved to handle those failures.
Recent CE tools realize the automated execution of predefined CE experiments.
However, defining these experiments and reconfiguring the system after the experiments still remain manual.

To reduce the costs of the manual operations, we propose ChaosEater, a "system" for automating the entire CE operations with Large Language Models (LLMs).
It pre-defines the general flow according to the systematic CE cycle and assigns subdivided operations within the flow to LLMs.
We assume systems based on Infrastructure as Code (IaC), wherein the system configurations and artificial failures are managed through code.
Hence, the LLMs' operations in our "system" correspond to software engineering tasks, including requirement definition, code generation and debugging, and testing.

We validate our "system" through case studies on both small and large systems.
The results demonstrate that our "system" significantly reduces both time and monetary costs while completing a reasonable CE cycle.
Our code is available in the Supplementary Material.

Decision: reject
Reviewer scores: 6, 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730108336245

thread_type:
official_review

<thread_object>
creator: reviewer_alzw

<content>
summary:
This paper proposes ChaosEater, an automatic framework for Chaos Engineering (CE) operators. In particular, each CE operator is automated using one or multiple LLM agents, equipped with carefully crafted system/user/AI prompts. The experimental results demonstrate that the proposed system significantly reduces both time and monetary costs within the CE cycle.

strengths:
- The paper presents an intriguing application of LLM agents in improving distributed system resilience.
- The framework effectively automates the CE process, minimizing the need for manual intervention.
- The authors provide the implementation of the proposed methods, which could be beneficial to the research community.
- The paper is well-organized, with a clear and logical flow.

weaknesses:
- It is unclear to what extent ChaosEater reduces the reliance on human expertise. For example, steady-state selection still requires experts to define measurable states, with the agent only used for state selection.
- Both steady-state selection and failure injection are determined by LLM agents, whose inherent biases could hinder the discovery of new issues. 
- Figure 7 is difficult to comprehend; it could be better to include a summarized version in the main text and move the detailed figure to the Appendix.
- The experiments seem limited to a toy example, which may not fully demonstrate the effectiveness of ChaosEater.

questions:
- Is the LLM primarily used in most phases to adjust parameters within predefined templates (e.g., fault scopes)? Could a smaller model be employed in the experiment instead?
- The prompts in ChaosEater appear to be carefully designed, resulting in well-structured LLM responses. I suggest providing some detailed prompts in the Appendix. 
- In addition, what is the cost of prompt tuning, and can the agent maintain robustness when using other LLM models or in different environments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730171569213

thread_type:
official_review

<thread_object>
creator: reviewer_iwjm

<content>
summary:
The author proposes a system called ChaosEater, primarily designed to automate the Chaos Engineering (CE) workflow using LLMs. It provides an efficient and low-cost solution for maintaining system resilience, offering potential for automated resilience testing and fault remediation in future complex distributed systems.

strengths:
The article leverages LLMs to automate various stages of chaos experiments, thereby reducing manual operations. This innovative application demonstrates the potential of LLMs in infrastructure-as-code and fault injection testing, showcasing a degree of novelty.

weaknesses:
1.The current version of ChaosEater relies on LLMs to handle complex input and output, particularly in multi-stage and multi-dependency chaos experiments, which can be susceptible to context length limitations. Even with models like GPT-4, longer contexts may still lead to information truncation, affecting the accuracy and comprehensiveness of the experiments.
2.The types of failures and injection methods currently supported by ChaosEater may be too limited to cover all potential faults in distributed systems. For example, the system may lack support for specific failure injections related to storage systems, such as disk latency or database locking, as well as certain network issues like packet loss and jitter. Furthermore, the paper does not mention support for various complex and dynamic failure scenarios, such as cross-injection of multiple faults, cascading failures, or the failure of partially dependent components, all of which are quite common in complex systems.
3.Chaos experiments typically require real-time monitoring and response to system states to promptly terminate the experiment in the event of severe anomalies or catastrophic failures. However, the paper does not mention that ChaosEater has such real-time monitoring capabilities, which limits its ability to dynamically adjust strategies during the experiment.

questions:
The main points and issues have been outlined in the "Weaknesses" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730414177821

thread_type:
official_review

<thread_object>
creator: reviewer_jtmf

<content>
summary:
The paper proposes a system, called CHAOSEATER, that automates the entire Chaos Engineering (CE) cycle using LLMs to enhance the resilience of distributed systems. It defines a structured CE process with five phases: hypothesis setting, chaos experimentation, analysis, improvement, and final review. CHAOSEATER using Infrastructure as Code (IaC) to manage system configurations. The paper show the proposed system help in reducing the time and cost compared to manual CE

strengths:
1- The paper demonstrates how LLMs contribute to reducing time and costs in Chaos Engineering (CE).

2- The paper is well-written and includes clear figures and visualizations.

weaknesses:
1- Clearly differentiate your fully automated technique for Chaos Engineering (CE) using LLMs from existing approaches like self-refinement and self-debugging. Provide specific details on what sets your method apart and its unique contributions.

2- Include accuracy metrics in Table 1 to present success and failure rates alongside time and cost. This will offer a more comprehensive view of the technique's effectiveness.

3- To position this work as a foundational effort in automating system resilience improvement, construct a larger dataset, establish robust evaluation metrics for correctness, and include baseline comparisons with prior automated approaches using LLMs.

4- Add a qualitative analysis and human evaluation component to assess the effectiveness of LLMs in Chaos Engineering tasks. This will strengthen the paper by showing how well LLMs perform in real-world scenarios.

questions:
If the paper proposes an automated system for Chaos Engineering (CE), consider constructing a benchmark to evaluate multiple LLMs and compare different automated approaches. This could provide a more comprehensive assessment of the system's effectiveness and highlight its unique contributions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648409450

thread_type:
official_review

<thread_object>
creator: reviewer_jkoh

<content>
summary:
The paper presents a framework for automating Chaos Engineering (CE) workflows using LLMs. The system, ChaosEater uses multi-step, multi-agent workflow to generate hypotheses, inject failures, run experiments, analyze output, suggest improvements and repeat until Validation as Code passes. Chaos Eater may automate CE in Kubernetes (K8s) environments managed with Infrastructure as Code (IaC).

**Architecture**: The system consists of five primary phases:
   - **Pre-processing**: Processes configuration dependencies.
   - **Hypothesis Definition**: Defines system resilience criteria (steady states) and specifies failure scenarios for testing.
   - **Experimentation**: Conducts chaos experiments by injecting pre-defined failures while validating steady states in real-time.
   - **Analysis**: Reviews the experiment results to assess whether the system meets the resilience hypothesis.
   - **Improvement**: If the hypothesis is not satisfied, reconfigures the system accordingly and repeats the experimentation phase.

strengths:
1. **CE Automation**: ChaosEater attempts full automation of Chaos Engineering, covering hypothesis generation, fault injection, and iterative system improvement using LLMs and agentic architecture.
2. **Well Defined Architecture**: Authors clearly define the architecture of the system and LLMs/agents that work in each of the parts.
3. **Demonstrated Case Study**: System is demonstrated on a case study with a Kubernetes-managed Nginx server system.
4. **Cost and Time Efficiency**: The system would significantly reduce the time and costs associated with manual CE processes.
5. **Validation as Code (VaC)**: Provides a transparent and consistent method for validating system resilience.

weaknesses:
### Weaknesses
1. **Demonstrated on Toy System Only**: System has been demonstrated on a toy system with a very simple failure. It is not known if system will work well on actual large systems. Could you show results of experiments on a benchmark set or on larger systems? Also see weakness 2.
2. **Challenge in Vulnerability Discovery**: The system may not be able to identify issues in already resilient systems, where fault discovery requires deeper analysis.
3. **Limited to Development Environments**: Currently operates only in development environments. Additionally, CHAOSEATER struggles to uncover vulnerabilities in systems that are already resilient.
4. **Limited to Configuration Improvements**: If I understand correctly, only K8s configuration scripts can be changed in the system improvement step. In other words, the system does not automatically change the tested system code. This is a significant limitation, although I understand that currently LLMs may not be able to automatically change the tested system code to improve resiliency.

questions:
Figure 7 is unreadable while taking a whole page – please replace with something that serves better in presenting the system.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742650672

thread_type:
official_review

<thread_object>
creator: reviewer_ujqs

<content>
summary:
The paper presents a framework for automated testing and improving Infrastructure-as-Code systems based on Kubernetes. It follows the Chaos Engineering (CE) approach, which observes how the system reacts to artificially injected deficiencies. Based on configuration files, manifests, etc., the framework automatically analyzes to identify the normal behavior, then modifies the configuration and checks whether the system behavior degrades significantly, and, if necessary, modifies the system configuration to make the system more resilient.

The approach essentially boils down to using scripts to connect existing CE tools and having LLMs figure out parameters and code modifications as required. I'm not an expert in CE, so I don't know how hard that is. I welcome that the authors did make an effort to insert verification and validation steps to ensure the soundness of the approach. All in all, this seems like a promising step in applying LLMs to an interesting problem in software engineering.

strengths:
The approach essentially replaces a human CE engineer with a combination of scripts and LLM input to implement the required tests and modifications. The entire workflow is highly automated so that with enough computational resources, one could imagine the system finding and fixing a variety of problems all by itself. This might bring considerable cost reductions, at least under the assumption that compute costs will decline rapidly in the future (and LLM power will increase).

weaknesses:
The paper says little about the exhaustiveness of the approach. Given infinite time, would the system be expected to find the majority of relevant faults? The case study seems relatively simple, and even in such an idealized case, two different solutions are proposed, and it does not seem clear which one is preferable. I can imagine that this problem of choosing between alternatives is much worse in more complex systems. The authors admit in the discussion that it is not trivial to scale this to challenging tasks, and I applaud their intellectual honesty.

questions:
- How are thresholds calculated from the inspected values (margin for natural fluctuations…)?
- Why is the temperature of the LLM set to zero? Doesn't this limit the creativity (or exhaustiveness, depending on the time budget) in devising chaos experiments?
- How can you be sure that VaC scripts work as intended (e.g., rather than just always giving positive results)? For example, you could voluntarily use thresholds that are too low to check that they give negative results when necessary.
- In the case study, the analysis phase presented two solutions: changing the restart policy or using a Deployment resource. Why was the latter solution chosen in the improvement phase? What are the perceived upsides/downsides? How about the actual upsides/downsides of this solution? E.g., does introducing three replicas increase the overall cost of operating the system?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732969830532

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response by Authors: Current Discussion Summary and Revised Manuscript [1/3]

comment:
Dear all the reviewers

Thank all the reviewers for their valuable time and constructive feedback!
We have uploaded the final revised manuscript.
In the following, we summarize:

- Selected discussion with the reviewers so far
- Changes in the revised manuscript

We hope this summary helps the reviewers catch up on the current state of our work.  
For details, please refer to the threads for each reviewer.  
If you have additional questions or concerns, please feel free to ask at any time!

---
## Discussion

**D1. Only Small System Evaluated**

The reviewers ```ujQS```, ```jkoh```, and ```aLZw``` were concerned that Nginx studied in the case study is too simple to sufficiently demonstrate the effectiveness of ChaosEater. To address this, we added the case study of SockShop [1], which is a practical and large-scale e-commerce system consisting of 29 manifests. Compared to Nginx, the number of manifests is approximately 15 times, the code lines 35 times, and the tokens 40 times greater (see Table 3 in Appendix C for this statistic). Additionally, the intentional resiliency issue included in SockShop is "Single replica of Deployment resource (Deployment has already restarting mechanism)" and is more challenging.

The additional case study showed that even for the large-scale system with relatively redundant configurations, ChaosEater can identify the downtime issue of the single replica and improve the system by increasing the number of replicas. The time and monetary costs are 25 mins and USD 0.8. Intuitively, they are still significantly lower than the costs incurred by human engineers.

The reviewer ```jtMF``` suggested an additional metric obtained from multiple runs. Therefore, we also ran the case studies for Nginx and SockShop five times and calculated the "completion rate" and "reconfiguration rate". These results confirmed that ChaosEater can stably complete reasonable single CE cycles as mentioned above.

We believe that these additional case studies enhance the demonstration of ChaosEater's effectiveness.
Additionally, the successes in SockShop support that ChaosEater can effectively mitigate long-context issues, a concern raised by the reviewer ```iWjm```.

[1] https://github.com/microservices-demo/microservices-demo

---
**D2. Support for various types of failures and their modes**

The reviewer ```iWjm``` was concerned that supported failure types and failure modes are too limited in ChaosEater.
However, we believe that the current version of ChaosEater already covers most types of failures and failure modes.
ChaosEater supports all failure types of Chaos Mesh except for kernelChaos. There are 7 general failure types\*, such as PodChaos, NetworkChaos, and IOChaos, covering the majority of failures.
ChaosEater also supports complex failure modes by flexibly combining multiple Chaos Mesh failures sequentially or in parallel. Such complex failure combinations are realized by the LLM agent planning and our rule-based algorithm to convert the plan to a Chaos Mesh workflow manifest.

Please see our response to reviewer ```iWjm``` (A2, A3) for more details on this topic.

\* Including subtypes and detailed parameters, an enormous number of failure types can be generated.

---
**D3. Can Reconfiguring K8s Manifests Alone Address All Failures?**

Regarding Discussion 2, the reviewer ```jkoh``` pointed out an important point that reconfiguring K8s Manifests alone would not be sufficient to address all failure modes. Systems are constructed based on not only K8s manifests but also lower-layer settings (e.g., cluster settings) and application code (e.g., HTML/CSS/JS/Python). Therefore, the reviewer ```jkoh``` was concerned that even if ChaosEater can inject most types of failures, it would not be able to provide solutions for the cases that require modification on layers other than K8s manifests.

We agree that reconfiguring other layers than K8s manifests is necessary to improve the system resiliency in an optimal way. 
On the other hand, we still believe that solely K8s manifest reconfiguration can somehow solve the majority of failures (even if it is not the optimal solution). K8s manifests manage system resource settings, and their settings can handle most failures by increasing the redundancy/resiliency and improving the error handling of the corresponding resources.

Therefore, the current version of ChaosEater supports only K8s manifest reconfiguration as the top priority feature. We believe that this is sufficient to improve the system’s resiliency in most cases. However, to optimize the system resiliency improvement, we plan to add the other layer reconfiguration to the next version.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310343460

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Comment by Authors

comment:
Dear AC and Reviewers

We greatly appreciate all the reviewers for their valuable time and constructive feedback.  
The feedback helps us not only improve our manuscript and demonstration but also identify our system’s limitations to be addressed in future work.

Our current system still has room for improvement, including support for various LLMs and code types, more comprehensive analysis, and enhanced diversity of hypotheses through multiple CE cycles.
However, we believe our work has successfully established an initial foundation by presenting a new system architecture that can fully automate CE cycles for both small- and large-scale systems at low cost. This non-trivial architecture design, along with the potential, limitations, and future directions suggested through case studies, provides sufficient guidance for subsequent works in the new field of LLM applications. This field is expected to become increasingly important not only for infrastructure but also for ML communities.

Finally, we would like to leave a comment to once again emphasize how our contributions are related to the ML community.  
We hope these additional comments and our discussion summary serve as helpful references for your final decision.  
We sincerely thank the AC and reviewers in advance for your valuable time and additional feedback during the final decision period.

Sincerely,  
Authors


---

### Relation between our contributions and ML community
This paper proposed an LLM-based system that fully automates CE cycles, thereby reducing the time and monetary costs of CE. We believe our system innovates the conventional operations in the infrastructure of software-based systems, enabling anyone to build highly resilient systems at low cost. 

On the other hand, why does it matter to the ML community? The answer is related to the broader impact section. In the ML community, the automatic generation of software applications using LLMs has been actively explored. However, few consider their infrastructure and resiliency, which are crucial for building practical services. Our proposed system addresses this gap. Therefore, we believe that our contributions to the ML community lie in demonstrating the importance of the applications’ infrastructure and suggesting that even improving its resiliency can be automated using LLMs.

Additionally, the automation of software engineering (SE) using LLMs has been actively studied. Since CE can be regarded as SE, our work is considered a part of this trend. Unlike existing benchmarks for solving GitHub issues, CE requires raising issues independently and resolving them. We believe our work suggests the existence of such a new and challenging SE task and demonstrates the potential of LLMs in addressing this task to the ML community.

Therefore, we believe our contributions are meaningful to both (software-based) infrastructure and ML communities.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper concerns automating Chaos Engineering (CE) workflows, and proposes an LLM-based framework ChaosEater. ChaosEater uses a multi-agent 6-phase workflow ranging from hypothesis generation, failure injection to improvement suggestion and validation as code. ChaosEater is specifically designed to automate CE in Kubernetes (K8s) environments. The problem of using LLMs to automate CE workflows is novel and interesting. The main concerns shared by the reviewers and the AC are twofold. First, whether sufficient contributions have been made to the machine learning community; the paper is written in simply describing a workflow of applying LLMs for automate CE in K8s, which certainly has values for CE community but is less clear to the ML community. CE workflows may serve as a valuable benchmark for LLMs, however, the current work is more like a system work rather than setting up a benchmark. Second, whether the evaluation of using a toy example (i.e., a simple Nginx-server system that consists of two K8s manifests) provides convincing conclusions.