OpenRCA: Can Large Language Models Locate the Root Cause of Software Failures?

Authors: Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, Qi Zhang
Primary area: datasets and benchmarks
Keywords: Language models, Natural language processing, Software engineering

Abstract:
Large language models (LLMs) are driving substantial advancements in software engineering, with successful applications like Copilot and Cursor transforming real-world development practices. However, current research predominantly focuses on the early stages of development, such as code generation, while overlooking the post-development phases that are crucial to user experience. To explore the potential of LLMs in this direction, we propose OpenRCA, a benchmark dataset and evaluation framework for assessing LLMs’ ability to identify the root cause of software failures. OpenRCA includes 335 failures from three enterprise software systems, along with over 68 GB of telemetry data (logs, metrics, and traces). Given a failure case and its associated telemetry, the LLM is tasked to identify the root cause that triggered the failure, requiring comprehension of software dependencies and reasoning over heterogeneous, long-context telemetry data. Our results show substantial room for improvement, as current models can only handle the simplest cases. Even with the specially designed RCA-agent, the best-performing model, Claude 3.5, solved only 11.34% failure cases. Our work paves the way for future research in this direction.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1730440622108

thread_type:
official_review

<thread_object>
creator: reviewer_rqhx

<content>
summary:
The paper preposes OpenRCA, a benchmark for root cause analysis that requires models to extract potential root cause from large quantities of telemetry data, logs and traces. The paper also proposes RCAagent as a potential method to solve this root cause analysis problem since vanilla models struggle. The paper uses three SOTA closed-source models and three open-source models for the evaluation on OpenRCA.

strengths:
- RCA remains an important problem in realistic scenarios.
- Good paper structure, explains the key problem, terminologies, and ideas very clearly.
- Clean and good dataset with clear structures, etc.
- A realistic RCA benchmark, with huge effort made towards ensuring fairness and denoising.

weaknesses:
### Major
- The “golden set” identified in the oracle sampling might be not realistic enough. In real-life tasks, it is also one of the key challenge for the model to understand and select the correct KPI in order to diagnose the problem.
- Using agent has been a widely adopted method in other research fields. This paper fails to provide any insight into the nature of the problem and merely uses a general planner-executor agent framework almost without applying any domain knowledge. The only domain knowledge is a 42-line long rule given to the planner, which basically outlines all the things that the LLM should do. As a result the agent part seems really redundant and unnecessary. The same argument applies to the case study.
- There is no point trying vanilla LLM on this task. The author acknowledges in the paper that it is virtually impossible to give the telemetry data to the LLM, which is why they adopt the sampling method. There are two flaws in this: 1. LLM is not designed to process huge amount of data, so an agent evaluation/previous baseline evaluation should be more proper, and 2. There is no tell that if the performance of the LLM is bounded by the sampling.

### Misc
- Some apparent spelling mistakes, like “Partitial” in table 2, or “Accuray” in table 3
- The instruction given to the baseline in the artifact is far less comprehensive than the one given to the agent. Maybe fairer to give more domain knowledge to the baseline models?
- The CSV file for the record in the artifact is confusing and consists lots of excess newlines?
- Rather than reducing the size of the benchmark, why don’t do a success rate calculation on each system first and then compute the final score on the three success rates?

questions:
- Does oracle sample generalize to general RCA tasks? Is it a realistic baseline?
- Does the 42-line rule in the system prompt given to the planner LLM generalize to other cases as well? What is the insight in designing an agent this way specifically for this task?
- Is the benchmark intended for LLM or agents? Why wasn't other more specialized tools/agents from previous work tested and reported on the benchmark? 
It would also be great for the authors to address the misc weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667630501

thread_type:
official_review

<thread_object>
creator: reviewer_41p3

<content>
summary:
This paper provides OpenRCA, a benchmark set for root cause analysis for software systems. The benchmark set is composed of 335 failures and the task is to identify the timing of the failure along with the component that failed and its reason. The paper also proposed RCA-agent, a multi-agent approach that utilizes a coe execution environment to effectively utilize telemetry data. While the best performing LLM was able to solve 6% of failure cases, RCA-agent increased it 11%.

strengths:
The paper introduces a novel benchmark dataset for root cause analysis. The benchmark set is carefully designed with emphasis on consistency and quality. This in my opinion is a significant contribution as it can likely help advance the field, motivating further contributions utilizing this dataset. The second contribution on RCA-agent is limited in its originality as multi-agent systems utilizing code execution are widely available. Nevertheless, showcasing the increase in accuracy through an agentic approach motivates further research on this. The paper is very well written and organized.

weaknesses:
The paper puts emphasis on creating a high quality dataset and one of the processing steps was the removal of failure records where the root cause cannot be identified using the given telemetry. Envisioning a system which acts based on the analysis from automated root cause analysis, it is vital to be able to flag if the root cause cannot be identified from given data. Therefore, the benchmark set should reflect that in my opinion. 

Due to the highly stochastic nature of agentic systems as well as LLMs, I think evaluations would be stronger if authors can provide error bars.

As the paper focuses on root cause analysis for distributed systems, one insight that would make the paper stronger is how LLMs and the proposed RCA-agent perform as the scale of the system increases. That would help the readers understand limitations and applicability of such approaches.

questions:
- The comparative analysis of LLMs are interesting but given the rather low performance of individual LLM’s, I’m curious to see if one would get any additional boost in performance when they are used in an ensemble fashion. This also helps to understand if the LLMs considered identify similar failures or not.
- Table 3. For the hard category, all LLM’s even in the agent setting gets 0. I’m guessing maybe identifying the reason is harder than identifying the timing of the issue. Following this intuition, I’m wondering if there is a task that is difficult across the board?  
- Figure 5. The accuracy distribution for RCA-agent is completely different than the sampling based ones. For instance, using GPT-4o, we see that the accuracy is reduced for Telecom and bank datasets but increased for Market dataset. Similarly, using Llama3.1, we see that the accuracy is similar for Telecom and Bank but reduced for the Market dataset. I’d love to hear more insights on these behaviors.
- Could you clarify which model size you used for Llama3.1-instruct?
- Nit: Table 4. Explain what drop ratio is in the title.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720337132

thread_type:
official_review

<thread_object>
creator: reviewer_rsze

<content>
summary:
The paper introduces OpenRCA, an innovative benchmark designed to evaluate large language models in identifying root causes of software failures using diverse, long-context telemetry data. The authors evaluate state-of-the-art LLMs on this benchmark, offering a high-level analysis of their performance. The authors also introduce RCA-Agent as a tailored solution for OpenRCA task. The findings suggest significant room for enhancing LLM capabilities in solving real-world service reliable problems.

strengths:
- OpenRCA pioneers the application of LLMs in the post-development phase of the software life-cycle, highlighting a new direction to use LLMs for identifying and analyzing real-world software issues.

- The authors’ rigorous data-cleaning process in building OpenRCA is valuable, involving system selection, data balancing, data calibration and query synthesis.

- The goal-driven task design covers various aspects of RCA tasks, framing them in a generalized, natural-language query format.

- The authors conduct a comprehensive experiments of popular closed and open LLMs with different strategies (Balanced Sampling, Oracle Sampling and RCA-Agent). And the authors also provide high-level comparative analysis for each setting based on experiment results.

weaknesses:
- OpenRCA includes only three distributed software systems, which is small and may limit the benchmark’s generalizability. As noted by the authors, accuracy appears to correlate with system complexity, and the telemetry distribution varies across systems. Although OpenRCA provides 335 failure cases, the limited number of software systems could introduce significant variance.

- The benchmark does not fully disentangle the capabilities of long-input retrieval, non-natural language understanding, and software problem identifying. The long, complex, non-natural language input introduces additional difficulty for LLMs, potentially limiting performance beyond the task of software failure identifying. Adding an ablation study could help isolate the challenges introduced by long, non-natural language input.

- The data collection approach seems to highly reliant on human effort, which may cause difficulty in updating and scaling. Collecting software telemetry and manually annotating Oracle sampling seems to be labor-intensive. Actually, frequent updates are crucial for keeping LLM benchmarks quality, particularly in consideration of potential data leakage.

- As the authors note, the current benchmark includes only distributed systems. Expanding the benchmark to incorporate systems from other domains would provide a broader evaluation of LLM capabilities.

- Typo: In line 035, "post-development phrases" --> "post-development phases."

questions:
- The authors categorize tasks into three levels: Easy, Mid, and Hard, based on the number of elements. Given that system complexity varies among repositories (for instance, Telecom has the smallest telemetry volume), could the authors further explain on why element count was chosen as the primary criterion for task difficulty? How did the authors account for the impact of system complexity on task difficulty?

- The authors state in Line 363 that "models generally follow the same trend: accuracy increases with more reasoning steps." However, Figure 6 does not appear to strongly support this, as accuracy varies significantly across different models and lengths. Could the authors provide further clarification or evidence on how this conclusion was reached?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731051099938

thread_type:
official_review

<thread_object>
creator: reviewer_kxev

<content>
summary:
This paper introduces OpenRCA, a novel benchmark and evaluation framework aimed at assessing the ability of large language models (LLMs) to perform root cause analysis (RCA) in real-world software systems, specifically focusing on the post-development phase, which has been largely overlooked in current research. The benchmark consists of 335 failure cases from three enterprise software systems, accompanied by over 68 GB of telemetry data (logs, metrics, and traces), challenging LLMs to identify the root causes of software failures by reasoning across complex, multi-modal data. The paper also proposes RCA-agent, a specialized tool to help LLMs process large volumes of telemetry data. The evaluation results show that current models, even with RCA-agent, struggle to handle these tasks effectively, with the best-performing model (Claude 3.5) solving only 11.34% of the failure cases. This paper makes a valuable contribution by highlighting the challenges and potential of LLMs in post-deployment software maintenance and lays a foundation for future research in automated root cause analysis.

strengths:
The paper demonstrates several key strengths. First, it offers a clear and detailed evaluation, presenting comprehensive results that highlight how different models perform on the OpenRCA benchmark. The inclusion of various methods, such as RCA-agent and different sampling techniques, alongside a breakdown of performance by system complexity and task difficulty, provides valuable insights. Second, the model comparison between proprietary and open-source LLMs is highly informative, showcasing the limitations of current open-source models like Llama 3.1 compared to more advanced proprietary models like Claude 3.5 and GPT-4o, effectively setting the stage for future improvements. The paper also includes a strong task complexity and system analysis, showing how model performance correlates with system complexity and the number of root cause elements, offering practical insights for future research. The reasoning length and error tolerance analysis provides an interesting perspective on how LLMs' performance can be improved with longer reasoning and better error handling. Finally, the real-world applicability is demonstrated through a detailed case study, illustrating how the OpenRCA benchmark can be used in practice to solve root cause analysis tasks, thus bridging the gap between theory and real-world challenges.

weaknesses:
Insufficient Contextualization of Problem Importance:
Lack of Quantitative Evidence: The paper mentions that software failures cost "billions of dollars," but does not provide specific references or data to substantiate this claim. For instance, Figure 1 cites the financial impact without attributing it to any source.

Actionable Insight: Provide concrete data and citations from industry reports or academic studies that quantify the financial and operational impact of software failures. For example, referencing studies from Gartner or IEEE could strengthen the argument and give readers a clearer understanding of the problem's magnitude.

Inadequate Literature Review on Existing RCA Benchmarks:
Questionable Claim in Section 2.2: The assertion that "RCA is a critical step in the software development lifecycle but often lacks practical benchmarks" may not fully reflect the current state of research. There are existing benchmarks and tools for RCA in software engineering.

Actionable Insight: Expand the literature review to include a comprehensive analysis of existing RCA benchmarks and tools. Discuss works like the MIMIC dataset in healthcare analytics or other RCA frameworks in software engineering to highlight how OpenRCA differentiates itself and fills existing gaps.

Limited Discussion on Broader Impact and Applications:
Underdeveloped Broader Impact Section: The paper does not sufficiently elaborate on how OpenRCA can improve software maintenance, quality assurance, or developer workflows.

Superficial Analysis of Low Model Performance:
Lack of Deep Dive into Results: The observation that "current models can only handle the simplest cases" lacks a thorough analysis. The paper does not explore why models performed poorly—whether it's due to limitations in model architecture, data complexity, or other factors.
Actionable Insight: Conduct a detailed error analysis to identify the root causes of low model performance. Discuss whether issues stem from the models' inability to process long-context data, difficulties in reasoning over heterogeneous data types, or shortcomings in handling complex software dependencies. Providing specific examples where models failed can offer valuable insights.

Overreliance on Synthetic Queries and Lack of Real-World Data:
Synthetic Nature of Failure Reports: The queries are synthesized based on common failure reports, which may not capture the complexity and nuances of real-world scenarios.

Complex Tables and Figures Without Adequate Explanation: Some tables (e.g., Table 2) and figures (e.g., Figure 5) are dense and may be difficult to interpret without additional context. References to Appendix A suggest that crucial information is relegated to the appendices.

Limited Diversity of the Dataset:
Narrow Focus on Distributed Systems: The dataset comprises failures from only three enterprise distributed software systems, potentially limiting the applicability of the benchmark to other system architectures like monolithic or embedded systems.

Absence of Ethical Considerations and Potential Risks:
No Discussion on Ethical Implications: The paper does not address potential ethical concerns, such as the consequences of incorrect root cause identification in critical systems, which could lead to misdiagnosis and exacerbate issues.

questions:
Can you provide more details on the selection process for the failure cases in OpenRCA?

Clarification Needed: The paper mentions 335 failure cases from three enterprise systems but does not describe how these cases were selected. Were they chosen based on frequency, severity, or another criterion?
Suggestion: Please clarify the criteria used to select these cases and explain how representative they are of real-world failure patterns.

What are the specific challenges faced by models when handling multiple root cause elements?

Clarification Needed: The paper observes a significant drop in accuracy as the number of required root cause elements increases. Is this due to limitations in context window size, inability to handle heterogeneous data, or something else?
Suggestion: A more detailed explanation of the challenges could help readers understand the underlying issues and suggest pathways for improvement.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311592128

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal

comment:
We would like to thank all the reviewers for their detailed review. We summarize the key clarifications&modifications and additional experiments&analysis as follows:

### Clarifications&Modifications

1. **[Reviewer RqHx]** Clarified the insight of OpenRCA’s baseline choice. We also provided a [general response](https://openreview.net/forum?id=M4qNIzQYpd&noteId=KxYY0dx1Wy) to corresponding questions.
2. **[Reviewer KxEv, RSze]** Highlighted the size, scalability, diversity, and the nature of complex data input of OpenRCA.
3. **[Reviewer KxEv, RSze]** Explained the specific challenges of handling multiple elements and the reason for using element number to categorize task complexity.
4. **[Reviewer KxEv]** Explained the details of OpenRCA, including its real-world impact, performance, and criteria of data selection and query construction.
5.  **[Reviewer KxEv]** Explained the potential ethics concerns of OpenRCA.
6. **[Reviewer 41p3]** Explained the reason why OpenRCA does not require the LLMs to identify whether the root cause is identifiable.
7. **[Reviewer KxEv]** Added extra papers and reports as references in Sec. 1 and Sec. 2.
8. **[Reviewer KxEv, 41p3]** Polished the discussions and captions to the tables and figures in the manuscript.
9. **[Reviewer RSze, RqHx]** Corrected the typo in the manuscript.

### Experiments&Analysis

1. **[Reviewer 41p3]** Provided the raw results and discussions of GPT/Claude’s performance on both oracle and agent settings.
2. **[Reviewer 41p3]** Conducted the experimental analysis of majority voting on both oracle and agent settings.
3. **[Reviewer 41p3]** Conducted the experimental analysis of individually localizing each root cause element across all tasks.
4. **[Reviewer RSze]** Provided the raw results and discussions of agent’s performance with different reasoning steps.
5. **[Reviewer RSze]** Provided the statistical data to illustrate the size and diversity of OpenRCA.
6. **[Reviewer RqHx]** Conducted the comparison between agents and vanilla LLMs with the exactly same prompts.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a benchmark dataset, OpenRCA, for evaluating the ability of LLMs to identify the root cause of software failures. Various LLMs are evaluated and the results show current models can only handle the simplest cases. The proposed benchmark dataset is interesting and it may be a valuable resource for future research. The experiments are comprehensive. The reviewers raised a few concerns about presentation and evaluation, and the authors addressed most of the concerns during the rebuttal period.