OpenRCA: Can Large Language Models Locate the Root Cause of Software Failures?
Authors: Junjielong Xu
Qinan Zhang
Zhiqing Zhong
Shilin He
Chaoyun Zhang
Qingwei Lin
Dan Pei
Pinjia He
Dongmei Zhang
Qi Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Language models
Natural language processing
Software engineering
Primary area: datasets and benchmarks
TL;DR: A novel benchmark for evaluating language models that introduces real-world root cause analysis as a task.

Abstract: Large language models (LLMs) are driving substantial advancements in software engineering, with successful applications like Copilot and Cursor transforming real-world development practices. However, current research predominantly focuses on the early stages of development, such as code generation, while overlooking the post-development phases that are crucial to user experience. To explore the potential of LLMs in this direction, we propose OpenRCA, a benchmark dataset and evaluation framework for assessing LLMs’ ability to identify the root cause of software failures. OpenRCA includes 335 failures from three enterprise software systems, along with over 68 GB of telemetry data (logs, metrics, and traces). Given a failure case and its associated telemetry, the LLM is tasked to identify the root cause that triggered the failure, requiring comprehension of software dependencies and reasoning over heterogeneous, long-context telemetry data. Our results show substantial room for improvement, as current models can only handle the simplest cases. Even with the specially designed RCA-agent, the best-performing model, Claude 3.5, solved only 11.34% failure cases. Our work paves the way for future research in this direction.

Published as a conference paper at ICLR 2025
OPENRCA: CAN LARGE LANGUAGE MODELS LO- CATE THE ROOT CAUSE OF SOFTWARE FAILURES?
Junjielong Xu
Qinan Zhang'
Zhiqing Zhong'
Shilin He
Chaoyun Zhang'
Qingwei Lin? Dan Pei
Pinjia Helf
Dongmei Zhang² Qi Zhang?
School of Data Science. The Chinese University of Hong Kong. Shenzhen
"iicosooft
*Tsinghua University
cuhk edu..edd.coom en, {shilin. he, chaoyun. zhang, qlin, dongmeia 12, qi qizhang peidanetsinghua eertshuuueedu.c.. en edu.ccn
(junjielongxu, qinanzhang, iiri,ziioo zhiqgingzhong]e1in
com.
ABSTRACT
Large language models (LLMs) are driving substantial advancements in software engineering, with successful applications like Copilot and Cursor transforming real-world development practices. However, current research predominantly fo- cuses the early stages of development, such as code generation, while overlook- ing the post-devclopment phases that are crucial to user experience. To explore the potential of LLMSs in this direction, we propose OpenRCA, a benchmark dataset and evaluation framework for assessing LLMs' ability to identify the root cause of software failures. OpenRCA includes 335 failures from three enterprise soft- ware systems, along with over 68 GB of telemetry data (logs, metrics, and traces). Given failure case and its associated telemetry, the LLM is tasked identify the root cause that triggered the failure, requiring comprehension of software de- pendencies and reasoning over heterogeneous, long-context telemetry data. Our results show substantial room for improvement, current models can only han- dle the simplest cases. Even with the specially designed RCA-agent, the best- performing model, Claude 3.5, solved only 11.34% failure cases. Our work paves the way for future research this direction.
INTRODUCTION
Large language models (LLMs) have recently driven significant advancement of software engineer- ing, with numerous research works and real-world applications impacting both the methodology and practice in software development, such MetaGPT (Hong et al., 2024). SWE-agent (Yang) 2opilol jet al. 2024). OpenDevin (Wang al., 2024b), Copilot, and Cursor However, existing efforts focus mostly on the early stages of Software Development Life Cycle (SDLC) while ignoring the post- development phases. In practice, maintaining software services and debugging issues during online operations are labor-intensive and error-prone tasks that often require 24/7 on-call support. Online incidents can cost service providers billions of dollars (CrowdStrike) UniSuper). highlighting the urgent need for more effective solutions in root
analysis (RCA) to mitigate software issues.
In recent years, Al researchers have explored various learning-based methods for RCA with tech- niques such as causal discovery (Arnold et al. 2007 Li et al. 2022a Chakraborty et al. 2023 Bi
et al. 2024), dependency graph analysis (Wang et al. 2023a Zheng et 2024)) and other neural networks Wang et al.] (2023b): Yu et al. (2021). However, RCA remains challenging due to the immense complexity of real-world software systems, which require multi-step reasoning capabili- ties over vast and heterogeneous data to identify root causes across diverse failure patterns. As the success of LLMs in software development (Jimenez. et al. 2024 Ni et al.) [024]] Chen et al.] [202]] 2024a) an important question is: Can current LLMs be effective in solving RCA challenges? The answer is critical to further enhance the automation of the entire software lifecycle via LLMs.
Work was done when Junjielong Xu was interning at Microsoft DKI.
1 Shilin He and Pinjia He the corresponding authors.

---

Published as a conference paper at ICLR 2025
Software System
Telemetry Data
oD
Large Language Model
GB G8
Log
Trace
Generated Root Cause
roroct cause 1":
( Query
("component") "tomcat01",
Services
On March 25, 2021, between 08:00 and 09:30AM, there was a fallure observed system X, plesse identily the root cause componsnt the toot causs resson.
"reason": "high dIs read usage")
Nodes
"root cause
Failures often propagate between services, requiring extensive telemetry (metrics, logs, traces) to identify the root cause. OpenRCA collects real-world failures and corresponding teleme- try, framing root cause analysis as a goal-driven task: the model must identify the root cause ele- ments (time, component, reason) specified in the query.
Figure I:
To answer this question, propose OpenRCA, public benchmark dataset and evaluation frame- work for assessing LLMs' root cause analysis ability in a practical software operating scenario. OpenRCA consists of 335 failure cases collected from three heterogeneous software systems de- ployed in the real world, accompanied by over 68 GB of de-identified telemetry data. Specifically, as shown in Figure [] each failure case paired with a query in natural language, requiring LLMs to analyze massive telemetry data to generate the corresponding root cause elements, including time, component, and reason. The process demands LLMs understand intricate system dependencies and conduct complex reasoning across telemetry data of diverse types, such as time series, dependency graphs, and semi-structured text.
Evaluating state-of-the-art LLMs on OpenRCA reveals significant challenges: these models are currently only capable solving parts of the simplest tasks. For example, Claude 3.5 only resolved 5.37% OpenRCA tasks when oracle telemetry using balanced sampling strategy to extract the possibly related telemetry.
given. This results further drops to 3.88% when
To outline possible direction for solving OpenRCA tasks, we further developed RCA-agent, multi-agent system[Qian Qiao et al. (202)); Zhang et al. (2024a) based on program synthesis & execution. By utilizing Python for data retrieval and analysis, the model is freed from processing large telemetry as an overly long context. This allows the model to focus solely on reasoning and makes it scalable for massive telemetry. With RCA-agent, the accuracy of Claude 3.5 is further improved to 1 11.34%.
We believe OpenRCA will serve as a foundational benchmark, driving future research at the inter- section of Al and Software Engineering, and allowing the community explore the true potential of LLMs in solving real-world service reliability problems
OPENRCA
2.1 PRELIMINARIES OF RoOT CAUSE ANALYSIS
Root Cause Analysis (RCA): In the software development lifecycle, root cause analysis refers to the process of identifying the underlying causes of failures, such as service unavailability, in a soft- ware system. On-call engineers must gather relevant telemetry data and other pertinent information to understand how the failure occurred.
Typically, a root cause should consist of the originating component (i.c., which part of the system failed), the start time (i.e., when the failure occurred), and the failure reason (i.e., why it failed, such as CPU overload or excessive disk throughput). Furthermore, a failure the originating component propagate to other components through service dependencies (e.g., payment service relying on a database) or deployment configurations (e.g., containers on the same server). This propagation can lead to broader system failures, complicating the identification of the exact root cause.
'The OpenRCA code and data available GitHub
2

---

Published as a conference paper at ICLR 2025
Telemetry:
Telemetry refers to the data used to monitor the internal status of software systems, encompassing metrics, traces, and logs. Metrics represent time series data points that track key performance indicators (KPIs), such as CPU usage response time. Traces capture the interactions among multiple system components, illustrating their dependencies, and often structured as a graph. Logs record runtime events and messages for each component, with verbosity levels such as info, warn, and error. Examples of telemetry data are provided in Appendix
A.3 A.3
2.2 FEATURES OF OPENRCA
OpenRCA is benchmark designed to evaluate the capability of LLMs perform RCA in practical software operation scenarios. The benchmark comprises 335 failure cases along with associated telemetry data, collected from three real-world software systems. Each failure case is structured as a goal-driven RCA task, where a natural language query serves as the input, and the objective is to identify the root causes of the failure. OpenRCA offers the following unique features:
Real-world Software Development Scenarios: RCA is a critical step in the software development lifecycle. Current RCA datasets Li et al. (2022a); Ikram al. (2022) are either synthetic or small- scale. OpenRCA addresses this gap by providing hundreds of failures collected from three real-
world software systems. This paves the way for solving more practical RCA problems scale.
Goal-driven Task Design: Traditional RCA datasets Li et al. (2022c; Lee et al. (2023) Yu et al. (2023) often focus on single goal (e.g., identify the originating component only), resulting in RCA methods tailored to each dataset with low generalizability. OpenRCA adopts goal-driven approach to cover various aspects of RCA by synthesizing queries natural language, making RCA a unified task and more accessible for language models. In addition, OpenRCA includes numerous real-world failures, addressing the limitations of traditional synthetic or small datasets used for specific tasks.
Extensive and Heterogeneous Data: The failure cases in OpenRCA encompass diverse patterns, such as CPU/memory/networt issues across comtainer/nodeesevvicc levels (see Table Appendix). Each case involves vast and heterogeneous telemetry data: metrics are time series of numerical val- ues, traces use a graph structure to show dependencies, and logs are semi-structured text, requiring LLMs make reasoning across diverse data formats.
Comprehensive LLM Assessment: OpenRCA requires LLMs to understand the software archi- tecture, interpret various types of real-world telemetry data, and correlate clues and observations from different data pieces. This process assesses LLMs' ability in understanding, reasoning, and decision-making extending beyond the scope of many existing software engincering tasks.
Benchmark Updatable: Our framework for constructing the benchmark allows new labels and telemetry data to be easily integrated into OpenRCA as additional datasets. We also plan to keep OpenRCA updated to maintain its challenge and prevent data contamination.
2.3 TASK FORMULATION
Task Input & Output: As mentioned earlier, root cause can be depicted with three elements: originating component, start time, and failure reason. In OpenRCA, we formulate seven tasks (or goals) by combining subsets of these three elements as the target output, which are common in RCA scenarios Lin et al.) (2018;; Amar & Rigby (201); Li et al. (2022b). Among the seven tasks, three focus on identifying only a single element, three on identifying two elements, and one on identifying all three root cause elements. Detailed input-output specifications are provided in Appendix A.6 As shown in Figure 1 for each failure case in the benchmark, the input consists of natural language query and the associated telemetry data, while the output can be one of the seven goals, i.c., subset of the three root cause elements, in a structured JSON format.
Evaluation: For each failure case in OpenRCA, it receives point if all generated root cause ele- ments match the ground truth ones, and 0 points if any mismatch was identified. The overall accuracy is the average score across all failure cases. To avoid evaluation errors caused by differences in tex- tual expressions from LLM generation and ground truth, all possible failure reasons and originating components are provided in the prompt beforehand. Further details provided in Appendix A.7
3

---

Published as a conference paper at ICLR 2025
Summary of OpenRCA datasets regard- ing the data size, number of unique root causes, i.e, component (C. for short) and reason (R. for short), and telemetry data size.
Table 1:
Tore 30A55
bESs Dark Make
Cases Count
#Unique RC
Telemetry Data
Dataset
C.
R.
Size
Lines
DA
Telecom
17.6G
154M
Bank
136
26.4G
248M
Market
148
44
15
24.5G
121M
Total
335
28
68.5G
523M
Figure 2: Composition of OpenRCA (by size).
2.4
DATASET CONSTRUCTION
OpenRCA consists of three diverse datasets originating from the widely-recognized AIOps Chal- lenge series, held annually since 2018 (ATOps 2018). Each raw dataset was collected from a real-world software system. However, the raw data is noisy, poorly maintained, and has different input-output specifications, making it unsuitable as a benchmark. Specifically, some datasets lack detailed records of failures like failure reasons, while others are not formulated into input-output pairs required for the RCA task. In addition, disorganized telemetry, inconsistent naming, and fail- ures across systems further reduce data usability. More importantly, there is significant amount of "dirty" data, such missing telemetry data, mismatched failure reasons, inconsistent failure times, and incorrect root cause labels. To address these issues, we implemented a four-step data processing detailed below:
procedure with human engagement (Figure
1D Fallure
Originating Component
Fallure Resson
System Selection
shrppinparri
records
Data Balancing
20
335 records
Data Calibration
online market wendoe system douthesst experienced Iilure titto 00:30, March 2022, badked to identill companent reiponuble for this lune the underiytng Label: ('root cause 1" "componentt" "shipiineerdoe11"
335 records [3 datasets)
Query Synthesis
A
LLMs
"reason" Toontainer nead UD loarf
Figure 3: The workflow of OpenRCA benchmark construction.
Stage 1: System Selection. Although numerous software systems emerged from past AIOps chal- lenges, many of them either lack labels only indicate that failure exists without detailing the root cause. In addition, some systems contain solely metric data, making them not suitable for RCA tasks. After excluding these unqualified systems, telecom database system, a banking system, and an online market system, totally comprising 1.753 failure records and their corresponding telemetry. Details of the three systems are in Appendix [A.2
retained data from three systems, including
Stage 2: Data Balancing, Among the three datasets, the data volumes varied significantly with differences up 100-fold in scale, which could lead biases in benchmark evaluation. To alleviate the issue, we downsampled the larger datasets to approximate the scale of smaller ones. Finally, 412 failure records remained, where each dataset contains several tens to a few hundred failure records.
Stage 3: Data Calibration. We calibrated the data from two perspectives. First, the raw data followed varying naming conventions as they were collected independently. We standardized the names in all telemetry data and labels, and reorganized the data into unified format for easier access by models. Second, we employed three experienced engineers to manually verified whether the root cause labels could be pinpointed using the associated telemetry. Failure records were removed if (1) no root cause could be identified from telemetry, (2) telemetry for the failure period was missing, or (3) the root cause inferred from the telemetry misaligned with the labeled one. After filtering, 335 failure records remained with over 68GB telemetry, with each containing about 20GB (distribution
shown in Table and Figure 2 More details about data calibration are presented in Appendix A.5
4

---

Published as a conference paper at ICLR 2025
Stage 4: Query Synthesis. Following the goal-driven task design, OpenRCA aims to comprise various goals and their combinations to better generalize to practical scenarios. In this step, we syn- thesize queries in natural language from failure records because (1) real failure queries are usually unavailable, (2) synthesizing queries ensures diversity and closely resembles human queries, and (3) it is easy scale up after accommodating new datasets. Specifically, seven goals be derived from three root cause elements (C3 + C3 + + = 7), covering various RCA scenarios.
As illustrated in Figure 3 for each failure record, first randomly select one of the seven goals and combine it with the failure time range (a 30-minute window), and the underlying system as specification. In some cases, multiple root causes may occur sequentially within the same time range. To address this, the number of actual failures is also included in the specification. Each spec- ification is then fed into LLMs like GPT-4 to generate a natural language query (see Appendix A.6 for prompts), which is further verified by human annotators.
3 RCA-AGENT
To resolve an OpenRCA task via LLMs, the first key challenge is how to process large volumes of telemetry data. An intuitive solution is chunk the data into smaller pieces and feed them into the LLMs sequentially. However, this approach is inefficient, costly and sacrifices the global view. Another method is to sample subset of data, which is more cost-effective but risks losing critical information. The second challenge is that telemetry data is predominantly non-natural language, consisting of numbers and many rare encoded tokens (e.g., GUIDs, not well-equipped to handle. An alternative way to process the vast amounts of data is by executing code Qiao al. (2023); Zhang et al. (2024b), which allows all data operations be conducted programmatically. This approach climinates the need to embed raw telemetry data into the LLM context, thereby significantly reducing token consumption while maintaining the global view.
codes), which LLMs
User
Controller
Executor
Telemetry
Query
Instruct
Reasoning
Read
Python Kernel
Answer
Action
Metric
Trace
Log
Figure 4: The structure and workflow of RCA-agent.
Based on this idea, we developed RCA-agent, multi-agent system that integrates decision decision-making, program synthesis, and execution. RCA-agent comprises two LLM agents (prompts in Ap- pendix B as described below:
Controller:
The Controller serves as the main decision- maker, responsible for directing the overall process, analyzing results, and determining next step iteratively. Based on common practices in RCA, it provides high-level guidance for L.LMs to follow: anomaly detection + fault identification + root cause localization. Additionally, Controller is instructed to analyze in the order of metric + trace log. To help LLMs understand how the telemetry data is organized, the directory structure and telemetry data schema are dynamically injected into the prompt.
Executor: The Executor tasked with writing Python code based on the Controller's instructions. executing the code, and reply back to Controller for further analysis. It comprises two parts: LLM- code generator responsible for synthesizing code and a Python kernel environment for executing it. Both the model and the Python environment are stateful, meaning that the generated code and executed variables are cached in memory until the query is resolved.
Workflow of RCA-agent: When receiving the input query, Controller follows the general guid- ance to carry out series of ReAct reasoning steps. Initially, Controller instructs the Executor to load the relevant telemetry data, and the Executor generates and executes the code. If the execu- tion fails, Executor will reflect itself using the error stack. Otherwise, Executor responds with the results to Controller, who then observes, gains insights, and decides the next action. This iterative process between Controller and Executor continues until all reasoning steps are completed and a final conclusion is reached.
5

---

Published as a conference paper at ICLR 2025
It is important to note that while the RCA-agent uses code generation avoid processing large amounts of context, allowing for scalability and handling vast amounts of telemetry, its performance is limited by the model's enror handling ability. If the model struggles with handling errors, the RCA-agent's effectiveness might be constrained, as will be discussed Section 5
EXPERIMENTAL SETUP
In this section, we describe the experimental setup used to evaluate LLMs on OpenRCA problems.
4.1 SAMPLING-BASED METHOSS
Given the vast volume of telemetry data, it is impractical to feed all telemetry into the LLMs due to their limited context window. A common strategy to reduce telemetry volume in RCA is sam- pling Huuag et al. 2024]; He et al. (2023). Thus, we downsample all telemetry data (including trace, log, and metric) each minute, regardless of the original freeuency. Meanwhile, we relax the accuracy criteria for predicting failure start time to within one minute of the actual event. However, this sampling is still insufficient as the metrics consist of a large number of KPI types (e.g., memory usage, network delay). necessitating further sampling of KPI types. Thus, we consider two sampling strategies:
frequency of minute by selecting the first recorded value within
To investigate the upper bound of the sampling- based method's performance, we introduce the oracle sampling. During benchmark construction, engineers identified a fixed set of "golden" KPIs that are helpful for identifying the root cause. In the oracle sampling, we filtered these "golden" KPIs as our target. Although this approach unrealistic, significantly reduces the number of KPIs by 95% (from 1263 to 53), thereby lowering the task complexity. It is important to note that these "golden" KPIs are not the only possible indicators that could point out the root cause. Different KPIs within the same metric file often correlated; for example, "CPU-used "CPU-used-percent" and "CPU-used-Me "MB tend to show similar trends. Therefore, even KPIs not included in the fixed set (non-oracle KPIs) may also potentially uncover the failure causes.
Oracle Sampling:
Balanced Sampling: We use stratified sampling by iteratively selecting one random but unique KPI type from each metric file until the number of sampled KPIs matches that in the Oracle setting. is a practical and intuitive approach to adapt LLMs to OpenRCA queries, which also ensures balanced representation across all KPI types. Due to correlations between KPIs, balanced random sampling usually produces stable statistical results, with a low variation of 1 % in our experiments. To ensure reproducibility, we tested each balanced sampling method three times and reported the median result.
In addition to sampling, we also removed uninformative data columns like service alias, and com- pressed data that could consume excessive tokens like trace IDs. After sampling, the prompt of these two methods contains around 1 100K tokens (tokenized by GPT-4o).
4.2 LANGUAGE MODELS
To solve OpenRCA tasks, which require processing long contexts, we selected six models with at least 128K token context windows, including three proprietary models: Claude 3.5, GPT-4o, and Gemini 1.5 Pro; and three open-source models: Mistral Large 2, Command R+, and Llama3.1 Instruct. The model checkpoints are shown in Appendix C.I
5 EXPERIMENTAL RESULT
This section presents the evaluation results, followed by an analysis and key insights. Table sum- marizes the accuracy of different methods on the OpenRCA benchmark, where the bold font rep- resents the accuracy of the best model in each column. Overall, proprietary models consistently outperform open-source models across all methods. In addition, RCA-agent performs better than sampling-base basedd approaches, with oracle sampling outperforming balanced sampling. This is pected as oracle sampling leverages the ground-trun -ruth information. However, all models face signif- icant challenges in resolving OpenRCA tasks. The best-performing model, Claude 3.5, achieved

---

Published as a conference paper at ICLR 2025
Table 2: Accuracy comparison of models using sampling and agent-based methods on OpenRCA (%). Correct denotes the percentage of fully solved queries, and Partial indicates the percent- age where least one required element correct.
Balanced
Oracle
RCA-agent
Model
Correct
Partial
Correct
Partial
Correct
Partial
Closed
Claude 3.5 Sonnet
3.88
18.81
5.37
17.61
11.34
17.31
GPT-4o
3.28
14.33
6.27
15.82
8.96
17.91
Gemini 1.5 Pro
6.27
24.18
7.16
23.58
2.69
6.87
Open
Mistral Large 2
3.58
6.40
4,48
10.45
N/A
N/A
Command R+*
4.18
8.96
4.78
7.46
N/A
N/A
Llama 3.1 Instruct
2.99
14.63
3.88
14.93
3.28
5.67
Due to budget constraints, we evaluate only Llama RCA-agent among open-source models.
Balanced Sampling
Oracle Sampling
RCA-agent
15.0
Cause
15.0
15.0
GPT-4
Lieme
Amuney
10.0
10.0
10.0
50
5.0
5.0
0.0
0.0
Telocom
Bank
Market
Telecom
Bank
Market
Telecom
Bank
Market
Figure 5: Accuracy distribution across three software systems (%).
only 11.34% accuracy even when using RCA-agent. A detailed discussion of these results is pro- vided below. Due to space constraints, we focus our analysis on two proprietary models (Claude 3.5 and GPT-4o) and one open-source model (Llama 3.1) representatives.
Accuracy correlated with system complexity. We observed that models consistently perform bet- ter on less complex systems. As shown in Figure all models achieve higher accuracy on Telecom system compared to others. This is primarily due to the lower complexity of the Telecom data. As indicated in Table 1 the Telecom system has the smallest telemetry volume and significantly fewer unique root cause types, which simplifies the root cause analysis process. Moreover, it lacks the fault tolerance mechanisms used in Bank and Market, and has the fewest pods among systems. Thus, in Figure5] Telecom's is accuracy improved most significantly when using Oracle sampling and agent.
Models struggle to identify multiple root cause elements. We found that model performance declines significantly as the number of required root cause elements increases. As shown in Table OpenRCA tasks were categorized into three levels: Easy (1 element), Mid (2 elements), and Hard (3 elements). Accuracy drops by at least half when the number of required elements increases from one two. Furthermore, none of the current methods successfully resolve queries requiring all three
Table 3: Accuracy w.r.t task categorization, Easy (I element), Mid elements), Hard (3 elements)
Category
Model
Method
Easy
Mid
Hard
Oracle
8.72
3.50
0.00
Claude
RCA-agent
16.78
9.09
0.00
Oracle
9.40
4.90
0.00
GPT
RCA-agent
13.42
6.99
0.00
Oracle
7.38
1.40
0.00
Llama
RCA-agent
6.71
0.70
0.00

---

Published as a conference paper at ICLR 2025
elements, where provide a further analysis of these Hard queries in Appendix D This finding suggests that most correctly answered queries by current models are relatively simple. Even with simpler queries, the models correctly solve only small fraction of them. Additionally, it indicates that while OpenRCA is inherently challenging, it presents a clear difficulty gradient across tasks.
Table 4: Accuracy of
GPT-4n
12
swers involving failed execu- tion. Drop indicates the per- centage of accuracy reduction compared to Table
3.5 Sonnet
Stonnct Lamo
Li
10
R0k0 (%)
8-
Acsunec
6-
20
4-
RCA-agent
Model
Correct
Dropl
0
Claude
9.31
17.90
GPT
7.56
15.63
Figure 6: Left figure: the reasoning length among all queries. Right figure: the accuracy of queries with different lengths.
Gemini
0.85
68.40
Llama 3.18
18.04
Agents prefer shorter reasoning but perform better with longer reasoning. As shown in the left panel of Figure O about half of the responses fall within 10 steps, regardless of correctness, indi- cating that the agents tend perform shorter reasoning. However, perform better when they engage in longer reasoning. As shown in the right panel of Figure 6 mod- els generally perform better when the reasoning length exceeds 10 steps. Combining both results, we hypothesize that current LLMs might be "lazy" in reasoning. Moreover, we observed an interesting finding with Claude. Although only few of the model's 's responses are within 5 steps, achieves a high accuracy of around 20%. To investigate the reason, we manually reviewed these cases, finding that they tend be the simplest tasks without significant failure propagation and Claude properly resolved these cases with the short reasoning.
also found that LLMs could
Agent's performance constrained by the model's error tolerance capability. We found that model's inability to effectively handle errors can significantly limit its performance as an RCA- agent. As shown in Table 2 while Gemini achieves the best performance among all sampling- based methods, its performance agent's 's intermediate steps, we found that although all models may make mistakes when generating code, GPT and Claude can effectively utilize execution feedback, such empty output or exception traceback, to revise their code or adjust their reasoning to bypass errors. In contrast, Gemini rarely does so. To quantify it, we further analyzed the accuracy of each model in cases involving failed executions. As shown in Table 4 under this setting, Gemini's accuracy sharply drops from its original accuracy of 2.69% 0.85%. %, marking significant reduction of 68.4%. In contrast, Claude and GPT exhibit only minimal declines of 15.6% and 17.9%, respectively. This suggests that the use of an RCA-agent adds an extra requirement to a model's capabilities, representing a trade-off: enhancing the scalability of language models for handling massive telemetry necessitates stronger error tolerance during reasoning.
RCA-agent is the worst. Through manual review of the
6 CASE STUDY
We present a case generated by RCA-agent (Claude) illustrate a potential direction to solve Open- RCA queries. As shown in Figure 7 this case is a query from the bank dataset, where the failure query requested the root cause component between 23:00 and 23:30 on March 6. To address this, the RCA-agent first analyzed metrics by extracting KPI sequences for the given time frame and cal- culating the P95 threshold to filter out anomalous data points. After identifying a large number of components with anomalies, the RCA-agent filtered out isolated noise points to reduce false posi- tives. Components with persistent anomalies were considered genuinely affected by the failure. The RCA-agent then extracted traces involving the affected components, ranked the traces based com- mon call dependencies, and identified Tomcat 01 as the root cause due to its frequent self-calls in the downstream chain. Further log analysis pointed to network instability and memory leaks, lead- ing the RCA-agen to conclude that packet loss was the failure cause. The agent correctly identified the root cause component required by the query, and the answer was deemed correct.
8

---

Published as a conference paper at ICLR 2025
G Query
Reasoning Chain (Thought (T) and Action [A]) Jurmoonied versice
Wiiti in the time sange of 5 5, 2021, from 28.00 23:30, there was crensperted tnlture in the spenem. Your to determine the root couse compo iom"."
None
[A Risemies miee 2021 oos.mmttoo and abtsinal all volld KIft namer
hamms nelde
[T] oa CPU Load, NEH fien, We should porfo es anomaly detoction firtt
JA2 bue-act cata between 22000 to 23:30 and catculate PSD thresholda for these KPta.
[]] Data estmetionn success. The threch olus are Wie should continue incmmaly distect on.
Telemetry /ol the whole bank wotemm
JA] Pibtr the anemaaoo data points exceed the threamouths in gPts
Myrrjit
2000 CTD. 3cads
11.32 24,4 2.35, a
dhil
Faults confirmed: Hyeabot Migopt Tomcatt1, Via should identity
towo.
Metric
Hopepooa ES Capeciege
[-7 10000- 2003
M4 Reard tiee '202 olstracao sctract traces inaching these fault components.
[ Several co responiiing traces are extrected, We sholdd ondluce that depponeencc nont.
[A] Shrtihe traces by eccurences and en the most common call chein in the tree structure.
[T] The most common chainis Tommattt ooomannt Need log enalysie on Tomconton
[A] tlea 2202 dotoggi' extrect logs generated by Tomeaton.
We loge
memary
Inoffoment Ei E1
I->7omout E1 -ot----tttt---teeo -Toenetttt
Concluaion: Trmneed1 bacne network paccet patvental marrery lanks.
Structured Prediction
8B Ground-Truth
Evaluation Result
cierenssiee miweeeee 08724, sous
"comppeenn" "homcatt""
(limml 222 Ba00rr "porpponenn "Somnatt"" "reasson"" "rantrnns infency
"ooureot elomentt" componen Twenrng elamant": hans
"raason" " "ww.eerk
"ignonedi menn" reason
[Inde] CNS coseuccent pepet abact
1002 poparo .00,
X
Figure 7: A case of RCA-agent (Claude) in Bank dataset. The reasoning chain is summarized.
7 RELATED WORK
LLM for Real-world Software Engineering:
LLMs have demonstrated strength in various real-
life application scenarios (Wang al., 2025 2024a), particularly within the SDLC: coding (Hong
et al. 2024; Shi et al. [202]] Dinh et al. 2024] Li et al. 2025 Cao et al., [2025] Wan et al. 22024)
testing (Chen et al.
2023: Pei et al.) 2023; Wu et al. 2022 Xia et al. (024b), logging (Xu et al.
2024a c Xie et al. 2024). and debugging (Chen et al. 2024a; Jimenez et al., 2024 Xia et al.
2024a Xu et al., 2024b| Lee et al. 2024). However, the post-deployment stage has been largely overlooked. While the software researchers have recently acknowledged the potential of LLMs in this phase (Ahmed et al. [023] Chen et al. (2024b), they mainly use LLMs as summarization tools for service failures, underutilizing their reasoning capabilities. Moreover, they rely proprietary datasets and manual evaluation within corresponding organizations, limiting scalability. OpenRCA addresses this gap by introducing an open-access benchmark and evaluation framework for root cause analysis, leveraging LLMs' advanced reasoning abilities in post-deployment scenarios. This paves the way for further research and supports the automation of the entire SDLC.
Root Cause Analysis (RCA): The goal of the RCA task varies depending on the available teleme- try and application scenario. If only metrics and service topology are available, the task locate the failure originating component (Yu et al. [2221) or its failure reason (Bi et al. 2024). If only logs are available, the goal is identify the relevant logs generated when failure occurs (Amar& & Rigby) 2019 Rosenberg & Moonen 2020). If only traces are available, the goal is to identify the failure service operation (Yu et al. 2021). scenarios where multiple telemetry types are available (Li et al., [022b) |ee et al.) 2023 Yu et al.] (023), the goal is various based on the needs (e.g.. identi- fying the faulty component and the relevant KPIs). However, existing evaluation datasets (Li et al.) 2022c Lee et al. [2023] Yu et al. (2023) focus on single goals, providing labels for ad-hoc scenarios. Openk addresses this by framing RCA as goal-driven task, covering diverse scenarios of RCA.
8 DISCUSSION
Limitation: First, the failures collected by OpenRCA are all from distributed software systems. In the future, we aim to expand the dataset to include failures from systems with other architectures, such as monolithic systems, and use them to continuously update OpenRCA. Second, due to privacy and security concerns, OpenRCA is currently unable to gather first-hand failure reports from users or engineers. To approximate real-world scenarios, OpenRCA has synthesized queries based on com- mon failure reports in goal-driven manner. In the future, we hope to collaborate with companies to obtain real, anonymized failure reports for use as queries. Third, while our proposed RCA-agent makes models scalable for handling large volumes of telemetry without consuming extensive con- text, it increases the demands on the model's eттоr tolerance capability. In the future, we aim to explore more robust methods to reduce these additional requirements for handling OpenRCA tasks.

---

Published as a conference paper at ICLR 2025
Conclusion: We present OpenRCA, benchmark and evaluation framework designed to assess lan- guage models' performance in real-world software engincering, which particularly fills the gap in post-deployment phase of software development lifecycle. By targeting the root cause analysis tasks, OpenRCA requires LLMS to perform advanced reasoning across diverse data types and structures. We also designed RCA-agent, execution-basee multi-agent system trying solve the challenges in OpenRCA. The results expose current limitations in LLMs' ability to address these practical software tasks while opening new research opportunities in Al-driven software maintenance and reliability. As LLMs improve, success on OpenRCA could lead to significant advancements in auto- mated troubleshooting and system reliability, potentially transforming the management of complex software systems.
ETHICS STATEMENT
The OpenRCA is an open-sourced benchmark. Its telemetry data is sourced from the AIOps Chal- lenge series AIOps 2018 Data under CC BY-NC 4.0, which permits the non-commercial use, redistribution, and adaptation of the material, provided appropriate credit is given and any deriva- tives are shared under identical license terms. The adaptations of OpenRCA the original data detailed in Sec. [24] No privacy or enterprise-sensitive information remains due to prior rigorous anonymization the original challenges. The RCA-agent is strictly intended for academic bench- marking purposes. The RCA-agent, or any other LLM agent designed for solving OpenRCA tasks, should not be deployed in real- world production environments without a comprehensive safety as- sessment. Note that the authors hold sole legal responsibility for OpenRCA's data use, while users bear responsibility for any consequences of deploying RCA-agent in real- world scenarios.
ACKNOWLEDGEMENTS
This paper was supported by the Guangdong Basic and Applied Basic Research Foundation (No. 2 024A1515010145) and the Shenzhen Science and Technology Program (No. 2DSYS20230626091 302006).
REFERENCES
Toufique Ahmed, Supriyo Ghosh, Chetan Bansal, Thomas Zimmermann, Xuchao Zhang, and Sar- avan Rajmohan. Recommending root-cause and mitigation steps for cloud incidents using large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1737-1749. IEEE, 2023.
A1Ops. A1Ops Challenges. competition] 2018.
Anunay Amar and Peter C Rigby. Mining historical test logs to predict bugs and localize faults in the test logs. In 2019 IEEF/ACM 41st International Conference on Software Engineering (ICSE), pp. 151. IEEE, 2019.
Andrew Arnold, Yan Liu, and Naoki Abe. Temporal causal modeling with graphical granger meth- ods. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 66-75, 2007.
Tingzhu Bi, Zhang Yang, Yicheng Pan, Yu Zhang, Meng Ma. Xinrui Jiang, Linlin Han, Feng Wang, Xian Liu, and Ping Wang. Faultinsight: Interpreting hyperscale data center host faults. In Pro- ceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 141-152, 2024.
Jialun Cao, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li. Mingwei Liu, Ruixi Qiao, Yuting Han, Chaozheng Wang, Boxi Yu, et al. How should i build a benchmark? revisiting code-related benchmarks for llms. arXiv e-prints, pp. arXiv-25011 2025.
Sarthak Chakraborty, Shaddy Garg, Shubham Agarwal, Ayush Chauhan, and Shiv Kumar Saini. Causil: Causal graph for instance level microservice data. In Proceedings of the ACM Web Con- ference 2023, pp. 2905-2915. 2023.
10

---

Published as a conference paper at ICLR 2025
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian Man-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, 2023.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Represeniations, 2024a.
Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, et al. Automatic root cause analysis via large language models for cloud incidents. In Proceedings of the Nineteenth European Conference on Computer Systems, pp. 674-688, 2024b.
GitHub Copilot. GitHub. http:://gghhhb.cccm//aaaaterrrep
CrowdStrike. Microsoft.
https://log...tcrrooff...o.o/o/p
helping our customers-teeeeeeee ohw.
Anysphere Cursor. Cursor.
com/
AIOps Data. AIOps Challenges
Data Source.
https://ww..iopscccc//i....
aiops nankai/data/tzaae
Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. Large language models of code fail at completing code with potential bugs. Advances in Neural Information Processing Systems, 36, 2024.
Shilin He, Botao Feng, Liqun Li, Xu Zhang, Yu Kang, Qingwei Lin, Saravan Rajmohan, and Dong- mei Zhang. Steam: observability- Joint European Software Engineering Conference and Symposium on the Foundations of Soft- ware Engineering, pp. 1750-1761, 2023.
trace sampling. In Proceedings of the 31 ACM
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024.
Haiyu Huang, Xiaoyu Zhang, Pengfei Chen, Zilong He, Zhiming Chen, Guangba Yu, Hongyang Chen, and Chen Sun. Trastrainer: Adaptive sampling for distributed traces with system runtime state. Proceedings of the ACM Software Engineering, 1(FSE):473-493 2024.
Azam Ikram, Sarthak Chakraborty, Subrata Mitra, Shiv Saini, Saurabh Bagchi, and Murat Kocaoglu. Root cause analysis of failures in microservices through causal discovery. Advances in Neural 2022.
Information Processing Systems,
Carlos E Jimenez, John Yang, Alexander Wettig. Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2024.
Cheryl Lee, Tianyi Yang, Zhuangbin Chen, Yuxin Su, and Michael R Lyu. Eadro: An end-to-end troubleshooting framework for microservices on multi-source data. In 2023 IEEE/ACM 45th International Conference Software Engineering, pp. 1750-1762. IEEE, 2023.
Cheryl Lee, Chunqiu Steven Xia, Longji Yang, Jen-tse Huang, Zhouruixin Zhu, Lingming Zhang, and Michael R Lyu. A unified debugging approach via Ilm-based multi-agent synergy. arXiv preprint arXiv.2404.17153 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel. et tion for www.lldge-intensiiee nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
genera-
Cheryl Li, Tianyuan Xu, and Yiwen Guo. woapniing----oii--nniiiii Scaling test-time reasoning in large language models through logic unit alignment. arXiv preprint Siti::502.07803 2025.
11

---

Published as a conference paper at ICLR 2025
Mingjie Li, Zeyan Li, Kanglin Yin, Xiaohui Nie, Wenchi Zhang, Kaixin Sui, and Dan Pei. Causal inference- based root cause analysis for online service systems with intervention recognition. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 0233230-33 3230-3240, 2022a.
Zeyan Li, Nengwen Zhao, Mingjie Li, Xianglin Lu, Lixin Wang, Dongdong Chang, Xiaohui Nic, Li Cao, Wenchi Zhang, Kaixin Sui, et al. Actionable and interpretable fault localization for recurring failures in online service systems. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 996-1008, 2022b.
Zeyan Li, Nengwen Zhao, Shenglin Zhang, Yongqian Sun, Pengfei Chen, Xidao Wen, Minghua Ma, and Dan Pei. Constructing large-scale real-world benchmark datasets for aiops. arXiv preprint 2rXi::2008.03338, 2022c.
JinJin Lin, Pengfei Chen, and Zibin Zheng. Microscope: Pinpoint performance issues with causal graphs in micro-service environments. In Service-Oriented Computing: 16th International Con- ference, ICSOC 2018, Hangzhou, China, November 12-15, 2018, Proceedings 16, pp. 3 20.
Springer, 2018.
Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. Next: Teaching large language models to reason about code execution. arXiv preprint ArX:::244..1 14662, 2024.
Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. Can large language models reason about program invariants? In International Conference Machine Learning, pp. 27496-27520. PMLR, 2023.
Bo Qiao, Liqun Li, Xu Zhang, Shilin He, Yu Kang, Chaoyun Zhang, Fangkai Yang, Hang Dong, Jue Zhang, Lu Wang, et al. Taskweaver: A code-first agent framework. arXiv preprint
arXiv:2311.1754, 2023.
Carl Martin Rosenberg and Leon Moonen. Spectrum- log diagnosis. Proceedings of the 14th ACM/IEEE International Symposium Empirical Software Engineering and Measurement (ESEM), pp. 1-12, 2020.
SDLC. Software development life cycle (SDLC).
Kensen Shi, Joey Hong, Yinlin Deng, Pengcheng Yin, Manzil Zaheer, and Charles Sutton. Exedec: Execution decomposition for compositional generalization in neural program synthesis. In The Twelfth International Conference on Learning Representations, 2024.
httpp://ww..unispperr. com UniSuper. ortronnn--saatttttot--oooooo-ee-
centree/
Yuxuan Wan, Yi Dong, Jingyu Xiao, Yintong Huo, Wenxuan Wang, and Michael R Lyu. Mrweb: An exploration of generating multi-page resource-aware web code from ui designs. arXiv preprint arXiv:2412.15310, 2024.
Dongjie Wang. Zhengzhang Chen, Yanjie Fu, Yanchi Liu, and Haifeng Chen. Incremental causal graph learning for online root cause analysis. In Proceedings of the 29th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining, pp. 2269-2278, 2023a.
Lu Wang. Chaoyun Zhang, Ruomeng Ding, Yong Xu, Qihang Chen, Wentao Zou, Qingjun Chen, Meng Zhang, Xuedong Gao, Hao Fan, et al. Root cause analysis for microservice systems via hier- archical reinforcement learning from human feedback. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5116-5125, 2023b.
Wenxuan Wang. Yihang Su, Jingyuan Huan, Jie Liu, Wenting Chen, Yudi Zhang, Cheng Yi Li, Kao- Jung Chang, Xiaohan Xin, Linlin Shen, et al. Asclepius: A spectrum evaluation benchmark for medical multi-modal large language models. arXiv preprint arXiv:2402.1 /2/7, 2024a.
12

---

Published as a conference paper at ICLR 2025
Wenxuan Wang, Zizhan Ma, Zheng Wang. Chenghan Wu, Wenting Chen, Xiang Li, and Yixuan Yuan. A survey of 11m-bar based agents in medicine: How far are we from baymax? arXiv preprint 11111:2502211211, 2025.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Opendevin: An open platform for ai software developers generalist agents. arXiv preprint
(tf:::400.1167411 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. Autoformalization with large language models. Advances in Neural Informa- tion Processing Systems, 35:32353-32368, 2022.
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying Ilm-based software engineering agents. arXiv preprint arXiv:2407.0// 01449, 2024a.
Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. Fuzz4all: Universal fuzzing with large language models. In Proceedings of the IEEE/ACM 46th Interna- tional Conference on Software Engineering, pp. 1-13, 2024b.
Xiaoyuan Xie, Zhipeng Cai, Songgiang Chen, and Jifeng Xuan. Fastlog: An end-to-end method to efficiently generate and insert logging statements. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 26-37, 2024.
Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, al. Unilog: Automatic logging via I1m and in-context learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1-12, 2024a.
Junjielong Xu, Ying Fu, Shin Hwei Tan, and Pinjia He. Aligning the objective of blmedased program repair. arXiv preprint arXiv:2404.08877, 2024b.
Junjielong Xu. Ruichun Yang, Yintong Huo, Chengyu Zhang, and Pinjia He. Divlog: Log parsing with prompt enhanced in-context learning. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 2024c.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprini 2rXiv:2405. 2024.
Guangba Yu, Pengfei Chen, Hongyang Chen, Zijie Guan, Zicheng Huang, Linxiao Jing, Tianjun Weng, Xinmeng Sun, and Xiaoyun Li. Microrank: End-to-end latency issue localization with extended spectrum analysis in microservice environments. In Proceedings of the Web Conference 2021, pp. 3087-3098, 2021.
Guangba Yu, Pengfei Chen, Yufeng Li, Hongyang Chen, Xiaoyun Li, and Zibin Zheng. Nezha: Interpretable fine-grained root causes analysis for microservices multi-modal observability data. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 553-5553 2023.
Chaoyun Zhang, Liqun Li. Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. UFO: A Ul-Focused Agent for Windows OS Interaction. arXiv preprint 242::2402..0939, 2024a.
Chaoyun Zhang, Zicheng Ma, Yuhao Wu, Shilin He, Si Qin, Minghua Ma, Xiaoting Qin, Yu Kang, Yuyi Liang, Xiaoyu Gou, et al. AllHands: Ask me anything on large-scale verbatim feedback via large language models. arXiv preprint arXiv:2403. 240...453..... 2024b.
Lecheng Zheng, Zhengzhang Chen, Jingrui He, and Haifeng Chen. Mulan: Multi-modal causal structure learning and root cause analysis for microservice systems. In Proceedings of the ACM Web Conference 2024, pp. 4107-4116, 2024.
13

---

Published as a conference paper at ICLR 2025
A
BENCHMARK DETAILS
Table 5: Basic composition of datasets (with coarse-grained taxonomy)
Dataset
Component level
Failure resource
Data type
Telecom Bank
os.pod,db
cpu,ner,db cpu,mem,io, neet. cpu.memmio.nct..rooess
trace.metric log.trace,metie log log.trace.metric
pod node.pod.
Market
A.1 LICENSE
All telemetry data is originating from AIOps Challenge series A1Ops (2018): Data which is open- sourced and licensed under Creative Commons AttributionNNon Commercial 4.0 International:
https:// / http:///ccreativecoooooooo www.:oogggrllo.e. http//icennsss//y// nc/4.07
A.2 INTRODUCTION OF SERVICE SYSTEMS
The telecom system consists of 22 virtual machine operating systems, 8 pods, 13 database services, 12 Redis middleware services, and multiple business services. The database services and Redis are directly deployed the operating systems, while other business services hosted by the operating systems. The potential root cause components this system include all operating systems, pods, and database services. There five potential failure causes, categorized into CPU, network, and database failures. The telemetry data for the telecom system includes traces and metries but excludes logs.
Telecom:
pods
The bank system consists of 14 pods, 6 services, and multiple nodes. Five of these services service deployed across four pods. This setup provides fault tolerance; when pod hosting a service fails, another pod with the same service can handle the requests. The potential root cause components in this system include all 14 pods, with eight possible causes across five categories: CPU, memory, disk, network, and JVM failures. The telemetry data for the bank system includes metrics, traces, and logs.
Bank:
deployed across two pods each, while
The market system comprises 6 server nodes, 40 pods, and 10 services. Each service is deployed across different pods, similar to the bank system, providing fault tolerance. The potential root cause components in this system include all 6 nodes, 40 pods, and 10 services. Notably, if all pods hosting a particular service fail, this is considered a service-level failure rather than an individ- ual pod failure. There are 15 possible causes of failure, spanning CPU, memory, disk, network, and process termination. The telemetry data for the market system includes metrics, traces, and logs.
Market:
Please note that all telemetry data is collected in the UTC+8 time zone. Therefore, when converting between timestamps and datetimes, ensure you specifically use the UTC+8 time zone.
A.3 INTRODUCTION OF TELEMETRY
The telemetry data in OpenRCA encompasses metrics, logs, and traces, all structured in CSV files. Below is brief introduction each type of telemetry:
Metric files contain time series data representing key performance indicators (KPIs) for different components. Each data point in series is associated with component (identified by cmdb d) and KPT type (kpi name). By analyzing these metrics, one can detect anomalies in component performance. Below is example of 10 rows from metric file, which contains data points from different KPI:
Metrics:
timestamp cmdb_ id, ltipkippieepppaaeee value 1614787200, 1614787200, oygg Imn data pending writes, 0.0 1614787200, wyop
wwt__pUCCpUUt_1111 26.26.
Innodb data pending reads, 0.0
161 1
0. 0.3158
161 46847770 oo
25.545.
14

---

Published as a conference paper at ICLR 2025
1614787200, hrg.::00.....aaa.......oooooaoa. ens160 NeTackttttt 100.0 1614787200, http://www.......a.o.oooaooaaaaa 0.0 1614787200, wttp::/ww.......oooooooo.ooooooa
(1atest_fork_usee) 0.0 1614787200, 2 o4
1614787200, rolds (latests_esteeee
(loading), 0.0
All data points for specific KPI of a component form a complete time series. For example, the OSLinuuK-CPLCCPUCCCUUUUUU for Tomcat 03 can be represented as:
timestamp value
1614787200 25. 9228
1614787260 29.4098
1614787380 25.6756
1614787440 25.5445
1614787560 25.9666
1614787680 26.0624
1614787740 26.0624
1614787860 26.0859
1614787920 25.9044
1614788040
26.4985
Tomcat03 OSLinux-CPU, BAU____U__CPPPPP CPUCCPUUl
25.8
Vo Wh | M MM
25.7-
value
25.5
25.4
796000
798000
800000
802000
804000 tmmestap
B06000 808000 810000
Figure 8: Time series of SESLLNux-CUU.CU.CCUUUCCO
for Comcat03.
Trace: Trace files record the call chains between services, where each trace consists of multiple spans. A span represents a single communication event, capturing the interaction when one service makes a request to another. These spans are organized hierarchically, with each span having a parent span, which links it to the preceding span, forming complete trace. By examining the relationships between spans within a trace, one can understand the dependencies between services and trace the path of potential failures across them. Below are 24 rows from trace file, where each row represents a span, and together, they form a complete trace. (trace span_id, parent.ic simplified to four digits)
span timestamp, cmdb id, id, parent ptt.rrrrra.i.ss.........reeeeeee id, duration
1614787511536,,601,, I603, 8603, gw9703, 1614787515633,,600 1601, 944, gw9703, (141777719800,,mmcccao00 3432, 9209, gw9703,
16147875119880, Tomcatt0 9209, 9210, 9209,,2200,,997033,00 gw9703, 717/7/7517781,,m1c12222,,,, 9209, 9209,9211,,997033
1211 922,
6521 6004,
161 (B
161 16
15

---

Published as a conference paper at ICLR 2025
1614787200095, 1611187200095,cccee1lll. 663,, 6639, g
161478720011, docker/l,,635,,.....11 668,,
1614787200119, docker1,,cccccceccc.c. 6635, 66,,,,,,,,,,,,70000
1614787200120, o,0p 992,, 1407,,15,333,,,00,199,,,,,,,, g
2331199770011,,c0ccksB,,,,,,,, 6644, 6644, gw9703,,
162217//117798..mmcmt2222222222 921, 0 161478751 8058, Tomcat02, 9209, ,,14, gu9703,,
Specifically, the call chain can be represented as a tree structure:
IG01, 21messtanp::61118883121
IGV1,
Tomcat (
Tomcat02, 1immstaampp111177771113333
Tomcat02, Timmstaap::11147775511388
Tomcat02, Timmsttam::111777777772
Tomcat T1mmsst8mm:111114773177333
MG01, f1mmsstamp:1111477771880000
MG01, rimesttam-:1614477511300
dockerB1, Sociecahhh..iinercaann:333222223
dockerB1, Tinesttm:::111477720000
dockerB1, Timesstamm:11111777200000
dockerB1, Wimesttamm::1111787720000
dockerB1, Tinesttamp:1611777700009
dockerB1, Timestamp:161188/2200
dockerB1, Timestamp:161478:222
dockerB1, Timestaapp:1111177220112
MG01, Tinnestamp:111147771883333
M601, TInesttamm:11114775518833
dockerB1, Timestamp:1111477220111
dockerB1, Timestamp:111147772001::
dockerB1, Timestanp:1111777200011
Tcmcat02, Timestamp: 1 Tomcat02, 1imestamp:161477751115
It also be represented dependency graph:
#7 (x7)
dockerB1
IG01
#10 (x3)
#8
#5
n2
MG01
Tomcat02
H3 (x4)
H11 (x2)
Figure 9: Dependency graph of the trace example, with (xN) indicating self-requests.
Log: Log files capture runtime messages from components, often providing insight into the inter- nal state or behavior system. Each log entry consists of a timestamp, a verbosity level, and message. Logs particularly useful for understanding the details of service operations and diag- nosing issues. Below are rows of log entries representing messages from different components: (Log id are simplified to four digits)
16

---

Published as a conference paper at ICLR 2025
log id, timestamp, cmdb id, log www.ooogggggaaaaaaeeeeeeeeeeee value
|t 1f13,,61448722 http://pww...............oooaaa.
171,7161478 g,, ott.://ww...aiaaooooooooooooooor
secs]
c87b,
sdbf,1614787202,
re00000ee0000000 secs]
secs]
E138,1614792893
9959,1614792994, Tomcat02, ge, [Times: www. -y rea==5.0ee50 secs
0114,16147, Tomcatt2, |rg.a//aaaaaaraaaaaaoooooaaa.ann org.a//doo...ooooooooooooootttnn
A.4 INTRODUCTION OF ROOT CAUSE ANALYSIS
Failures in service systems are typically triggered by anomalies in specific components (e.g., nodes, containers) due to issues like disk saturation. These anomalies can propagate through service in- teractions-for instance, full disk on node may cause all services on containers deployed on that node to become unresponsive. This unresponsiveness then affects other services relying on them, leading broader system failures at the application level. Timely and effective root cause localization is therefore critical issue in software engineering to maintain software service system stability.
However, current research of RCA does not follow unified task definition of the task output, typi- cally due to the different requirements of specific scenarios. For example, if engineers only wonder which container is failure and want to redirect the traffic to other container, it does not need to know the in-depth reason of failure, i.e., only the failure components are needed. Thus, OpenRCA is considered to construct the practice of RCA into goal-driven manner, where each goal refers to a combination of three key elements of the root cause, i.c., the failure originating component, occurrence time, and reason.
A.5 DETAILS OF DATA CALIBRATION
We hired three SREs with over three years of RCA experience to calibrate the data. We provided the hired engineers with standardized procedure to perform individual data calibration and verification. Specifically, they first extracted all telemetry data generated by the root cause component within 30 minutes before and after the labeled failure event. Next, they visualized KPIs for the relevant source types based on the description of the root cause since we found such type-related KPIs (c.g.. CPU, memory, network, disk I/O) are the most primary evidence for verification. In most cases, we can solely rely on type-related KPIs to verify the root cause time/componen//rasson. For example, if the failure was due high CPU usage container, they examined all CPU-related KPIs within the time window. The engineers then identified anomalous data points in these KPIs, such as values exceeding the mean + standard deviations. However, in some instances where type-related KPIs do not show explicit anomalies, we use general business KPIs ('golden signals" like rr, sr, mrt) along with logs and traces to support the verifications. For example, in the Market dataset, we used business KPIs (mrt) along with trace latency and proxy logs to verify network packet corruption fail- ures when packet transmit KPIs do not show explicit anomalies. In the Bank dataset, we combined business KPIs (mrt) along with service logs validate high memory usage failures when memory usage KPIs do not show explicit anomalies. We do not verify telemetry for non-root cause compo- nents. Even though root cause components may not exhibit clear anomalies while their downstream components (affected by propagated faults) show significant anomalies, people still cannot precisely determine the exact root cause elements (e.g., the exact time of occurrence) and can only infer or guess a range of potential answers. Therefore, records without clear anomalies on root cause com- ponent were removed. Additionally, we checked for consistency between the labeled failure time and the actual onset of the root cause. As shown in Figure 10] Figure [1] and Figure [2] the failure record will be removed if the nearest data point of the telemetry around the failure occurrence time is not the first data point of anomalous data duration. Finally, any failure records lacking telemetry data for the relevant time period or components were also filtered out. After completing individual calibration, the engineers disagreed on only 8 (less than 2%). Following thorough discussion, they reached a consensus on all cases.
validated each other's results. Out of the original 412 cases, they
17

---

Published as a conference paper at ICLR 2025
Figure 10: A case where the failure record matches the telemetry. Based on the record's guidance, for the root cause component frontend
engineers visualized the KPI wwwttaiinnrfssreeaeee within a 30-minute window, since the record suggests that the failure reason containex read I/O load. The red dashed line at the x-axis center indicates the failure occurrence time provided by the record, which is also the exact start time of the KPI spike. Given that the records align with direct evidence of the root cause, i.e., the corresponding KPI, we consider it possible identify the root cause. Thus, this record is retained.
teudd unlaa SU
Figure 11: A case where the failure record does not match the telemetry. The engineers visualized all KPIs related to the disc since the record illustrates the failure reason high disc I/O read usage. However, the exact failure occurrence time significantly deviates from the time provided the failure records, as the nearest data point around the failure occurrence time is normal point. Thus, this record is removed from our dataset.
C1R. awe onders M1 J00W
100, Ira under
A case where the failure record matches the telemetry. The engineers visualized all KPIs related to the dise since the record illustrates the failure reason hi gh disc I/O read usage. The exact failure occurrence time is aligned with the label, as the nearest data point around the failure occurrence time is the first anomalous data point among an anomalous duration. Thus, the record is deemed accurate and is retained.
Figure 12:
18

---

Published as a conference paper at ICLR 2025
A.6 DETAILS OF PROBLEM SYNTHESIS
To adapt to the different outputs of RCA, we use GPT-4o to synthesize the RCA problems for each failure record. We randomly select one task for each failure record, and then send the failure records with the corresponding task specifications to the model for query synthesis. A task specification a JSON object that determines the input and output information of the query to be synthesized. The detailed system prompt and corresponding task specifications used for program synthesis are provided below.
System Prompt
Your task to generate an issue related to DevOps failure disgnosis based on a given set of specifications. The goal is to make the issue realistic enough that an engineer could encounter at work.
The specifications provided to you include the following components:
known
(The known information explicitly provided in the issue.)
Quuery
(The target query that requi red the user to answer.)
Your response should follow the JSON format below:
Misssu
(The generated issuc based the spacifications.)
For example:
(IN-CONTEXT EXAMPLES)
Some rules to follow:
1. Do not tell the user "how sclve the issue" (e.g., retrieve the telemetry data like netrics/logs/tracess)
Now, let's a get started!
User Prompt:
Please generate an issue related to DevÖps failure diagnosis based the following specifications:
"known
imput_speciHccttion)
'query
cnttput_specification
Task Specification:
"task 1"; {
input":
"time range: (rime_pericd]""
"number of failures: (num)"
"output":
19

---

Published as a conference paper at ICLR 2025
"root occurrence time: Idatetime)"
"task_2;;
"input":
"time range: (time_pericd)",
"number of failures: (num)"
"oupput:
"root cause reason: [reasen)"
ttask3"; (
"input":
"time range: [time_ _esme__ericdd""
"number of failures: (num)"
outputt;
"root cause component:
(component)
"task_4" 4":
"input":
"time range: [time_ _eriidd_erddd""
"number failures: (num)"
1,
ooupput":
"roct cause occurrence time: (datetime)",
"roet cause reason: (reascn)"
ttask_5::
" "input":
"time range: (time_pericd)",
"number of failures: (num)"
1,
"output":
"roet cause oscurrence time: (datetime)",
"root cause component: [cemponent
6";;_6";
" "input";
"time range: (cime_pericdd",
"number of failures: (num)"
"oupput":
"roet cause component: [cemponent)
"root cause reason: (reasen)"
"task_7" {
"input":
"time range: (time pericc!i,
"number of failures: (num)"
ooupput":
"root cause component: (component)",
"root cause occurrence time: (datetime)",
"reet cause reason: (reason)
20

---

Published as a conference paper at ICLR 2025
A.7 DETAILS OF EVALUATION
OpenRCA requires all the methods to structure their final answer in the following JSON format:
"I":
"roct cause occurrence datetime"; (A time in - àd SH: 85: fornat),
root cause component"; (A component selected from the given possible root cause component'),
"roct reason":
selected from the given 'possible root cause reason'),
1,
During evaluation, the LLMs are tasked with correctly answering all the required elements all the failures that occurred within the given time duration. If the LLM provides answer to element that was not required, that element will not affect the correctness of the answer. OpenRCA only focuses whether the required elements generated by the model. All possible root cause components and reasons for each system is provided in each method's prompts. The prompt of these methods generally discussed in Appendix B.2 and
answered correctly and ignores the unnecessary fields
21

---

Published as a conference paper at ICLR 2025
B RCA-AGENT DETAILS
B.1 FEATURES OF RCA-AGENT
Scalability: RCA-agent not constrained by telemetry volume. By loading telemetry into memory via code execution rather than the LLM's context, it can process large datasets as long as memory allows. Expanding memory is far easier than increasing LLM context, ensuring scalability.
Clarity: RCA-agen avoids overwhelming the LLM with processing data analysis on large amounts of numbers, codes, and symbols in telemetry. By handling data analysis through code execution, the L.M can focus solely on the reasoning and decision-making process for the execution results.
Since telemetry is not directly fed into the LLM,, it minimizes unnecessary token usage, keeping the context length concise and reducing overhead from irrelevant data.
Efficiency:
Generalizabilityy RCA-agent does not require Nomain-speciinn knowledge beyond its telemetry schema. Instead, it only follows two general guidelines of root cause diagnosis. This allows RCA- agent to generalize effectively across various service systems.
B.2 AGENT PROMPTS
We provide the system prompts of the Controller and Executor here for reference.
Controller system prompt:
You are the Adminis st st rator of a DevOps Assistant system for failure diagnesis. To solve each given issue, should iteratively instruct Executor to write and execute Pythor code for data analysis telemetry files target system. By analyzing the execution results, you should approximate the answer step-by-step.
There is domain knowledge for you:
(BACKGROUND KNOWLEDGE OF SYSTEM)
## RULES OF FAILURE DIAGNOSIS:
What you SHOULD do:
1. #FFlloww the workflow of 'preprccess -> anomaly detection -> fault identification -> root cause localization for failure diagnosis.
What you SHOULD NOT do:
1. DO NOT include any programming language in your response.
The issue you are going to solve is:
(PROBLEM TO SOLVEI
Solve the issue step-by-step. In each step, your response should follow the JSON format below:
"analysis"; (Your analysis of the code execution result from Executor in the last step, with detailed reasoning of 'what have been done' and 'what can be derived'. Respond 'None' if it is the first step
completed"; ("True" if you believe the issue resolved, and an answer can be derived in the 'instruction' field. Otherwise "False "),
"inttonn"""" (Your instruction for the Executor perform via code execution in the next step. involve complex multi-step
22

---

Published as a conference paper at ICLR 2025
instruction. Keep your instruction atomic, with clear request of what do' and hoo do' Respond summary by yourself you believe the issuc is resolved.)
Let's begin.
Executor System Prompt
You are DevÖps assistant for writing Python code to answer DevOps questions. each question, you need write Python code to solve by retrieving and processing telemetry data of the target system. Your generated Python code will be automatically submitted to IPython Kernel. The execution result output in IPython Kernel will be used as the answer to the question.
## RULES OF PYTHON CODE WRITING;
1. Reuse variables much pessible for execution efficiency since the IPython Kernel stateful, i.e., variables define in previous steps can be used in subsequent steps.
There is some domain knowledge for you:
(BACKGROUND KNOWLEDGE OF SYSTEM)
Your response should follow the Python block format below:
" "python (YOUR CODE HERE)
Summary Prompt
Note that once the Controller believe the task is completed, a summary prompts will be provided to controller for summarizing and structuring its answer to the JSON format required by OpenRCA:
Now, you have decided to finish your reasoning process. You should now provide the final answer to the issue. The candidates of possible root cause components and reasons are provided to you. The root cause componentss and reasons must be selected from the provi ded candidates
[BACKGROUND KNOWLEDGE OF SYSTEM)
Recall the issue (PROBLEM SOLVE)
Please first review your previous reasoning process to infer an exact answer of the issue. Then, summarize your final answer of the root causes using the following JSON format the end your response:
(OPENRCA ANSKER FORMAT)
B.3 BACKGROUND PROMPTS
We also designed three background prompts to introduce the schema of telemetry, i.e., Telecom, Bank, Market, and the possible failure components and reasons in the system. Note that these prompts are provided not only to the RCA-agent but also included the prompts for sampling- based methods provide basic system knowledge. All background prompts for all systems follow the format below:
DATA SCHEMA
23

---

Published as a conference paper at ICLR 2025
1. **Metric Files**: (METRIC FILE SCHEMA)
2. .+Trace FilesAM: (TRACE FILE SCHEMA)
2. **Log Files**: (LOG FILE SCHEMA)
## POSSIBLE ROOT CAUSE REASONS: [PATLURN REASONS)
FF POSSIBLL ROOT CAUSE COMPONENTS: FAILURE COMPONENTS)
24

---

Published as a conference paper at ICLR 2025
C EXPERIMENTAL SETUP
C.I CLARIFICATION
Why not retrieval-based methods: Unlike tasks such code generation summarization, which can leverage natural features (e.g., class names, function names, or keywords) for retrieval- augmented generation (RAG) Lewis et al. (2020). RCA faces challenges in identifying effective retrieval strategies due to the absence of such features in telemetry data. Actually, identifying faulty telemetry is key challenge in RCA, as failures typically occur without clear indicators pointing to specific KPIs, failure logs. or anomalous call chains. To address this, we employed common sampling strategies from traditional RCA methods construct our baseline.
Why not Shain-offthougeee reasoning Wei et al. (2022), as generally resulted in poorer performance in our repetitive exper- iments. Table 6 compares the effect of explicitly requiring CoT versus not doing so. The results consistently show that CoT underperformed compared to prompts that did not require it. After manually reviewing both settings, found that CoT often led models to focus anomalies listed in its thought, overlooking the given diagnostic guidance to explore the deeper fail- propagation chain among these anomalies. We also report brief case study in Appendix C.3
We did not explicitly instruct LLMs to perform chain-of- oftttuugttthh (CoT)
few obvious
Table 6: Repetitive comparison between our original prompt (Original) and CoT prompt (CoT) (%)
Balanced
Oracle
GPT-4o
Original
CoT
Original
CoT
Try-1
3.28
2.39
6.27
4.48
Try-2
3.28
2.99
6.27
5.37
Try-3
3.58
2.69
5.37
5.37
Median
3.58
2.99
6.27
5.37
The models in experiments were accessed via APIs, with the open- source models using services from Mistral, Cohere, and Together.AI, as shown in Table 7 Due to differences in tokenizers, the open-source models often generated more tokens than GPT-4o, even with the same 128K token limit, causing some prompts to exceed the context window. To ensure an accurate evaluation, we reduced the number of KPIs in the prompt by eliminating interdependent KPIs in the oracle setting while keeping the total KPI count consistent between the oracle and sampling settings. Additionally, we selected the 70B version of Llama3. rather than 405B version since Together.Al does not support 128K context in 405B version.
Selected Language Models:
Table 7: Checkpoint of each model
Name
Checkpoint
Claude 3.5
claude- 1llaud-33-5-somee--222444
GPT-4o
gpt 2ot-4o-20220513
Gemini 1.5 Pro Mistral Large Command R+
0801 misttral-Larre--nsstrc-----2
command-r-plus-08-2224
Ilaama 3.1 Instruct
meta
Despite KPI sampling, original telemetry files contain re- dundant columns. We carefully filtered out irrelevant columns (c.g., component hash IDs) and com- pressed meaningful, long-encoded fields.
Prompt sampling-based methods:
Evaluation for sampling-basee based methods: Since the sampling interval is limited to one minute, OpenRCA considers a failure time prediction correct it falls within a one-minute window. We also manually verified that telemetry from oracle sampling still reveals the root cause components and failure reasons after sampling. Despite KPI sampling, original telemetry files contain redundant columns.
25

---

Published as a conference paper at ICLR 2025
C.2 PROMPTS SAMPLING-BASEI METHODS
Original Prompts:
We first provide the prompt template used for both oracle sampling and balanced sampling in experiment:
You will be provided with some tel telemetry data and an issue statement explaining root
analyais problem to resolve.
[BACKGROUND KNOWLEDGE OF THE SYSTEM)
SAMPLED TELEMETRY DATA)
Now, I need you to provide an roct cause analysis to the following question:
(PROBLEM TO SOLVE)
Note: A root cause is the fundamental factor that triggers a service system failure, causing other system components to exhibit various anomalous behaviors. It consists of three elements: the root cause component, the start time of the root cause occurrence, and the reason for its occurrence. The objective of root cause analysis may vary, aiming to ident fy one or more of these elements based on the issue. Each failure has only one root cause. However, sometimes system' s abnormal state may be due multiple simultaneous failures, each with its own root cause. relationship between multiple components exhibiting abnormal behavior these anemalies originate from the same failure, with the component the downstream end of the call chain being the root cause caused by the
you find that there a call
component. The anomalies in the other components failure. there is no call relationship between the abnormal components, each component may be the root cause of a different. failure. Typically, the number of failures occurring within half hour does not exceed three.
Your response should be structured into a JSON format, itemising the relevant root cause information you fird. You only need to provide the elements asked by the issue, and ommited the other fields in the JSON, The overall fornat is as follows:
(OPENRCA ANSKER FORMAT)
Please follow the format above to provide your response of current issue. Response below:
In this prompt, provide the background knowledge of each system same what we did for RCA-agent. addition, we also summarized the methodology to perform root cause analysis from the system prompt of RCA-agent (i.c., "Note that ..."..
CoT Prompts:
We also provide the CoT prompts used in our repetitive experiment discussed in Appendix C. You will provided with some telemetry data and an issue statement explaining a root cause analysis problem to resolve.
[BACKGROUND KNOWLEDGE OF THE SYSTEM)
[AMMPLDD TELEMETRY DATA)
Now, I need you to provide an roct cause analysis to the following question:
26

---

Published as a conference paper at ICLR 2025
[PROBLEM TO SOLVE
Note: A root cause is the fundamental factor that triggers a service system failure, causing other system components to exhibit various anomalous behaviors. It consists of three elementa: the root cause component, the start tine of the root reason for its occurrence. The objective of root cause analysis may vary, aiming to identity one or more of these elements based on the issue. Each failure has only one root cause. However, sometimes a system's a abnormal state may be due muitiple simultaneous failures, each with its own root cause. relationship between nultiple components exhibiting abnormal behavior these anomalies originate from the same failure, with the component at the dewnstream end of the call chain being the root cause component. The anomalies in the other components failure. If there is no call relationship between the abnormal components, each component may be the root cause of a different failure. Typically, the number of failures occurring within half an hour does not exceed three.
occurrence, and the
you find that there a call
caused by the
Please first conduct a comprehensive analysis on the given telemetry data step-by-step in your response. Then, sunmari ze your findings using the following JSON format to provide a concise answer to the given the summarized ansewr, you only
issue at the end of your response. need to provide the elements asked by the issue, and ommited the other fields in the JSON. The overall format is as follows:
(OPENRCA ANSWER FORMAT)
Please follow the format above to provide your response current issue. Response below:
The only difference between the original prompt and the CoT prompt is that the CoT prompt explic- itly asked LLMs perform step-by-sted by-step analysis of the telemetry data.
C.3 CASE STUDY COT PROMPTING
We provide a case study to discuss the possible reason that the vanilla CoT does not work in RCA. In this case, we use Oracle sampling settings, and the model is GPT-4o. The RCA problem is:
On March 21, 2022, between 12:00 and 12:30, the cloud service system, cloudbed-1, experienced fault. The exact component responsible for this fault is currently unknown. Please identify the root cause component.
When using the original prompt, GPT-4o provides the correct answer, which is:
"1":
"root component": "productcata.oogeeviiee "root cause reason": "container read I/O load"
However, when using CoT prompt, the full response is: IF Step-by-Step Analysis
#### 1. Identify the Time Window and Cluster
The issue occurred March 21, 2022, between 12:00 and 12:30. The affected cluster is 'eloudbed-1',
#### 2. Analyze Service Metrics
Serviiee Metrics*:: Check for anemal ies in 'metric_service.cs for the specified time ndow.
27

---

Published as a conference paper at ICLR 2025
Frontend Service**:
'mrt' (Mean Response Time) spikes significantly during the time window
'mrt' values: 918.47, 13879.44, 13133.99, 10519.49, 33002.95, 12387.8,
13131.03, 03, 5234.86.
**Checkout Service::
"mrt"" values: 398.92 92, 2375.0, 6340.0, 4821.43, 5500.0, 0, 4950.0, 5075.0, 3331.5. **Recommendation Service**:
'mrt values: 64.81, 376.63, 416.07, 395.0, 395.0, 390.0, 395.0, 161.54.
-*Procuct Catalog Service:::
'mrt" values: 360.68, 1851.71, 1886.56, 1824.74 71, 1837.86, 1837.15, 1839.85, 573.48.
#### 3. Analyze Container Metrics **Frontend Containersex;
'Erontend-O', 'frentend-1', 'frontend-2', 'frontend2-0" show high CPU usage and network receive metrics.
*CCheckout Containers*::
"checkoutsservice-00, 'checkoutservice-11,, 'checkoutservvcce22) Seaeckoutseervice22000 show high CPU usage and network receive metrics. **Recommendation Containers*s:
"yecomennattioosevvicee [heommentttttteseeerreeee
recomeeddationeerree=>),, 'ecommeeddesoonserreeeee show high CPU usage and network receive metrics.
Proocctt Catalog Containers*::
"productcaaalllggeerie---- "prodictettllogeerr.eeel>
protucteattllogerriie== "produttcataloogeer.cee show high CPU
usage and network receive metrics.
#### 4. Analyze Node Metrics
**Node Metrics**: ''oode-'', 'node-2', 'node-3', 'node-4', 'nede-5', 'node-6' show high CPU usage and disk usage metrics.
#### 5. Analyze Trace Data
***Trace Data*:: -Multiple traces show high latency 'frentend', 'checkoutservice', recomendationeeriee. and "prodietcattalooseviiee
UUU 6. Analyze Log Data
-+Loog Data**;
- Loogs from 'frontend', 'eheckoutservice", 'ecomondatt.onervvlec) and
"productcatallogeriice show errors and high latency.
### Summary of Findings
The root cause component is likely to be 'frontend' service due to its significant impact on 'mrt and high CFU and network usage. and are likely propagated from the 'frontend"
The anemalies in 'dheckoutservice", trecomendationserviice productcatal service.
### JSON Output jjjoon
"1": "root cause component";
"frentend"
While LLMs with CoT prompting also identify multiple anomalous data points across the frontend, checkoutservice, recommendationserviie,, producteatalogservice in step 5, it directly determines the
28

---

Published as a conference paper at ICLR 2025
service with the most significant latency, frontend, as the root cause component, instead of follow- ing the guidance to analyze traces and identify the failure's 's originating component. We consider this may be because the model became lost in the massive context and complex reasoning process among heterogeneous telemetry data, leading to focusing only on superficial findings from its earlier analysis in stage 2. Therefore, better reflect the model's upper bound in solving OpenRCA, did not use CoT prompting as baseline.
D SUPPLEMENTARY ANALYSIS
As shown in Table 3 all large language models (LLMs), even in the agent setting, achieve 0% accuracy on Hard queries. To further investigate which element contributes to the difficulty of solving Hard queries, we conducted an in-depth analysis of the relative difficulty among different elements.
Our approach is to evaluate the partially correct accuracy (i.e.. Partial in Table 2 for each el- ement across all failure cases to infer their relative difficulty. Note that directly using the absolute value of each element's accuracy is insufficient, as LLMs may occasionally guess an element cor- rectly by chance (this is also the reason we place less emphasis on partially correct results when calculating accuracy in Sec. 5). To better reflect difficulty, we should first exclude the impact of random guessing by calculating the improvement over random accuracy. Specifically, we compute the difference (delta) between each method's 's absolute accuracy and the random guessing accuracy. The element with the smallest delta is the hardest to predict, as the method cannot improve upon its random guessing accuracy. Below, compute the random guessing accuracy for each element:
As shown in Table the number of reasons varies across systems. The random guessing accuracy is calculated as /51/5 + 136/8 + 148 (48/15)/338 1535 = 11.06%.
Reason:
Component: Similarly, the number of components differs by system. The random guessing accuracy is /55 + 136/14 4 148/ 1/////44//// 43)/ 4.92%.
Time: As detailed in Appendix A.5 and Sec. 4.1 each query involves identifying the root cause within 30-minute window, with predictions considered correct they fall within one minute of the actual failure time. The random guessing accuracy is 1/30- 3.33%.
The absolute accuracy and delta values of each element's accuracy are provided below.
Table 8: Comparison of partially correct accuracy across elements. R, C, and T refer to Reason, Component, and Time, respectively.
Type
Category
Absolute
Delta
R.
C.
T.
R.
C.
T.
Claude
12.56
13.54
11.30
+1.50
+8.62
+7.97
Oracle
GPT-4o
13.39
12.56
12.14
+2.33
+7.64
+8.81
Llama
11.30
11.72
6.70
+0.24
+6.80
+3.37
Claude
19.67
18.00
14.23
+8.61
+13.08
+10.90
Agent
GPT-4o
18.00
14.65
13.39
+6,94
+9.73
+10.06
Llama
7.53
3.35
2.51
-3.53
-1.57
-0.82
Random
11.06
4.92
3.33
N/A
N/A
N/A
The table shows that regardless of the method used, identifying the reason is consistently the most challenging task, while component and time are comparatively easier to determine. Specifically, Claude performs better at identifying components than time, whereas GPT exhibits the opposite trend. Llama, limited by its weaker coding and mathematical capabilities, frequently encounters exccution failures during reasoning, often preventing it from completing the task successfully. This observation aligns with the findings Table 3
29


Reviews:
[{"review_id": "BLnfl80NYj", "reviewer": "Reviewer_KxEv", "summary": "This paper introduces OpenRCA, a novel benchmark and evaluation framework aimed at assessing the ability of large language models (LLMs) to perform root cause analysis (RCA) in real-world software systems, specifically focusing on the post-development phase, which has been largely overlooked in current research. The benchmark consists of 335 failure cases from three enterprise software systems, accompanied by over 68 GB of telemetry data (logs, metrics, and traces), challenging LLMs to identify the root causes of software failures by reasoning across complex, multi-modal data. The paper also proposes RCA-agent, a specialized tool to help LLMs process large volumes of telemetry data. The evaluation results show that current models, even with RCA-agent, struggle to handle these tasks effectively, with the best-performing model (Claude 3.5) solving only 11.34% of the failure cases. This paper makes a valuable contribution by highlighting the challenges and potential of LLMs in post-deployment software maintenance and lays a foundation for future research in automated root cause analysis.", "questions": "Can you provide more details on the selection process for the failure cases in OpenRCA?\n\nClarification Needed: The paper mentions 335 failure cases from three enterprise systems but does not describe how these cases were selected. Were they chosen based on frequency, severity, or another criterion?\nSuggestion: Please clarify the criteria used to select these cases and explain how representative they are of real-world failure patterns.\n\nWhat are the specific challenges faced by models when handling multiple root cause elements?\n\nClarification Needed: The paper observes a significant drop in accuracy as the number of required root cause elements increases. Is this due to limitations in context window size, inability to handle heterogeneous data, or something else?\nSuggestion: A more detailed explanation of the challenges could help readers understand the underlying issues and suggest pathways for improvement.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates several key strengths. First, it offers a clear and detailed evaluation, presenting comprehensive results that highlight how different models perform on the OpenRCA benchmark. The inclusion of various methods, such as RCA-agent and different sampling techniques, alongside a breakdown of performance by system complexity and task difficulty, provides valuable insights. Second, the model comparison between proprietary and open-source LLMs is highly informative, showcasing the limitations of current open-source models like Llama 3.1 compared to more advanced proprietary models like Claude 3.5 and GPT-4o, effectively setting the stage for future improvements. The paper also includes a strong task complexity and system analysis, showing how model performance correlates with system complexity and the number of root cause elements, offering practical insights for future research. The reasoning length and error tolerance analysis provides an interesting perspective on how LLMs' performance can be improved with longer reasoning and better error handling. Finally, the real-world applicability is demonstrated through a detailed case study, illustrating how the OpenRCA benchmark can be used in practice to solve root cause analysis tasks, thus bridging the gap between theory and real-world challenges.", "weaknesses": "Insufficient Contextualization of Problem Importance:\nLack of Quantitative Evidence: The paper mentions that software failures cost \"billions of dollars,\" but does not provide specific references or data to substantiate this claim. For instance, Figure 1 cites the financial impact without attributing it to any source.\n\nActionable Insight: Provide concrete data and citations from industry reports or academic studies that quantify the financial and operational impact of software failures. For example, referencing studies from Gartner or IEEE could strengthen the argument and give readers a clearer understanding of the problem's magnitude.\n\nInadequate Literature Review on Existing RCA Benchmarks:\nQuestionable Claim in Section 2.2: The assertion that \"RCA is a critical step in the software development lifecycle but often lacks practical benchmarks\" may not fully reflect the current state of research. There are existing benchmarks and tools for RCA in software engineering.\n\nActionable Insight: Expand the literature review to include a comprehensive analysis of existing RCA benchmarks and tools. Discuss works like the MIMIC dataset in healthcare analytics or other RCA frameworks in software engineering to highlight how OpenRCA differentiates itself and fills existing gaps.\n\nLimited Discussion on Broader Impact and Applications:\nUnderdeveloped Broader Impact Section: The paper does not sufficiently elaborate on how OpenRCA can improve software maintenance, quality assurance, or developer workflows.\n\nSuperficial Analysis of Low Model Performance:\nLack of Deep Dive into Results: The observation that \"current models can only handle the simplest cases\" lacks a thorough analysis. The paper does not explore why models performed poorly—whether it's due to limitations in model architecture, data complexity, or other factors.\nActionable Insight: Conduct a detailed error analysis to identify the root causes of low model performance. Discuss whether issues stem from the models' inability to process long-context data, difficulties in reasoning over heterogeneous data types, or shortcomings in handling complex software dependencies. Providing specific examples where models failed can offer valuable insights.\n\nOverreliance on Synthetic Queries and Lack of Real-World Data:\nSynthetic Nature of Failure Reports: The queries are synthesized based on common failure reports, which may not capture the complexity and nuances of real-world scenarios.\n\nComplex Tables and Figures Without Adequate Explanation: Some tables (e.g., Table 2) and figures (e.g., Figure 5) are dense and may be difficult to interpret without additional context. References to Appendix A suggest that crucial information is relegated to the appendices.\n\nLimited Diversity of the Dataset:\nNarrow Focus on Distributed Systems: The dataset comprises failures from only three enterprise distributed software systems, potentially limiting the applicability of the benchmark to other system architectures like monolithic or embedded systems.\n\nAbsence of Ethical Considerations and Potential Risks:\nNo Discussion on Ethical Implications: The paper does not address potential ethical concerns, such as the consequences of incorrect root cause identification in critical systems, which could lead to misdiagnosis and exacerbate issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U0TRzqaLkS", "reviewer": "Reviewer_RSze", "summary": "The paper introduces OpenRCA, an innovative benchmark designed to evaluate large language models in identifying root causes of software failures using diverse, long-context telemetry data. The authors evaluate state-of-the-art LLMs on this benchmark, offering a high-level analysis of their performance. The authors also introduce RCA-Agent as a tailored solution for OpenRCA task. The findings suggest significant room for enhancing LLM capabilities in solving real-world service reliable problems.", "questions": "- The authors categorize tasks into three levels: Easy, Mid, and Hard, based on the number of elements. Given that system complexity varies among repositories (for instance, Telecom has the smallest telemetry volume), could the authors further explain on why element count was chosen as the primary criterion for task difficulty? How did the authors account for the impact of system complexity on task difficulty?\n\n- The authors state in Line 363 that \"models generally follow the same trend: accuracy increases with more reasoning steps.\" However, Figure 6 does not appear to strongly support this, as accuracy varies significantly across different models and lengths. Could the authors provide further clarification or evidence on how this conclusion was reached?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- OpenRCA pioneers the application of LLMs in the post-development phase of the software life-cycle, highlighting a new direction to use LLMs for identifying and analyzing real-world software issues.\n\n- The authors’ rigorous data-cleaning process in building OpenRCA is valuable, involving system selection, data balancing, data calibration and query synthesis.\n\n- The goal-driven task design covers various aspects of RCA tasks, framing them in a generalized, natural-language query format.\n\n- The authors conduct a comprehensive experiments of popular closed and open LLMs with different strategies (Balanced Sampling, Oracle Sampling and RCA-Agent). And the authors also provide high-level comparative analysis for each setting based on experiment results.", "weaknesses": "- OpenRCA includes only three distributed software systems, which is small and may limit the benchmark’s generalizability. As noted by the authors, accuracy appears to correlate with system complexity, and the telemetry distribution varies across systems. Although OpenRCA provides 335 failure cases, the limited number of software systems could introduce significant variance.\n\n- The benchmark does not fully disentangle the capabilities of long-input retrieval, non-natural language understanding, and software problem identifying. The long, complex, non-natural language input introduces additional difficulty for LLMs, potentially limiting performance beyond the task of software failure identifying. Adding an ablation study could help isolate the challenges introduced by long, non-natural language input.\n\n- The data collection approach seems to highly reliant on human effort, which may cause difficulty in updating and scaling. Collecting software telemetry and manually annotating Oracle sampling seems to be labor-intensive. Actually, frequent updates are crucial for keeping LLM benchmarks quality, particularly in consideration of potential data leakage.\n\n- As the authors note, the current benchmark includes only distributed systems. Expanding the benchmark to incorporate systems from other domains would provide a broader evaluation of LLM capabilities.\n\n- Typo: In line 035, \"post-development phrases\" --> \"post-development phases.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MvawOm2mlK", "reviewer": "Reviewer_41p3", "summary": "This paper provides OpenRCA, a benchmark set for root cause analysis for software systems. The benchmark set is composed of 335 failures and the task is to identify the timing of the failure along with the component that failed and its reason. The paper also proposed RCA-agent, a multi-agent approach that utilizes a coe execution environment to effectively utilize telemetry data. While the best performing LLM was able to solve 6% of failure cases, RCA-agent increased it 11%.", "questions": "- The comparative analysis of LLMs are interesting but given the rather low performance of individual LLM’s, I’m curious to see if one would get any additional boost in performance when they are used in an ensemble fashion. This also helps to understand if the LLMs considered identify similar failures or not.\n- Table 3. For the hard category, all LLM’s even in the agent setting gets 0. I’m guessing maybe identifying the reason is harder than identifying the timing of the issue. Following this intuition, I’m wondering if there is a task that is difficult across the board?  \n- Figure 5. The accuracy distribution for RCA-agent is completely different than the sampling based ones. For instance, using GPT-4o, we see that the accuracy is reduced for Telecom and bank datasets but increased for Market dataset. Similarly, using Llama3.1, we see that the accuracy is similar for Telecom and Bank but reduced for the Market dataset. I’d love to hear more insights on these behaviors.\n- Could you clarify which model size you used for Llama3.1-instruct?\n- Nit: Table 4. Explain what drop ratio is in the title.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper introduces a novel benchmark dataset for root cause analysis. The benchmark set is carefully designed with emphasis on consistency and quality. This in my opinion is a significant contribution as it can likely help advance the field, motivating further contributions utilizing this dataset. The second contribution on RCA-agent is limited in its originality as multi-agent systems utilizing code execution are widely available. Nevertheless, showcasing the increase in accuracy through an agentic approach motivates further research on this. The paper is very well written and organized.", "weaknesses": "The paper puts emphasis on creating a high quality dataset and one of the processing steps was the removal of failure records where the root cause cannot be identified using the given telemetry. Envisioning a system which acts based on the analysis from automated root cause analysis, it is vital to be able to flag if the root cause cannot be identified from given data. Therefore, the benchmark set should reflect that in my opinion. \n\nDue to the highly stochastic nature of agentic systems as well as LLMs, I think evaluations would be stronger if authors can provide error bars.\n\nAs the paper focuses on root cause analysis for distributed systems, one insight that would make the paper stronger is how LLMs and the proposed RCA-agent perform as the scale of the system increases. That would help the readers understand limitations and applicability of such approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TIpwPqmjYO", "reviewer": "Reviewer_RqHx", "summary": "The paper preposes OpenRCA, a benchmark for root cause analysis that requires models to extract potential root cause from large quantities of telemetry data, logs and traces. The paper also proposes RCAagent as a potential method to solve this root cause analysis problem since vanilla models struggle. The paper uses three SOTA closed-source models and three open-source models for the evaluation on OpenRCA.", "questions": "- Does oracle sample generalize to general RCA tasks? Is it a realistic baseline?\n- Does the 42-line rule in the system prompt given to the planner LLM generalize to other cases as well? What is the insight in designing an agent this way specifically for this task?\n- Is the benchmark intended for LLM or agents? Why wasn't other more specialized tools/agents from previous work tested and reported on the benchmark? \nIt would also be great for the authors to address the misc weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- RCA remains an important problem in realistic scenarios.\n- Good paper structure, explains the key problem, terminologies, and ideas very clearly.\n- Clean and good dataset with clear structures, etc.\n- A realistic RCA benchmark, with huge effort made towards ensuring fairness and denoising.", "weaknesses": "### Major\n- The “golden set” identified in the oracle sampling might be not realistic enough. In real-life tasks, it is also one of the key challenge for the model to understand and select the correct KPI in order to diagnose the problem.\n- Using agent has been a widely adopted method in other research fields. This paper fails to provide any insight into the nature of the problem and merely uses a general planner-executor agent framework almost without applying any domain knowledge. The only domain knowledge is a 42-line long rule given to the planner, which basically outlines all the things that the LLM should do. As a result the agent part seems really redundant and unnecessary. The same argument applies to the case study.\n- There is no point trying vanilla LLM on this task. The author acknowledges in the paper that it is virtually impossible to give the telemetry data to the LLM, which is why they adopt the sampling method. There are two flaws in this: 1. LLM is not designed to process huge amount of data, so an agent evaluation/previous baseline evaluation should be more proper, and 2. There is no tell that if the performance of the LLM is bounded by the sampling.\n\n### Misc\n- Some apparent spelling mistakes, like “Partitial” in table 2, or “Accuray” in table 3\n- The instruction given to the baseline in the artifact is far less comprehensive than the one given to the agent. Maybe fairer to give more domain knowledge to the baseline models?\n- The CSV file for the record in the artifact is confusing and consists lots of excess newlines?\n- Rather than reducing the size of the benchmark, why don’t do a success rate calculation on each system first and then compute the final score on the three success rates?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)