LEMMA-RCA: A Large Multi-modal Multi-domain Dataset for Root Cause Analysis

## Abstract

Abstract Root cause analysis (RCA) is crucial for enhancing the reliability and performance of complex systems. However, progress in this field has been hindered by the lack of large-scale, open-source datasets tailored for RCA. To bridge this gap, we introduce LEMMA-RCA, a large dataset designed for diverse RCA tasks across multiple domains and modalities. LEMMA-RCA features various real-world fault scenarios from IT and OT operation systems, encompassing microservices, water distribution, and water treatment systems, with hundreds of system entities involved. We evaluate the quality of LEMMA-RCA by testing the performance of eight baseline methods on this dataset under various settings, including offline and online modes as well as single and multiple modalities. Our experimental results demonstrate the high quality of LEMMA-RCA. The dataset is publicly available at https://lemma-rca.github.io/ .

## 1 Introduction

Root cause analysis (RCA) is essential for identifying the underlying causes of system failures, ensuring the reliability and robustness of real-world systems. Recent advancements in artificial intelligence and software development have led to increased complexity and interdependence in modern systems. This complexity heightens their vulnerability to faults arising from interactions among modular services, which can disrupt user experiences and incur significant financial losses. Traditional manual RCA, however, is labor-intensive, costly, and prone to errors due to the complexity of systems and the extensive data involved. Therefore, efficient and effective data-driven RCA methods are crucial for pinpointing failures and mitigating financial losses when system faults occur.

Root cause analysis has been extensively studied across various domains and settings [; ; ; ; ]. Based on the application scenarios, RCA can be carried out in *offline/online* fashion with *single/multi-modal* system data. Existing studies on RCA in these settings involve numerous learning techniques such as Bayesian methods [], decision trees [], etc. Particularly, causal structure learning technique [; ; ; ; ; ; ; ] has proven effective in constructing causal or dependency graphs between different system entities and key performance indicators (KPIs), thereby enabling the tracing of underlying causes through these structures.

Data is the oxygen of data-driven methods. Despite significant progress in RCA techniques, the availability of large-scale public datasets remains limited, often due to confidentiality concerns []. This scarcity hinders fair comparisons between RCA methods. Additionally, publicly accessible datasets often contain manually injected faults rather than real faults, and each dataset typically covers only a single domain. These limitations can prevent existing RCA methods from effectively identifying various types of system faults in real-world scenarios, potentially leading to regulatory and ethical consequences in critical sectors.

To address these limitations, we introduce LEMMA-RCA, a collection of Large-scalE Multi-ModAl datasets with various real system faults to facilitate future research in Root Cause Analysis. LEMMA-RCA is multi-domain, encompassing real-world applications such as IT operations and water treatment systems, with hundreds of system entities involved. LEMMA-RCA accommodates multi-modal data including textual system logs with millions of event records and time series metric data with more than $100,000$ timestamps. We annotate LEMMA-RCA with ground truth labels indicating the precise time stamps when real system faults occur and their corresponding root-cause system entities.

Table: Table 1: Existing datasets for root cause analysis. The top row corresponds to our dataset. The symbols ✓and ✗ indicate that the dataset has or does not have the corresponding feature, respectively.

A comparison between LEMMA-RCA and existing datasets for RCA is presented in [Table 1](#S1.T1). We briefly discuss the status of existing datasets:
1) NeZha [] has limited size and contains many missing parts in the monitoring data, and it is confined to one domain: microservice architectures.
2) PetShop [] has a small size. Additionally, the system comprises only $41$ components, limiting its complexity and reducing the practicality for real-world scenarios.
3) ITOps [] dataset is not public and contains structured logs that do not contribute to comprehending the underlying causal mechanism of system failures, making it difficult to conduct fine-grained RCA.
4) Murphy [] is collected from a simple system and also not public. In comparison to prior work, LEMMA-RCA demonstrates a comprehensive maturity on the accessibility, authenticity, and diversity.

LEMMA-RCA enables fair comparisons among different RCA methods. We evaluate eight baseline methods, with five suited for offline settings and the remaining three designed for online RCA. The quality of various data modalities is assessed in both online and offline setups. As previously mentioned, causal-graph-based RCA methods provide deeper insights into system failures; hence, all baseline methods fall into this category. The experimental results demonstrate the high quality of LEMMA-RCA and its extensive utility for advanced research in root cause analysis.

## 2 Preliminaries

Key Performance Indicator (KPI) is a monitoring time series that indicates the system status. For instance, latency and service response time are two common KPIs used in microservice systems. A large value of latency or response time usually indicates a low-quality system performance or even a system failure.

Entity Metrics are multivariate time series collected by monitoring numerous system entities or components. For example, in a microservice system, a system entity can be a physical machine, container, pod, etc. Some common entity metrics in a microservice system include CPU utilization, Memory utilization, disk IO utilization, etc. An abnormal system entity is usually a potential root cause of a system failure.

Data-driven Root Cause Analysis Problem.
Given the monitoring data (including metrics and logs) of system entities and system KPIs, the root cause analysis problem is to identify the top $K$ system entities that are most relevant to KPIs when a system fault occurs. RCA techniques can be implemented in various settings, where offline/online and single-modal/multi-modal are mostly commonly concerned. Offline RCA is conducted retrospectively with historical data to determine past failures, whereas online RCA operates in real-time using current data streams to promptly address issues. On the other hand, single-modal RCA relies solely on one type of data for a focused analysis, while multi-modal RCA uses multiple data sources for a comprehensive assessment. We illustrate the procedure of RCA in single-modal offline and multi-modal online settings in [Figure 1](#S2.F1).

Figure: Figure 1: Illustration of RCA workflow in the single-modal offline setting (top) and the multi-modal online setting (bottom). The other two settings can be viewed as an ensemble of corresponding components (data collection, detector, modality) and follow the same systematic procedure.
Refer to caption: /html/2406.05375/assets/Figure/rca-intro.png

## 3 LEMMA-RCA Dataset

This section outlines the data resources, details the preprocessing steps, and presents visualizations to illustrate the characteristics of the data released.

### 3.1 Data Collection

We collect real-world data from two domains: IT operations and OT operations. The IT domain includes sub-datasets from Product Review and Cloud Computing microservice systems, while the OT domain includes SWaT and WADI sub-datasets from water treatment and distribution systems. Data specifics are provided in [Table 2](#S3.T2) and [Table 3](#S3.T3).

Figure: Figure 2: Visualization of the microservice system platform and ElasticSearch log data.
Refer to caption: /html/2406.05375/assets/Figure/aiops_system_structure.png

In the IT domain, we developed two microservice platforms: the Product Review Platform and the Cloud Computing Platform. The Product Review Platform is composed of six OpenShift nodes (such as ocp4-control-plane-1 through ocp4-control-plane-3, ocp4-compute-1 and ocp4-compute-2, and ocp4-infra-1) and $216$ system pods (including ProductPage, MongoDB, review, rating, payment, Catalogue, shipping, etc.). In this setup, we simulated four distinct system faults, including out-of-memory, high-CPU-usage, external-storage-full, and DDoS attack, on four different dates. Each simulation operated the microservice system for at least $49$ hours with different pods involved. The structure of this microservice system with some key pods of one simulation is depicted [Figure 2](#S3.F2) (a). Both log and metric data were generated and stored systematically to ensure comprehensive monitoring. Specifically, eleven types of node-level metrics (e.g., net disk IO usage, net disk space usage, etc.) and six types of pod-level metrics (e.g., CPU usage, memory usage, etc.) were recorded by Prometheus [], and the time granularity of these system metrics is 1 second. Log data, on the other hand, were collected by ElasticSearch [] and stored in JSON files with detailed timestamps and retrieval periods. The contents of system logs include timestamp, pod name, log message, etc., as shown in [Figure 2](#S3.F2) (b). The JMeter [] was employed to collect the system status information, such as elapsed time, latency, connect time, thread name, throughput, etc. The latency is considered as system KPI as the system failure would result in the latency significantly increasing.

For the Cloud Computing Platform, we simulated six different types of faults (such as cryptojacking, mistakes made by GitOps, configuration change failure, etc.) on eleven system nodes. At each simulation, we gathered both system metrics and logs from various sources. In contrast to the Product Review platform, system metrics were directly extracted from CloudWatch Metrics on EC2 instances, and the time granularity of these system metrics is 1 second. Log events were acquired from CloudWatch Logs, consisting of three data types (i.e., log messages, api debug log, and mysql log). Log message describes general log message about all system entities; api debug log contains debug information of the AP layer when the API was executed; mysql logs contain log information from database layer, including connection logs to mysql, which user connected from which host, and what queries were executed. Latency, error rate, and utilization rate were tracked using JMeter tool, serving as Key performance indicators (KPIs). This comprehensive logging and data storage setup facilitates detailed monitoring and analysis of the system’s performance and behavior.

Table: Table 2: Data statistics of IT operation sub-datasets.

In the OT domain, we constructed two sub-datasets, SWaT and WADI, using monitoring data collected by the iTrust lab at the Singapore University of Technology and Design []. These two sub-datasets consist of time-series/metrics data, capturing the monitoring status of each sensor/actuator as well as the overall system at each second. Specifically, SWaT [] was collected over an 11-day period from a water treatment testbed equipped with $51$ sensors. The system operated normally during the first $7$ days, followed by attacks over the last $4$ days, resulting in 16 system faults. Similarly, WADI [] was gathered from a water distribution testbed over $16$ days, featuring $123$ sensors and actuators. The system maintained normal operations for the first $14$ days before experiencing attacks in the final $2$ days, with $15$ system faults recorded.

Table: Table 3: Data statistics of OT operation sub-datasets.

Figure: Figure 3: Visualization of KPI for system failure cases. Left: the first two sub-figures are from the Product Review sub-dataset; the third and fourth sub-figures are from the Cloud Computing sub-dataset; Right: the first two sub-figures are from the SWaT sub-dataset; the last two sub-figures are from the WADI sub-dataset.
Refer to caption: /html/2406.05375/assets/Figure/kpi_visualization.jpg

We visualized the key performance indicator (KPI) for eight failure cases in [Figure 3](#S3.F3), where sudden spikes or drops in latency indicate system failures. The first two sub-figures on the left show the KPIs for two faults in the Product Review sub-dataset, while the third and fourth sub-figures depict faults in the Cloud Computing sub-dataset. The first two sub-figures on the right display faults in the SWaT dataset, and the last two show faults in the WADI dataset. The x-axis represents the timestamp, and the y-axis shows the system latency.

### 3.2 Data Preprocessing

After collecting system metrics and logs, we assess whether each pod exhibits stationarity, as non-stationary data are unpredictable and cannot be effectively modeled. Consequently, we exclude non-stationary pods, retaining only stationary ones for subsequent data preprocessing steps.

Log Feature Extraction for Product Review and Cloud Computing. The logs of some system entities we collected are limited and insufficient for meaningful root cause analysis. Thus, we exclude them from further analysis. Additionally, the log data is unstructured and frequently uses a special token, complicating its direct application for analysis. How to extract useful information from unstructured log data remains a great challenge. Following [], we preprocess the log data into time-series format. We first utilize a log parsing tool, such as Drain, to transform unstructured logs into structured log messages represented as templates. We then segment the data using fixed 10-minute windows with 30-second intervals, calculating the occurrence frequency of each log template. This frequency forms our first feature type, denoted as $X^{L}_{1}\in\mathbb{R}^{T}$, where $T$ is the number of timestamps. We prioritize this feature because frequent log templates often indicate critical insights, particularly useful in identifying anomalies such as Distributed Denial of Service (DDoS) attacks, where a surge in template frequency can indicate unusual activity.

Moreover, we introduce a second feature type based on ‘golden signals’ derived from domain knowledge, emphasizing the frequency of abnormal logs associated with system failures like DDoS attacks, storage failures, and resource over-utilization. Identifying specific keywords like ‘error,’ ‘exception,’ and ‘critical’ within log templates helps pinpoint anomalies. This feature, denoted as $X^{L}_{2}\in\mathbb{R}^{T}$, assesses the presence of abnormal log templates to provide essential labeling information for anomaly detection.

Lastly, we implement a TF-IDF based method, segmenting logs using the same time windows and applying Principal Component Analysis (PCA) to reduce feature dimensionality, selecting the most significant component as $X^{L}_{3}\in\mathbb{R}^{T}$. We concatenate these three feature types to form the final feature matrix $X^{L}=[X^{L}_{1};X^{L}_{2};X^{L}_{3}]\in\mathbb{R}^{3\times T}$, enhancing our capacity for a comprehensive analysis of system logs and improving anomaly detection capabilities.

KPI Construction for SWaT and WADI. The SWaT and WADI sub-datasets include the label column that reflects the system status; however, the values within this column are discrete. To facilitate the root cause analysis, it is beneficial to transform these values into a continuous format. Specifically, we propose to convert the label into a continuous time series. To achieve this, we employ anomaly detection algorithms, such as Support Vector Data Description and Isolation Forest, to model the data. Subsequently, the anomaly score, as determined by the model, will be utilized as the system KPI.

### 3.3 System Fault Scenarios

We simulate $10$ different types of real system faults in Product Review and Cloud Computing platforms. Due to the space limitation, we select two representative cases (one from each) and provide the details below.
We also visualize the system fault of these two cases in [Figure 4](#S3.F4).

Figure: Figure 4: Visualization of two system fault scenarios. Left: Cryptojacking. Right: External storage failure.
Refer to caption: /html/2406.05375/assets/Figure/aiops_0215.png

- •
Cryptojacking. In this scenario, cloud usage fees increase due to cryptojacking, where a Coin Miner is covertly downloaded and installed on a microservice (details-v1 pod) in an EKS cluster. This miner gradually consumes IT resources, escalating the cloud computing costs. Identifying the root cause is challenging because the cost (SLI) encompasses the entire system, and no individual service errors are detected. Periodic external requests are sent to microservices, and after a day, the miner’s activity triggers auto-scaling in details-v1, increasing resource usage. Fargate’s impact on EKS costs is significant due to its resource dependency. KPI (SLI) is calculated from resource usage, with all pod and node metrics collected from CloudWatch. However, there are no node logs for Fargate, complicating diagnosis.
- •
External Storage Failure. In this system failure, we fill up the external storage disk connected to the Database (DB) pod (i.e., mongodb-v1) within Microservice A’s OpenShift cluster. When the storage becomes full, the DB pod cannot add new data, resulting in system errors. These errors propagate to pods that depend on the DB pod, causing some services (ratings) within Microservice A to encounter errors. We monitor changes in response and error information for Microservice A using Jaeger logs. Metrics for all containers and nodes, including CPU and memory usage, are obtained from Prometheus within OpenShift. Logs for all containers and nodes are retrieved from Elasticsearch within OpenShift. Additionally, we collect message logs from the external storage. We illustrate the metrics and log data of the root cause pod in [Figure 5](#S3.F5).

Figure: Figure 5: Visualization of root cause for one system failure case (i.e., External Storage Failure) on the Product Review Platform. Left: six system metrics of root cause. Right: the system log of the root cause pod (i.e., Mongodb-v1) with the x-axis representing the timestamp, the y-axis indicating the log event ID, and the colored dots denoting event occurrences. Sudden drops in the metrics data, as well as new log event patterns observed at the midpoint, indicate a system failure.
Refer to caption: /html/2406.05375/assets/Figure/1203_metric_visualization.jpg

## 4 Experiments

### 4.1 Experimental Setup

Evaluation Metrics. To asses baseline RCA method on LEMMA-RCA, we choose three widely-used metrics [; ; ] and introduce them below.

(1). Precision@K (PR@K): It measures the probability that the top $K$ predicted root causes are real, defined as:

$$ $\text{PR@K}=\frac{1}{|\mathbb{A}|}\sum_{a\in\mathbb{A}}\frac{\sum_{i<k}R_{a}(i)\in V_{a}}{\min(K,|v_{a}|)}$ (1) $$

where $\mathbb{A}$ is the set of system faults, $a$ is one fault in $\mathbb{A}$, $V_{a}$ is the real root causes of $a$, $R_{a}$ is the predicted root causes of $a$, and $i$ is the $i$-th predicted cause of $R_{a}$.

(2). Mean Average Precision@K (MAP@K): It assesses the top $K$ predicted causes from the overall perspective, defined as:

$$ $\text{MAP@K}=\frac{1}{K|\mathbb{A}|}\sum_{a\in\mathbb{A}}\sum_{i\leq j\leq K}\text{PR@j}$ (2) $$

where a higher value indicates better performance.

(3). Mean Reciprocal Rank (MRR): It evaluates the ranking capability of models, defined as:

$$ $\text{MRR@K}=\frac{1}{|\mathbb{A}|}\sum_{a\in\mathbb{A}}\frac{1}{\text{rank}_{R_{a}}}$ (3) $$

where $rank_{R_{a}}$ is the rank number of the first correctly predicted root cause for system fault $a$.

Baselines.
We evaluate the performance of the following causal discovery based RCA models on the benchmark sub-datasets: (1). PC []: This classic constraint-based causal discovery algorithm is designed to identify the causal graph’s skeleton using an independence test. (2) Dynotears []: It constructs dynamic Bayesian networks through vector autoregression models. (3). C-LSTM []: This model utilizes LSTM to model temporal dependencies and capture nonlinear Granger causality. (4). GOLEM []: GOLEM relaxes the hard Directed Acyclic Graph (DAG) constraint of NOTEARS [] with a scoring function. (5). REASON []: An interdependent network model learning both intra-level and inter-level causal relationships. (6). Nezha []: A multi-modal method designed to identify root causes by detecting abnormal patterns. (7). MULAN []: A multi-modal RCA method that learns the correlation between different modalities and co-constructs a causal graph for root cause identification. (8). CORAL []: An online single-modal RCA method based on incremental disentangled causal graph learning.

The fist four models can only learn the causal structure from time series data. Thus, we first collect monitoring data from the beginning until system failures occur as historical records. Then, based on the collected records, we apply the causal discovery models to learn causal graphs and leverage random walk with restarts on such graphs [] to identify the top $K$ nodes as the root causes. Besides we extend NOTEARS and GOLEM to the online learning setting, denoted by NOTEARS^∗ and GOLEM^∗, respectively(^1^11Other baselines are not extended to the online setting as they are time-intensive when there are multiple data batches.).
For the online setting, we use the historical normal data (e.g., $8$ hours for the Product Review sub-dataset, and $1$ hour for the SWaT and WADI sub-datasets) to construct the initial causal graph and update iteratively for each new batch of data. CORAL can inherit the causations from the previous data batch, while NOTEARS^∗ and GOLEM^∗ have to learn from scratch for each new data batch.

### 4.2 Offline Root Cause Analysis Results

Product Review and Cloud Computing.
We evaluate seven offline RCA methods including both single-modal and multi-modal methods on Product Review and Cloud Computing sub-dataset. The experimental results are presented in [Table 4](#S4.T4) with respect to Precision at K (PR@K), Mean Reciprocal Rank (MRR), and Mean Average Precision at K (MAP@K). Our observations reveal the following insights: (1) The REASON method demonstrates notable success in identifying the root cause in 75% of system fault scenarios, achieving a PR@1 score of 75%. This indicates the utility of metric data alone in facilitating root cause identification. (2) The performance of these RCA methods is diminished when relying solely on log data for root cause analysis. (3) Integrating both metric and log data enhances the performance of most RCA methods in terms of MRR, compared to using only metric data. This suggests that log data complements these methods, aiding in more accurate identification of potential root causes.

Table: Table 4: Results for offline RCA baselines with multiple modalities on the Product Review dataset.

Table: Table 5: Results for offline RCA baselines on the SWaT and WADI dataset.

Water Treatment/Distribution.
We employ five single-modal RCA methods to assess root cause localization performance on the SWaT and WADI sub-datasets. The comparative results, presented in Table [Table 5](#S4.T5), are evaluated in terms of PR@K, MRR, and MAP@K. Consistent with observations on the Product Review and Cloud Computing sub-datasets, REASON outperforms the other four baseline methods. However, a decline in performance for the best baseline method, REASON, is noted when compared to its results on the Product Review and Cloud Computing datasets. This decrease in performance can be attributed to the nature of the SWaT and WADI sub-datasets, where faults are brief and the intervals between them are short. These fleeting events can be easily missed by most RCA methods, thus posing a significant challenge in accurately identifying the root causes within these two sub-datasets.

### 4.3 Online Root Cause Analysis Results

We evaluate three RCA methods on all sub-datasets to assess the quality of the LEMMA-RCA sub-dataset in an online setting. Notice that due to the lack of multi-modal online RCA methods, we measure the performance of these single-modal baseline methods using only metric data shown in [Table 6](#S4.T6). By observation, we find that the online version of RCA models (e.g., GOLEM^∗) outperform their offline version (e.g., GOLEM) as online methods can rapidly capture the changing patterns of the metric data, thus learning a more accurate and noise-free causal structure for RCA. Among online methods, CORAL significantly outperforms NOTEARS^∗ and GOLEM^∗ due to the design of state-invariant and state-dependent representations learning tailored for the online setting. Notably, LEMMA-RCA is a large-scale real-world dataset, consisting of more than 100,000 timestamps across several days with various system fault scenarios, which can be naturally transformed to the online setting, compared with small datasets (e.g., NeZha []) with limited timestamps for online RCA.

Table: Table 6: Results for online root cause analysis baselines on all sub-datasets.

## 5 Discussions

Broader impact: To facilitate accurate, efficient, and multi-modal root cause analysis research across diverse domains, we introduce LEMMA-RCA as a new benchmark dataset.
Our dataset also offers significant potential for advancing research in areas like multi-modal anomaly detection, change point detection, causal structure learning, and LLM-based system diagnosis. Based on the thorough data analysis and extensive experimental results, we highlight the following areas for future research:

- •
Expanding Domain Applications: To enhance the LEMMA-RCA dataset’s versatility and impact, we plan to incorporate data from additional domains such as cybersecurity and healthcare. This integration of diverse data sources will facilitate the development of more comprehensive root cause analysis technologies, significantly extending the dataset’s applicability across various industries.
- •
Online Multi-Modal Root Cause Analysis: The majority of RCA methods are offline and typically single-modal. However, our research highlights a notable gap in online, multi-modal approaches. This presents an opportunity to develop real-time, multi-modal root cause analysis methods that can instantly process and analyze diverse data streams. Such advancements are crucial for dynamic environments like industrial automation and real-time monitoring services, where immediate response is essential.

Limitations: Despite its broad capabilities, the LEMMA-RCA dataset may have
limitations in terms of the dataset generalizability, as the system fault scenarios we created may not adequately reflect the diversity of conditions prevalent in broader real-world applications. Another limitation is the LEMMA-RCA dataset is associated with some missing data in the system metric readings. These issues can arise from factors such as system interruptions and other unforeseen circumstances.

## 6 Conclusion

In this work, we present LEMMA-RCA, the first large-scale, open-source dataset featuring real system faults across various application domains and multiple modalities. We evaluate the quality of LEMMA-RCA by testing the performance of eight baseline methodologies on this dataset under different settings, including offline/online modes and single/multiple-modality data. Our experimental results demonstrate the high quality of LEMMA-RCA. By making this dataset publicly available, we aim to facilitate further research and innovation in root cause analysis for complex systems, contributing significantly to the development of more robust and secure methodologies that ensure the high performance of modern systems, particularly those that are mission-critical.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could the authors consider including the dependency graph? Having this graph like in petshop seems like a deal breaker to me.

2. Could the authors benchmark the baselines using the dependency graph instead of the causal graph inferred by the PC?

3. For the CIRCA method as well, could the authors provide results based on the dependency graph?

4. The experiments section needs more and a systematic explanation on why each method performed better or worse.
soundness: 2
strengths: - Lemma-RCA is a large, multi-model and multi-domain dataset that includes data from both IT and OT domains. It has over 100,000 timestamps across several connected pods, with a rich mix of test logs and time-series data. This dataset will be valuable for testing and improving future RCA methods.

- Unlike most other datasets, Lemma-RCA provides exact ground-truth labels, showing both when faults happened and the specific components responsible. 

- The paper builds on past studies that highlight using causal structure-based methods for RCA. The authors compare Lemma-RCA with causal discovery methods and other recent RCA models.

- **Clarity and Presentation**: The paper is well-organized, with clear visuals and a smooth flow that makes it easy to understand in a single read.
summary: This paper introduces Lemma-RCA, a dataset designed for root cause analysis. Lemma-RCA has distinctive and appreciable characteristics like large-scale, multi-modal nature and spans two domains: IT and OT. It includes test logs and time-series data, capturing KPI metrics across several interconnected pods over more than 100,000 timestamps. Notably, the dataset provides ground-truth annotations for both the exact fault occurrence times and the root cause components. This level of detail makes Lemma-RCA a valuable resource for advancing research in RCA.
weaknesses: **Missing Dependency Graph**: A key limitation of Lemma-RCA is the absence of a dependency graph, which prior datasets like PetShop provided as a causal graph. This dependency graph is critical for RCA, as it allows more direct evaluations of causal discovery methods. The paper seems to already hint the partial dependency graph in Figure 1(a). I wonder if the authors could add the full dependency graph along with the datasets.

**Insufficient Explanation of Baseline Approaches:** The paper does not include explanations of the baseline approaches used, even in the appendix. Although prior work is cited, providing brief descriptions of each benchmarked approach, particularly the high-performing REASON method, would enhance the reader’s understanding of the comparative results.


**Limited Explanation of Experimental Results**: The experimental results focus primarily on causal discovery approaches, but they lack in-depth analysis of why these methods failed. The authors' insights and intuition about why each method achieved the numbers reported in the table could significantly enhance the understanding of the experiment section. For instance, suppose we assume that the dependency graph is the true causal graph as in PetShop. Then can the authors establish how far the PC's predicted causal graph is from the true dependency graph. This would at least give us a sense of the causal discovery performance and put the RCA results in context. For instance, if the causal discovery performance is very poor, there is no meaning in expecting the methds like PC, GOLEM, etc. to perform better in predicting root causes. Additionally, one interesting experiment to run would be evaluating the causal graph based baselines on the true dependency graph, instead of the one inferred from observational data by PC.

**Choice of Baseline Algorithms:** Given that the dataset is timestamped, it cannot be assumed that each record is i.i.d. Some causal discovery methods, like those in the Tigramite package (https://jakobrunge.github.io/tigramite/), are tailored for time-series data. It is unclear why the authors chose standard PC over these alternatives, which may be more suitable for time-dependent causal discovery.


Finally, some important prior RCA works appear to be missing among the benchmarked methods. For example, the paper by Pham et al. (2024) on BARO highlights that inaccurate RCA predictions can result when a method fails to learn the correct causal graph. Including such approaches would provide a more thorough baseline comparison and strengthen the evaluation.

[1] Pham L, Ha H, Zhang H. Baro: Robust root cause analysis for microservices via multivariate bayesian online change point detection. Proceedings of the ACM on Software Engineering. 2024 Jul 12;1(FSE):2214-37.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The authors claimed that the proposed datasets contain real system faults. If I understand correctly, the authors developed two microservice platforms and deployed them in production and collected the real system faults for 8 days when users are using these platforms. It is a bit surprising that so many faults happened in 8 days. Moreover, in the faults description section, it seems that these faults are simulated (e.g., External Storage Failure) to mimic the real world scenario. Could the authors clarify this?
2. It seems that SWaT and WADI are from existing work. The authors applied some anomaly detection algorithms on them to transform discrete labels into continuous ones. It is not clear why this is necessary. Moreover, SWat and WADI are already evaluated for RCA in the REASON (Wang et al. KDD2023) paper.  Since this is a dataset paper, including existing datasets into the proposed one should not be seen as the contribution. 
3. From experiments on existing methods, it seems that two IT ops datasets are not very challenging. For instance, REASON performs quite well in terms of PR@k, MRR and MAP@k on both of them with only the metrics data. What is the difference between the proposed datasets compared with existing ones, e.g., AIOps data in REASON and the popular train ticket datasets? When new datasets are proposed, they are expected to be more challenging, where current methods are failed on them. If current method can handle the proposed well with only metric modal, what is the meaning of including log modal?
4. The authors conducted some preprocessing to convert logs to time series for evaluation. But the open-sourced datasets do contain all the original logs, right?
soundness: 3
strengths: 1. New multi-modal datasets are collected for RCA problem.
2. Eight existing RCA methods are evaluated on the proposed datasets.
summary: In this paper, the authors proposed a new dataset with both metrics and log collected for the root cause analysis task. In addition, 8 existing RCA methods are evaluated on this dataset. The proposed datasets could be a good addition for evaluation of RCA methods for later research. However, it is not very clear what the benefit of including log modal data is. Existing methods work quite well on these datasets with only metrics modal.
weaknesses: 1. The description of the data collection is insufficient. See Q1.
2. Some subsets of the datasets are from existing work and have been evaluated before. They should not be seen as the contribution of this work. See Q2.
3. The proposed IT ops datasets seems to be less challenging for existing works. See Q3.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: (1)	The author should provide more valuable data rather than simply assembling data. Additionally, it is not clarified whether these platforms the data come from are sufficiently representative to ensure the quality of the data and the data collection period appears to be rather short, making it difficult to establish whether the dataset adequately captures a wide range of fault patterns and system behaviors.
(2)	The experiments designed by the authors do not seem sufficient to demonstrate the value of the dataset. I suggest that the authors select several widely recognized RCA methods with known performance differences and analyze whether these methods exhibit similar performance distinctions on this dataset.
(3)	The author can pay more attention to the readability of the figures in the paper and the normalization of the experimental results.
soundness: 2
strengths: (1)	LEMMA-RCA is the first public dataset specifically designed for root cause analysis, covering two domains—IT and OT. 
(2)	The paper thoroughly tests multiple existing RCA methods on LEMMA-RCA, demonstrating the dataset’s quality and multi-modal value.
(3)	By making LEMMA-RCA freely accessible, the paper lowers research barriers, encouraging collaboration between academia and industry and enhancing the generalizability and practical impact of RCA methodologies.
summary: This paper presents LEMMA-RCA, a large-scale, multi-modal, and multi-domain dataset specifically designed for Root Cause Analysis (RCA) in complex systems. The dataset includes real-world fault cases from IT and OT operational systems, covering microservices, water treatment, and distribution systems to support a wide range of RCA tasks. To validate the effectiveness of LEMMA-RCA, the authors evaluated various RCA methods on this dataset, demonstrating its diversity and utility across offline and online settings as well as single and multi-modal data scenarios.
weaknesses: (1)	One contribution of this study is the introduction of LEMMA-RCA, the first multi-domain dataset for RCA. However, the dataset includes only IT and OT domains, which appear to be a simple combination of two unrelated domains, thus raising questions about the solidity of this contribution. Additionally, the limited data collection period, such as the 11 days for the OT system, may not capture long-term trends, potentially limiting its applicability to broader fault analysis scenarios.
(2)	The figures in this study are unclear, heavily relying on screenshots. 
(3)	The experimental analysis tables lack consistency in reporting, with varying decimal places and an absence of standard deviation reporting.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 2
questions: 1. **Clarification:**
- The current data collection section seems tailored for domain experts and could benefit from clarification for a general audience. For instance, what do "IT" and "OT" refer to? Are "Prometheus" and "ElasticSearch" tools or companies? Clarifying the meanings of such terms would improve the accessibility. 
- Figure 2(a) would benefit from a more detailed explanation in its caption.
- In Figure 3, what is the KPI being referenced? Should it be assumed that all KPIs in this figure relate to system latency? Please specify the y-axis further.
- How were the root causes $V_a$  for each system fault $a$ labeled? The authors  may include a section in the paper detailing their labeling methodology.
2. **Evaluation Metrics:** The evaluation metrics appear to be sample-independent. Why did the authors not consider sample-dependent metrics? For example, over a 10-day system run yielding 1000 faults, the accuracy of the prediction algorithm could be tested against the actual root cause labels.

3. **Data Quality Claims:** The authors suggest high data quality based on baseline comparisons. This conclusion seems somewhat overstated, as the main insight from these experiments appears to be that "MRR performance improves when considering two modalities jointly."

**Comment on Missing Data:** While the authors view missing data as a limitation, I consider it a realistic aspect of real-world data, which poses a meaningful challenge rather than a flaw.
soundness: 2
strengths: - The dataset is open-source, multi-modal, and well-suited to RCA, making it both timely and relevant.
- The authors have provided a thorough review of existing baseline approaches.
summary: The paper presents Lemma-RCA, a novel dataset and benchmark designed for root cause analysis (RCA). This dataset includes four sub-datasets: two from IT environments and two from OT environments, offering a large-scale, multi-modality dataset (with both KPI and log data) that captures real-world system faults. The authors validate the dataset’s quality by testing it with eight baseline models across offline single-/multi-modality and online single-modality settings.
weaknesses: - The dataset description could be more accessible to a broader audience, as suggested in the questions below.
- Reproducibility is limited due to insufficient implementation details for the baseline models.

Meta-review: The paper contributes 
(1) large dataset for root cause analysis
(2) 14 baselines evaluated .

The reviewers feel that if ICLR had a separate Dataset track this would be a sure accept. 
The methodological contributions are modest and hence it is not clear on how this paper will stand with other ICLR papers which have more methodological contributions. At this point the paper is, at best, borderline.

Decision: Reject