Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Due to the complex and dynamic inter-module correlation structure, the microservice system in the cloud-native environment produces complicated failure behaviors. To efficiently diagnose the fault and identify the root cause, this study draws on the idea of the $\psi$-FCI algorithm, treating faults as interventions on root causes to learn the underlying causal structure. With this, the authors design a causal graph construction algorithm, called $\psi$-PC, which degenerates from the $\psi$-FCI algorithm based on the assumption of no confounding variables, randomly grouping all nodes, and using a two-step hierarchical test to optimize computational efficiency. Finally, the generated and real data are used to verify the algorithm's advantages in terms of diagnostic accuracy and computational efficiency.
strengths_and_weaknesses: Strengths:
- The problem has real research significance and economic value, especially for large-scale cloud system failure operation and maintenance
- The efficiency advantage of the proposed solution makes it more adaptable to real system scenarios

Weaknesses:
- Lack of clarification on the primary observation/motivation of the paper
- Running cases are not provided to show many details and results of the proposed method
- Insufficient theoretical contribution as $\psi$-PC is a simplification on the basis of $\psi$-FCI
questions: The abstract fails to highlight the problems of existing research and the essential difference between this method and previous studies. Likewise, while Section 2 comprehensively covers the relevant concepts, it does not explain the inadequacies of previous studies and what is the specific observation / motivation of this study. 

The authors mention in Section 3: “The key benefit of modeling the failure as an intervention is that it enables us to translate the problem of finding the root cause of failure into finding the interventional target. In addition, this view allows us to leverage the recent developments in the causal discovery literature that can identify intervention targets. ” I disagree that this is a convincing statement, as the authors only say the benefits but do not justify the validity of underlying assumptions.

Furthermore, the paper lacks real-world case studies to aid in understanding the algorithmic processes and details. For example, how to identify the intervention target in $\psi$-PC, and what is the relationship between the intervention target and the root cause. It is recommended to use real-world running examples and causal e-and-effect graphs in real diagnostic scenarios to illustrate which nodes are (or could be) interventions and which are intervention targets? Figure 1 does not help to evaluate the key observations’ validity / reasonableness in microservices troubleshooting scenarios. Besides, the author mentions an assumption (observations) without any explanation: “a fault changes the generative mechanism of the failing node.” I would like to see why this assumption or observation holds true with examples. 

In the first phase of $\psi$-PC, is it a random division? Or should refer to any prior knowledge? How to ensure that the division place no adverse effect on the result? Since it also divides the causal structure into local graphs, how to theoretically ensure that the local graph will always contain the correct root cause compared to the global graph.

The section analyzing the algorithmic complexity mentions: " By decoupling learning the causal graph from finding the interventional targets, we can improve runtime by significantly reducing the number of CI tests." However, even if we use an iterative PC algorithm, the complexity can be controlled by limiting the depth of CI tests. In addition, adopting the heuristic search algorithm can also generate more accurate results on specious causal graphs. In other words, performing the CI tests thoroughly and iteratively is unnecessary. Therefore, it is recommended to discuss and demonstrate the computational complexity by comparing with various optimization baselines (such as $\psi$-FCI, PC, or other related methods such as PCMCI and Granger's test, etc.).

Lastly, I noticed that the authors compared ϵ-Diagnosis and AutoMAP using synthetic data. The diagnostic goals for these baselines appear to be different from $\psi$-PC. They aim to find the deepest nodes in the failure propagation chain and consider whether they are highly correlated with failure patterns. This idea stems from engineers' intuition in manual troubleshooting. It is suggested that the author analyze the adaptability of different diagnostic ideas and simulation methods.
limitations: Not applicable.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper formulates the root cause analysis problem as a causal inference task by modeling fault as an intervention on the failing node. A Root Cause Discovery algorithm inspired by existing FCI algorithm for detecting interventional targets is proposed. Additional node called F-node is introduced to represent the effect of the intervention. The proposed approach relies on the PC algorithm to learn causal graphs. It   reduces computation overhead of conditional independence tests by using a hierarchical approach and a localized learning algorithm that focuses on only learning the neighborhood of the F-NODE. Evaluation of this algorithm with synthetic and real-world dataset shows that the proposed approach generally outperforms the baseline methods.


strengths_and_weaknesses: Strengths

- The proposed algorithm addresses a challenging problem: quick and accurate root cause diagnosis of failures in complex microservice systems. The localized learning for causal discovery approach is a good fit for this use case. 

- Experimental evaluation on synthetic and test dataset show improvement in runtime and detection accuracy compared to the baseline approaches. A real-world failure is also diagnosed using the proposed approach to demonstrate the algorithm's superiority over the baseline methods.

Weaknesses

- Although the addition of a hierarchical approach reduces overall execution time with reduced conditional independence tests, it is not clear how to select the subset size or how the subset size impacts the accuracy of the algorithm. Experimental results on different subset sizes would provide useful insights.

- The evaluation metrics include precision numbers but no measurement of false negatives are presented.

- Inclusion of SOTA ML based techniques like Sage[1] in the comparative study would strengthen the overall evaluation of the proposed approach.

[1] Yu Gan, Mingyu Liang, Sundar Dev, David Lo, and Christina Delimitrou. 2021. Sage: Practical and Scalable ML-Driven Performance Debugging in Microservices. In ASPLOS. 135–151.

questions: - How is the value of subset size determined?
- How significant are the false negative rates and/or recall numbers?
- A recent work [2] published in KDD’22 also models failures as interventions in their proposed approach. Could you briefly provide insights on additional benefits of the proposed approach?  

[2]  Causal Inference-Based Root Cause Analysis for Online Service Systems with Intervention Recognition: https://arxiv.org/pdf/2206.05871.pdf

limitations: The proposed approach ignores the existence of latent variables. Also, the algorithm cannot process time-series data. These limitations are mentioned in the paper as future works. 

There is no societal impact of this work that needs to be addressed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a causal discovery algorithm for quickly detecting the root cause of failure in microservice systems. The proposed algorithm treats a failure as an intervention on the root cause to learn the underlying causal structure. It only learns the portion of the causal graph related to the root cause, instead of relying on conditional independence tests. The authors compare their proposed solution with a modified version of the PC algorithm and some existing work on root cause analysis. The results show the effectiveness of the proposed algorithm (in terms of top-k precision) while keeping the execution time reasonable. 
strengths_and_weaknesses: Microservices become popular in recent years. This paper addressed an important problem of diagnosing failures of microservices systems. The proposed approach seems outperform the existing methods and is scalable.

The evaluation of the proposed approach is limited. The comparision with existing work is only conducted on a synthetic dataset. No real data is used in quantitative comparisons. It would be better to compare the proposed approach with existing work using large-scale, real-world data. The evaluation on Sock-shop is a small-scale test bed.  It consists of 13 microservices and only two failures (CPU hog and memory leak) are simulated. Therefore, the quantitative evaluation is rather limited. 

Furthermore, there are many existing causal discovery-based methods for root cause analysis of failures in large-scale service systems.  For example, the following work also adopts a PC-based method for causal inference:

[Ref-a] Satoru Kobayashi, Kazuki Otomo, Kensuke Fukuda, and Hiroshi Esaki, Mining causality of network events in log data, in TNSM 2018.

The following work also adopts a FCI-based method for diagnosing failures of cloud service systems:

[Ref-b] Yujun Chen et al., Outage Prediction and Diagnosis for Cloud Service Systems, Proc. The Web Conference 2019, San Francisco, May 2019.

It would be better if the authors can discuss and compare with the above closely related work.

In the proposed Algorithm 1 (Root Cause Discovery Algorithm), it is not clear how F-Node is set. More details could be given here.

questions: . Did you compare the proposed approach with existing work using large-scale, real-world data? 

. How does your method compare with [Ref-a] and [Ref-b] above?


limitations: No, the authors didn’t adequately address the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The paper provides a new approach for learning root causes of failures and is targeted to micro-services. One of the main shortcomings of the paper raised by reviewers was in the evaluation. On the one hand, reviewers pointed out a number of very relevant related works that are not compared with this work, and on the other hand, the bulk of the quantitative assessments are done using synthetic data, and there were some questions about the level of realism of the Sock-store experiment. That said, the rebuttal had a good analysis of the relationship with those related works and argued for why they are not entirely comparable. I do think the evaluation could be stronger; in fact, I think the paper could have more impact and more visibility in a systems conference, but that would require a more complete evaluation on more real systems (it is instructive to compare the evaluation of this paper with that of the Sage paper brought up by one of the reviewers). However, I think the combination of an interesting algorithmic contribution, strong results on simulated data and a compelling real-world case study puts this paper above the bar. 
confidence: Less certain
Decision: Accept