A Robust Method to Discover Causal or Anticausal Relation

Authors: Yu Yao, Yang Zhou, Bo Han, Mingming Gong, Kun Zhang, Tongliang Liu
Primary area: causal reasoning
Keywords: Trustworthy Machine Learning, Semi-Superivsed Learning, Causality

Abstract:
Understanding whether the data generative process follows causal or anticausal relations is important for many applications. Existing causal discovery methods struggle with high-dimensional perceptual data such as images. Moreover, they require well-labeled data, which may not be feasible due to measurement error. In this paper, we propose a robust method to detect whether the data generative process is causal or anticausal. To determine the causal or anticausal relation, we identify an asymmetric property: under the causal relation, the instance distribution does not contain information about the noisy class-posterior distribution. We also propose a practical method to verify this via a noise injection approach. Our method is robust to label errors and is designed to handle both large-scale and high-dimensional datasets effectively. Both theoretical analyses and empirical results on a variety of datasets demonstrate the effectiveness of our proposed method in determining the causal or anticausal direction of the data generative process.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730302165819

thread_type:
official_review

<thread_object>
creator: reviewer_ndts

<content>
summary:
This paper proposes a robust estimator, the Robust Causal and Anticausal (RoCA) Estimator, to identify causal or anticausal relationships in datasets, even when label noise is present. Traditional methods for causal discovery struggle with high-dimensional data, especially when label noise disrupts the feature-label relationships. The RoCA estimator's key innovation is a noise injection technique that exploits an asymmetry between causal and anticausal relationships. In causal datasets, instance distribution P(X) should not help predict the label distribution P(Y∣X), while in anticausal datasets, P(X) can be informative for predicting P(Y∣X). Key to this method is the identification of an asymmetric property: in causal datasets, the instance distribution P(X) lacks predictive information about the noisy label distribution P(Y~∣X), while in anticausal datasets, P(X) does provide such information. By injecting varying levels of instance-dependent label noise and measuring shifts in the disagreement rate between pseudo-labels (generated through clustering) and noisy labels, RoCA detects causality based on whether disagreement trends change across noise levels. Unlike traditional methods that can struggle with perceptual data or noise, RoCA remains accurate across various noise types and dataset scales, as validated in experiments on both synthetic and real-world datasets.

strengths:
The paper introduces the RoCA (Robust Causal and Anticausal) Estimator, an innovative approach to causality detection that is both noise-robust and adaptable to complex, high-dimensional data. This originality stems from its focus on the asymmetric relationship between instance distributions P(X) and noisy label distributions P(Y~∣X), rather than relying on cleaner, idealized data. By targeting causality detection under instance-dependent label noise, the paper introduces a novel technique to address causal or anti-causal discovery in reql-world noisy data. This subimission os overall high-quality. RoCA’s robustness across multiple noise types is carefully validated, including tests on both synthetic and real-world datasets. Moreover, the paper's detailed experiments showcase significant improvement over baseline methods, lending credibility to RoCA’s real-world applicability. Furthermore, the proposed method is backed-up by sound theoretical analyisis, showing that RoCA can indeed detect causal and anti-causal relations. The paper is structured and articulated in a clear and accessible manner. Technical details about clustering, disagreement measures, and the rationale behind examining instance distribution asymmetries are explained systematically. Overall, this work is significant for the field of Causal Inference. RoCA is potentially applicable to fields requiring robust causal inference from noisy data, such as healthcare, economics, and social sciences, where data noise is often unavoidable.

weaknesses:
The robustness of RoCA relies heavily on certain parameters, such as those governing the clustering process and the threshold for disagreement trends. If these parameters are not tuned appropriately, the method’s performance may degrade, particularly in cases of subtle causal effects or in datasets with less distinct noise patterns. This reliance on careful parameter tuning may limit the method’s effectiveness in some real-world applications where optimal parameter settings are difficult to identify. Furthermore, RoCA’s reliance on clustering as a step in handling noisy labels presupposes that the dataset is large enough to form meaningful clusters. In cases of limited or highly sparse data, this dependency may affect the method’s reliability, as insufficient data could lead to unstable or non-representative clusters, impacting the overall accuracy of causality detection.

questions:
Regarding the weaknesses mentioned above, can you comment on the robustness of RoCA to parameters tuning? Also, what is the performance of RoCA in low-sample regimes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730432452869

thread_type:
official_review

<thread_object>
creator: reviewer_pe7x

<content>
summary:
This paper aims to discern if a data generation process is causal or anti-causal. It proposed the Robust Causal and Anticausal (RoCA) Estimator, where the authors attempt to differentiate the two by investigating if the instance distribution, $P(X)$, offers information about the prediction task, $P(Y|X)$. They opted for the noisy class-posterior distribution, $P(\tilde{Y}|X)$, to surrogate for $P(Y|X)$ and used clustering to generate pseudo labels. Their discussion suggests that there should be no correlation between mismatch and noise levels in a causal scenario, while in an anti-causal context, a correlation should exist. Experiments are conducted on several causal discovery datasets and several image datasets.

strengths:
1. The problem of interest is an important topic in casual discovery.

2. The idea of using injected label noise to discover causal vs. anti causal relationships is novel.

3. Experiment results show significant improvement over existing causal discovery approaches.

weaknesses:
1. The proposed algorithm crucially depends on the idea that $P(\tilde{Y}|X)$ is a surrogate of $P(Y|X)$. Unfortunately, it is not clear what information the surrogate must preserve for the $P(Y|X)$. An intuitive algorithm for constructing pseudo label and injecting label noise is used in the implementation, but I think the paper lacks a formal discussion of the identifiability. The authors can provide a precise definition on under what conditions RoCA can identify the direction.

2. In the instance-dependent noise injection, the "dependence" between the instance and its noisy label is created based on the l1-norm. Because instances with different $X$ can have the same l1-norm, the noisy label only depends on the l1-norm but not the original instance. For the noise to be instance dependent, the noisy label should be related to the semantic meaning of the instance, e.g., a husky dog is incorrectly labelled as a wolf, or a lion cub is incorrectly labelled as a cat. The authors should make a better explanation of how the noise-generation process relates to the noise being instance-dependent. 

3. It seems that the validity of the algorithm will largely depend on $P(X)$ as a crucial step in the reasoning is that "under the causal setting, $P(X)$ does not contain information about the surrogate distribution $P(\tilde{Y}|X)$." and thus $P(X)$ cannot help predict observed labels. But, if we consider a dataset where there are 10 classes, and the samples of these 10 classes happen to reside in well-defined clusters. In other words, a clustering algorithm (e.g., k-means) can easily separate these 10 clusters. Then, the pseudo label obtained from $P(X)$ will not be random guesses of the observed labels. Even if noise is introduced to the observed labels, the pseudo label could continue to correspond to the observed labels unless the noise rate is extremely high.

This does not suggest an algorithm should work on all scenarios. But rather, as a causal discovery algorithm, the authors should be clear regarding when the algorithm is expected to work, and when the algorithm is expected to not work/fail. Currently, this is not clearly discussed in the paper.

questions:
Besides the weaknesses, there are also some questions regarding the clarity of writing. 

1. For the expected noise level injection using the truncated distribution. What is the specific procedure of truncation? If you set the mean to 0.2 and truncate at 0 by setting all negative values to 0, then the actual expected noise level will be smaller than 0.2. How is this accounted for in the experiments?

2. What is the chosen unsupervised algorithm for clustering the instances to generate pseudo labels? Also, if the clustering algorithm changes, then the cluster results will also change. How will this affect RoCA? 

3. In the real-world noisy image datasets, how to validate whether the results are correct? CIFAR-10N is a dataset injected with human-annotated label errors. As the human will annotate the image by looking at the image, the data generation process seems to be causal $X\to \tilde{Y}$ instead of anticausal. For Clothing1M, the dataset contains images crawled from the web, and the labels are generated from the text description accompanying the image, it also seems to be causal. How do we determine whether a real-world noisy dataset is causal or anti-causal for the sake of evaluating performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620146321

thread_type:
official_review

<thread_object>
creator: reviewer_gqjm

<content>
summary:
To address the limitations of existing methods on high-dimensional data (such as images), this paper proposes a novel noise injection method that determines the causal direction by examining the asymmetry in data distribution. The proposed RoCA estimator uses unsupervised clustering to generate pseudo-labels and determines the causal direction of the dataset by observing the divergence between pseudo-labels and observed labels under varying degrees of noise injection. This approach performs well in both theoretical analysis and empirical experiments, effectively distinguishing causal and anti-causal data generation processes across multiple datasets.

strengths:
1. The method proposed in this paper differs from traditional causal discovery algorithms, providing a new direction for high-dimensional causal relationship detection.

2. The effectiveness of the RoCA method is demonstrated through theoretical analysis and validation on multiple datasets.

3. The appendix provides a detailed review of relevant techniques and datasets in the field, offering strong reference value.

4. By generating pseudo-labels through unsupervised clustering, RoCA can effectively determine causal direction without relying on true labels, making this method more adaptable and flexible for various application scenarios.

weaknesses:
1.	How can we ensure that the pseudo-labels generated by clustering effectively represent the true distribution of the data? The paper lacks analysis or at least a clear statement of assumptions in this regard, which would strengthen the theoretical support for the method’s pseudo-label generation process.
2.	Although the paper compares several causal discovery methods, it lacks comparison with some of the latest methods based on causal inference theory, such as neural network-based causal discovery algorithms. It is suggested to include comparisons with additional advanced causal discovery methods.
3.	The RoCA method involves multiple steps of noise injection and pseudo-label generation, which may lead to substantial computational costs on large-scale datasets. It would be helpful if the authors could discuss the model's complexity in more detail to assess its feasibility in practical applications.

questions:
1.	The paper uses K-means and SPICE* clustering to generate pseudo-labels and achieves promising results. Would the method's stability be maintained if other unsupervised clustering algorithms were used for pseudo-label generation?
2.	If there is class imbalance in the dataset, the generated pseudo-labels might skew toward the larger classes, potentially affecting the accuracy of causal direction detection. In such cases, would the performance be impacted? If so, how could the imbalance issue be addressed?
3.	For the poor performance on Pair0071, the authors explain it as due to the presence of confounders affecting both features and labels. However, have they attempted to analyze the reasons for RoCA not achieving the best results on the Mushroom dataset across all methods?
4.	The paper does not discuss cases with confounding variables, but have the authors considered the robustness of the current method in environments with confounders? Alternatively, how might the method be improved to enhance its robustness in the presence of confounding variables?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a noise injection method to address the limitations of existing methods on high-dimensional data (such as images). The proposed method determines the causal direction via examining the asymmetry in data distribution. It uses unsupervised clustering to generate pseudo-labels and determines the causal direction of the dataset by observing the divergence between pseudo-labels and observed labels under varying degrees of noise injection. Both the theoretical and empirical studies provide solid results and insights, effectively distinguishing causal and anti-causal data generation processes across multiple datasets.

Most questions raised by the reviewers are for clarification of important details of the proposed methods, and some questions request for extra experiments. The authors have provided detailed rebuttal, providing sufficient clarification and extra experiments. The reviewers have either increased or retained their acceptance ratings.

Overall, all reviewers are in favor of accepting this paper. I agree with this consensus and recommend an acceptance accordingly. I encourage the authors to incorporate all the extra details (e.g., extra experiments) of the discussion into future revision of the paper.