Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper focuses on the problem of causal structural learning from heterogeneous populations, where the label or index of the population is unknown. The main contribution of this paper is on developing a distance covariance-based kernel to measure the similarity between the underlying nonlinear causal structures of different samples, so that we can first perform clustering to identify the homogeneous subpopulations and then use existing methods to learn a causal structure for each of these subpopulations. The proposed method is tested on genetic data.

main_review: This paper studies an interesting problem—how to do causal structural learning when the data may be from different populations, without knowing their labels. However, overall, the writing needs to be polished to improve the readability, including the definitions and theorems. 

My main concern of the proposed kernel metric is as follows. In Theorem 10 and Corollary 11, the authors show that if the kernel metric is negative, then X and X’ have different causal structures. The problem is that the theorem does not show the if and if only property; when the kernel metric is positive or zero, how should the conclusion be? Without the if and only if property, I do not think it can be applied to do clustering. Another issue is that Theorem 10 requires to know which n samples are in X and which n’ samples are in X’, and Theorem 10 holds asymptotically; but at the beginning of clustering, we need to start from n=n’=1, where the asymptotical result does not seem applicable.

Regarding the experiments, it would be more convincing if the authors can perform some simulation studies, since we can know the ground truth and estimate the accuracy of the proposed method. In addition, it is important to compare the results with different sample sizes and graph structures. For example, when some populations have very few samples, then how is the accuracy?

Update:

The author's response does explain some of my concerns about the if and only if property. However, I still have concerns about the writing and the experimental results. Also, the authors did not compare with any other methods.

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors proposed the dependence contribution kernel to resolve the homogeneous clusters in the projected feature space. This kernel is able to cluster samples based on difference in nonlinear independence structures.
main_review: The manuscript, especially the theory section, is well organized. Comparably the application section needs to be further polished.
limitations_and_societal_impact: (1)	It would be helpful to perform a simulation study to demonstrate the power of the distance covariance-based kernel. The dataset (Iyer, 1999) is quite outdated as there are only 517 genes available. Nowadays it’s very common to have over 10,000 genes from the experiment. I’m curious how the kernel behaves when the number of genes scales up. The other possible source of heterogenous data is the whole-blood/PBMC gene expression as blood is a heterogenous collection of different blood cells. Maybe this kernel is helpful to discover the cell markers for different cell population which is an important question in biology.

(2)	Authors should at least include one baseline model/kernel, e.g., plain k-means with Euclidean distance to highlight the novelty of the distance covariance-based kernels.

(3)	There is ground truth based on biological knowledge regarding TF regulation. It’s impossible to make sense of the results without knowing what members are in each TF cluster. The other thing authors can try is to perform gene set enrichment analysis to make sense of the TF clusters.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Paper proposes a new distance covariance based kernel for causal clustering of heterogenous population data. It presents theoretical results  on how kernel it can be used as a two-sample statistical test for causal data and induces a metric space on ancestral graphs. Empirical results show leaned causal graphs for gene expression data.
main_review: Idea of a distance covariance kernel for two-sample statistical testing of causal data is novel and interesting and is able to check for non-linear dependencies which follows from properties of distance covariance. Hilbert Schmidt Independence criterion (HSIC) which is an equivalent criterion of distance covariance could also used to defined a similar kernel on causal structures but not addressed on the paper.

Results on two-sample statistical testing and the induced metrics space on ancestral graphs provide a strong theoretical justification for the proposed kernel. 

Empirical results on learned causal structures from gene expression data show the learned latent causal models but they still seems qualitative and verification on simulation data would be valuable here. Paper is well written and provides theoretically strong justification of the work which would be interested to the community.
limitations_and_societal_impact: Paper describes limitations of the causal structures learned and states assumptions clearly. Societal impact is not direct as it is mostly a theoretical work.
review_title: null

Meta-review: This paper proposes a distance covariance-based kernel designed specifically for measuring the similarity between the underlying nonlinear causal structures of different samples, in order to deal with the problem of causal structural learning from heterogeneous populations without known labels, in which a single causal structure does not adequately describe all population members. According to the reviews, the paper would greatly benefit from simulation studies to demonstrate the power of the distance covariance-based kernel, and the authors should at least include one baseline model/kernel, e.g., plain k-means with Euclidean distance, to highlight the novelty of the distance covariance-based kernels.
Decision: Reject