Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes strategies for learning the structure of multiple sets of data observed over a common set of variables which may exhibit distribution shift. The authors address this problem by augmenting the dataset with an indicator variable which indicates membership to  dataset. After augmenting the dataset standard algorithms for structure learning are applied, with the additional restriction that the indicator variable may only be an ancestor. The authors provide theory that shows the procedure consistently estimates the local structures. The authors then show how the additional information obtained from the structure learned with the context variable can be used to disambiguate directions. Experimental results show the efficacy of the proposed approach. 

Overall, I think this is a sensible idea and contains some nice results. I do have a couple of questions: 

(1) In the paper the authors explicitly limit the algorithm to the case where all domains observe the same variables, however it seems like this need not be the case?
(2) The authors should cite the literature on learning with mixtures of Bayesian networks. While the aims are different, the model representation is very similar. 
(3) Can the authors provide intuition on whether this degeneracy will also become an issue if there are errors in the skeleton detection algorithm?
review_title: Review for Score-based Causal Discovery from Heterogeneous Data

final_score: rating: 3: Clear rejection
review_content: The paper presents a score-based approach for causal graph discovery from heterogeneous data. The approach is claimed to be guaranteed to find the correct graph skeleton asymptotically, and can detect more causal directions than previous algorithms designed for single domain data.  

Pros:
-The paper addresses an important problem of causal discovery from heterogeneous data.

Cons:
I have doubts about the correctness of many conclusions in the paper. For example, it looks to me that for Theorem 1 to be true, the pooled data $D_C$ must be i.i.d. samples from distribution P(V,C). However, $C$ is defined as the domain index. The problem setup simply assumes we are given $n$ data sets from $n$ domains. Then it’s natural to assume the domain index “C” will stand for 1, 2,  …, n. Under this setting, the pooled data are not i.i.d. samples, Theorem 1 will not hold, and I believe some other conclusions in the paper will not hold either. For Theorem 1 (and other conclusions) to hold, I believe one has to assume the domain index C is a random variable, such that the data sets from different domains are drawn randomly based on some P(C) and P(V|C). However, this may not be a realistic setting, and is not how the problem is set up in the paper. I don’t think the real data sets in the experiments satisfy this setting (not clear how the synthetic data sets are generated).

Overall, I think the paper has mistakes and vote for reject. 

review_title: The paper contains mistakes

final_score: rating: 5: Marginally below acceptance threshold
review_content: Strengths:

1. This paper studies an interesting problem of causal discovery from heterogeneous data, and does a comprehensive survey of related literature.

2. This paper proposes a novel Multiple-Domain Score Search algorithm for score-based causal discovery from different domains.

3. This paper conducts experiments on both synthetic and real-world data with comparison to state-of-the-art baselines, and the results seem promising.

Weaknesses:

1. The difficulty of causal discovery in multiple domains is not clearly shown in the experiments.

2. Some parts of the writing can be improved.

3. There might still be some important aspects of the method left to evaluate.

This paper studies a problem of causal discovery from heterogeneous data, the problem is well-motivated, and the authors give a comprehensive survey of the related work. This paper proposes a novel algorithm Multiple-Domain Score Search (MDSS) to address the problem, which searches over the space of augmented graphs to avoid the spurious edges, characterizes distribution shift with the additional domain index, and uses a novel Multiple-Domain Score (MDS) to identify causal directions. The paper evaluates the proposed method on synthetic and real-world datasets, it achieves better performance than baselines.

Here are some concerns about this paper:

1. In the experiment, it is not very clear how difficult it is for causal discovery in multiple domains (e.g. the number of spurious edges, the difference of data distributions in multiple domains, especially in the real-world dataset), which brings challenges to convincingly evaluate the superiority of the proposed method.

2. As for the writing, some parts of the paper are not very easy to follow for non-experts. It would be better if the terms (e.g., causal sufficiency, causal modules) can be briefly introduced when first mentioned.

3. Also, I think there may be many interesting aspects to evaluate in the experiments, e.g., how robust the algorithm is under different levels of domain shift; how each component in the method contributes to the performance (some ablation studies may be considered).
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary of the paper: The paper introduces a novel score-based approach for causal discovery from heterogeneous data. The novel score is called Multiple-Domain Score, and can be incorporated into existing search strategies. It is shown that the MDS is guaranteed to find the correct graph skeleton asymptotically.

Strengths: The paper is rather well written. The related work is properly addressed. The proposed criteria are interesting. 

Weaknesses: The results are quire incremental. The paper takes a lot from Huang et al., 2019. Assumptions 1 and 2 are somewhat obvious. The paper discusses criteria based on covariance and independence tests, however, their relation to the causal inference is not so straightforward. Algorithm 2 and Section 2.4 need more details.  

Questions: The authors tell that the proposed criterion can identify more causal directions compared to the state-of-the art. I did not really find any confirmation of this statement.

Section 2.3: Why do we need $\theta$? I see that it is also the formulation adopted by Huang et al., 2019, but why the 'augmented' graph is needed? Why $\theta$ are separate nodes if they are (eq. 2 and 3) just parameters of distributions? 

Section 2.3: "the dependence can be described with covariance": I agree but what about causality? 

The criteria eq. 3 and 4 contain several terms. I wonder, what is the performance (and whether it is very different) of the criteria containing the last terms (the results of the covariance and HSIC) only. What is the impact of other terms?

Probably I missed it: what is h (what operation applied to the adjacency matrix)?

How the policy-gradient-based search integrates the MDS? (Section 2.4) 

In experiments: is a CPDAG a causal graph? Is it always so? 
The same for the Bayesian networks (the hill-climbing mentioned in section 2.4). Can we consider that a BN is a causal network? We need at least some assumptions. 

I also doubt that a reinforcement learning algorithm without necessary assumptions can be considered as a causal discovery method.


review_title: The paper is well-written and proposes novel causal discovery criteria. The results are rather incremental.

Meta-review: This submission tackles an important problem and presents interesting ideas. I am confident that the research will lead to good publications. However, in the particular situation here, AnonReviewer2 had serious concerns that are shared by me. The authors made a great effort to clarify the situation, but the current situation still leaves me uncertain about the presentation and correctness of everything. Because some issues were major, it is not easy to re-evaluate and take new conclusions in the short time of this process. I hope the authors do not take this too negatively, but given all the comments and discussions, it is best that another round of improvements and reviews be conducted.
Decision: Reject