Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of conditional causal effect identification, which asks to recover P(y|do(x), z) from the observational distribution. There is a sequence of works that propose an algorithm for this problem under the assumption that the underlining causal graph is known. This paper relaxes this assumption by assuming that only the Markov equivalence class of the underlining causal graph is known. More specifically, the paper assumes that the so-called partial ancestral graph (PAG) is known. The paper proposes new calculus, which is a modified version of Zhang's [2007] Calculus. For the new calculus, the authors prove that it is complete and atomic complete. Moreover, they propose an algorithm for the identification of conditional causal effects given a PAG and show that the algorithms is complete, that is if some causal cannot be identified by this algorithm then it is not identifiable in some causal graph in the equivalence class of a given PAG.
strengths_and_weaknesses: *Significance:* The problem of identifiability of interventional causal effects is an important and long-studied problem. This paper improves over the state-of-the-art result by relaxing the assumption that the ground truth causal graph is known. Instead, the paper assumes that only the partial ancestral graph is known. This is a significant relaxation of the assumption as the true underlining causal graph is frequently not known, while PAG can be recovered from conditional independencies of variables. Zhang [2007] proposed a calculus to solve the problem considered in this paper which relies on the notion of ``possibly m-connected paths". However, Zhang's calculus is not complete, i.e., it is possible to contract a PAG in which the effect is identifiable but is not deducible from Zhang's calculus. 

The key observation of this paper is that if one relies on the notion "definite m-connected paths" instead of "possibly m-connected paths", then the minor modification of Zhang's calculus becomes a complete calculus. Moreover, the paper proposes an algorithm that uses the rules of the newly introduced calculus to recover causal effect and shows that this algorithm is complete.

*Clarity:* Despite the fact the paper is reasonably well-written it is extremely technical, and as a result quite hard to read. I was not able to verify the correctness of the proofs in the appendix within a reasonable time.  

*Originality:* The paper relies mostly on ideas that are well-known in the field. The proposed algorithm appears to be a minor modification of the prior algorithm by Jaber et al. 2019a. However, the proposed calculus and its completeness seem to be novel.
questions: 1) What is the main difference between the algorithm proposed in by Jaber et al. 2019a and your algorithm? If the difference is insignificant, I believe the algorithm of Jaber et al. 2019a deserves more credit in the discussion, in particular, it should be acknowledged that your algorithm is a minor modification. Alternatively, if the difference is significant, I would kindly suggest including a more detailed discussion of the distinction between these algorithms. 

2)  Are there any runtime or space complexity guarantees for your algorithm? Is it polynomial time?

3) Is anything known about the approximate version of the problem: how many samples are sufficient to identify the causal effect?

4) Are there any open problems left related to conditional effect identification (that authors consider interesting)? 

4) I would kindly suggest renaming Proposition 1 (by Zhang 2007) into a Theorem, as calling it a proposition appears to be a bit judgemental. 

5) typo in line 213: X, Y, Y

6) I believe the notation in line 265 is poorly chosen, y serves the role of both a fixed variable and a summation index (which makes no mathematical sense). I kindly suggest renaming the summation index. The same notation problem occurs in other places.  
limitations: I believe the authors should comment on the runtime and sample complexity guarantees of the proposed algorithm. If the problem of establishing such guarantees is not resolved I kindly suggest that the authors acknowledge this. Alternatively, if such guarantees (lower or upper bounds) follow from the prior work it will be useful if they are included in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work studies the question of identification of conditional causal effects given a PAG. Authors propose sound and complete algorithm for the identification of the causal effects of the form P(y|do(x), z). 

strengths_and_weaknesses: It studies an important and quite interesting problem in causality which is causal effect identification in PAGs. 
 
Authors propose new set of do-calculus rules for PAGs and prove their soundness and completeness in terms of identifiability of conditional causal effects. Also they propose sound and complete algorithm for the identification of causal effects P(y|do(x), z) given PAG. 


There are statements that require more references. For example, the IDP algorithm is similar (except slight changes in one line)  to the algorithm in “Causal Identification under Markov Equivalence: Completeness Results” by A. Jaber. This needs to be clarified in the main text. 

Results resemble the ones presented by Shpitser for identification of the conditional causal effects in DAGs. Moreover, the problem of identification of causal effects in PAGs has been addressed before. Therefore, the natural question is the contribution of this work. It is very important to clearly mention what is the main difference between the proposed method and the other work and what is the significant of this work compared to the others, e.g., proofs are different and more involved, assumptions have been relaxed, ...



questions: It is important to provide high-level intuition about the difference between the algorithm for identifying conditional causal effects proposed in “Identification of Conditional Causal Effects under Markov Equivalence” by A. Jaber and the algorithm proposed in this work.


It will improve the readability to  add more details of what has been already done (like Algorithm 1) and what is newly proposed in the paper.

Papers on identifiability commonly have an assumption that the underlying DAG is semi-Markovian. 
I would like to ask whether it is also needed here and whether this has any effect (limitation) on the set of PAGs which are considered in this work? Will the proofs still hold if the underlying DAG of the corresponding PAG was not semi-Markovian?

limitations: Please see above comments.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This is a paper about the identification of causal queries in structural causal models. If most of the existing works assume the availability of the causal graph, here only the PAG (partial ancestral graph) structure (corresponding to a collection of CGs) is assumed to be available. The authors derive an analogous of do-calculus for such setup. This is an extension of Zhang's calculus, but here completeness guarantees are provided. Algorithms for the identification of causal effects are provided.
strengths_and_weaknesses: Strength

- Although representing an evolution of existing results (Zhang's results and calculus for DAGs and Jaber's algorithms and completeness result) the results in this paper are clearly novel and authors' final claim about their paper "clos[ing] the problem of effect identification under Markov equivalence" is hard to question. This makes the contribution highly significant.

- The paper is very technical, but the results and the proofs seem to be correct (I only partially checked to long proofs in the supplementary material). The the derivations and the notation is quite accessible to people working with SCMs. I also appreciate the insights provided by the different examples reported in the paper.

Weaknesses

- While the statements of the theorems and the examples are generally very clear, the presentation of the algorithms can be probably improved.



questions: - It is not perfectly clear to me whether or not these results can be also applied to counterfactual queries (I think so but this point might be made explicit).
- The considered approach allows to address identifiability when the causal graph is not known and only the PAG is available. As I understand, PAG identifiability corresponds to identifiability on all the CGs compatible with the PAG and the fact that the query would give the same result on each CG. What I don't understand is whether non-identifiability in PAGs, means that there is at least a non-identifiable CG compatible with the PAG or it might be the case that all the CGs are identifiable but leading to different values. A clarification on these points would be helpful, at least from my point of view. 
limitations: I don't see critical issues in terms of societal impact.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The manuscript provides a sound and complete calculus and algorithm for identifying conditional causal effects of the form $p(\bf{y} | do(\bf{x}), \bf{z})$ in partial ancestral graphs (PAGs), i.e. a Markov equivalence class of maximal ancestral graphs (MAGs). This generalizes prior work on causal effect identification in different kinds of graphs and equivalence classes thereof.
strengths_and_weaknesses: Completeness results for conditional effect identification in PAGs are notably lacking, despite strong progress in this area by Zhang (2007) and Jaber et al. (2019a, 2019b). Providing such a result – not just for an abstract set of rules but in an efficient algorithm – constitutes a major contribution. Since the FCI algorithm outputs a PAG, CIPD could plausibly be incorporated into a causal discovery pipeline for effect estimation over a learned Markov equivalence class. This could be of great use to practitioners in a variety of fields. The manuscript is well-researched and well-written, quite clear and easy to follow despite considerable technicalities. I commend the authors for their strong and original work.

If anything, I wonder if a conference submission is really the right venue for this, given how many follow up questions I have! I fully appreciate that space constraints prevent the authors from tackling all the problems that arise in causal reasoning with PAGs, but I strongly encourage them to consider a journal length follow up article, along the lines of Zhang (2007) or Shpitser & Pearl (2008). Short of that, some revisions to the present manuscript at least addressing these concerns would be welcome: 

-The do-calculus is famously complete for effect identification across all three levels of the causal hierarchy (Shpitser & Pearl, 2008). How, if at all, could the methods proposed here be leveraged for counterfactual inference in PAGs? While a complete answer may lie beyond the scope of this manuscript, some discussion of the problem would be interesting, if only as a direction for future work. 

-There is some discussion in the manuscript of an R implementation. Mentioning this without providing any empirical results or supplemental code is a bit frustrating, as it’s unclear how reviewers are meant to evaluate the performance or usability of the CIDP algorithm. I would either include this material as part of the submission or cut the reference altogether. 

-There has been considerable work in recent years on CATE estimation. Could methods like inverse propensity weights or double machine learning be used to estimate conditional causal effects in PAGs, at least under certain conditions? Again, perhaps beyond scope to go too deep into this, but a natural follow up that will likely occur to many readers.

questions: See above.
limitations: I am not convinced that the brief reference to an independence oracle constitutes a thorough discussion of the method’s limitations. Some elaboration would be welcome here, as well as further comments on if/how the method can be used for counterfactual effect identification, computational efficiency, etc.
review_title: null

Meta-review: metareview: The reviewers are all in agreeement that the paper constitutes a fundamental advance in the theory of causal inference. The authors responded to the reviewers' remaining questions in a detailed way, and there is no further issue with the paper being accepted. 
confidence: Certain
Decision: Accept