Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents identification algorithms for counterfactuals  and conditional counterfactuals. The counterfactuals can be nested. The input consists of observations and experiments were the interventions can be arbitrary. Proofs for soundness and completeness are provided. No software is included.
main_review: [Note: I am an additional reviewer and have not seen the earlier comments or the authors' rebuttal.]

This is a potentially strong paper but I would not like to see it published in the current form because I see that the presentation could be improved in many ways. I am afraid that publishing the paper in the current form would create confusion. On the other hand, I think it is relatively easy to fix the paper and resubmit.

1) The authors do not explain how their proposal differs from the current state-of-the-art.  For years I have believed that the algorithm published by Shpitser & Pearl (2007,2008) completely solves the counterfactual identification problem. The manuscript implicitly indicates that this is not the case but do not present the novel contribution in clear and direct way. It is should not be readers' task to find the subtle differences between the proposed algorithm and the algorithm published by Shpitser & Pearl. I would like to see the differences explicitly stated and demonstrated in an identification problem that is solvable with the proposed algorithm but not with the existing algorithms.

2) The area chair has expressed a concern that a query could be identifiable from marginalization of  some other form than the one considered (e.g. the last line of Algorithm 1). I share this concern because I cannot see an immediate reason why some other marginalization could not be used.

3) The  notation gets really heavy in some places like in line 9 of Algorithm 1.

4) I traced algorithm 2 in Fig. 5(a) and obtained the same result as in equation (23). I ran a simulation to check the result numerically. So I can confirm that the algorithm works correctly in this example.  Proofs are of course needed but these kind demonstrations could serve the average reader who does not have the perseverance to study the proofs in detail.
limitations_and_societal_impact:  I would say that "Arbitrary surrogate experiments" in the title is a too strong statement. Although the interventions can be arbitrary, all variables are always measured in the experiments.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper aims to use the theory of causal graphical models to yield a sound and complete identification algorithm for identification of a joint probability of counterfactual events from a combination of observed and interventional distributions.
main_review: 
The paper is very difficult to follow, as written.  The paper would definitely benefit from worked examples of the operation of the algorithms, and at least an outline of the proof strategy for soundness and completeness.  The proofs are very difficult to follow otherwise.

It appears to generalize results in [10].  However, the paper completely fails to mention ("Complete Identification Methods for the Causal Hierarchy", Shpitser and Pearl, 2008), which gives an algorithm for identification of counterfactual events in terms of all interventional distributions P* -- the same problem the authors consider.

Many of the concepts the authors introduce, e.g. counterfactual factors, eliminating redundancy, and detecting inconsistency of events, and the problem of identifying a joint probabilities of counterfactual events and conditional probabilities of counterfactual events (at the third level of the ladder of causation) in terms of interventions distributions (at the second level) first appeared in the above reference.
Specifically, identification of counterfactual joints and conditionals is given by the ID* and IDC* algorithms, respectively.

In addition, the above reference gives a graphical view of the problem based on counterfactual graphs (a generalization of Balke and Pearl's twin network graphs) that shed light on many concepts the authors use that are typically graphical (e.g. ancestors of counterfactuals -- in what graph?, counterfactual factors -- corresponding to districts in what graph?, ancestral set factorization -- with respect to what graph?) but appear not to be in the authors' paper.

Given that the omitted paper deals with exactly the same problem, the omission is quite disappointing, and leads to serious worries regarding novelty, and the true relationship of the authors' work to prior work.  Are the authors' algorithms effectively the same as ID* or IDC*?  Did these algorithm "miss something" that the authors caught?

Other comments:

Theorem 1: this appears to be definition of notation, rather than a theorem.  In other words, the semantics the authors use give the definition of the joint distribution over counterfactuals using the structural model (via equation (1)), while Theorem 1 is a definition of the "capital letters in the subscript" notation, which is otherwise undefined via (1).

Minor comments:

"Further, there are also result on the identification of path-specific effects, which correspond to counterfactuals that isolate specific paths in the graph. In particular, [1] provides a complete algorithm for identification from observational data."

Unlike other results the authors cite, [1] appears to only apply to fully observed models.  The extensions to hidden variable models appear in ("Identification of Personalized Effects Associated With Causal Pathways" Shpitser and Sherman, 2018).

---

"the composition axiom [13, 7.3.1]"

To most readers, this is probably known as consistency.  It might be worth pointing this out to improve readability.

limitations_and_societal_impact: This is a methodological paper causal inference: no potential negative societal impacts are foreseen by this reviewer.

For limitations, please see the main body of the review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors study the identification of nested counterfactuals from an arbitrary combination of observations and experiments. They prove the counterfactual unnesting theorem, which allows for the mapping of nested to unnested counterfactuals. Then they introduce 1) sufficient and necessary graphical conditions for counterfactual identification from an arbitrary combination of observational and empirical distributions, and 2) an algorithm for identifying nested counterfactuals, the failure of which implies non-identifiability. 
main_review: The paper is well written and the motivation for the presented theoretical work is well justified. Notation, definitions and assumptions are clearly stated. Though the contributions of the proposed theoretical results have important implications for the analysis, understanding and identifiability of structural causal models, the practical implications beyond of the analytical tools in Algorithms 1 and 2 remain to be seen.

There are few typos beyond those presented in Section F of the SM, thus the paper will benefit from careful proofreading.
limitations_and_societal_impact: The limitations and assumptions of the proposed theoretical work are fully described.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This submission presents two contributions to the theory of counterfactual inference from observational and experimental data. First, it reduces nested counterfactuals (which are often required for standard estimates in the potential outcomes literature) to their non-nested counterparts. Second, it introduces a necessary and sufficient graphical condition under which these (conditional and marginal / nested and non-nested) counterfactuals are non-parametrically identifiable from observational and experimental data.
main_review: This is overall a very strong submission, tackling an important problem with what appears to be a technically correct approach. 

In particular, I found the examples throughout the paper to be particularly helpful in understanding (i) how certain estimands could be connected to nested counterfactuals, and (ii) how the various theorems imply useful conclusions towards identification and estimation. This is particularly important in technically dense theoretical work such as this, and I would even encourage the authors to expand on the existing examples to further improve the paper's readability.

I would appreciate if the authors could respond to a few minor questions about the results.

1. Theorem 5 and Theorem 7 are described as "completeness" results, but the statements of the theorems appear to provide soundness and completeness. Could the authors clarify this?

2. It appears that this submission only provides identification results for counterfactual queries that are averaged over units, and not unit-level counterfactual identifiability results. Could the authors clarify how unit-level identification results may or not differ? 
limitations_and_societal_impact: The authors have not addressed limitations and societal impact, but because of the theoretical nature this is not necessary.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this causality paper the evaluation of nested counterfactuals is reduced to that of unnested ones together with an algorithm for identification. This is a dense technical contribution, proofs are clearly detailed in the supplementary material. 
main_review: The contribution is clearly an advance for the state of the art in causality research. I personally missed some attention in better motivating the importance of nested/conditional CFs in practical application, but apart from that I see no reasons for not accepting the paper.
limitations_and_societal_impact: Nothing.
review_title: null

Meta-review: This paper was discussed at length between the AC and SAC, and between the SAC and program chairs.  The AC was strongly against publication due to the lack of appropriate engagement with prior work.  The SAC and program chairs agree with this concern.  The discussion of the relationship with prior work is insufficient and absolutely must be addressed in the final version.  However, in light of the positive sentiment from four out of five reviewers, we believe the paper should be accepted.

(We are counting as "positive sentiment" one of the expert reviewers who gave the paper a low score, based on their comment: "This is a potentially strong paper but I would not like to see it published in the current form because I see that the presentation could be improved in many ways. I am afraid that publishing the paper in the current form would create confusion. On the other hand, I think it is relatively easy to fix the paper and resubmit.")

We expect the authors to address the points raised by the expert reviewers.  We would not be accepting this paper if we didn't think this was something the authors were capable of doing.

The original meta-review provided by the AC follows.

----

A summary from one of the reviews:

"The authors study the identification of nested counterfactuals from an arbitrary combination of observations and experiments. They prove the counterfactual unnesting theorem (CUT), which allows for the mapping of nested to unnested counterfactuals. Then they introduce 1) sufficient and necessary graphical conditions for counterfactual identification from an arbitrary combination of observational and empirical distributions, and 2) an algorithm for identifying nested counterfactuals, the failure of which implies non-identifiability."

The paper was initially received fairly positively. However, some positive reviews were very short and lacking detail, and all were relatively low confidence. As a result, an additional expert reviewer's opinion was solicited, and (at the behest of the senior AE), another expert reviewer.

The paper received an extensive discussion, both with the authors and among reviewers. The result of this discussion was an overall reduction of scores. The concerns were as follows.

First, the authors' contribution builds on prior work on identification of joints and conditionals of counterfactual events. While the authors do cite this work, both expert reviewers independently felt that there was far from a sufficient comparison to it. In particular, while it is clear that the authors consider a generalization (where only some interventional distributions are allowed, in the spirit of (Lee et al, 2019)), it is not at all clear if sufficiently novel methods, in light of existing prior work, are needed to address this generalization. The authors' manuscript, as it is currently written, does not make it clear what is novel developments unique to the paper, and what is a reformulation of known prior work.

Second, the authors claim to consider identification of nested distributions, and discussion of nested distributions forms much of the paper (and indeed appears in the title). However, the authors do not consider a complete algorithm for the problem of identifiability of nested counterfactuals (and indeed did not seem to answer a direct question about this). Thus, reviewers felt the contribution, as written, did not properly emphasize the actual target they considered. In other words, it is true that the algorithm considered nested counterfactuals as valid targets, but only in a weak sense that such a counterfactual is a (marginal) function of a joint set of events (via what the authors call the CUT). But it is surely the case that in some cases marginals are identified while joints are not.

As a result, the reviewers felt the paper needs a restructure and another round of reviews.

Feedback for authors.

Two independent expert reviewers agreed that the appear needs a more extensive discussion of prior work, in order to make it clearer how novel the contribution is. The authors also need to make it clearer what the completeness argument applies to. It appears not to apply to nested distributions (a class of queries appearing in the title). This is quite confusing. Please address these issues when preparing your revision.
Decision: Accept (Poster)