Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the identification and estimation problem of the joint probabilities of potential outcomes. To solve this problem, this paper proposes two sets of identification conditions using covariate information. Furthermore, this paper proposes a novel statistical estimation method based on the augmented Lagrangian method to evaluate the joint probabilities of potential outcomes under the proposed identification conditions. 
main_review: This paper studies the problem of evaluating the joint probabilities, which is important in causal inference for two reasons. First, if the joint probabilities are identifiable, then the causal risk is also identifiable. Second, the joint probabilities are critical to evaluate the "Probability of Sufficiency", “Probability of Necessity”, and “Probability of Necessity and Sufficiency”.

The two sets of identification conditions for the joint probabilities seem to be novel. However, I remain concerned about the practicality of these conditions. It would be nice to give a few examples to illustrate when Conditions 1-3, and Conditions 4-6 could hold in practice. Moreover, it will be helpful to clarify the conceptual challenges in proposing these conditions.

The estimation approach based on the augmented Lagrangian method is a nice solution to estimate the joint probabilities, given that the standard statistical estimation methods do not work. I appreciate that the authors provide theoretical guarantees for their estimation approach. It would be nice if the authors could provide a formal and rigorous statement of their theoretical results (e.g. a theorem or proposition) with the clearly stated assumptions (as opposed to just a sentence in line 286 and a section (i.e. B.3) in the appendix to show the proof).

Regarding numerical simulations, I appreciate that the desired properties are verified through simulations, but I remain concerned about the simulation setup. It would be nice to clarify why the numbers chosen in Table 1 are of practical relevance. In addition, it would be nice if the authors could demonstrate the performance of their estimation methods on real data sets. 

limitations_and_societal_impact: Identification with more than two covariates: The authors provide two sets of identification conditions. One is based on one covariate with causal risk, and the other one is based on two covariates without causal risk. What happens if there are more than two covariates, and what happens if there is more than one covariate with causal risk? It is worth explaining how the identification conditions can be generalized to these more general cases if these conditions are generalizable. Otherwise, it is also worth clarifying why these conditions are not generalizable. 

Numerical simulation, practical relevance, and presentation of this paper: The authors mention some healthcare examples in the introduction, which are helpful to motivate the problem. If the authors could explain and interpret Theorem 1, Conditions 1-3, Theorem 2, and Conditions 4-6 using the examples in the introduction, the identification section will be much easier to read. In addition, this paper could have a broader impact if the authors could demonstrate the practical relevance of their methods through experiments on real data sets (e.g. the real data related to the heathcare examples mentioned in the introduction).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, authors propose a mechanism to calculate joint probabilities of potential outcome under the assumption that certain covariate information is available. The first part of the paper identifies conditions under which these probabilities can be estimated and the second part aims to use augmented Lagrangian method to calculate the probability.


main_review: The problem of relaxing assumptions of prior techniques would be an impactful contribution for the literature on explainable AI.

Weak points:
1. Motivation: It is unclear how the presented techniques are applicable in practice. For example, the covariate assumption is not available in most real-world settings.

2. Theory: It seems like the conditional independence assumption X\indep Z | U makes the analysis straightforward. I am not sure how the analysis would extend if this condition does not hold. 

3. The intuition behind the proposed method in section 4 is unclear. Please add more details and discuss the proof techniques.

4. The proof is discussed with respect to two cases. How does it generalize it to other graph structures?

5.  Evaluation: The empirical evaluation of the presented techniques is very superficial. 
a) The experiment is conducted on a very small synthetic causal graph. Please consider a real-world dataset that motivates the motivation and consider more than 4 variables.
b) There is no comparison with prior techniques that make monotonicity assumptions. It is unclear if the proposed method really performs better.
c) In Figure 3, it maybe better to compare PNS scores of different attributes and validate its correctness with respect to what prior techniques report.

6. Paper is not well written. The notation is not defined clearly which makes it very hard to validate the correctness of some results.

7. Also, certain claims about medical results, e.g. lines 187-192 are not well justified. Please update the introduction to discuss these use cases and define formal notation.

limitations_and_societal_impact: None identified
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper solves the identification of the probability of necessity and sufficiency in the case of unobserved confounder and the existence of proxy variable(s). Authors provide a novel formulation to solve the problem (presenting an unusual system of linear equations. 

main_review: 
quality
The paper is in good shape (see clarity)

originality
The paper provides an answer (under a certain condition) for an important causal question in an original way (although solving system of linear equations have bee investigated in different contexts)

clarity
The paper is in general pleasant to read but there are several places where improvements can be made.

For example, conditions for Theorem 1 are too shallowly explained. How should we interpret P or Q? Why do we have a marginal probability at the top row and leftmost column? Why there is no probability with x0 and y0 in P? etc. It seems that one can replace the top and the leftmost with those unused probabilities. But I am not sure. There is too much gap unexplained. The same applies to Theorem 2. Px1 and Px2 (which can be represented in a single expression with just x\in x1, x2). Is invertibility implies that proxies (i.e. covariates) Z and W contain sufficient information to relate U (similar to (Yx, Yx’)) and X? 

In the estimation section, it is difficult to read through lines 274—282, which refer to the results in supplementary materials extensively. 

What is a singular model?

Empirical results section can be reduced (with Table 2 in Appendix). What can we do with outliers when Condition 6 doesn’t hold (or the matrix is close to singular)? It would be nice to plot invertibility test results together with the estimates (PNS and causal risk difference) to ensure that those outliers are mainly due to near-singular matrices.


significance
In the context of the theory of causal effect identifiability, this paper provides a novel identification result where non-trivially constructed matrices and their inverse is used. 


limitation
Although some of the reviewers may point out the stringent assumptions this paper relies on. I do not consider them as the limitation but the ones that ‘enable’ the identification.


minor
As far as I know, the paper needs to use the basic font provided by NeurIPS.
Line 227, ito → it
limitations_and_societal_impact: .
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes two nonstandard identification strategies for the joint distribution of counterfactuals. One is based on a single proxy $Z$ for the unknown confounder with at least $4$ levels, such that $p(y|do(z))$ is identifiable. The other one is based two proxies $Z$ and $W$ with at least $4$ levels for each. Both identification strategies are fully nonparametric. They are given by solutions of a complex system which are nonlinear in the observed distributions. Beyond identification, the paper also proposes estimation procedures and proves the asymptotic normality of the estimators. The estimation procedures are essentially the plug-in version of the system involved in the identification strategy. 
main_review: The identification strategies are mind-blowing. Most (partial) identification strategies are based on linear programming. But the proposed ones are certainly beyond that. I really enjoy the smart ideas. Furthermore, the assumptions for those two strategies appear to be reasonable in practice. Therefore, I think this work is both intellectually and practically interesting. 
limitations_and_societal_impact: All my comments are minor. 

(1) For case 1, don't you need $(X, Y)\perp Z | U$ instead of just $X\perp Z | U$?

(2) It would be more clear if $S$ and $M_x$ can be defined in Section 4. Otherwise, Section 4 is not readable enough. If the page limit is a concern, perhaps Section 3.1 can be trimmed a bit because the main text only discusses the estimation for Case 2. 

(3) Suppose $Z$ (and $W$) have more than four levels, is it possible to combine all levels instead of just choosing three of them? A naive idea is to ensemble (9) with all subsets of three levels. But can we expand $P_x, Q_x$ by including all levels? 

(4) Could you provide some heuristic explanation on how you get these strategies? More broadly, is it possible to get some generic principle from these ideas (like linear programming for other problems).
review_title: null

Meta-review: All reviewers have a favorable opinion about this paper after discussion.

Paraphrasing one of the reviewer comments, identification results for joint probabilities of potential outcomes are "very rare" and definitely crucial for many important quantities like probability of sufficiency, necessity etc., individual treatment effect. This paper provides a novel conditions and an estimation procedure for solving it. 

So clearly the contributions are fundamental, original and timely.


There is a reviewer comment asking for formal statement for the asymptotic normality, some empirical comparison and further examples for the theorems. I really hope authors can revise accordingly.
Decision: Accept (Poster)