Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: Bounding unknown counterfactual probabilities using observational data and assumptions in a causal graph is the core problem addressed here. The main result is formally presented in theorem 1 which states that all counterfactual distributions in an arbitrary causal diagram could be generated using discrete exogenous variables taking values from a finite domain.

main_review: Too many results crammed in together makes the paper extremely hard to understand and follow. Too few examples and complicated notations make it harder. Broadly speaking, there are solutions to two problems that are jointly presented here, the first is identification and the second is estimation.  Also, in this work, proofs are the main contribution. Proofs are in the appendix and the appendix is filled with errors and typos. 

An example for equation 2 where V is a small set V will be helpful. I found one in the appendix but there are issues with it that I detail below. 

There is an ‘f’ at the end of line 112 on page 3.

(3) in definition 3: Please clarify what you mean. Is it the case that an U variable can only occur once in the path?

“SCM lives can be mapped… well-behaved space”: Try to avoid using such vague and ill-defined terms. 

“Since G has a single c-component...family of discrete SCMs N”(page 4). It is not immediately clear why U1 and U2 must share the same cardinality. 

Appendix A

Why is there a difference in the definitions of SCM in section 1.1 and definition 5?  h_{u_v} in definition 5 appears to be essentially the same as f_v in section 1.1.  Also in the first line of definition 5, you have M=<V,U,F,P>. ‘F’ is not mentioned anywhere in the rest of the definition. In line 557, the authors revert to the use of ‘f_v’. 

“Since Y is not a descendant of X” (line 531): Are you still referring to Fig. 1d? If so, Y is a child of X and hence a descendant.

In 533, the notation pertaining to ‘h’ changed. I am assuming the ‘h’ referred here is the same as in line 3 of definition 5. What does the superscript denote? Also shouldn’t the subscript denote the exogenous variable that is a parent of X? The notation of h in line 540 makes better sense. This example described from lines 535-545 is crucial. It ought to be corrected and moved to the main paper. 


There are similar issues throughout the paper. For the results to be useful to the community this paper needs to be split into two, carefully revised (rewritten with more examples and by including more content from the appendix) and reviewed again. 

limitations_and_societal_impact: None
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The main result of this paper is to establish that any SCM with discrete endogenous variables can be transformed into a counterfactually equivalent SCM where also the exogenous variables are discrete (finite domains). This result is used to find exact bounds on counterfactual probabilities based on the observational distribution, or, using an MCMC procedure, approximate bounds based on observational data.
main_review: The main theorem is an impressive result that was apparently very challenging to prove. Knowing that infinite domains need not be considered for exogenous variables is of great theoretical interest, and could open the way to computational approaches for problems that were previously not accessible in such a way.  Two such new approaches are given in the paper, and the empirical results confirm their usefulness. But to me, Theorem 1 is the main contribution of this paper. I did not check the proof, though.

Minor comments:

Line 112: runaway "f"
Line 113-114: If I'm not mistaken, this condition follows from the general definitions of SCMs + the preceding condition + the assumption that endogenous variables have finite domains (which was stated earlier). It does apparently serve the role of introducing the $\xi$-notation, and maybe to put the finiteness assumption in a clearer place. But as it is, it looks like a fairly complicated condition when the actual message is much simpler.
Line 144: Why mention $2d=64$ here? Wouldn't $d^2$ be more relevant computationally?

UPDATE: I thank the authors for engaging in the discussion. Based on the other reviews and the discussion, I have revised my score to a 6: while there is a novel contribution in this paper, the result is in fact closer to existing results than I thought at first.
limitations_and_societal_impact: The exact method is computationally prohibitive, possibly to the point of not being practical, but this is not quantified (at least not in the main paper)
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a methodology to obtain bounds on counterfactual probabilities from unknown SCMs using observational data. The authors consider a setting where the causal structure (DAG) of the data-generating model is known and all the endogenous variables are discrete and finite.
 

In this setting, the authors show that the set of all SCMs compatible with the known DAG, G, and observational data can be mapped to a special class of SCMs, which they call discrete SCMs. These discrete SCMs have the nice property that they are compatible with DAG G and the observational distribution and all exogenous variables are discrete and finite (with known maximum value). The authors show that there must exist a discrete SCM within this family which agrees with the original (unknown) SCM on all counterfactual probabilities. Using this as the main idea, the authors attempt to bound the counterfactual probabilities by maximising/minimising the counterfactual probabilities over all discrete SCMs in this family.
 

The paper makes no assumptions about the missing exogenous variables in the SCM model under question. This makes for a very powerful idea which can be used to bound counterfactual queries.
main_review: 
The paper tackles the question of gaining insights into the counterfactual world using the factual observed data with missing confounders. This is a fundamentally challenging question to answer without making additional assumptions. The authors propose a methodology which does not make any assumptions about the missing (exogenous) variables, which makes this methodology quite appealing.
 

Setting:
However, the setting under consideration is very specific: the authors are only considering SCMs with discrete and finite endogenous variables. This rarely occurs in most real world problems. Additionally, the authors work on the assumption that the DAG of the data-generating model is known (including causal links between unobserved variables and observed variables). This may be a reasonable assumption in some settings, but depending on the problem at hand, we may not know the number of missing (exogenous) variables and how they causally affect endogenous variables in great granular detail. In such cases, we may have to resort to a very general DAG, which could potentially come at the cost of the tightness of the bounds on counterfactual probabilities.
 

Notation:
The work is generally high quality, however, in my opinion the notation gets very dense at times and could be further simplified. While the authors generally did a good job of defining the notation, it was sometimes difficult to get an intuitive feel of what a concept meant for a model under consideration. For example, it would have been nice to have a visual representation of c-components within a DAG, as it would allow readers to immediately see why they form partitions over both endogenous and exogenous variables. Moreover, the notation U_V is unclear as this is the first time the authors use endogenous variables as subscripts for exogenous variables. I believe either the definition of this notation is missing, or this is a typo.
 

Method:
With regards to the methodology itself, I think the general idea is very nice, and the authors have provided a reasonable theoretical backing to show that it works. However, one of the questions that has not been addressed theoretically in sufficient detail is: how good is the Bayesian approach in finding the family of discrete SCMs that agree with the observational distribution? While the authors show that the problem of finding the bounds can be reduced to a polynomial program in Appendix E, they do not use this approach to find the counterfactual bounds. I think it is very important to include a comparison of credible intervals obtained using Bayesian approach with the bounds obtained by solving these polynomial programs. This would be useful to gauge the reliability of credible intervals against the ground truth.
 

Experiments:
The experimental results generally show that the credible intervals obtained contain the ground truth, which is promising, and in the case where the counterfactual probability is identifiable, the credible interval is very tight. However, one thing which is concerning is that the observational data size used in first three experiments is very high (N=10^5) even though the endogenous variables are binary and models are very small (with at most 3 endogenous variables). I would like to see the effect of having smaller datasets on credible intervals. Additionally, the credible intervals in experiment 2 and 3 are very wide (and largely uninformative). In experiment 4, the authors have used age as a confounder, however, having looked at the IST details on https://pubmed.ncbi.nlm.nih.gov/9174558/, it seems that the trial is randomized ("Half the patients were allocated unfractionated heparin (5000 or 12,500 IU bd [twice daily]), and half were allocated "avoid heparin"…"). This suggests that age is not a confounder. Please clarify this if I am wrong and have overlooked something.
 

Overall, the idea is original and nice, but I believe more needs to be done to justify the use of this methodology in partial identification of counterfactual outcomes.
 

I have given it a rating of 6 because of some of the above questions. I will be happy to reconsider my rating if these questions are addressed. 
limitations_and_societal_impact: The authors have stated the assumptions clearly, however I believe more can be done to justify the Bayesian approach for obtaining Credible Intervals (see main review for more details).
 

I do not see any potential negative impacts of their work on the society.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: EDIT:  I thank the authors for their clarifications and corrections to my previous line of thought, and I am prepared to revise my score up to 5 as a result.  I take the point about polynomial vs linear programming, and I can see that Lemma 4 is a non-trivial result.  The authors, however, make no effort to point to Lemma 4 as being the primary source of the novelty in Theorem 1!

Overall, the content of the paper seems too rich for a conference with a page limit like this one, it might be better off submitted to a journal with much more explanation and more examples.

----

This paper gives a Monte Carlo method for obtaining bounds from any DAG model with hidden variables.   The authors illustrate the algorithm on the front door model, as well as the instrumental variables model; they give several more examples in the supplementary material.  

I think the paper could be improved by explaining more explicitly how the Bayesian algorithm for obtaining samples helps, when presumably one could solve the linear program (6).
main_review: ### Main Points

1. I do not think that Theorem 1 is quite as much of a contribution as the authors claim.  First, note that Rosset et al. do not explicitly exclude edges within the graph, and I don't think their result is restricted in the way that the authors here imply.  Secondly there is an unpublished manuscript of Sainz and Wolfe that shows it is sufficient to consider graphs where the random vertices (i.e. those inside a given C-component) do not have children (it is referenced in Navascués and Wolfe, 2020).  

    The addition of counterfactually equivalent distributions seems like only a small technical tweak.  I also note that the proof is now essentially identical to that of Rosset.

2.  The MCMC algorithm seems quite good, though it is never tested on a really high dimensional example.  In fact the only example given seems to be the instrumental variables model.  In addition, does the tighter credible interval you obtain simply reflect a more concentrated prior distribution?

3.  I note that there is no comparison with Richardson et al (2011), who do something very similar. 

4.  I am curious about the authors' statement that the Balke-Pearl bounds do not apply to Figure 1(b).  Can you elaborate? 

### Minor Points

 - l14: 'finite number of observational data' $\rightarrow$ 'finite quantity of observational data'?  Or 'observational data points'?

 - l41: 'as the principal stratification'- should this just be 'as principal stratification'?  (I am genuinely not sure.)

 - l345: '100% credible interval' - should this be '95%'?

 - l354: 'counterfactual equivalent' $\rightarrow$ 'counterfactually equivalent'

### References

Navascués and Wolfe. The inflation technique completely solves the causal compatibility problem. _Journal of Causal Inference_ 8(1)70-91, 2020.

Richardson, Evans and Robins.  Transparent parameterizations of models for potential outcomes. _Bayesian Statistics 9_, 2011.
limitations_and_societal_impact: I'm not sure what to write here!  The paper seems fine other than the remarks I have made above.
review_title: null

Meta-review: A summary from one of the reviews:

"This paper gives a Monte Carlo method for obtaining bounds from any DAG model with hidden variables. The authors illustrate the algorithm on the front door model, as well as the instrumental variables model; they give several more examples in the supplementary material."

The paper received extensive discussion, both with the authors, and among the reviewers.  This discussed alleviated some of the concerns reviewers had, which resulted in overall improved scores.  Nevertheless, concerns about an appropriate way of explicating the novelty of the contribution given constraints of conference proceedings, remained.

Theorem 1, prior work and overall length.  The author's manuscript, as currently written, made it very challenging to see the distinction in the proof strategy between their contribution (via Lemma 4 in the supplement), and related prior work.  While discussions with the authors clarified this to some extent, making this connection clearer would entail nontrivial changes.

In general, while many NeurIPS submissions have extensive appendices, at the conclusion of the discussion reviewers felt this paper in particular relied on results and narrative in the supplement in a way that made reading challenging without expanding the main body of the paper.  Thus, it was felt that the paper in its current was better suited for a journal.

Decision: Reject