Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This work studies the identification and estimation of treatment effects (with special attention on the CATE) under "weak" overlap. 
It proposes a theory of treatment effect identification when the overlap assumption does not hold for the entire domain of $\mathbf{x}$ using the prognostic score. 
It offers counterfactual generalization bounds on treatment effect estimates.
It develops the Intact-VAE as a regularized treatment effect estimation model that accounts for identification and balancing.
It offers proof that their generative model p(y, x | x, t) is identifiable.
It offers proof that CATE is identifiable via their generative model.
The methods are validated via a comparison to existing methods using synthetic and semi-synthetic datasets.
main_review: ## Originality
- The intact VAE is a novel combination of CVAE and iVAE.

## Quality
- [Line 64] Counterfactual outcome is not a synonym for potential outcome. The description of $\mathbf{y}(t)$ corresponds to a potential outcome. Also, word choice on introduce, sounds like you are saying the definition of potential outcome is something you introduce. Could change to "Following [22], the potential outcome of a treatment is given as ..."
- [equation 1] the expression for $\tau(x)$ is the CATE given the sufficient conditions you describe later on line 77 and consistency. I would suggest moving the conditions before equation 1.
- [line 77] I'm not sure that this is a valid way to express exchangeability. Is $\mathrm{y}(1) \perp \mathrm{t} \mid \mathrm{x}$ and $\mathrm{y}(0) \perp \mathrm{t} \mid \mathrm{x}$ equivalent to $\\{\mathrm{y}(0), \mathrm{y}(1)\\} \perp \mathrm{t} \mid \mathrm{x}$? Similarly for [line 112].
- Section 3.1: the formulation of IntactVAE looks to be a sensible combination of CVAE and iVAE
- Lemma 1: Is correct, as far as I know. It is not immediately obvious to me why D1 holds easily in practice, maybe an illustrative example would help.
- [line 163] $k(x) \to 0$? Lambda is infinite for $k(x) = 0$.


## Clarity
In general I found this submission quite difficult to read. The majority of my time reviewing was spent on trying to parse the work, and I have not had the time to go deeply into the methods or results. Below, I have given some recommendations to make the work more readable. I hope that they are helpful.

- In the introduction you define *weak overlap* on line 32 and give citations to related work on line 34. Each of those works use the term *limited overlap.* [9] do use the term weak overlap, but give a different definition. Why did you choose to use weak instead of limited?
- In my opinion there are too many acronyms. BRL, PGS, PPS, PS, PtS, ANM, and DGP are non-standard acronyms and it is a big ask of the reader to keep in mind 7 of them. PGS, PPS, PS, PtS are all visually quite similar, which makes things challenging for the reader. I would suggest maybe 2-3 max (e.g. Pt-score, P-score, and Intact-VAE), and I would have had an easier time reading the paper if you just used the terms themselves for the rest.
- What do M, G, and D stand for?
- [line 16] "counterfactual outcomes, which would have been" -> counterfactual outcomes: outcomes that would have been
- [line 32-33] "is that" -> is when. "are all belong" -> all belong
- [line 36-37] What is a rich covariate? An explanation of the term or just descriptors of what you mean by rich covariate would be preferable as rich covariate is ambiguous. 
- [line 38-44] It is nice that you give an example of a prognostic score (the propensity score), but an intuitive definition of prognostic scores or  a description of how the propensity score is a prognostic score would be helpful.
- [line 72] The difference between roman fonts and italic fonts is subtle. Another common option is to use capital roman fonts for r.v.s and lower-care roman fonts for realizations.
- You use $\mathbf{e}$ for noise on [line 107] and $\mathbf{\epsilon}$ on [line 134]. Please use one or the other consistently.
- Equation (4) $\theta$ is missing. Suggest an equation $p_{\theta}(\mathrm{y}, \mathrm{z} \mid \mathrm{x}, \mathrm{t}) = p_{f}(\mathrm{y} \mid \mathrm{z}, \mathrm{t}) p_{h, k}(\mathrm{z} \mid \mathrm{x}, \mathrm{t}),$ then define component distributions. Maybe keep in terms of h, k rather than lambda?
- [line 134] density for $\epsilon$?
limitations_and_societal_impact: The authors have clearly defined their problem setting and the assumptions under which they are working. Namely that they are relaxing the overlap assumption, and assume unconfoundedness and consistency. They state the linear independence assumption of Lambda (though it is not clear to me why this holds easily in practive), and the consequences of this assumption failing.  There is a sentence in the conclusion recommending that practitioners judge whether the conditions hold under which their method is valid.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed a novel method for conditional average treatment effect (CATE) estimation under weak overlap with theory of individualized treatment effect (TE) identification and counterfactual generalization bounds on TE errors.
main_review: The paper is well structured and written with decent clarity in theory explanations, experiment settings and results discussion. Code and supplementary materials are available to reproduce the experiment results. 
 The proposed method is novel in terms of treatment effect identification without overlap, which is also supported by thorough reference and comparison to related work.
Technical analysis of synthetic dataset and benchmark dataset is sound with state-of-art baseline models and results with confidence intervals. It might be better to include more empirical experiment results from the supplementary materials in the main paper if feasible. 

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This paper proposes a generative prognostic model for identifying and estimating causal effects under weak overlap. The proposed method uses a latent variable to model a prognostic score (PGS). The authors prove that the latent variable recovers a PGS, and the model identifies individualized treatment effects. The model is learnt as an intact-VAE. The authors provide a theory of treatment effect identification and counterfactual generalization bounds on the treatment effect error. The generalization bounds motivate conditional balanced representation learning (BRL). Then they propose a new regularized VAE to estimate treatment effects. In experiments, they compare the proposed method with some state-of-the-art benchmarks on some synthetic and semi-synthetic datasets.
main_review: Poor presentation with inaccurate statements. I thought some statements in the introduction are misleading for readers who are not familiar with causal inference.  

For example, the description of RCTs in line 18 is incorrect. It is well known that RCTs play a crucial role in the process of drug development. RCTs are indeed expensive but not necessarily prohibitive. Is observational data free of ethical issues?  The statement in line 19, “causal inference from observational data is indispensable” exaggerates the importance of observational data. The advantage of RCT is that it controls certain biases through randomization. Experimental and observational studies have their own strengths and weaknesses. The authors should discuss both in short and provide references to support the claims.

In lines 20-21, you claim the most crucial challenge in observational data is confounding, spurious correlation arises from confounding. This statement is inaccurate. If the confounder is observed, there is no spurious correlation conditioning on the confounder. If the confounder is unobserved, there will be a spurious correlation. This is also the reason why we need the unconfoundedness assumption in observational studies. The explanation of unconfoundedness in lines 22-24, “the confounding can be control by conditioning on covariates” does not clearly capture the meaning of unconfoundedness.

In the second paragraph of the introduction, more works in semiparametric statistics and representation learning should be cited. In the third paragraph, you mention your approach to the weak overlap issue is a prognostic score (PGS) but do not clearly explain what PGS is and why it can address the overlap issue.  Then in the fourth paragraph, you start to discuss the problem of recovering PGSs without telling the reader what a PGS is. 

There are existing works on each of the main contributions of the paper (Identification theory of treatment effects, (i)VAE, counterfactual generalization bounds and balanced representation learning). The introduction should make it clear what is the difference between your work and the existing works. Perhaps the related work section should come earlier for this purpose. 

Section 2. Please use upper-case letters for random variables and lower-case letters for fixed values. It is difficult for the readers to distinguish Roman and italic fonts because using Roman and italic fonts together is not a standard practice.

In lines 77 and 87, p(t|x) and p(t|v) should be also be less than 1.

In line 98, you mentioned overlapped b(x) implies overlapped x, overlapped x implies overlapped PGS. But to show PtS is more applicable, you also need to prove the other way around, PGS-> x -> b(x), does not hold. I don’t see why PtS is lower-dimensional and more overlapped? What is the definition of more overlapped? PtS is defined in lines 94-95. I can't see why it maximizes overlap.

In line 102, conditionally balanced representation is better for what? I am confused here, 
Conditioning on x, the only randomness in P_t(x) is t. It is impossible that P_t(x) \indep t | x.
Do you mean x \indep t | P_t(x)? I see this is what you have in Definition 2. A balanced representation may also lose important information of the outcomes. It is not better surely.

In Proposition 1, mu_t (x) is the outcome function, not the CATE function.

Section 3. The comparison between Intact-VAE with CVAE and iVAE is clear. However, the novelty of Intact-VAE is limited.  Are the conditions in Lemma 1 and Theorem 1 all verifiable in practice? Is identifying f_t up to some affine transformation or some invertible mapping sufficient for consistent CATE estimation?

Section 4. The generalization bounds in this section are not novel. Similar results are already obtained in the existing works [A, B, C]. More importantly, does minimize the  VAE objective function lead to minimizing all the empirical terms in the generalization bound so that the CATE estimation is consistent (i.e. the expected risk can be as small as the variance of the random noise in the ANM) as the sample size goes to infinity? BTW, why the error from random noises does not exist in the generalization bound? Rigorously, the generalization bound in Theorem 2 is not exactly a generalization bound in learning theory, because the terms in the upper bound are expected risks rather than empirical risks.

[A] U Shalit, FD Johansson, D Sontag. Estimating individual treatment effect: generalization bounds and algorithms, ICML (2017).
[B] Alaa, Ahmed, and Mihaela Schaar. Limits of estimating heterogeneous treatment effects: Guidelines for practical algorithm design, ICML (2018).
[C] Yao Zhang, Alexis Bellot, Mihaela van der Schaar. Learning Overlapping Representations for the Estimation of Individualized Treatment Effects, AISTATS (2020).

Section 6.  The results do not convince me that the proposed method is useful in practice. The method gives worse PEHE or \sqrt{PEPE} than CFR, as shown in Figure 2 and Table 1. There is no experiment provided to back up the identifiability theory. For example, could you empirically show the function f_t in Lemma 1 is identified up to some affine transformation?

Overall, I find the paper has a large room for improvement in terms of presentation, theory and experiments. 





limitations_and_societal_impact: Yes, they do.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies identification and estimation of treatment effects (TEs) under weak overlap of covariates (Definition 1, probability of treatment conditioned on covariate has different supports for some covariate, which is closer to the real world). For identification of TEs, the paper exploits the prognostic score (PGS) and the identiﬁable representation, and as a result, proposes a novel, VAE-based generative prognostic model. Prior studies have seen good practical performances of PGS in many settings and hence, the paper expects it to be a good choice and to outperform other candidates in joint use with additional techniques. The identifiable representation is essential to guarantee the identification of TEs. The estimation of the VAE model is via a $\\beta$-regularized-ELBO which helps to keep the posterior distribution of latent variables of VAE ''balanced'' (defined in line 219). Motivations for e.g. balanced representation are provided later in the theoretical analyses on the counterfactual risk bound and the generalization bound. The proposed method is compared with baselines on (semi-)synthetic datasets.
main_review: 1. Originality

The paper combines several recent techniques for TEs: generative learning (VAE), identifiable representation, PGS, and balanced representation learning. The combination was not explored before. The theoretical analyses are new under the setting of joint use of multiple methods.

2. Quality

The VAE based method with theoretical guarantees and experimental validation is technically interesting. I haven't realized any technical issue in the main paper. However, it is slightly cumbersome to stack so many techniques.

3. Clarity

The paper is dense and needs to provide e.g. examples or remarks to help readers understand concepts and assumptions. For example, the concept of ''balanced'' distribution non-rigorously introduced on line 219 is not easy to understand; the importance of the assumption G1' in the theorems needs explaining (if just to make dimensionality explicit, can G1 and G1' be merged as just one assumption in the paper?).

4. Significance

The identifiable and balanced representation learning approach for TEs will be interesting for many readers in the field. The practical outperformance and the theoretical guarantee are the essential reasons.
limitations_and_societal_impact: The paper has proposed sound suggestions to solve the limitations and potential negative societal impact in the conclusion section.
review_title: null

Meta-review: The authors study estimation of causal effects under weak overlap---when certain features have values that occur only for one treatment group. They propose a latent variable model of a prognostic score, a variant of the variational auto encoder, showing that it can identify conditional average treatment effects under certain conditions. A basic generalisation bound is provided and the approach is evaluated on a standard benchmark dataset. 

The reviewers found the proposed use of latent variable models for this purpose novel and appreciated the identification results. However, they found that proposed generalisation bound should be contextualised better in previous work and possibly renamed. This was partially addressed by the author response, but would require rewriting certain parts of the paper. Further, the empirical results could be improved by choosing more appropriate benchmarks for the proposed methods (which highlight the strengths and weaknesses of the idea) and by including further simulations which illustrate the proven identification results empirically. While the latter point was partially addressed by the authors in Figure 3, as pointed out in their response, this is a fairly anecdotal result. The synthetic results are also somewhat inconclusive with regards to the parameter \beta. In addition to this, several of the reviewers found that the text itself needs a large revision to reach the standards required for acceptance, beyond what can be expected for a camera-ready update. 
Decision: Reject