Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on estimating treatment effects under a limited overlap, i.e., subjects with specific characteristics might only belong to one treatment group. To solve this problem, the authors propose a variational autoencoder (VAE) called $\beta$-Intact-VAE, which extends the framework of CVAE and iVAE.  The authors prove that the identifies the treatment effects. The proposed model is compared with other methods on synthetic datasets.

main_review: Strengths:

I believe the $\beta$-Intact-VAE is novel and is useful in treatment effects estimation. 

The authors have provided theoretical guarantees for the proposed method. 

Experimental results also demonstrate that $\beta$-Intact-VAE outperforms other generative models. 


Weaknesses:

I am able to understand the general idea of the method but have difficulty understanding some notations in the paper. The authors might want to define some of the symbols before using them; for example,  $f_t$, $k_t$, $h_t$, $diag$ and $diag^{-1}$ in Section 3.1 and $f_t$, $k_t$, $h_t$ in Section 4.1. The notation $f_t$ indications that there are two different functions for $t=0$ and $t=1$, respectively, is it true? 

I do not understand the differences between $p_\theta(y|x, t)$ and $p(y|x, t)$ in Section 3.2. 

In the caption of Table 1, What is the unconditional balancing hyperparameter?

In Section 4.2, are the parameters $\Lambda$ learn from the data? In Equation (6), $ q_{\phi} (x, y, t) $ is still a function of $t$. How is balanced PtS achieved in this case? In the second term of Equatnion (7), it looks like $g_t^2(z)$ should be $g_t(z)$.
summary_of_the_review: In general, I believe the $\beta$-Intact-VAE is novel and is useful in treatment effects estimation. The authors have provided theoretical guarantees for the proposed method. Experimental results also demonstrate that $\beta$-Intact-VAE outperforms other generative models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The present paper proposes to address the issue of limited overlap in treatment effect estimation by investigating the identification assumptions of prognostic scores which, in certain contexts, are less restrictive on overlap than other methods such as propensity score based methods.
The main contributions of this work are presented in Sections 3.2 and 4.2, where they derive identifiability of the treatment effect via prognostic scores and propose a generative prognostic model based on variational auto-encoders (VAE).
The theoretical results are complemented with several synthetic and semi-synthetic experiments which also show that the proposed models can compete with and in certain cases improve upon state of the art.
main_review: ## Strong points
- The authors extend the idea of prognostic scores to the context of machine learning and with the additional difficulty of limited overlap. Indeed this approach that can be seen as an alternative to the more common propensity score based approaches in causal inference is indeed interesting in cases of limited overlap which is relevant especially in high-dimensional settings.
- The authors provide a broad range of theoretical results, ranging from treatment effect identification via prognostic scores to a novel estimation strategy with error bounds for the CATE estimation.
- The article is well written, especially the first two sections lay out the context and motivation. The authors provide an adequately succinct yet sufficient list of references on related works to position their proposal.


## Issues/Points that require clarification
- As a general comment, the claims and statements of the main text, both in the identification and the estimation sections, are in part hard to follow without checking in the Appendix which makes the reading difficult. The content being very dense, I would suggest to either split the work into two articles, one for identification and the other on estimation or to submit this work under a more readable format or to defer the experiments to the appendix instead of the theoretical aspects and small examples which would help to follow the justifications in the identification and estimation sections.
- In definition 1 the authors talk about overlapping random variables ($V$) and in definition 2 they talk about functions. Do they mean functions of $x\in\mathcal{X}$ and when they say overlapping functions $\mathbb{P}_t(X)$, do they mean the random variables or do they make an implicit definition of overlapping functions?
- The authors claim that $\mathbb{M}(X)$ is an effect modifier. Since this relies on a definition of effect modifier from Hansen (2008) which is much less common than standard definitions of effect modifiers as interaction terms between covariates and the treatment assignment variable, it would be helpful to provide a concrete example (if not in the main text, then at least in the appendix) of such an effect modifier that does not involve $T$.
- At the end of Section 2, the authors provide an informal definition of _balanced PtS_. Why this non-rigorous definition instead of a formal one?
- Maybe I'm confusing the Intact VAE and the $\beta$ Intact VAE but from the graphical model of Intact VAE in Figure 1 I don't understand the connection between $Z$ and a PS $\mathbb{P}$. The definition of conditionally balanced representation requires that $Z\perp T|X$ which does not seem to be possible under the model of Intact VAE. 
- The choice of the acronym "PS" for prognostic scores could be reconsidered since the target audience of this work seems to be rooted in causal inference where "PS" usually stands for the propensity score in many works.
- I have several questions about the experiments
    - Why the notation switch from $Z$ to $W$ in the experiments? Is there a conceptual difference between the two?
    - Why the choice of linear functions $h,k,l$ and how are they chosen?
    - Why isn't the proposed method from D'Amour and Franks (2021) also included in the comparative study? Even if their method relies on linear assumptions, it would be interesting to see a comparison in this context.
    - Since the paper contains a derivation of error bounds on the CATE (in Theorem 2), it would be interesting to assess them in the simulations and compare the empirical results to these bounds.
    - For the IHDP dataset, it is written that in the used model, $\beta=1$ (referenced as _Ours_ in Table 1) but then there are also modified versions that are reported with other values of the hyperparameter $\beta$. Shouldn't _Ours_ and _Mod.1_ show the same results or am I misunderstanding the table's description?


### Minor comments (that did not impact the score)
- p.1: Confounders are not necessarily high-dimensional. I would reformulate the last sentence of the 2nd paragraph to nuance the claim: rather say that the more covariates are collected the more likely unconfoundedness is to hold but that this can quickly lead to issues in separation between treatment and control groups.
- p.1: I would suggest adding the work on overlap weights by Li and Li (2019)
- p.3: In Section 2.1, I would recommend adding the SUTVA assumption for completeness (either by mentioning it directly or by adding the exclusion of interference in the assumptions).

######################################################################################################
### Post-rebuttal update
I thank the authors for their detailed and timely responses to all reviewers. Their efforts to clarify and reformulated assumptions on the DGP, model and data, as well as their concession to adapt their notations following the reviewers' remarks and the additional experiments related to the theoretical bound have helped addressing most of my concerns. However, I still think the positioning of their work relative to related works is difficult to follow or find which makes it difficult to compare its contributions with previous works. I have therefore decided to only increase my rating by one level.

######################################################################################################

### References
[1] Alexander D'Amour and Alexander Franks. Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap. _arXiv preprint arXiv:2104.05762_, 2021.

[2] Ben B. Hansen. The prognostic analogue of the propensity score. _Biometrika_ 95(2): 481-488, 2008.

[3] Fan Li and Fan Li. Propensity score weighting for causal inference with multiple treatments. _Annals of Applied Statistics_, 13(4):2389-2415, 2019.

[4] Christos Louizos, Uri Shalit, Joris Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_, pp. 6449-6459. 2017.

[5] Uri Shalit, Fredrik D. Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In _International Conference on Machine Learning_, pp. 3076-3085. PMLR, 2017.
summary_of_the_review: In summary, the aim and approach of this paper are interesting and I believe the approach could be a methodological gain for the causal machine learning audience, but its underlying assumptions are presented in a way that makes it difficult to link or compare them to other works and its significance for real-world examples is therefore difficult to judge. Especially, the details of the underlying causal model are not clear to me and make it difficult to conceptually compare it with other methods such as CFR (Shalit et al., 2017) or CEVAE (Louizos et al., 2017).
I will read the rebuttal carefully and am willing to increase the score if the authors address the raised concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of estimating causal effects under partial overlap conditions. Overlap is a common assumption in causality, so the problem is hard and relevant. The authors present a theoretical analysis using PtS, an adaptation of prognostic scores. The estimation is carried out with a modified version of a Variational Autoencoder (VAE) called \beta-intact-VAE. In addition, the authors present bounds on the error of the Conditional Average Treatment Effect (CATE) using their framework. Their paper ends with experimental results of the proposed method and a comparison with other recent methods for CATE estimation.
main_review: Strengths:
- With the exception of a few errors, the paper is well written and has an understandable language. The motivation of the work is clear and relevant.
- The theoretical analysis in the paper is serious, as far as my understanding of the topic goes. 
- The proposed method seems reasonable, and justified by the theoretical analysis in the paper.

Weaknesses:
- I found the paper to be a bit information dense. In fact, I am unsure about what could be some of the most subtle assumptions of the method. Clearly there is the assumption that X is a valid adjustment set, but what about (G1, additive noise models)? If this is necessary, then why use the machinery of a neural network? It seems that in order to make the theoretical analysis easier you have to assume a noiseless prior. Is there any chance you can make a list of the required assumptions, and a sentence or two of why they are needed? Maybe you can consider having such an explicit version of the assumptions in the main document or a separate section in the appendix.

- Along the same line of the previous point, I would invite the authors to think about scenarios in which such assumptions are fulfilled. The examples of this paper, in which we want to adjust a causal estimate using high dimensional covariates has always struck me as a bit unrealistic. As the authors might know, adjusting for all pre-treatment variables is not always the best option because that might introduce M-bias. I would like to say, though, that as the authors correctly point out, the more variables you have, the lower the chance of having overlap of the treatments. Can you please comment on what could be some possible scenarios in which the method could be applied?

- Finally, if known to the authors, I would like to know the relation of this work between work on generalization of neural networks. It seems, from my point of view, that the problem of limited overlapping of conditioning variables can be fixed by having estimators of the counterfactuals that generalize well beyond the support of the combination of covariates, and treatment. Could it be possible that what is driving most of your results is the regularization of the estimates? In general regularization is a way of achieving out of support generalization. Can you comment on this?

summary_of_the_review: I believe the paper solves the interesting problem of estimating CATE with limited overlap. It seems to me that most of the theory, and the estimation method were developed before this article, so in a certain way it is an incremental contribution. Nevertheless, both theoretical, and empirical analysis of the article seem serious.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the setting of limited overlap of covariates and studies identifying and estimating the causal effects. This setting is difficult to deal with because it is impossible to estimate causal effects at non-overlapping values due to a lack of data. To this end, the paper proposes an idea based on the prognostic score and identifiable representation (Intact-VAE). The prognostic score is an appropriate tool for limited overlapping because it can map some non-overlapping values to an overlapping value in a space of lower dimensions. Intact-VAE is a natural combination of iVAE and CVAE and is proposed to help to identify the causal effect. The paper implements this idea by proposing a new regularized VAE, called \beta-Intact-VAE, and further analyzes the treatment effect error bounds. In the end, the paper compares the model with recent methods in the experiments.
main_review: Strengths:

1. Authors contribute a novel approach for limited overlapping of covariates. The setting of limited overlapping is interesting. The proposed approach combines the prognostic score and Intact-VAE, which are appropriate and novel under this setting. Although the \beta variational lower bound was proposed in the previous methods, it plays a role of controlling balance and it can be viewed as a novel application.

2. The experiment results are extensive. The authors follow some existing data generation processing or data and show that the balanced prior is important in practice compared to non-balanced ones and their models can obtain better performance.

3. The theoretical analyses are novel under the setting of this paper. These results follow those of iVAE and adapt to the new settings. The authors elaborate on some conditions to help readers to get them, which is great.

Weaknesses:

1. The paper is dense and difficult to follow. It introduces many concepts from different areas. It takes much time to understand and check the paper. I suggest authors add some plots to help readers to faster understand some definitions, e.g. overlapping. The paper involves and requires readers to have wide knowledge, so I think that a good presentation is difficult. Maybe moving some contents to the appendix and illustrating some concepts could be a choice. In addition, characters with the same style are used for different types of subjects, such as $\mathbb{R}$ for real numbers and $\mathbb{M}$ for a function. I strongly suggest using consistent notations.

2. Experiment results measured by pehe are worse than CFR. ATE and PEHE reflect different information. ATE focuses on the mean and PEHE reflects the stability because if there is a large CATE error, PEHE can be large (as mentioned on the page 9). Large CATE errors mean that the model performs unstably at some values. So, it is better to discuss the results more properly.

summary_of_the_review: I vote for weak acceptance. The paper proposes a technically and theoretically sound approach and it studies an interesting setting -- limited overlapping, though it has some presentation issues.
review_title: null

Meta-review: In this paper, the authors proposed a method for causal inference under limited overlap -- an important and understudied complication.  The authors propose to recover a prognostic score using a variational autoencoder, and thereby map a higher dimensional set of covariates with limited overlap to a lower dimensional set where overlap holds, and such that ignorability is maintained.

The paper was reviewed quite favorably by reviewers, and the authors updated the manuscript to address specific issues raised by reviewers.
Decision: Accept (Poster)