Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a method for using real-world patient data to generate (semi-)synthetic privacy-preserving data on which to evaluate methods for causal effect estimation. For this purpose, they adapt the ADS-GAN (Anonymization through Data Synthesis using Generative Adversarial Networks) model introduced by Yoon et al. (2020) to produce a synthetic dataset that is identical in distribution to the original dataset, yet cannot be used to identify any of the original patients. To ensure the former desideratum, the authors calculate the Wasserstein distance between $P_{\hat{X}}$ and $P_X$; to ensure the latter, they use $\epsilon$-identifiabilty (Yoon et al., 2020). The authors show empirically that the synthetic dataset satisfies these desiderata, and then evaluate a few causal effect estimators on the synthetic data using the generated ground truth.
main_review: **Strengths**:
- The problem tackled is very relevant for causal inference problems, as there is almost always a lack of ground truth available to assess how well different causal inference methods perform.
- The paper is well-written and virtually error-free.

**Weaknesses**:
- When the inclusion and exclusion criteria are applied to the original dataset, the authors are potentially introducing selection bias by conditioning on downstream variables such as hypertension. This can then introduce spurious correlations between any common causes of the selected variables, which may for example include treatment variables if one of the treatments has an effect on hypertension. I was surprised to see that the authors have not at least discussed a potential selection bias issue. Furthermore, other data preprocessing steps such as filling in missing values or standardizing the values have the potential to make or break existing causal relationships.
- The experimental section is rather limited. The authors only consider a single starting dataset, and evaluate only a few methods for causal effect estimation on the synthetic data.

Other comments:
- It is possible I am missing something, but it seems to me that the generator and discriminator roles are switched. Based on the explanation before Definition 1, it appears as though the generator is maximizing the loss so as to ensure that the synthetic dataset is not too close to the original one in terms of $\epsilon$-identifiability, while the discriminator is minimizing the loss to ensure that $P_{\hat{X}}$ and $P_{X}$ are indistinguishable. Shouldn't it be the other way around?
- The ADS-GAN model and $\epsilon$-identifiability should have corresponding citations to Yoon et al. (2020) when first mentioned. Furthermore, it would be helpful to write what the acronym ADS-GAN stands for.
- I share the authors' concern that propensity score stratification performs better than the doubly robust methods. Is the propensity model the same for both methods? Perhaps there is some particular structure in the data that favors the simpler propensity score stratification.
- page 6, subsection 4.3, second paragraph: Treatment domain is introduced as $X_T$, yet is almost immediately replaced with $\mathcal{T}$? Are these two sets the same? If not, what is the difference? I cannot find where $\mathcal{T}$ is defined.
- page 6, last paragraph: New domain set $\mathbb{Y}$ for $Y$ is introduced out of the blue. Perhaps the authors meant $\mathbb{R}$ instead?
- page 8, third line after Figure 3: small typo, "We" should not be capitalized
summary_of_the_review: The authors have presented a simple and potentially interesting approach for generating privacy-preserving synthetic data from a real-world dataset, but I am concerned by the fact that selection bias was introduced when the authors applied inclusion and exclusion criteria on the patients from the original dataset. Furthermore, the method is only applied to one real-world dataset, so it remains unclear to me how well it can be applied to datasets with different characteristics (e.g. size, amount of missing data, data types).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a method to generate a set of binary datasets that are realistic and include counterfactual outcomes so that they can be used for evaluating causal inference algorithms. Moreover, the proposed method ensures that the original patients’ privacy is preserved.

main_review: The paper is well-written and easy to read. It also addresses a very important, long-lasting problem in the literature; that is, generating datasets that include realistic counterfactuals. I really liked the idea of generating pseudo-patients to preserve privacy. 
That being said, I think there is a fundamental problem with generating the counterfactuals from a model trained on data: this is the whole reason why we develop causal inference methods. If we already have an algorithm that can predict the true counterfactuals, then the problem has been solved (and we no longer need datasets for evaluating causal inference methods)! So we agree that we cannot be sure that the proposed algorithm predicts the true counterfactuals. Then, training causal inference methods using the generated dataset means they merely try to learn the underlying function that the proposed outcome neural network models. I’m not sure how this is different from adopting any arbitrary response surface as Hill et al. and ACIC organizers have done. 
Please see my other comments below:
- Please provide a reference for the statement that the ACIC dataset is “limited by non-representative populations”. As far as I know, ACIC datasets are all adopted from large medical studies.
- Please define “identifiability” and “realisticity” formally.
- For calculating the weights, the authors have used the inverse of H. I think it’s wrong though: imagine a feature is the same for all patients, then the entropy for that feature would be zero, and the inverse of zero is infinity. So a feature that doesn’t matter at all (for identification) gets a large weight. Or am I missing something here?
- The paper provides little discussion on many design choices (e.g., why ADS-GAN or WGAN-GP was selected, etc.). 
- I couldn’t understand why “the neural network weights for treatment connections can be interpreted as the causal treatment effects”. Also, the true individual treatment effects are often different from the average treatment effect (think personalized medicine)!

Minor comments:
- The abstract is too long; consider reducing it to one third.
- Wrong citation format; in most cases, both name and year must be inside parentheses.
- Citation for ADS-GAN is missing in the text.
- Page 5, line 2: don't → do not
- The “subset of” sign in page 5, lines 9, 10, and 15 should be reversed.

summary_of_the_review: Neither of the two components of the paper (i.e., privacy preservation and counterfactual prediction) are novel (they are adopted from previous works). The claim that the proposed counterfactual prediction component generates true counterfactuals is not supported (and my understanding is that it cannot be supported either). For these reasons, I think the proposed algorithm is not practical and therefore, I recommend rejecting the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of generating synthetic patient data for the evaluation of causal inference models. The generated patient data is expected to highly mimic the distribution of the original dataset while also taking patient privacy into consideration. Experiments are conducted on the synthetic dataset with three well-established causal inference models.
main_review: Strenth: 
+ The paper is easy to follow and codes are provided for review.

Weaknesses:
- The proposed problem is not well addressed in this paper. The purpose to simulate patient data is to evaluate causal inference models with generated ground truth. However, it is hard to guarantee that the generated treatment effects are close to the original treatment effects (both the effect size and sign): one can only guarantee the similarity between observed outcomes but not counterfactuals. 
- The technical contribution of this paper is not enough. The entire generation framework is based on the existing ADS-GAN model. The only difference is an additional contrastive loss to add penalty on the original loss function when the generated sample is much closer to x' than to x. The following potential outcome generation is based on a feed-forward neural network, which is a widely used backbone in causal effect prediction. 
- The potential outcomes are generated directly through a neural network without any adjusting for confounding variables. Since the data is generated from real-world observational data that includes various confounding factors. 
- Only three causal inference models are included in the experiments and two of them are from the same family (DR). More baselines (e.g., IPTW, matching-based methods) should be considered to examine the goodness of created dataset. 
- Why propensity score stratification is much better than DR methods? When the results are not consistent with most studies, can we still trust the generated data as well as the ground truth?
- Only Wasserstein distance is used to measure the distance between original and synthetic dataset distribution. More metrics such as KL-divergence should be considered for comprehensive evaluation. 
- What's the performance of three causal inference models on the original dataset? Are the results on the original dataset consistent with the results on the synthetic dataset?
- What's the influence of \beta in Eq.(3) to model performance? An ablation study should be conducted to show this influence.
summary_of_the_review: An interesting paper, but it seems that the proposed problem is not well-addressed by the method.
review_title: null

Meta-review: In this paper, the authors propose a method for generating high quality synthetic datasets, and use their methods to evaluate a variety of causal effect estimators.

In general, the paper was not received very favorably by reviewers.  The primary concerns were: (a) issues with built-in bias in the algorithm that generates synthetic data (due to collider stratification bias induced by conditioning on causally "downstream" variables, (b) issues with "replicating underlying counterfactuals," which indeed is a difficult problem, and (c) lack of "technical novelty."

First, I am personally very sympathetic to what the authors are trying to do.  Regardless of current reviewer reception, I think the causal inference community really needs more high quality benchmarks, and (semi)synthetic datasets, and validation approaches.  I urge the authors to continue this line of work.

That said, I think it is important (for causal benchmarks) to be clear about the distinction between the observed data distribution (e.g. p(C,A,Y) for the backdoor model), and the full data distribution (e.g. p(C, A, Y(0), Y(1)) for the backdoor model with a binary treatment).

Generally what makes a benchmark interesting is preserving some features of the _full_ data distribution, and allowing "knobs" that make the problem easier and harder.  Much of what the ACIC competition organizers did was provide such knobs.  Mimicking features of just the observed data distribution, even if they are complicated, isn't enough to make a causal benchmark interesting, since the problem is all about how full and observed data relate.

When revising the paper, please keep this difference in mind, and consider what features of p(C, A, Y(0), Y(1)) (or more complex versions of this) make for an interesting benchmark, while also generating p(C,A,Y) that "mimics observed data" in some way.
Decision: Reject