Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a data augmentation method for learning the simultaneous effects of many causes in settings where treatments manipulate many causes at once. Specifically, the authors consider a case where the treatment manipulates a K-dimensional binary vector of variables that cause a single outcome Y. The problem being addressed is sparsity in observed treatments: because there are $2^K$ possible treatments, as $K$ grows, it is less and less realistic that we will observe all levels of the treatment.

The proposed SCP method suggests an approach where the potential outcome distributions for single-cause interventions are learned first, then these are used to generate synthetic data with single-cause perturbations to augment the dataset. The augmented dataset is then used to estimate the potential outcome distributions for all treatments. The authors make several heuristic arguments for why this might work, referencing sample size and covariate balance. They then demonstrate on fully simulated data, and semi-synthetic data using covaraites and treatment indicators for COVID-19 treatment with simulated length-of-stay outcomes.
main_review: **Per discussion below, authors' addition of failure cases and general guidance, as well as agreement to switch some framing has moved my recommendation to "Accept"**

# General Feedback

The paper is written clearly, and the problem is well-introduced. However, I remain skeptical of the approach because it is not clear where the information about sparsely-observed treatment combinations is coming from. Because there is no theoretical treatment in the paper or even demonstrations on simple, transparent examples, it is difficult for me to endorse the approach as presented. However, I do think there are some interesting possibilities here that the authors should do more to explore. In my opinion, the current "reasons" for SCP working that are considered are not sufficient to justify the approach, and there are more interesting potential justifications that would be more compelling.

The problem of combinatorially exploding cause combinations has a long history in the statistical experimental design literature, going back to Fisher (see, e.g., https://digital.library.adelaide.edu.au/dspace/bitstream/2440/15191/1/48.pdf). The main issue with so-called factorial experiments is figuring out how to estimate the effects of interactions between causes. For this reason, analysts will often make assumptions about certain orders of interaction being zero (e.g., assuming that there are no 3-way or higher interactions between causes). There are then experimental designs (so-called fractional factorial designs) that can identify the remaining low-order interaction effects (see, e.g., Box, Hunter, and Hunter Chapter 6, hosted here: http://pages.stat.wisc.edu/~yxu/Teaching/16%20spring%20Stat602/%5BGeorge_E._P._Box,_J._Stuart_Hunter,_William_G._Hu(BookZZ.org).pdf).
An extreme case of this is assuming that on a certain scale there are no interactions between causes (e.g., the expectation of the treatment is just a linear combination of the causes). In this case, one could estimate the potential outcome distribution for all 2^K treatments by simply observing K treatment levels, so long as those treatment levels are linearly independent of each other.

This paper seems to aim at addressing a similar issue (at least per the introduction), but now in a more challenging observational context. However, it is not clear how this method addresses the fundamental question of having an exploding number of interaction effects to estimate. The empirical exploration is no help here. Note, for example, that the synthetic experiment only includes first-order effects of causes (there are no interactions between causes, only between causes and covariates), and the simulated outcomes in the "real data" experiment only include, at maximum, second-order interactions between causes. If we knew at the outset that there were only second-order interactions between causes, then we could estimate the effects of all treatment levels with many fewer observed treatment levels (see https://link.springer.com/chapter/10.1007%2Fb105081_8). Thus, SCP could succeed on these experiments by simply implicitly regularizing the estimated treatment effects toward low-order interactions between causes. This is not necessarily a bad thing, but is very different from the current framing of the paper that repeatedly emphasizes that SCP makes no assumptions about the DGP.

There are some ways that I could imagine SCP could help. For example, in cases where there is non-trivial causal structure between the causes, SCP might enforce certain marginalization constraints between causal effects, i.e., that the causal effects of downstream variables should average out in a particular way to match the causal effect of intervening on an upstream variable. These kind of moment constraints could be helpful regularizers for CATE estimators.

There is also a possibility that SCP does nothing that is actually causally interesting, but applies useful regularization to the neural networks used to estimate P(Y | X, A) by self-distillation (see, e.g., https://arxiv.org/abs/2002.05715).

To make their analysis of SCP clearer, I would suggest that the authors examine the method in the following contexts:
* Consider an ideal experimental setting where there is no confounding, and we are just interested in the average treatment effect at each level $\bm a$. This separates out the treatment sparsity problem (the main motivation of this work) from the confounder adjustment problem.
* In the experimental context, show how SCP would work with a simple model that is not a neural network (e.g., a linear model with higher-order interactions with K relatively small). A transparent toy example worked out analytically would be extremely useful, even if it does not prove general properties of the approach. If SCP does very little here, it would suggest that there is something special about the self-distillation.
 * Consider a range of settings with different orders of important interactions. I suspect that SCP will perform worse when higher-order interactions play a major role.
 * Consider a range of cases where the causal structure between the causes is changed systematically. It may be the case that SCP performs better when there is detailed causal structure, as this could correspond to a system of moment constraints that regularize the CATE estimator.

Importantly, when suggesting a causal procedure that relies on unverifiable assumptions, or for which there is no general theoretical justification, it is important to characterize cases where the method breaks, as well as those cases where it succeeds. I suspect that the above analyses would reveal such cases, and this is fine. There is no free lunch when it comes to sparsely observed data, so the method must have limitations. The authors should articulate them clearly.

I consider these to be more essential explantations for SCP's performance than the "balance" and "sample size" considerations listed in the paper, because these would also be reasons that SCP would help with a single binary treatment; they are not unique to the multi-cause context. (As an aside, it would be interesting to see whether SCP also seems to help with estimation in the standard setting. If it does help, then that would again suggest an alternative explanation for why SCP might help with many treatments as well.)

# Detailed Comments

## Notes on Related Work

In Table 2, G-computation and covariate adjustment should have their intermediate estimands listed as P(Y | A, X) since they both both estimate outcome models as nuisance functions. G-computation is also usually done by simply taking the empirical measure of X to be P(X) and P(Y | A, X) is just averaged over the observed values of X. Deconfounder methods generally do not rely on weighting, but instead involve an outcome model P(Y | A, Z).

## Careful with the "Real Data" Example
Section 5.2 uses a dataset with simulated outcomes, so should not be called a real data experiment, especially for such a sensitive topic (COVID-19 Treatment). This should be called a "semi-synthetic" experiment.
limitations_and_societal_impact: As stated above, given that this is a method without any theoretical characterization, it is important to highlight cases where the method would fail. As of now, there are only two simulated settings where the method works well; I would encourage the authors to do more stress testing. The ablation study is a useful first step.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a data augmentation approach for conditional average treatment effect (CATE) estimation with multiple treatments. Under the assumption that the multiple treatments follow a DAG, sequential ignorability, and sequential overlap, the conditional average of each (multi-cause) potential outcome can be identified through the conditional average of the corresponding single-cause potential outcome. The proposed method starts by fitting models for (1) how each treatment affects its causal descendants, and (2) how covariates, each treatment and its causal descendants affect the outcome. Then it generates $K$ pseudo-samples, where $K$ is the number of treatments, from the estimated data generating process, each flipping one of the treatment. Finally, it fits a regression model on the augmented data. The proposed method outperforms a handful of competing methods on extensive numerical experiments. 
main_review: The single-cause perturbation-based data augmentation is novel and interesting for causal inference with complex and high-dimensional treatments. Generally speaking, I feel that data augmentation should be used more in causal inference, because it helps regularizing the estimation under data scarcity. On the other hand, the proposed single-cause perturbation appears to be an effective augmentation scheme because it exploits the DAG structure. 

Furthermore, the paper has a very impressive experimental assessment --- it not only shows that the proposed method has a superior performance (i.e., RMSE), but also presents various robustness checks to investigate why it works. For example, Figure 4 and Figure 5 are very informative. They exactly answer some of my questions when I read the previous sections. 
limitations_and_societal_impact: (1) The paper claims that "instead of making additional assumptions on the DGP, we exploit the connection between a single-cause intervention and a multi-cause intervention". This is an overstatement. The proposed method does impose more restrictions on the DGP that is not required in other works (e.g., deconfounder). It requires a DAG structure among the treatment, which is a big assumption. In many applications involving multiple treatments, such as those in two-sided markets, it is unclear whether the acyclic assumption holds among the treatments. Even if it holds, it is not always easy to draw a reliable DAG. 

(2) Line 120-123: what does "the model for $A_{-k}^{\downarrow}(a_k)$" and "the model for $Y(a_k)$" mean? In order to generate the pseudo-samples, it seems that they should be the whole distribution instead of just the conditional mean. The algorithm 1 in Appendix does not seem to clarify this. 

(3) Line 130-132: similarly, the word "obtain" is confusing. Do you mean sampling from the estimated distribution or plugging in the estimated conditional mean or something else?

(4) Does the proposed method have any connection with the marginal structural models (MSM)? 

(5) Line 235: should the metric be $\sqrt{\frac{1}{N_t}\sum_{i=1}^{N_t}\sum_{a\in \Omega}(y_i(a) - \hat{y}_i(a))^2}$? Why do you use $y(a)$ instead of $E[y(a)|x]$?
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The authors propose a data augmentation method for CATE estimation in the setting of multiple causes. In this setting, there is sparsity in the observed treatment assignments, which makes estimation harder. The authors address this sparsity by aggregating multiple datasets together, generated by estimating single cause potential outcomes, which are theoretically shown to be equivalent in expectation to the multi-cause PO's of interest. The method is agnostic to the methods employed for CATE estimation and covariate adjustment on the adjusted data.
main_review: **Originality**: Novel method for causal estimation in multi-cause settings. Dispenses with certain assumptions made by past works on multi-cause CATE estimation. In theory, multi-cause methods for ATE estimation could be adapted, but this is discussed. To the best of my knowledge, related work is well cited. 

**Quality**: Submission was technically sound and overall of a very high quality. I was particularly impressed by how well the method was explored: discussion of assumptions, possible limitations, avenues for future work outlined, investigation into the inner workings of the method, etc. There is empirical support for the method, as well as theoretical support for why estimating single-cause PO's and using them to augment data is a valid thing to do. 

**Clarity**: I thought the paper was very well written and organized. 

**Significance**: CATE estimation in multi-cause settings is undoubtedly an important problem. I find this work very compelling because it is a data augmentation scheme that can work in tandem with a wide variety of CATE estimation / covariate adjustment methods, which greatly extends its applicability. The method is scalable but also seems to work well on relatively small datasets. High significance overall. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a novel method for estimating conditional average treatment effect (CATE) of multiple treatments (or a multi-cause treatment). The research is motivated by data scarcity in multi-cause CATE estimation, and  a single-cause perturbation (SCP) procedure is proposed to augment the observational dataset to improve the estimation.  Detailed theoretical analysis has been provided in the paper (main text and appendices) to support the development of SCP. A comprehensive evaluation of SCP with synthetic data has been conducted, and an application of SCP to a real-world COVID-19 dataset (with simulated outcomes) is presented too. The experiment results have demonstrated the effectiveness and superiority of SCP in comparison with the benchmark methods.  
main_review: Please find below the detailed comments on the originality, technical quality, clarity and significance of the paper:
1. Originality:

It is a very interesting and effective idea to make use of single cause CATE estimation for multi-cause CATE estimation. The authors have done a great job in establishing the theoretical link between single cause CATE estimation and multi-cause CATE estimation, and then utilizing this link and the idea of perturbing individual causes to create augmented and more balanced datasets for better multi-cause CATE estimation. The proposed approach has also provided the opportunity to use off-the-shelf single cause CATE estimation algorithms for the multi-cause problem.

From the paper, the originality of the work is clear overall, but to strength the originality further, it would be helpful to clarify the following points:

(a) For single-cause CATE estimation and ATE estimation, is there any existing research which employs data augmentation? If yes, how is the existing research related to SCP proposed in this paper?

(b) Are there any existing research which links single cause CATE or ATE estimation  and multi-cause CATE or ATE estimation? If yes, how is the existing research related to the idea proposed in this paper?

(c) Although in the Appendix, the authors have clarified that the setting of their work is different from mediation analysis, I have a feeling that the analysis in Section 2.2 is related to mediation analysis (or direct vs indirect causal effect), although the overall approach and aim of SCP is different from mediation analysis. It would be good if authors could elaborate on this a bit more.

(d) Related t multi-cause ATE estimation, the joint-IDA method should be mentioned:
Nandy, P., Maathuis, M., & Richardson, T. (2017). ESTIMATING THE EFFECT OF JOINT INTERVENTIONS FROM OBSERVATIONAL DATA IN SPARSE HIGH-DIMENSIONAL SETTINGS. The Annals of Statistics, 45(2), 647-674. 

2. (Technical) quality

The proposed method is sound, and detailed discussions (and proofs) have been presented to support the correctness of the SCP algorithm. The experiments are comprehensive and the results are promising. I appreciate the authors' thoroughness in developing, evaluating and presenting the proposed method. 

Below are some points that may need clarification to improve the paper:
(a) In terms of partitioning the causes, what about if A_k and some of the other causes are siblings? That is, unlike what's illustrated in Figure 2(B), some other causes are neither A_k's parents/ancestors or children/descendants? I don't think this would change the results the authors have presented, but it would be good to include this case more explicitly.

(b) Lines 120- 121 on page 4: Are the two models trained definitely the predictors for the "potential" treatments or outcome? That is, there is no formal analysis showing that the two factors on the RHS of equation (4) are the potential treatments (A^{\downarrow}_{_k}(a_k)) and the potential outcome (Y(a_k)) respectively. I think you can still get the potential treatments or outcome from data based on (4), but the statement here needs clarification/correction or analysis to support.

(c) Section 3.2: It sounds ok to pool the augmented datasets, but what about if the estimations of the expected outcome (i.e. the RHS of (5) estimated from data) for two different k values are not consistent? Have you tried estimating the RHS for different k values and check their consistency?

(d) Page 7, about the benchmarks: It would be good if more details could be added on the benchmarks, in particular, how those single cause CATE methods are used as benchmarks (i.e. how are they applied for multi-cause CATE estimation)?

3. Clarity/presentation

Although as mentioned above, some technical details might need some clarification/improvement, this paper is well written and very easy to follow. It is enjoyable to read it.

One point though (on Appendix A.1): the sequential ignorability assumption does not seem to appear in reference [50]? Also, it would be good to provide some intuition on this assumption. 

4. Significance

Multi-cause CATE estimation is an important and challenging task, and data scarcity is a practical issue for the task. The proposed SCP method could a very useful tool for tackling the challenge. Although strong assumptions are required by SCP, comparing to existing methods (which make similar assumptions), from the experiments, SCP and causal data augmentation look like a promising direction to follow. 

limitations_and_societal_impact: The authors have addressed some specific limitations of their work (e.g. in Section 3.4). It would be ideal if the authors could elaborate a bit more on the restrictions created by the assumptions, e.g. unconfoundedness and sequential ignorability.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors aim to the estimation of CATE when there are multiple treatment variables and the causal relations between them are known in advance. The main innovation in methodology is an additional step of flipping the value of each variable and updating the corresponding descendants according to the estimated causal mechanisms. This step leads to that more data could be obtained and the data are more balanced, which two steps could benefit the CATE estimation. 
main_review: After the rebuttal, the authors have addressed my main questions. I am willing to increase my score.

------
This paper is well written and organized well. The problem the authors tackle is common in reality. And for the notations or concepts that possibly cause ambiguity, the authors provide detailed illustrations. I enjoyed reading this paper. 

My main concern is about the novelty. Although I do not see it as a drawback, I correspondingly pay more attention to details. For example, as the discussions on Line 177-181, I am curious about the experimental result if we perturb two variables at once. In that way we could have more data than the current method. I think it will be more persuasive if the authors could provide the support from the theoretical perspective or empirical perspective for the balance between the first step error and the sample size. 

In addition, I have one more question. What method did the authors adopt to estimate the first term on the right hand of Eq. 4?

I am willing to increase my score if the authors could provide more details about the balance between the first step error and the sample size, or some further theoretical guarantee for the effectiveness of the proposed method. I also possibly change my score according to other reviewers' comments.

Line 136: It should be ``a, \tilde{a}^1, \ldots, \tilde{a}^K''. If $k$ denotes an index from [K], there should be an ellipsis ahead of \tilde{a}^k.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The authors propose a data augmentation method for learning conditional average treatment effects with multiple treatments. Assuming sequential ignorability and sequential overlap, the authors propose to first fit models that describe how changing a single cause affects causal descendants; then to augment the data set with new observations; and finally to fit a regression model on the augmented data. This is a strong paper. All reviewers agree that it should be accepted. Reviewer yyUp praises how well the method was explored, including discussion of limitations. Reviewer f9co states that the paper has "a very impressive experimental assessment" that not only shows superiority to competing methods but also investigates why the method works. Reviewer 1Xus states that the paper is “enjoyable to read”. Reviewers 1Xus, sokR and YNQn had some reservations about the method, in particular how well it performs under higher-order interactions. The authors have provided thorough responses, with additional simulations and high-level guidelines to understand failure modes. The authors state that they will expand the discussion with the additional insights.
Decision: Accept (Poster)