Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:  
Main review: An estimated CATE function could potentially reveal individual covariates, treatment status, or potential outcomes. The authors propose a general approach for multi-stage learning under differential privacy guarantees and apply it to CATE estimation. They show empirically the trade-off between privacy and estimation accuracy.

The paper is well-written, however familiarity with the differential privacy is assumed by the authors.

The only concern about this manuscript is that it does not have much of theoretical contribution, but on the other hand, it is has great practical value and should be of interest to applied data scientists. 

The authors perform a thorough evaluation of the proposed approach. The results show the surprising different behaviors of DP-EBM-S-learner compared to DP-EBM-DR-learner and DP-EBM-R-learner. This shows that choosing a flexible CATE estimator versus a simple estimator is not a trivial decision.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: Summary: Review of Paper#67
Main review: 
This paper studies the problem of estimating conditional average treatment effects (CATE) with differential privacy guarantees. It proposes a general meta-algorithm that can work with simple, single-stage CATE estimators such as S-learner and more complex multi-stage estimators like DR and R-learner. The authors perform a privacy analysis utilizing the parallel composition property of differential privacy. Finally, experiments demonstrate how differential privacy noise affects the estimated CATE function.

Overall, I think the author considers an interesting problem of addressing privacy concerns in estimating causal effects. Such a problem could be prevalent due to the sensitivity of healthcare data. I appreciate the authors for starting this conversation. However, I do have some questions and concerns, summarized as below.

- This paper focuses on estimating conditional average treatment effect (CATE) while ensuring differential privacy (DP) constraints. This is a bit funny and seems ill-posed. For instance, suppose one wants to compare the CATE of a drug on male and female patients, and the actual CATEs of the male and female population diverge significantly. In this case, ensuring the privacy constraints on gender requires one to add DP noise on the CATE function until one cannot differentiate the CATE on the male and female population. There exists no ideal CATE function in this setting. It makes me wonder if DP formalism is the best tool to address the privacy challenges in causal inference tasks.

- I would suggest writing Sec 2.1 using the formal language of causal inference, e.g., Pearl's structural causal model and the backdoor criterion.
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper studied the indices of DP for the entire procedure of estimating CATE when each component of the procedure is made DP and suggested some interesting directions based on results of numerical experiments.
Main review: Strengths:

1. Conditional, or heterogeneous, average treatment effect estimation (CATE) is a natural problem that differential privacy is of great concern, given the essential role of CATE in individualized decision making. Therefore, this paper is indeed trying to make progress on an important problem. The topic is very fresh -- there are indeed only a few papers that consider differentially private causal inference.

2. The section before numerical experiments is relatively well-written, not dense with technical jargon, and accessible to people working on either causal inference or privacy data analysis.

Weaknesses:

1. The insight offered by the only theorem proved (Theorem 6) is quite limited. It is good to calculate the "tight" DP indices for the entire estimation procedure, but the only concrete example in the paper is about the trivial scenario: if all sub-routines are $(\epsilon, \delta)$-DP, then the entire procedure is $(\epsilon, \delta)$-DP. The motivation for proving this theorem is not strong, in my opinion.

2. The numerical experiment section lacks sufficient detail. The authors did not provide any details on the DP-nuisance parameter estimation algorithms. It is difficult to judge the value of the experiments without knowing these details.

3. The potential research questions suggested in this paper are only based on one simulation study. Since this paper is not about proving important theoretical results, more comprehensive simulation experiments should be performed. A pure "intuition + simulation studies" based paper, even without any theoretical guarantees, can still be very valuable to the community. But the experiments must be very comprehensive and aim at pushing the boundaries of the data generating mechanisms as hard as one could.

4. The insights offered by the experiments are not very pedagogical. When the sample size is large, complicated methods can perform better than simpler methods; when the sample size is low, simpler methods might be more stable so one might be willing to sacrifice some bias. Isn't it true for almost any statistical practice? How much more could we learn by following these directions, except that we add the DP component?

5. The type of differential privacy that the authors tackled is black-box. For instance, probably a most natural DP-related setting is what if some of the key confounders have to be taken to be differentially private while modeling the nuisance parameters and the final CATE function. A more concrete example will be more useful to illustrate the significance of either the theoretical or the empirical contribution of this paper.

Overall, the paper is trying to tackle an interesting problem. Unfortunately, the current "signal" in the paper is quite low: the only theorem is not a core contribution to the field; the simulation experiments are not solid or comprehensive. As a result, I would not recommend acceptance of this paper.
review_title: null

Meta-review: In this paper, the authors provide a meta-algorithm that uses models with differentiable privacy (DP) guarantees applied to nuisance functions such that a target causal parameter of interest (that uses nuisance functions for estimation) "inherits" those guarantees.

While reviewer opinion was initially split, subsequent discussion addressed most concerns expressed by reviewers. In particular, while the perception of the paper was that it did not have a large theoretical contribution, relative to what is known about DP its contribution in practice was deemed quite important.

As a result, the paper merits publication in CLEAR.
Decision: Accept (Oral)