Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a novel regularization term for designing loss functions to estimate outcome and propensity score models, where the end goal is to estimate ATE.  The regularizer is derived from the assumption of conditional independence of potential outcomes and treatment given covariates (i.e. the no hidden confounding assumption).  The authors observe that this assumption implies that residuals of potential outcomes and treatments are orthogonal.  The authors derive a loss function which yields this orthogonality at the optimum.

The Theorem 1 shows asymptotic normality and a double robust property. The authors also perform extensive empirical comparisons to other causal inference methods on 4 standard benchmarks, as well as on the ACIC challenge. The method performs admirably: it is competitive on IHDP, and in a statistical tie with the best-performing methods on Twins and Jobs.

Unconfoundedness is an assumption that ensures that an estimator of the form of equation (3) is unbiased for a causal effect. Without the assumption, an estimator may ascribe a causal effect to the treatment that really comes from a common cause.  To use it to inform a penalty on the estimator is an unusual step, as it seems unrelated to the question of efficiency.  However, the results are compelling.

The first question that arises is: have the authors presented a clear ablation analysis, similar to the TARNet vs CFR analysis of Shalit et al.?  The authors state that they use the TARNet architectures for the outcome models.  In that case, can we interpret DONUT vs TARNet results in Table 1 as an ablation analysis on the $\Omega_{OR}$ regularizer? The numbers for TARNet in this submission are identical to those in Shalit et al., so perhaps not all experimental variables have been exactly matched such that we can consider it a true ablation analysis?

I have some technical questions that I believe the authors should address:
*    Moving from equation (31) to (32) seems to require omitting any randomness in the denominator.  Why is this justified?
*    Since $\hat f = f^{\epsilon}$ in equation 31, can we ever expect that $\bar f$ is equal to $f_0$?
*   What are the numbers in table 2?  The authors state they evaluate on 97 models from the ACIC challenge. Is table 2 presenting an average of ATE estimation errors?
*   In describing the Pseudo Outcomes, the authors state that $\psi(x)$ is the true treatment effect at X.  However, it is the expected treatment effect, and hence can’t be used to exactly impute an unobserved outcome, which may have some noise in general.  This also affects the argument in Appendix A (steps 18 and 20), where this imputation is performed.  Are the authors assuming throughout that the treatment effect is deterministic given $X$?  In Theorem 1, a constant treatment effect is assumed, but this seems to be an assumption to make the theory go, and the method is designed to handle heterogenous treatment effects.  If such an assumption is at play in the entire paper, it should be presented earlier.
*   Can the authors address how to construct confidence intervals, which are generally always reported for ATE estimation?  Can one plug in quantities for equation 12 from in-sample estimates?  Can the authors compare widths and coverage properties of confidence intervals constructed in this way?
*    The authors are noncommittal about regularizing orthogonality of both $Y(0)$ and $Y(1)$: appendix A indicates it’s not necessary if plugging the true outcome and propensity score model: however, in practice, does it make a difference?
*    A small complaint about terminology: the authors state in 3.1 that “the inner product in (5) is the empirical covariance between $Y(t)$ and $T$ given $X$.” However, equation (5) is not a function of X so this statement cannot exactly hold.  The orthogonality constraint requires some removal of conditioning, which the authors should address.

review_title: A novel method that performs admirably; some clarification needed in places

final_score: rating: 3: Clear rejection
review_content: Overall, I found the main idea of this paper very interesting and the experimental results promising; however, there were several major and minor technical issues with the work that need to be resolved.

--- Major comments ---

1. There appear to be at least two major technical issues:

    a. A substantial portion of the work is based on the author's assertion that $Y_i(0) = Y_i - \psi(X_i)T_i$ which is not, in general, true. We can see this with a simple counter-example. Let $Y(0)$ be a binary variable. Then $\psi(X) = E[Y(1)|X] - E[Y(0)|X]$ will be some value in $[-1,1]$, but if $\psi$ is any value other than $1$, $0$, or $-1$, then $Y_i - \psi(X_i)T_i$ will be non-binary and thus not equal to $Y_i(0)$. This assertion only holds if $X$ and $T$ uniquely determine $Y$, which is not generally the case. Thus, it remains to be shown that enforcing equation (8) is equivalent to enforcing equation (5).

    b. In the first condition of Theorem 1, the authors assume that either $\hat{f}$ or $\hat{\pi}$ is consistent. If $\hat{f}$ and $\hat{\pi}$ were estimated separately and used as plug-in estimators, as in Chernozhukov et al. (2018), this would be a reasonable assumption, as it would be up to the user to show that their nuisance parameter estimates are consistent. However, $\hat{f}$ and $\hat{\pi}$ are estimated using Equation (6) and thus it is up to the authors to show that solving (6) gives consistent estimates of $\hat{f}$ or $\hat{\pi}$ under correct model specification. This very well may be the case, but it cannot simply be assumed.

2. I found the introduction very hard to follow. In particular, it is never really made clear what the motivation for the work is. Extrapolating from the experiments, it appears that the goal is to derive an estimator with lower variance than existing estimators, but that is not stated in the intro. I would recommend restructuring to something like: CI from observational data is important because XYZ. It is desirable that estimators for the ATE have the lowest possible variance. We propose a new estimator that, empirically, has lower variance than a variety of state-of-the-art causal estimation methods. We do this by translating the exchangeability assumption into an explicit constraint on the estimator objective. This constraint reduces variance because ABC. Additionally, lit review portion of the intro reads as a random list of methods with no clear connection between them or to the proposed method. Some specific issues are:

    a. It is incorrect that "the outcome of an alternative treatment has to be estimated". In particular, IPW methods do not do this. In fact, methods based on estimating the conditional expected outcome do not do this either since, as stated above, the expected outcome is not equal to the outcome. 

    b. "to find similar subjects" --> "to find similar subjects who received different treatments"

    c. "seek weights such that the treatment assignment is unassociated with the covariates" --> "reweights the data so that treatment assignment is unassociated with the covariates in the reweighted distribution"

    e. "However, they do not require the...": None of the methods described here require this, but they do require that they be conditionally independent. Also, potential outcomes have not yet been defined.

3. Page 2, Contribution section, "Compared to other estimators, its asymptotic variance is strictly smaller.": It is my understanding that TMLE is also semi-parametrically efficient, so I believe this statement is incorrect as are similar statements in the "Comparison to other estimators" section. Further, both Chernozhukov et al. (2018) and the work on CV-TMLE show $\sqrt{N}$-consistency without relying the Donsker class assumptions.

--- Minor comments ---

1. In equations (4) and (5), the authors jump from a constraint on the true distribution to a constraint on the empirical distribution. I recommend adding a statement indicating this.

2. What score was used to select the best $\lambda$ on the validation set? 

3. RHS of Equation (11): $\epsilon$ --> $\hat{\epsilon}$ or drop the "evaluated at" bar.
review_title: Very interesting idea, but some technical issues

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a regularized framework for estimating the average treatment effect. They assume unconfoudedness and show that it implies a specific orthogonality constraint. The main idea is to use this orthogonality constraint during estimation of the model parameters as a regularizer. On the theoretical side, the authors provide sufficient conditions under which the regularization yields an asymptotically normal estimator for the average causal effect. Based on the regularization framework, an estimator for average causal effect via feedforward neural nets is developed.

- The motivation for the regularization is not clear. It is not clarified in the paper that why the proposed regularization should in fact improve the estimation bias and variance. It is only shown that the resulting estimator is asymptotically normal (the authors also show efficiency but it is under the strong assumption 2 in the theorem), but it is not clear that why this regularization would be particularly helpful.

- Page 4, the authors say "if we had access to the treatment effect \psi(X)=f(X,1)-f(X,0), we would also have access to the untreated outcome Y(0) even if we did not observe it. This does not seem to be true. (same for the footnote) f(X,0) is a function of X and if we take its expected value, under positivity, consistency and ignorability, it gives us the expected value of Y(0) not the random variable itself. 

- Then \psi(X) is replaced by its empirical mean and the authors say "Under sufficient conditions, this converges to the true outcome (see Section 4)". This does not seem to be what is proven in Theorem 1.

- How can we show that equation 11 implies equation 5, which was the original goal?

- Assumption 2 in Theorem 1 is very strong and I suggest removing it from the manuscript.
review_title: The motivation and some parts about the method are not clear

final_score: rating: 7: Good paper, accept
review_content: Summary:
The present paper introduces a new approach, deep orthogonal networks for unconfounded treatments (DONUT), that allows to estimate (average) treatment effects exploiting an orthogonality property implied by the classical unconfoundedness assumption. The authors propose a regularization framework based on the orthogonality constraint and prove that a resulting estimator is doubly robust, asymptotically normal and with efficient variance. They supply multiple simulations to demonstrate their theoretical claims and to show state-of-the-art performance of their estimator.

Recommendation:
Clear accept. In summary, I am convinced that the this paper would be a valuable addition to this year's conference. It considers a novel approach to improve average treatment effect estimation on observational data using combining classical causal inference assumptions and predictive power of deep learning.

Strong points:
 - The authors propose a new methodology that seems theoretically solid and that has an implementable estimator for ATE estimation, exploiting a necessary condition implied by a standard causal inference assumption.
 - The article is well written and easy to read.

Weak points:
 - The code for their simulations is not accessible (broken/incorrect url?).
 - A discussion about the impact of the hyperparameters, especially the orthogonality regularization parameter $\lambda$, would give more insight into the importance of the contribution of the regularization term.

Questions/Issues:
 - The provided url to access the anonymized code did not work (at least for me), would it be possible to fix this or to provide the complete code as supplementary zip file?
 - How sensitive are the results to the hyperparameter choices, especially the $\lambda$ parameter?
 - The orthogonality constraint is implied by the unconfoundedness assumption, but it is not a sufficient condition for unconfoundedness. Have the authors studied the behaviour of their method and its performance in (simulated) cases where unconfoundedness (4) does not hold but the orthogonality constraint (5) holds?
 - The authors mention the R-learner (Nie \& Wager, 2017) which uses the notion of Neyman orthogonality and use the R decomposition to propose an estimator of treatment effects. Would it be possible to add this method to the list of compared methods?
 - Since the authors theoretically compare their estimator to the IPW estimator, it would be interesting to add this to the experiments to confirm the theoretical results.
 - For the ACIC datasets, how does DONUT compare to BART, which is known to perform well on these data (Dorie et al., 2018)?

Minor comments (that did not impact the score):
 - p. 2/3: equations (2) and (4) are the same.
 - p. 5: yield by our $>>$ yielded by our

References:
 - Dorie V, Hill J, Shalit U, Scott M, Cervone D. Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science. 2019;34(1):43--68.
 - Nie X, Wager S. Quasi-oracle estimation of heterogeneous treatment effects. arXiv preprint arXiv:1712.04912, 2017.

===================

Post Rebuttal Update:

After the discussions and reading the other reviews, I lower my score by one point. I could not find the changes to the manuscript announced by the authors during the discussion, especially the additional intermediate results necessary for the theorem's proof pointed out by reviewer 1.
review_title: Interesting novel approach to combine deep learning and causal inference by leveraging implications from the classical unconfoundedness assumption.

Meta-review: This paper proposes a regularization term that enforces the orthogonality between (i) a residual between the observed outcome and its estimator and (ii) the treatment and propensity score. The method empirically performs competitively. However, there seems to exist a gap between the proposed method and the assumptions made to provide theoretical guarantees (e.g., R3, R2). R4 was also concerned about the issue and adjusted his/her score accordingly. Even though the authors provide a detailed discussion on most of the reviewers' concerns, some of the problems remain unresolved. Further, unlike other papers submitted to ICLR, the authors did not actually update the paper such that we could check whether the revisions were adequately made. As such, I believe this paper is not quite ready for publication in its current form.

Decision: Reject