Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies a multiply-robust estimate of the conditional average treatment effect (CATE), $ \tau(x) = \mathbf{E}[Y(1) - Y(0) \mid X=x] $ using a binary instrumental variable (IV) for identification. The approach is to construct a transformed outcome, $\hat{Y}_0$ such that $E[ \hat{Y}_0 \mid X=x] = \tau(x)$, and then use a machine learning regression model to estimate the mean of $\hat{Y}_0$ using $X$. The paper shows that such an approach can improve convergence rates over naive approaches such as plug-in estimates typically used for IVs.
strengths_and_weaknesses: The paper is reasonably well written. While it focuses a lot of attention on the convergence rates, which require following a lot of subtlety that makes it difficult for the reader to fully understand the main message, it is reasonably high quality and appropriate for publication on the topics of clarity and quality.

Unfortunately, the authors seem to have missed a key paper on this topic. Their abstract says "[t]o the best of our knowledge, our MRIV is the first multiple robust machine learning framework tailored to estimating ITEs in the binary IV setting." This is strange, as they cite  Syrgkanis et al., 2019 [29 in the paper], who provide almost an identical method for inference (up to some algebra and mild re-parameterization), show that it is multiply robust, and provide convergence rates under certain conditions, similarly emphasizing the dependence on the nuisance parameter convergence rates.

Edit: it appears that "multiple" (meaning more than double) robustness is the key innovation here, and that there is a nontrivial practical difference between it and double robustness. See the conversation in the reviews for more details. While the transformed outcomes are very similar, and have the same conditional mean, suggesting some way of transforming from one to the next, the differences are nontrivial, and allow for construction of different nuisance parameter estimates that allow for improved performance. The authors have clarified this in their updated version, and therefore, I believe the paper is appropriate for publication.
questions: How is this work different from DRIV method of Syrgkanis et al., 2019?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an algorithm for estimating ITE for binary instrument and binary treatment. The procedure has three steps.
-implement some initial estimator of ITE
-construct pseudo-outcome as sum of the initial estimator and a term combining several nuisances
-regress the pseudo-outcome on covariates
For this algorithm, the authors prove excess risk that scales as the sum of an oracle rate and several product rates.
strengths_and_weaknesses: Originality

I will raise the score if the authors improve the framing of the contribution in the context of previous work.

1. The pseudo-outcome technique is relatively popular in causal inference; see e.g. Kennedy et al.’s work on dose response (2019) which Semenova and Chernozhukov (2021) extend to ITE. This paper extends those techniques, so I would like them to be cited.

2. The main theoretical result extends an analogous result by Kennedy (2020), which somehow doesn’t get mentioned until Section 4.2. Moreover, citations for ITE without unobserved confounding could be more generous; it is a vast literature.

3. To distinguish the originality of this paper relative to Syrgkanis et al (2019), I would like to see a discussion that compares the estimation procedures and their guarantees. If I understand correctly, the Syrgkanis et al (2019) estimator proposes a Neyman orthogonal loss while this work used the pseudo-outcome technique, yet both provide excess risk rates that involve products of nuisance rates. 

4. Estimation that uses the efficient influence function to study ITE in the IV setting has been proposed by Ogburn et al (2015). Estimation that combines the efficient influence function with sample splitting and machine learning has been analyzed in the semiparametric IV setting, e.g. Singh and Sun (2019). This paper goes further than these previous works in proposing an ML estimator for ITE. I would like these previous works to be cited as well.

Quality

1. I will raise the score if the experiments are made to align more closely with the theory.

-line 284. Is there any sense in which the proposed nuisance estimators would satisfy Assumption 3?

-line 294. I would like to see the experiments with sample splitting since that is the proposal in the theory section

-line 313. The comparisons are hard to understand. The various estimators have different estimands. For example, DeepIV estimates a function that is not the ITE. So what is being reported here? These comparison estimators also involve sample splitting in some cases, to align with their theory, while the proposed estimator is not implemented with sample splitting (though its theory requires it)

2. I would like more discussion in the case study. In line 386, at what values are these variables fixed? Simply fixing values means that the quantity visualized in Figure 1 is not the ITE. Nor is it the ITE conditional on some subset of covariates. Please interpret what is being displayed.

3. The theoretical result, stated in Section 4.2 and explained in Section 4.3, is well written and interesting. I would like to see the additional assumptions of Theorem 2 stated in the main text, rather than pointing to another paper. Then the reader can assess whether they are “mild” or not.

Clarity

1. There were several instances in which the authors made statements that were either unclear or untrue as written. I will increase the score when these are addressed.

-line 25. The definition of unconfoundedness is incorrect. The given definition is an example of when unconfoundedness holds.
-line 92. Exclusion is not unrestrictive.

-line 107. The descriptions in Assumption 2 are misleading. The assumptions state that U can essentially be differenced out. It is an additive equi-confounding.  

-line 117 (footnote). The definition of compliers is incorrect. It is the subpopulation for whom A(1)>A(0).

-line 142. This statement is misleading. Arguably this paper requires a type of additive separability of confounding as well; see the comment above.

-line 223. This is a misleading citation. Cross-fitting is a type of sample splitting that has the same theoretical analysis. It is not a practical shortcut. The cited paper does not advocate fitting all nuisances on the same data set; that approach is only valid when the function spaces have limited complexity.

2. Please provide some intuition for the first term in equation 3. It helps to group \hat{\tau}_{init} and look at what residual is being extracted from Y before being reweighted.

Significance

My assessment of significance will hinge upon the authors’ improvements described above. There are currently too many issues for this paper to be a significant contribution, but I am optimistic that these issues can be addressed in the revision.

questions: Please address the items in Originality, Quality, and Clarity and I will increase the score
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a two-stage regression method: In stage 1, they estimates nuisance components $\{\pi(x), \mu^Y_0, \mu^A_0, \delta_A(x)\}$; In stage 2, given an arbitrary initial ITE estimator ${\tau}_{\text {init }}(x)$ and  nuisance estimates $\{\pi(x), \mu^Y_0, \mu^A_0, \delta_A(x)\}$  (obtained from stage 1), they define a pseudo outcome for ITE estimation.  Theoretically, the framework proposed in this paper yields multiple robust convergence rates. 

----
I thank the authors for addressing my comments. I increase my score to 6.
strengths_and_weaknesses: This paper builds upon the theoretical results and propose a deep neural network architecture called MRIV-Net for ITE estimation. However, the proof of the theory and Appendices A-G are missing, which I cannot find in the supplemental material. This makes me doubt the solidity of the method and results. I will adjust my score according to the authors’ responses to my comments.

**Strengths**

- This paper considers a very important and interesting problem in Treatment Effect and proposes a novel estimator using binary instruments. 
- The paper is very well written and provides extensive evaluation and discussion/comparison of past work. 
- Theoretically, I believe that Theorem 1 is correct, and MRIV has Multiple robustness properties. But I think it's necessary to give a brief description or simply express the process with an example: 

(1) Assuming $\hat{\mu}_0^Y=\mu_0^Y, \hat{\mu}_0^A=\mu_0^A, \hat{\delta}_A=\delta_A, \hat{\pi}=0.5, \hat{\tau}=0$: 

$\hat{Y}_0 \leftarrow\left(\frac{Z-(1-Z)}{\delta_A(X)}\right)\left({Y-{\mu}_0^Y(X)}\right)$;

(2) Assuming $\hat{\mu}_0^Y=0, \hat{\mu}_0^A=0, \hat{\delta}_A=\delta_A, \hat{\pi}=\pi, \text{and } \hat{\tau}=0$: 

$\hat{Y}_0 \leftarrow \left(\frac{Z-(1-Z)}{\delta_A(X)}\right)\left(\frac{Y}{Z {\pi}(X)+(1-Z)(1-{\pi}(X))}\right)$;

(3) Assuming $\hat{\mu}_0^Y=0, \hat{\mu}_0^A=0, $ $\hat{\delta}_A=1, \hat{\pi}=\pi, \text{and } \hat{\tau}=\tau$: 

$\hat{Y}_0 \leftarrow\left({Z-(1-Z)}\right)\left(\frac{Y-A \tau}{Z {\pi}(X)+(1-Z)(1-{\pi}(X))}\right)+{\tau}$. 

$\mathbb{E}[\hat{Y}^0∣X=x]=τ$. 

**Weaknesses**

- For Eq. (3), at least two models need to be correctly specified to guarantee robust learning, which is a stronger condition than robust learning. The latter only requires either the propensity model or the potential outcome estimation model to be specified accurately. 
- The proof of the theory and Appendices A-G are missing, which I cannot find in the supplemental material. This makes me doubt the solidity of the method and results. 
- This paper should introduce the concept of multiple robust in the Introduction Section, and roughly explain that the original problem is simplified to 5 nuisance estimators —— the solution is still robust even if some models are misspecified. 
questions: In Figure 1, is it necessary to assume that X and U are independent? 

Lines 79-82: Does this article focus on observational data or experiments data? 

$\hat{Y}_0$ is usually used to denote $\mathbb{E}[Y|T=0]$ instead of ITE $\tau$, which is confusing. 

Typos: Line 157: “We then derive we derive …”

Maybe the author can give a brief conclusion and future exploration. 
limitations: I do not foresee any major limitations and/or societal impacts.
review_title: null

Meta-review: metareview: The authors offer a methodologically novel and very interesting approach to an important problem of CATE (or conditional LATE) with binary instrument/treatment. The paper should be published at a good ML venue.

Unfortunately, I need to recommend rejection primarily because the key proofs supporting their main theorems were missing from the original supplementary material and were only submitted after the rebuttal. Given that these were crucial elements supporting their main results and that they were submitted post the original submission deadline it seems unfair (even if it was most probably an honest mistake of the authors). 

Another issue that came up in the discussion phase that seems crucial to revise before a resubmission is that the authors main estimation rate result relies on a main theorem of the unpublished work of Kennedy 2020, which has since been revised and the new theorem in Kennedy 2020, is technically very different and requires different assumptions. Hence the authors need to revise their main estimation rate result accordingly. 

However, I acknowledge that this is not the main contribution of this work and a simple invocation of past work, but rather the main contribution is to formulate a loss for CATE using the idea of Want and Tchetgen-Tchetgen. So this wouldn't most probably be a reason for rejection.
confidence: Certain
Decision: Reject