Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors introduced flexible and scalable deep learning methods to estimate causal effects in the presence of unmeasured confounding using proximal inference.
strengths_and_weaknesses: Strengths: The authors proposed a new deep learning method for proximal inference: neural maximum moment restriction; the authors show desirable numerical performance of their methods; the authors provide a consistency result as a theoretical justification. 
Weakness: The authors could give some practical guidance on how to choose their proposed two methods; the authors could potentially strengthen their theoretical results. 
questions: * The authors proposed two methods, U-statistic and V-statistic, for proximal inference. The authors should comment on the pros and cons of each method. For example, in simulations, is there a reason why we would expect V outperforms in the former experiment and U outperforms in the latter one? 

* Recently there was inverse probability weighting and doubly robust estimators proposed for proximal inference (Cui et al, 2020). (Kallus et al, 2021; Ghassami et al, 2021) then considered kernel methods for doubly robust estimators. Would the proposed deep learning approach extend to inverse probability weighting and doubly robust estimators? Some comments on this would make the proposed approach more general.

* The authors provide a consistency result for their deep learning approach. What kind of convergence rate results such as that of generalized method of moments (Dikkala et al, 2020) could be expected? 
limitations: See weakness and questions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new estimator for the proximal inference problem. When the treatment is continuous, the object of interest is a function of treatment. The most closely related work appears to be [9, Zhang et al 2020], which studies the related problem of nonparametric instrumental variable regression. Like [9, Zhang et al 2020], the “first stage” is eliminated by introducing a kernel matrix into the loss—operationalizing an insight from [8, Muandet et al 2020]. All that remains is the “second stage” of minimizing the loss, for which the authors propose a neural network. The loss either contains cross terms (V statistic) or omits them (U statistic). The authors demonstrate state of the art performance in simulations. Theoretically, the authors outline the argument for consistency, but some terms in the bound are not analyzed. 
strengths_and_weaknesses: Originality

The authors emphasize that their implementation and analysis of the U statistic loss with neural networks departs from previous work. Indeed, the performance in simulations is impressive relative to previous work. However, the algorithmic innovation is not very substantial given that previous works mention the U statistic loss (though without fully developing the results like this work).

The authors also emphasize that the theoretical proof of consistency is original. I agree that a strong guarantee is missing from the previous work on U and V statistic losses for neural networks, and such a result would be original and significant. However, I have some issues with the quality of the result the authors provide.

Quality

I have some substantial concerns about the main result, Theorem 1, and some minor concerns about the subsequent result, Theorem 2.

In Theorem 1, has M_k been defined? Is ||h||^2_2 the empirical norm E_n[h(…)^2]? And what norm is ||…||_{P}. Is this the L2 norm?

The bound of Theorem 1 contains Rademacher complexities. The authors write “we cannot compute the scaling of the Rademacher complexity directly”. Do I understand correctly that we therefore do not know that the bound vanishes as n increases? What worries me is the possibility that the quantities in the bound are may not be the right ones for analysis. In ill posed inverse problems, finding the right quantities for analysis is the heart of the problem. Can the authors provide at least some sketch of why they are the right quantities?

In Theorem 2, are the norms of convergence L2? I am surprised that this result has no mention of ill posedness or properties of the kernels for A,X,Z, which are described in [17, Dikkala et al 2020], [11, Ghassami et al 2022], and [Kallus et al 2021]. Without any further assumptions, are we allowing the rate to be arbitrarily slow for h, e.g. even logarithmic rates? This point warrants discussion.

Clarity

Lines 9-10 describe how prior work has pre-specified kernel functions which are not data-adaptive and which struggle to scale. Doesn’t the proposed method also have pre-specified kernel functions that manifest in the kernel matrix in the loss? 

Line 30 says positivity can be empirically verified. This is only partially true; please revise the language.

Line 39 says the groups are mutually exclusive. This is not precisely true; some variables in the graph may be classified as either Z or X, for example. See [5, Tchetgen Tchetgen et al 2020]

In the related work, please include the additional references of [Kallus et al 2021] in the paragraph that mentions doubly robust methods for proximal inference and [Singh 2020] in the paragraph that mentions two-stage procedures.

Line 177 “has a slight amount of bias, but in practice is much less biased…” this is a bit imprecise. Do you mean in simulations?

Line 180 is very nicely written.

Significance

On the one hand, this paper provides convincing simulations that demonstrate that a U statistic loss for neural networks outperforms previous work. On the other hand, the proposed algorithm is a straightforward extension of previous work, and the main theoretical question in this literature remains unanswered. If my main theoretical concerns can be ameliorated, then I will increase the score.

questions: Please address the issues under the heading Quality for an improved score
limitations: --
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes the extension of PMMR, called NMMR-V and NMMR-U, which learn the bridge function in the proximal inference represented by neural net models. Although NMMR-V minimizes the same empirical loss as PMMR, which can be understood as the V-statistic of population loss $R(h)$,  NMMR-U minimizes U-statistics of $R(h)$.  The paper provides the theoretical results and demonstrates the superior performance over existing methods.
strengths_and_weaknesses: Strength:
- The extension of PMMR to NMMR is novel though it seems slightly straightforward.
- The analysis of consistent results seems novel and important to me. This might be of independent interest since it can be used in the setting where the U-statistic loss is used, such as preference learning.

Weakness:
- Assuming "discriminator" function $g$ to be RKHS seems slightly restricting to me. When the data becomes high-dimensional, it is known that kernels based on Euclid norm can be highly sensitive to the noise and degenerate the performance.
- It seems that the different network structures are used in the experiments, and it might be difficult to assess whether the NMMR method really contributes to the performance.
- The presentation can be improved. For example, it repeats the long inline equations in the text, which makes me difficult to go through the paper. Also, in the Demand design setting, it reports the performance of methods that uses Z information in the test time, which is not supposed to be available in the proximal setting. This might mislead the reader, who might think non-causal methods perform better than baseline methods in the proximal method.
questions: - Why the function $g$ is assumed to be RKHS?  I wonder whether we can learn $g$ by maximizing the loss $R_U(h)$. A similar trick is used for the IV setting in the following work.

Deep Generalized Method of Moments for Instrumental Variable Analysis
Andrew Bennett, Nathan Kallus, Tobias Schnabel

- It seems that convolutional NN is used for NMMP methods in dSprite setting, but not for the DFPV method. How does DFPV perform when it is run with the CNN structure?


limitations: I think the adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: Reviewers agreed that the paper proposes a new and valuable method for proximal causal inference with a solid theoretical analysis. The reviewers pointed out several ways in which the theory might be improved, some of which have already been undertaken by the authors, while some remain open. A potential drawback is that the authors were originally unaware of highly related work (especially Cui et al. 2020 and Kallus et al. 2021); while the authors have now addressed this work, I suggest that in the final version they add further details about the differences between their results and those of the above papers. 
Having said all that, we all view the contribution positively and believe its merits outweigh its drawbacks .
confidence: Certain
Decision: Accept