Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the problem of estimating an expectation over covariates of some functional of an unknown regression function. The authors propose two estimators, one based on neural networks and one based on random forests. In contrast to many (all?) previous estimators which were derived for specific functionals, these estimators are applicable to general functionals. They employ a (new?) debiasing technique. Moreover, they use a novel multi-task architecture based on the observation that it suffices to estimate the regression function as a function of the Riesz representer. The paper demonstrates improved accuracies compared to prior work and good coverage on semi-synthetic tasks derived from two data sets.
main_review: **Strengths:** The method presented in the paper is applicable to a broad class of problems. The main idea presented in Lemma 1 is easy to understand and is applicable to other machine learning methods as well. The presented methods perform well in experiments, exceeding the performance of two alternative methods, although this comparison is only performed on synthetic variations of one real-world data set. Code is provided.

**Weaknesses:** As someone who is not familiar with the type of problem discussed in the problem, I found some parts of the paper confusing (see below). Some explanations might be missing altogether unless I missed them. I think the paper could benefit from investing more time in the writing, including the notation. Additionally, in my opinion it could be stated more clearly what is and what is not novel about the presented methods (see below). Moreover, it appears to me that the paper hints at some theoretical guarantees in the beginning but then does not really discuss what theoretical guarantees can be given for the presented methods.

**Specific comments / questions:**
1. In the first eq. on page 1: The notation is confusing to me. It is not clear which variables the expectation is taken over. I assume the expectation is over $Z$. Also, it is not clear that $m$ even depends on $Z$. Moreover, in the following examples it is not clear whether $W = w$ or $W = w(Z)$ or something else. It would probably more clear to write something like $\theta_0 = \mathbb{E}_Z [m(Z; w, g_0)]$, or is it even $\mathbb{E}_Z [m(Z; w(Z), g_0)]$?
2. It seems to me that $w$ is always fixed throughout the paper except for the second function space definition in Eq. (3). Hence, it is not clear to me why it would not just be absorbed into the definition of $m$, such that we can write $m(g_0)$. Is the explicit dependence on $w$ in Eq. (3) needed to show that the resulting constant $C$ is uniform over different choices of $w$?
3. The semicolon notation $m(W; g_0)$ is not used consistently throughout the paper.
4. Sec 2: "This framework gives rise to a score": Which framework? Is it new or from the literature? In general, the steps here are not cited and the formulation does not make it clear whether they are novel. 
5. Eq (1): What does the index i mean in this context? Is it fixed or summed over? And why does $W$ even have an index? Is it because $W_i = w(X_i)$ ($W$ is not explicitly defined anywhere)? Explaining this might also make the paragraph below more understandable.
6. Theorem 1: Why does $\delta_n$ depend on $n$ even though the function spaces do not depend on $n$? Why is the assumption on all $f$ in the spaces of Equation (3) not formulated directly after the definition of the function spaces? Why would it be reasonable to assume that all functions satisfy $\|f\|_\infty \leq 1$ in the considered settings?
7. Theorem 1: In the mean-squared continuity property equation, is the square root intentionally only taken on the left-hand side and not on the right-hand side? In this case, I would not call it "mean-squared continuity" because then it is not the same as in the beginning of Section 2.
8. Sec 4: I don't understand the first paragraph. "One could potentially minimize the Riesz loss ..." - Why would an empty node change the Riesz loss? In general, the Riesz loss is formulated in terms of $Z = (X, T)$, making no distinction between $X$ and $T$. Why would there arise problems in $T$ that do not arise in $X$? Also, in the statement "Such discontinuous in $T$ function spaces will typically not satisfy the mean-squared continuity property", you could refer to the corresponding equation in Theorem 1, or even explain why this is the case.
9. Eq. (9): What does conditioning on $X = x$ mean for the empirical expectation $\mathbb{E}_n$? Is it only allowed for $x$ from the training set?
10. Figure 1: It seems (at least by text search) that the abbreviation IPS is never explained. Ideally, the abbreviations would be explained in the Figure caption. Is there any reason why only MAE is used for evaluation even though the methods are trained on RMSE? If you want to make Figure 1 look more like a typical comparison table in the Deep Learning literature, you could consider the following suggestions:
- Put both random forests and NNs in a single column, name them RieszNet-DR, RieszNet-Direct etc., and cite the sources (e.g. RieszNet-DR (Ours) or Dragonnet (Shi et al., 2019))
- Mark the best-performing method(s) in bold (the differences seem to be at least partially significant)
- Change the LaTeX environment from figure to table

Why is it not possible to have coverage estimates in this setting? What does it mean to "redraw the treatment" to get coverage estimates?

11. Figure 2: I think you should label the axes, and maybe not put experimental results in text form above the plots.
12. Why do you use the auxiliary loss in eq. (7) if you could also use the debiasing term from Eq. (1)? Do I understand it correctly that neither of the debiasing methods are novel?
13. You could mention the experimental evaluation at the end of Section 1, to have a more complete exposition of the contributions of the paper. 
14. Sec 2, 3rd paragraph: Why does it debias the naive plug-in estimator? Is it explained anywhere, or in the literature?
15. Could you discuss what theoretical guarantees can be given for (idealized versions of) the presented methods? What can be said about the infimum in Eq. (4)? The asymptotic statement $\delta_n = O(d_n n^{-1/2})$ is not helpful if it is not clear how $d_n$ can behave. Moreover, the constraint $\|f\|_\infty \leq 1$ seems to be restrictive, and even in the best case, the conclusion of the theorem gives what I suppose is meant by parametric $\sqrt{n}$-rates only for $\alpha$ and not for $\theta$.
16. How are the confidence intervals computed?
17. In my opinion Table 2 and Table 1 might be better placed in the Appendix, with perhaps some smaller summary table/figure in the main part of the paper. Also, making the best values bold might aid interpretation of the results. The gained space in the main paper could be used for explaining some other parts in more detail.
18. I did not find any details on the training of the neural networks. Although they are of course contained in the supplementary code, maybe they should be briefly described in the appendix. (Number of layers, optimizer, activation function, etc.)
19. Since the method is not that easy to summarize in the introduction, the paper might benefit from a conclusion section.


**Minor comments:**
1. Example 3: It should be $Z = (T, X)$ instead of $X = (T, X)$
2. Sec 2: "mean-square continuous functional" -> "mean-square integrable continuous linear functional"?
3. Sec 2, second paragraph: maybe it should be $f_0$ instead of $f$
4. Sec 2: Could mention here that $\alpha_0$ is in general unknown and will be estimated.
5. Sec 2: "This framework gives rise to a score": One line uses $\theta$ and the other one $\theta_0$. Is this intentional?
6. Eq (3): should this be a $\mapsto$ instead of a $\to$? Why $a \in \mathcal{A}$ and not $\alpha \in \mathcal{A}$?
7. Sec 2: "focus on objects" -> "focus on problems"?
8. Figure 3: Orthogonality has never been discussed in the paper, or is this standard to report in the literature?
9. Lemma 1: You could place the qed symbol in the same line as the last equation (using \qedhere).
10. The terms "mean-independent" and "parametric rates" were unclear to me. Also, for non-mathematicians, it might be helpful to explain a bit more about Riesz representers.
summary_of_the_review: The paper contains interesting new methods which exhibit very good results in a limited experimental evaluation. However, the exposition is lacking in clarity to me. Therefore, I would currently classify the paper as borderline, but I am open to adjust my score based on the authors' response.

**Edit**: Due to the author's response, I raise my score from 6 to 8.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors address the problem of estimating the average value of a moment function that depends on an unknown regression function, which is commonly used in causal inference. 
By the Riesz representation theorem, the authors design a loss function where the Riesz representer turns out to be the minimizer of the proposed loss. 
The authors propose both a Neural Network method (RieszNet) and a random forest method (ForestRiesz), i.e. a multi-tasking Neural Network method where the loss function is a combination of the Riesz representer and regression loss, and a random forest method that leans representation of both the regression function and the Riesz function. 
The authors conduct experiments of estimating the average treatment effect and average marginal effects, and show that the proposed RieszNet and ForestRiesz beat the state-of-the-art methods.



main_review: 
Strengths
- The proposed methodology that the estimation problem of the moment function is converted to the problem of learning the Riesz representer seems to be novel. To learn the Riesz representer, the authors introduce a loss function according to a theoretical result of recent work.
- The author proposed two methods to learn Riesz representer, including a Neural Network method (RieszNet) and a random forest method (ForestRiesz). The experimental evaluation seems to be very strong and can achieve state-of-the-art performance in estimating the average treatment effect.

Weakness
- The notations in this paper are very confusing. For example, $W$, $W_i$, $w_{1:k}$, $w_{1:d}$, $w_{k:d}$ are never defined in this paper. The notation $w$ is abused, referring to a function in Examples 2 and 4, but refering some certain values with a subscript such as $w_{1:k}$.
- The contributions of this paper is unclear. It would be better to clarify which parts are the contributions of this paper and which parts are results from previous works. Some parts of this paper are the conclusions of previous works, while some parts of this paper are methodologies and discussions of comparing methods, but these parts are all mixed together.
- The detail steps of the proposed RieszNet and ForestRiesz should be clarified. For RieszNet, if the network architecture and training details of RieszNet are similar to Shi et al. (2019)? What $w_{1:k}$, $w_{1:d}$, $w_{k:d}$ are referred to, and after the minimization of equation (7) what should we do? For ForestRiesz, if readers are not familiar with Athey et al. (2019), it would be very hard for readers to follow the idea of ForestRiesz.
- Although the paper aims to solve a general problem of estimating the average value of a moment function, the focus of this paper is still causal inference (example 1 average treatment effect). In the experimental part, the paper only evaluates the average treatment effect and average marginal effects, which are both problems in causal inference.



summary_of_the_review: The idea of this paper is original, and both RieszNet and ForestRiesz show strong experimental evaluations. But the paper needs to be well-organized. Therefore, I would recommend acceptance of this paper if the paper was well-written.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors study the nonparametric estimation of an important class of (causal) estimands that includes the average treatment effect (ATE) in experiments and observational studies under unconfoundedness. The main innovation is the "automatic" nature of the procedures, one based on deep learning and one on random forests. While previous work has developed hand-tailored constructions for the ATE (a traditional problem in statistics), the authors show that existing constructions can be generalized considerably based on recent advances in semi- and nonparametric statistics based on Riesz representers. The key challenge this work addresses is the fact that the Riesz representer is typically unknown.

The main proposed method based on deep learning, RieszNet, may be seen as a generalization of the DragonNet procedure by Shi et al. (2019) from the ATE to more general estimands. Even in the case of the ATE, DragonNet and RieszNet are not the same and RieszNet outperforms DragonNet in simulations: RieszNet directly targets the inverse propensities, while DragonNet estimates the propensities and then plugs in their inverse.
main_review: I like the premise of the paper. I think it is an important step towards making the estimation of policy-relevant causal estimands more accessible to practitioners working in data-rich environments in which nuisance functions are expected to be complicated. Although conceptually and theoretically most of the content of the paper is already known / described elsewhere, its novelty lies in combining multiple ingredients to propose practical methods and provide a concrete implementation.

I have some suggestions for improving the paper:

1) Compared to other papers in this line of work, it appears to me that the scope of the paper is relevant to a broader audience and especially so for applied data scientists. Yet, 
the paper feels rushed (more on this below), e.g., with a lot of notation remaining undefined. The consequence is that the paper is not self-contained and would be very hard to understand, even for somebody with background knowledge in causality (who is not aware of the latest theoretical developments). Exposition is very important for a paper that could reach a broad audience! 

2) I felt the description of the methods is a bit too short and probably one would not be able to reproduce the results based on the text description. This is particularly the case for the ForestRiesz procedure which seems to me to be less "automated" than promised.

3) There are many ingredients to the proposed procedures (beyond what I described above) and from the experiments it is not clear which ingredient is relevant (or if all ingredients are relevant). Ablation studies would be very helpful here for the reader.

Below I summarize these three points in more detail:

# Exposition

1) The introduction is very short and one immediately is faced with a formula, the components of which are never defined. It would be very helpful to at least define potential outcome notation early. $W$ is also never defined.

2) I think it would be very helpful to a reader to discuss the assumptions of Theorem 1 in more detail. What is the take-away for a practitioner? What should they be careful about?

3) After Lemma 1, please make sure to cite Rosenbaum and Rubin (1983). I think it would also be helpful to make a remark of how Lemma 1 is used. The "naive" (and perhaps traditional) approach would be to just nonparametrically regress on $\hat{\alpha_0}$. But this requires very small estimation errors for $\alpha_0$. Instead the innovation is the use of e.g., the last layer of the neural network, which may still be capturing all relevant conditioning information!

4) The first paragraph of page 4 talks about cross-fitting and double cross-fitting. What is actually being used in the implementation? For example if double cross-fitting is being used, then I think the reference to that could be removed (the audience for whom this point is relevant would likely already be aware of this point and practitioners would use the "automatic" software).

5) In Section 3, I think that having a figure on the architecture (similar to Fig. 1 of Shi et al.) could be very helpful for a reader.

6) Page 8, Riesz representer with Gaussian errors: It could be worthwhile to point out here that this is the case by Stein's identity (and point a reader e.g. to a supplement that explains this).


# Details for procedures

1) I think it is very difficult to follow what ForestRiesz is doing, e.g., it is not explained what is being computed in the leaves. 

2) Also the caveats and description in the first paragraph make the whole procedure seem less "automatic" and the implementation appears to need substantial additional modifications to account for treatments. This makes the procedure less appearling than RieszNet.

3) Please provide more details for the simulations and implementation of procedures, e.g., how many folds are being used? Where is IPS defined? Could there be more details for architectures and tuning parameters chosen? What is the "RF Plug-in" benchmark? Is it the default recommended method in the GRF package?

# Ablation studies

For example for the ATE simulations I would be interested in ablation studies showcasing the following aspects:

1) What is the impact of the regularization term for RieszNet? 

2) What is the impact of cross-fitting? Shi et al. (surprisingly to me) claim that cross-fitting does not help for their DragonNet architecture.


# Some potential Typos / minor points

* Page 2, "integration by parts gives" -> Is a negative sign missing?
* Page 2, Linearity after the first eqn. of Section 2?
* Page 3, after eqn. (3), "square" on the RHS?
* Page 4, before eqn. (6), $(k+1):d$ instead of $k:d$?
* Check formatting of references, e.g., Grathwohl et al. after example 4 should presumably be in a parenthesis?






summary_of_the_review: This paper provides a solid contribution towards the automatic estimation of policy-relevant causal estimands. I think this work should be accepted. However the paper could be substantially improved by improving exposition and making the paper accessible to a wider audience.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper shows novel methods based on the recent findings on the Riesz representation in econometrics and machine learning, such as Chernozhukov et al. (2021). While the proposed applications make sense and are persuasive, the contributions of this paper are a bit limited. Although the authors support the soundness of the proposed algorithms in experiments, the experiments are a bit simple and may not be sufficient for the justification. 
main_review: Main review:
- The proposed method is very innovative and important in the field. This paper will be of great value to the field. 
- On the other hand, it is difficult to evaluate the novelty of the paper because many of the results rely on existing studies, such as Chernozhukov et al. (2021). 
- This paper may be more of a summary of the existing studies.
- If the contribution of this paper is the proposal of the methods, then the experiments to show the validity of the methods will be important as the contribution of this paper. 
- However, the experiments are very simple, and it is not clear whether they are superior to other methods. 
- Even though the performance depends on the architecture and hyperparameters of the model, there is a little description of them.
- Besides, although the proposed method is persuasive, it seems to me that the paper is not well organized. There were several points that I couldn't catch up on. For instance, 
   - What are the definitions of $Y$, $X$, $Z$, $W$, and $T$? We can guess the meanings, but the rigorous definitions are needed.
   - In Example 3,  $X=(T, X)$ should be $Z=(T, X)$?
   - Theorem 1 is cited from Chernozhukov et al. (2021), but there is no description about which theorem is used from Chernozhukov et al. (2021).
   - In the first set in Equation (3), the authors might have forgotten to add $\gamma \in [0,1]$.
   - In page 4, what are the definitions of $\beta$, $w_{1:k}$, $w_{1:d}$, $d$, and $k$? I can guess about them, but the clear definitions are needed.
   - Equation (5) and (6) are correct? The LHS of these equations are not functions of $w$ because they are minimized over $w$. I think that $min$ operator should be removed.
   - In the RHS of Equation (5), $\theta$ is $\beta$?
   - What are the definitions of $f_2$ and $f$?

Questions: 
1. Can we apply the proposed method to estimate the conditional (potentially heterogeneous) average treatment effect?
2. Can we check that the conditions of Theorem 1 hold? In particular, the multi-task neural network architecture can satisfy them?
3. The estimator has $\sqrt{n}$ consistency? I think that we can derive the convergence rate from Theorem 1, which states the convergence rate of $\|\hat{\alpha} - \alpha_0\|^2_2$, but it is not clear, at least for me.

Minor points:
1. Figures 2 and 3 are a little difficult to see because the histograms overlap. How about using boxplots?
summary_of_the_review: I find it difficult to evaluate the contribution of this paper because it is not clear what the contribution of this paper is. The method is novel, but the theory relies on existing different work, so I think the evaluation of the method highly relies on the novelty of experiments. However, the experiments are very simple and not enough to claim the usefulness of the method. If the methodological and theoretical contributions are significant, the paper should be highly evaluated even if the experiments are simple. However, in this case, because this paper has less novelties in  the theoretical part, the experiments are important. In summary, the method in this paper is very novel and interesting, but the experiments to support it are lacking. Besides, there are several issues in writing even though we can guess what the authors insist. This is the reason why I vote for weak rejection.
review_title: null

Meta-review: In this paper, the problem of estimating the average of a moment function that depends on an unknown regression function.
It heavily relies on prior papers by e.g. Chernozhukov et al. and the actual novel material consists of making these theoretical 
results more practical. Experiments for two practical approaches based on neural networks respectively random forests are also reported.

Initially, the presentation of the paper was heavily criticized by the reviewers, but during the rebuttal phase at least some of the issues were removed. Together with some other improvements this lead to an increased average score. However, it seems fair to say that reading the other papers first, is still kind of necessary.

Despite the still unclear novelty the paper has some merits, which in principle make it acceptable. Compared to the other good papers in my batch, however, it is more incremental and the overall contribution is not as strong. For this reason I vote for rejection, but a comparison to other papers outside my batch is probably a good idea.
Decision: Reject