Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper examines a semi-synthetic data generating process, referred to as IHDP, that is frequently used to benchmark estimators of the Conditional Average Treatment Effect (CATE), which is a common target in causal inference. The authors compare several existing estimators/algorithms and show that their relative performance depends on the parameters of the data generating process, which vary randomly over simulation runs. They argue that IHDP naturally favors some estimation approaches over others, and that over-reliance on this benchmark may give the false impression that some estimators are uniformly better than others. They summarize other proposed approaches to benchmarking CATE estimators and discuss their advantages and disadvantages.

The primary contribution of this paper, in my view, is the empirical results, which yield some insight into how different algorithms perform on IHDP (although plenty of empirical results using IHDP exist already). However, the conclusions drawn with respect to IHDP in particular seem too strong, while the broader conclusions--that any benchmarking approach has limitations--seem fairly self-evident. Overall, I do not think the contributions are substantial enough to stand as a full paper at NeurIPS. [EDIT: following the author's remarks and revisions, I'm raising my rating from a 3 to a 5.]
strengths: The paper is well written and well organized, and it is clearly motivated. The empirical results are compelling in that they directly address the motivating example, and they clarify and extend the previous results that are cited there. I think researchers working in this area, including those who are tasked with crafting future causal inference estimation challenges (which is one of the settings the authors cite), will be interested in these results. I think that Section 3 almost could stand alone as a research note in an appropriate publication venue.
weaknesses: While the empirical results are compelling, they point to a conclusion that seems to me fairly self-evident, which is that any given data generating setting will favor some estimators over others.

Finding (i), that the histogram of RMSE values over simulation runs has a long right tail, does not seem surprising, and plenty of papers CATE papers (e.g. Hill, 2011, which first proposed IHDP) also display the distribution of performance values over simulation runs rather than merely reporting averages.

Finding (ii) seems to me to be overstated. We can't conclude that "indirect" (aka plugin) estimators are better than "direct" (e.g. two-stage/doubly robust) estimators on IHDP, and we can't conclude that neural nets are better than random forests in this setting. we can only conclude that the particular learners the authors used (random forests and neural nets, with some set of hyperparameters) performed worse in the particular two-stage estimators they examined. In particular, the performance of the indirect estimators decomposes entirely into the performance of the constituent regressions, a point that the authors make in Section 2. So, when TNet outperforms TRF, for example, it presumably just means that a neural net did better at estimating one or both of the regression functions. Changing the hyperparameter settings could potentially reverse this relationship, so the statements about neural nets vs random forests don't seem warranted.

Findings (iii)-(iv) are interesting and useful, if not necessarily surprising. While they shed light on the previous findings wrt neural nets and random forests on IHDP that are cited as motivation, I don't think they yield any insight into any unique limitations of IHDP, or of (semi-)synthetic datasets in general. They simply show that different algorithms work well on different datasets, which is true of any real or synthetic data. The authors make this point themselves in the paper, but the abstract seems to promise a critique that is unique to semi-synthetic data.

The remainder of the paper is largely a summary of existing proposals for ways to construct benchmarks and to evaluate CATE estimators. Re: Section 5, the optimal treatment regimes literature typically uses measures other than RMSE to evaluate estimators, and, though I'm less familiar with this literature, I believe many papers also evaluate performance with respect to predictive or prognostic value specifically, when those are the target quantities. The use of RMSE as a performance metric naturally isn't appropriate to all scenarios, but it is certainly of interest in some scenarios. I'm not quite convinced that the literature has failed to distinguish these cases.
correctness: I believe the claims in the paper are all correct, with one exception.

Namely, I don't follow the logic behind the claim that the as the two regression functions become less additively similar, the direct estimators will increasingly outperform the indirect estimators. The CATE is at least as smooth as the less smooth of the two component regression functions, which is why direct estimators are advantageous in many settings (in an asymptotic sense). Rather, it may be that the asymptotic advantages of the direct estimators over indirect estimators diminish as the difference in smoothness between the CATE and the outcome regressions decreases. Things may of course behave differently in a finite-sample sense, but the claim that "...this is expected given the DGP..." under Finding (ii) does not seem correct.
clarity: The paper is very clearly written and easy to read. My only comment here is that acronyms like DGP and PEHE should be spelled out when they are first introduced.
relation to prior work: Yes, the work is clearly contextualized in the literature.
additional feedback: No additional feedback
review_title: Some useful empirical results, but many conclusions are stronger than warranted

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: "Really Doing Great at Estimating CATE? A Critical Look at ML Benchmarking Practices in Treatment Effect Estimation" is an eminently practical and extremely readable paper documenting and theoretically grounding many issues with benchmarking practices for ML-based CATE estimators. It is critical of current practices, but also constructive.

I found the paper to be extremely readable and am sure I will refer back to it in part as a literature review and guide to CATE estimation across fields. I particularly appreciated the synthesis and perspective the authors provided across disciplines, allowing the reader to gain a more unified understanding of CATE estimation across econometrics, statistics, and ML as a part of reading the paper. 

The central findings of the paper, that popular benchmark datasets will favor certain algorithms over others in a way that is problematic for the field if it continues to blindly use these benchmarks, are clearly presented and persuasive. The authors provide the reader with enough theory to grasp exactly where and how those issues arise from the data generating process, and thus provide a framework for researchers who still wish to use a popular benchmark to properly contextualize their findings, tweak certain experimental knobs to check robustness, etc. 
strengths: As noted above, the paper has many strengths. I appreciated the lengthy discussion of alternatives to synthetic benchmark data (Section 4), in which the authors give a clear discussion of strengths and weaknesses to proposed alternative benchmark dataset generating approaches, allowing the researcher a detailed guide to find an approach most suited for their setting/question.

I also appreciated the attention paid to alternative evaluation metrics (Section 5). Better metrics would help bring the evaluation of CATE estimators potentially more in line with the purpose of those estimators, for example in identifying individual characteristics with prognostic versus predictive value, and  looking at how well the estimator performs at making real-world predictions.

I appreciate the guide the authors provide for how a researcher might continue to use existing benchmarks with (semi-)synthetic data in a more thoughtful way, for more limited, carefully-chosen applications, and with an awareness of and discussion about the limitations.

Finally, I appreciated the suggestions for how benchmarking and dataset-generating practices could be improved by collaboration with domain experts.
weaknesses: Authors should be careful to define acronyms before using. 

Given its importance to the paper, should define the acronym IHDP (Infant Health and Development Program) when it is first used, rather than on page 4 after the acronym has been used multiple times.

Similarly, "DGP" is used before it is defined, and it is defined incorrectly: Line 114: "The most common approach for simulating outcomes seems to be to rely on data generating processes (*DPGs*) with additive effects"
correctness: They appear to be correct.
clarity: Extremely well written; I found it pleasurable to read. The one major caveat is the issue with jargon-y acronyms like DGP being used before being defined. Because I'm a researcher with work in this space, this did not affect readability for me much, but I imagine it could affect the paper's readability for some others.
relation to prior work: Prior work is discussed. The authors could perhaps be clearer on which of their results, for example on data generating processes being highly influential to what kind of algorithm performs well on a popular benchmark, have been documented elsewhere in any form. For example, I was curious about how much overlap [23] has with their work.
additional feedback: None
review_title: Excellent, useful, critical, and constructive paper on current practices in evaluating CATE estimators

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper presents a benchmarking study for estimating conditional average treatment effect. Experiments are conducted on IHDP, and several useful conclusions are presented. The paper emphasizes synthetic datasets have problems in providing fair and insightful benchmarking results.
strengths: (1)	Experiments are conducted on IHDP, and several useful conclusions are presented.
(2)	The paper emphasizes synthetic datasets have problems in providing fair and insightful benchmarking results. This is a key problem of existing research.

weaknesses: I think this paper provides limited contribution as a benchmarking paper:
(1)	Experiments are only conducted on one dataset. This seems not thorough enough for a benchmarking.
(2)	This paper presents the disadvantage of using synthetic datasets. But no solution or new datasets are provided. This limit the contribution the this paper, and this key conclusion is somehow straightforward.

correctness: Claims apears to be correct in this paper.
clarity: The paper is well-written and easy to follow.
relation to prior work: It is clearly discussed how this work differs from previous contributions.
additional feedback: How to construct a better non-synthetic dataset for the CATE problem?
review_title: Useful results but with limited contribution.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors argue that CATE estimation algorithms have been evaluated only on a very limited set of semi-synthetic benchmark datasets, even though the literature grows rapidly. So the authors investigate current benchmarking practices for ML-based CATE estimators, with special focus on empirical evaluation based on the popular semi-synthetic IHDP benchmark. The case studies found problems with current practice and highlight that inherent characteristics of semi-synthetic benchmark datasets can favor some algorithms over others. There are also some other issues that should be taken into account in empirical studies. Finally, the authors discuss alternatives for current benchmark datasets, and implications of our findings for benchmarking in CATE estimation.
strengths: I think the biggest strength of the paper is to provide in-depth analysis on the dominant benchmark semi-synthetic dataset, IHDP data (and ACIC data in the appendix). In particular, the authors argue that the DGP underlying the IHDP benchmark indeed favours both (i) NNs over RFs and (ii) indirect over direct estimators. There are also some other issues, such as the long and heavy tailed metric score and difference of relative performance across runs. The discussions surrounding these investigations are summarized in Section 3. Given that the current CATE estimation community heavily depends on IHDP, I think this analysis (Section 3) is worth publishing. The work will make the researchers in the area rethink how to evaluate/compare the empirical performance of CATE algorithms.

Other contributions include the alternative benchmark guidelines and evaluation metrics. Having re-evaluation based on these proposals could strengthen the contributions more.

weaknesses: The authors summarize alternative evaluation metrics for CATE estimation methods in Section 5, which is itself great. However, I think that most of the readers expect the additional benchmark results on conventional estimators based on the proposed metrics.

Some figures are very small (e.g., Figures 1 and 4) and I was not comfortable when trying to understand what they were saying. I suggest the authors improve the presentation of the simulation results leveraging the one additional page in the camera-ready version (if the paper gets accepted)

One thing I want to ask just out of my curiosity is about one of the alternative metrics, the decision quality (“Do estimators lead to correct treatment rules?”). I think the authors are aware of another related literature on batch bandit learning from logged bandit data, which aims at directly optimizing the decision quality from observational data (or logged bandit data). The batch bandit learning algorithms are evaluated based on their decision quality in nature. So, I would like to ask how the authors differentiate these two literature (CATE estimation and off-policy learning). Given that the authors think that one of the goals of the CATE estimation is to optimize decision making, then do you think the off-policy learning algorithms should be included as baselines when the decision quality metric is used? Or, do you see that these two approaches have completely different motivations?
correctness: The analysis on the semi-synthetic data seems to be well-designed by the authors and the experiments are able to capture the flaws and concerns of the existing benchmark semi-synthetic datasets.
clarity: The contents of the paper are easy to follow and the motivation is convincing. Additional results are summarized in the supplementary materials in detail.
relation to prior work: The paper has an in-depth investigation of the current simulations study, which is one of the main contributions. The paper thus has sufficient explanation of the current literature and conventional way of benchmarking CATE estimation methods. The citations also look sound. Section 2 and Appendix A well summarizes the basic formulation and background of the literature. 

As I noted above, it might be better to add some explanation about the connection with the off-policy learning literature in "Do estimators lead to correct treatment rules?" in Section 5. Would love to see the authors' thoughts on this.
additional feedback: For suggestions for improvement, please see the “Weaknesses” section.
review_title: Review of the paper

Meta-review: The final scores for this paper are 9, 7, 6, 4. There were some concerns about whether this paper was sufficiently significant given that it analyzes only one dataset, but the authors released results on Twins, and now the majority of reviewers recommends acceptance and there is a champion. The reviewer who recommended 4 also has not responded and has not commented on the new results on Twins. I have to say I disagree with the authors about their claim that the negative reviewer has issues with the fit of the paper for this track (I think their main criticism is about using only one dataset). In any case, I feel like there is enough support for this paper.
Decision: Accept