Kati8/ICLR_2023_2025 huggingface dataset:
authors: Vibhhu Sharma
reviews: 
Comments: This paper studies a common risk-based approach to treatment allocation applied to a diverse range of treatments with randomized controlled trial data available. The authors find that allocating treatment according to risk (ie, bad outcomes under the no-treat baseline) produces worse outcomes than allocating according to conditional average treatment effect estimates, even when CATEs are biased or the decision maker has a preference for treating high-risk individuals.
Strengths: This is a simple, well-executed paper making a very important point. As the authors note, most algorithmic decision making approaches are based on risk or some manipulation of risk in pursuit of fairness goals. This paper demonstrates the problem with that approach across a wide range of interventions.

The authors anticipate the key criticisms of the policy based approach (that RCT data is hard to find and so CATEs may be biased, and that equity preferences might provide a non-utilitarian reason to prefer risk-based targeting), and provide compelling evidence that even under significant confounding or equity preferences one should still prefer allocations based on CATEs.
Weaknesses: The weakest part of this paper is probably the discussion of Fig 1. The authors cite a "unique trend for each dataset" (in CATE conditional on risk score) as evidence that the risk-based policy is flawed. Visually, to me, the trends look pretty similar: they look like there is basically no relationship between CATE and risk for most RCTs studied. This is still evidence for the risk-based policy being flawed (so it doesn't invalidate the claim being made), but I think it's a more accurate way to describe the results. I think Fig 2 is more compelling than Fig 1 anyway, so it probably make sense to lead with that result for the most impactful paper.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: The paper explores targeting strategies for allocating limited resources to maximize social welfare in areas like
human services, healthcare, and education. Specifically, it compares risk-based targeting, which prioritizes
high-risk individuals, with treatment-effect-based targeting, which uses machine learning models to estimate who
might benefit the most from interventions. Using five real-world randomized controlled trials (RCTs) across diverse
domains, the paper concludes that even biased estimates of treatment effects generally outperform risk-based
targeting. This finding suggests in addition to the widespread reliance on risk-based approaches, policymakers
could incorporate treatment effect estimation when feasible.
Strengths: Scope: The work’s target is evident in its cross-domain empirical evaluation, making it applicable to multiple policy
areas. The paper introduces innovative use of biased treatment effect estimates to assess targeting efficacy.
Quality: Methodologically, the paper is solid, employing credible RCT data and a robust approach to measuring
treatment effect heterogeneity. The use of doubly robust estimation and varying social welfare functions adds
depth to the analysis.
Weaknesses: Grammatical errors: there are notable grammatical errors in spelling and following the formatting of ICLR
instructions.

Scope of Treatment Effect Estimates: The reliance on simulated confounding could be more thoroughly justified;
real-world application requires consideration of domain-specific biases that could vary across different policy areas
Potential Ethical Implications: Since the work could influence resource allocation in sensitive domains, further
discussion on ethical considerations regarding inequality and bias would strengthen its impact and application.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: The authors compare the utility of targeting interventions based on estimated treatment effects with the utility of targeting based on predicted risk. The former is the method of choice from a utilitarian perspective. But practitioners and policy makers often choose the latter due to its simplicity and in cases where the normative goal is to assist those in greatest need.
Strengths: (1)

The comparison of these two targeting strategies is an important problem. I'm glad to see that the authors study this question. Despite its significance, there hasn't been much reliable insight so far. I'd love to see more work in this direction!  I'm weighing this strongly in my evaluation.

(2)

The paper is very clearly written. The authors narrate a compelling story about the advantages of targeting based on treatment effect estimates, even if they are biased.
Weaknesses: (1)

As compelling as the story is, I find the results less than conclusive. Looking at Figure 1, it looks like the confidence intervals around treatment effects are generally strongly overlapping across the entire x-axis of baseline risk. The one exception is the STAR dataset, where the lowest and highest point estimate have barely non-overlapping confidence intervals. As a result, another story consistent with the data is that effects are generally not so heterogeneous as conventional wisdom from the econometrics literature has it.


(2)

The results in Figure 1 are actually a fair bit more favorable towards risk-based targeting than the introduction of the paper had me believed. Treatment effects generally increase with risk. Targeting the 80th to 90th percentile of risk generally seems to capture high treatment effects across all datasets. So, another story could be that we should exclude the most extreme values of risk from targeting, but other than that risk-based targeting sort of works.


(3)

I found it rather confusing to have unnormalized utility values on the y-axis. After all the sample size is rather arbitrary and does not correspond to the population-level utility obtained if the policy maker were to implement the given approach. Along the same lines, I found it difficult to reconcile Figure 1 and Figure 2. See question below.


(4)

It would've been great to include datasets with real world confounding rather than the simulated confounding. My experience is that existing CATE estimation methods don't do very well in non-RCT settings. Might there be an advantage to risk-based targeting in non-RCT data?


Suggestion:

It seems to me that the story is much less certain than the abstract and introduction make it sound. I would therefore appreciate it if you could indicate a greater level of epistemic uncertainty in the writing throughout. I don't think this would hurt the paper at all. As is, though, I'd worry that your writing suggests the question is essentially closed conclusively which would actually discourage additional work in this direction.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3