Comparing Targeting Strategies for Maximizing Social Welfare with Limited Resources
Authors: Vibhhu Sharma
Bryan Wilder
Venue: ICLR 2025 Poster (2025)
Keywords: social welfare
causality
treatment
treatment effect
targeting
risk
policymaking
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Machine learning is increasingly used to select which individuals receive limited-resource interventions in domains such as human services, education, development, and more. However, it is often not apparent what the right quantity is for models to predict. In particular, policymakers rarely have access to data from a randomized controlled trial (RCT) that would enable accurate estimates of treatment effects -- which individuals would benefit more from the intervention. Observational data is more likely to be available, creating a substantial risk of bias in treatment effect estimates.  Practitioners instead commonly use a technique termed "risk-based targeting" where the model is just used to predict each individual's status quo outcome (an easier, non-causal task). Those with higher predicted risk are offered treatment. There is currently almost no empirical evidence to inform which choices lead to the most effect machine learning-informed targeting strategies in social domains. In this work, we use data from 5 real-world RCTs in a variety of domains to empirically assess such choices. We find that when treatment effects can be estimated reliably (which we simulate by using direct outcome observations), treatment effect based targeting substantially outperforms risk-based targeting, even when treatment effect estimates are biased. Moreover, these results hold even when the policymaker has strong normative preferences for assisting higher-risk individuals. However, when treatment effects must be predicted from features alone (as is always the case in practice), performance can degrade significantly due to limited data making it difficult to learn accurate mappings from features to treatment effects. Our results suggest treatment effect targeting has significant potential benefits, but realizing these benefits requires careful attention to model training and validation.

Published as a conference paper at ICLR 2025
COMPARING TARGETING STRATEGIES FOR MAXIMIZING SOCIAL WELFARE WITH LIMITED RESOURCES
Vibhhu Sharma Bryan Wilder Machine Learning Department Camegie Mellon University Piusburgh, PA 15232, USA {vibhhus, bw Ider)dd ) @éccc. cs. cmu. edu
ABSTRACT
Machine learning is increasingly used to select which individuals receive limited- resource interventions in domains such human services, education, develop- ment, and more. However, it is often not apparent what the right quantity is for models to predict. Policymakers rarely have access data from randomized controlled trial (RCT) that would enable accurate estimates of which individuals would benefit more from the intervention, while observational data creates sub- stantial risk of bias in treatment effect estimates. Practitioners instead commonly use a technique termed "risk- based targeting" where the model just used pre- dict each individual's 's status quo outcome (an easier, non-caue non-causal task). Those with higher predicted risk are offered treatment. There is currently almost no empirical evidence to inform which choices lead to the most effective machine learning- informed targeting strategies in social domains. In this work, we use data from 5 real-world RCTs in a variety of domains to empirically assess such choices. We find that when treatment effects can be estimated with high accuracy (which we simulate by allowing the model partially observe outcomes in advance), treat- ment effect based targeting substantially outperforms risk-based targeting, even when treatment effect estimates when the policymaker has strong normative preferences for assisting higher-risk individuals. However, the features and data actually available in most RCTs we examine do not suffice for accurate estimates of heterogeneous treatment effects. Our results suggest treatment effect targeting has significant potential benefits, but realizing these benefits requires improvements to data collection and model training beyond what is currently common in practice.
biased. Moreover, these results hold even
INTRODUCTION
Policymakers often face the difficulty of allocating a resource-limited intervention with the goal of targeting the intervention towards those who will benefit most from it. Indeed, the causal inference literature documents that any given treatment may not have the same effect on every individual that receives it [Wager & Athey 2018, Künzel et al., 2019 Varadhan Seeger 2013). When there observable features that correlated with greater benefit from the treatment, such variation can be used for targeting. Heterogeneity of treatment effect (HTE) refers to this nonrandom, explainable variability in the direction and magnitude of treatment effects for individuals within a population. Given this variability, policymakers often face the problem of selecting who to treat when having to assign a particular treatment to group of people under fixed budget. Machine learning methods seem to offer the promise of discovering richer forms of heterogeneity, allowing more effective targeting of interventions in practice.
The main challenge is that heterogeneous treatment effects are difficult to lear: doing so requires a potentially large amount of data that is unconfounded, i.c., where treatment is assigned in manner (conditionally) independent of each individual's 's outcomes. Conducting a randomized controlled trial (RCT) achieve this is often infeasible due to time constraints, resource limitations, and ethical

---

Published as a conference paper at ICLR 2025
concerns. Policymakers may prefer to prioritize access to treatment via already-availate proxy metric that believed to align with need if experimenting to gather additional evidence would be seen as unethical.
One particularly common proxy is to target according to the baseline risk each individual faces, i.c., their expected outcome in the absence of treatment (as opposed to the treatment effect, which is the difference in outcomes induced by treatment). This strategy has been referred to as risk-based targeting [Wilder & Welle| 2024). Individuals with poor predicted baseline outcomes may be seen as needing assistance the most. Policymakers may also believe that these individuals also stand the most to benefit since their status-quo prognosis is the worst. Importantly, baseline risks can often be learned using existing administrative data (from before a treatment was introduced) instead of re- quiring a new experiment, making this strategy easily implementable in many practical settings. For all of these reasons, risk-based targeting has seen widespread use by policymakers in wide range of domains, including targeting humanitarian assistance (Aiken et al.) 2021). allocating homelessness services via vulnerability scores (Shinn & Richard 2022), and the use of "early warning systems" in education (Perdomo et al. 2023).
Despite this widespread usage, there is only limited amount of work which empirically assesses the effectiveness of risk-based targeting: do individuals with the greatest baseline risk actually tend to benefit the most from intervention? Existing studies speak only to a single specific applica- tion domain each, for instance, targeting students for sending reminders for financial aid applica- tions (Athey et al.) 2024), preventing customer churn (Ascarza 2018), targeting cash transfers (Haushofer et al. 2022). (Kube et al. (022)) examines this from a human perspective by empiri- cally eliciting decision-maker preferences for whether to prioritize more vulnerable households or households who would best take advantage of more intensive interventions when determining which homeless households to serve with limited housing assistance. (Ascarza 2018) encourages practi- tioners to use RCTs to better inform their decision, however, as discussed above, running RCT may be infeasible in many settings. The alternative more likely to be available to practitioners is to simply estimate treatment effects using observational data which likely suffers from confounding, potentially leading to biased estimates of treatment effects.
How should practitioners navigate this tradeoff between more easily-leamabble label that may not always correlate with benefit from an intervention (baseline risk) and a difficult-to-leam quantity (heterogeneous treatment effects) that captures the impact of the intervention? This corresponds to the choice of the appropriate target for prediction, as opposed to the specific model used to make the prediction. The choice of outcome variable has been observed to exert a disproportionate influence on the impacts of machine learning systems in many settings (Obermeyer et al. 2019 Coster 2013] Gerdon al., 2022). In the setting of targeting interventions for causal impact, practitioners have little empirically woroccalllgrooddeedd guidance. Our goal in this work is to inform the selection of an objective function for machine-learning based targeting of scarce interventions. We make three contributions towards this goal:
First, we assess the efficacy of risk-based targeting on a wider variety of real RCT datasets encom- passing settings in economics, healthcare and education in contrast to prior studies that generally focus on one dataset. We find a generally noisy and variable relationship between baseline risk and treatment effects: high-risk individuals seem to benefit more on average in most domains, but with substantial variance in treatment effects which is not explained by baseline risk. When treatment effects can be estimated reliably (which we simulate by allowing the model to partially observe outcomes in advance), targeting based on these estimates produces substantially better results com- pared to risk-based targeting. However, in practice, treatment effects must be predicted from features alone, and limited data can make difficult to learn accurate mappings from features to treatment effects, potentially degrading the performance of treatment effect based targeting.
Second, we compare risk-based targeting to targeting policies based on biased estimates of treat- ment effect obtained from confounded data. Such biased estimates are likely when conducting full-fledged RCT is infeasible and policymakers have to rely on available observational data alone. Accordingly, potentially-biased causal estimates represent the likely alternative to risk-based target- ing in many domains of practical interest. To our knowledge, these two strategies have not been explicitly compared. We find that across even relatively severe levels of confounding, a utilitarian policymaker often prefers targeting according to biased estimates of the treatment effect rather than baseline risk when treatment effects can be estimated reliably at no confounding.
2

---

Published as a conference paper at ICLR 2025
Finally, we analyze the setting where policymaker has inequality- preferences: oftentimes, policymakers may prefer interventions which benefit those who are worse-off to begin with even if they produce less aggregate impact. Such normative goals are one possible justification for risk- based targeting, even risk-based targeting is less attractive in standard utilitarian terms. We com- pare the two targeting strategies under popular classes of social welfare functions which capture inequality-averse preferences. We find that when treatment effects can be estimated reliably, treat- ment effect based targeting is typically favorable to risk-based targeting, even in scenarios where policy makers are inequality-
and the only available data is confounded to some degree.
All together, our results suggest that the barrier to targeting on treatment effects more likely to lie in the collection of large and rich enough datasets to support accurate prediction of outcomes, while confounding or inequality-avers
preferences have more limited impact. This implies that large- scale administrative datasets may be a more attractive avenue for policy learning than previously thought, even if they lack the guaranteed validity of randomized controlled trial.
Measuring heterogeneity in treatment effect and choosing which subpopulations to assign a treatment to has long been an active avenue of research in causal inference literature with
Related Work:
variety of methods proposed to solve this problem. (Green & Kern 2012, Hill 2011 Hill & Su,
2013 Foster et al. 2011, Wager & Athey 2018) use forest-based algorithms to identify groups that show heterogeneity in treatment effeet with other identified groups. (Tian et al. 2014) proposed to measure the interaction between treatment and covariates by numerically binarizing the treatment and including the products of this variable with each covariate in a regression model. (Künzel al.] 2019) uses meta-learners that decompose estimating the CATE into several sub-regression problems that can be solved with any regression or supervised learning method. The problem of choosing who to treat is closely related to identifying the heterogeneity in treatment effects. This often vns balancing policies based solely on estimates of conditional average treatment effect (CATE) with ad- ditional prioritization rules set by the policymaker. (Yadlowsky et al. (202]) proposes rank- waighhethtee average treatment effect metrics for testing the quality of treatment prioritization rules, providing an example involving optimal targeting of aspirin to stroke patients. Meanwhile, (Gupta et al., 2020) provides a thcoretical analysis of risk-based targeting strategies, establishing performance bounds for scoring rules when treatments are benign.
2 PRELIMINARIES
Consider a setting where there is a population of individuals who are candidates for a treatment or intervention. Each individual has feature vector X Rd. Here we are concerned with binary treatments. Following Neyman and Ruben's (Splawa-Neyman et al. 1990 Rubin 1974) potential outcomes framework, we use (!) to denote the outcome that an individual would experience under treatment and ( to denote the outcome they would experience if not treated. Their individual treatment effect y()) y(0), We assume that (X, ( y() are drawn i.i.d. for each individual from some joint distribution. In order to identify individuals who are likely to benefit, common strategy is use individuals' observed covariates to predict the expected treatment effect. The conditional average treatment effect (CATE) is defined as:
Txx) = E Y (1) (0) X =
(1)
Estimating the CATE order to target based on treatment effects is a difficult statistical problem. Suppose we have access to data corresponding to people, labeled 1,... ..... consisting of fea- tures X,, treatment assignment W; E (0, 1}. and the observed outcome Y, = (WW), Importantly, for each individual, we can observe only the outcome corresponding to the treatment they were actually assigned. Accordingly, identifying treatment effects typically requires a a no-unobserved- confounders assumption (Rosenbaum Rubin 1983): (Y((), y() W X. This assumption is most credible in the context of randomized controlled trial (RCT). In an RCT, the assignment of treatment, represented by W., is assigned independently of the potential outcomes Y; (potentially after stratification on covariates X:). When data is purely observational, practitioners typically try to select a sufficiently rich set of covariates X such that all potential confounders between outcomes and treatment assignment measured. However, ensuring that all confounders are completely captured is notoriously difficult in practice, creating the likelihood that some bias in the estimated
CATE remains Londe 1986. Pearl 2009 [Skelly et al. 2012 Milli al. 2022
3

---

Published as a conference paper at ICLR 2025
As an alternative to targeting on treatment effects, policymakers often decide treat people who more vulnerable or worse-off at present, without attempting to quantify the benefit these individuals receive from treatment. This is quantified via 'baseline risk'; refer to the resulting allocation strategy as 'risk-based targeting" "eereingg(WWieee""" & Welle, 2024). Baseline risk may sometimes be directly measured quantity (one of the covariates in X, for example baseline test scores in an educational context). In many settings though, it is estimated using a predictive model that uses the covariates as input. Let be function that maps a set of covariates to baseline risk measurement such that 6(X) represents the baseline risk and b( (X;) denotes the baseline risk associated with i. Then this method involves selecting individuals with the highest values of for treatment, implying that these individ- uals have the highest 'risk" associated with them, which needs immediate resolution. It is important to note that this strategy requires only data on baseline outcomes prior to program implementation, with no information about the treatment's s effect being incorporated in the policymaker's is decision.
METHODS
We compare risk-based and treatment effect-based targeting strategies using real-world RCTs, which allow credible estimation of heterogeneous treatment effects because of the assumption of no un- measured confounding. We simulate these targeting strategies under varying degrees of confounding and under different social welfare functions for policymakers, evaluating their effectiveness against randonization-enabled treatment effect estimates. We now detail the methodology used each step of this process, starting with the datasets used.
3.1 DATASETS
We conduct experiments on a variety of RCT's across different domains as detailed below:
Targeting the Ultra Poor (TUP) in India (Banerjee al.] 2021)): This RCT conducted to study the long-term effects of providing large one-time capital grants to low income-families and observing how family income and overall consumption changed over a period of 7 years. We consider a family's 's total expenditure as the outcome, which is positively affected by treatment.
NSW (National Supported Work demonstration) Dataset (Dehejja & Wahbal 1999 2002 LaLonde 1986): This study estimated the impact of the National Supported Work Demonstration, a job training program, on beneficiaries' income in 1978. We consider an individual's 's income in 1978 as the outcome, which is positively affected by treatment.
Postoperative Pain Dataset: Patients undergoing operations like tracheal intubations often experi- ence throat pain following treatment (Mchardy & Chung] 1999). This RCT was conducted to test the efficacy of licorice solution at reducing postoperative sore throat. The outcome we focus on patient's ss throat pain 4 hours after surgery. Here, the effect of the treatment is to reduce the amount of throat pain, hence the treatment effect is negative. In order to maintain consistency with other plots, we present results with the sign for treatment effect reversed.
Acupuncture Dataset: (Vickers et al. 2004) This RCT aimed to determine the effect of acupuncture therapy headache severity in patients with chronic headaches. Our outcome variable is headache severity year rost-randomizztion. Here again, the effect of the treatment is to reduce the severity of headaches, hence the treatment effect is negative. In order to maintain consistency with other plots, we present results with the sign for treatment effect reversed.
Tennessee's 's Student Teacher Achievement Ratio (STAR) project (Achilles al. 2008): This four-year study by the Tennessee State Department of Education examined class size effects on stu- dent performance. The design compared: 1) Small classes (13-17 students per teacher), 2) Regular classes (22-25 students), and 3) Regular classes with a teacher's aide. We focus only on the first two configurations to maintain binary treatment consistency with other RCTs. Our outcome measure is kindergarten students' cumulative test scores.
For each of these datasets, we estimate ( X using a machine learning model applied to the RCT's control group and set b(X) = [| or b(X) = - (E [X] as appropriate so that larger values of b indicate worse outcomes. Additional details included in Appendix A
4

---

Published as a conference paper at ICLR 2025
3.2 ESTIMATING HETEROGENEOUS TREATMENT EFFECTS
We estimate heterogeneous treatment effects on each dataset using a doubly-robust estimator (Kennedy 2023a). The DR estimator splits the data into separate folds. For each fold, we esti- mate models for both the expected outcome and the treatment variable (estimating the latter even when the propensity scores are known can increase statistical efficiency Hirano et al. (2000)). Let A(X, A) be the estimated mean outcome for an individual with covariates X and treatment assign- ment A, and (X) be the estimated propensity score. For each individual in the held-out data for the fold, we estimate their pseudo-outcomes, defined as
1W; = A]2Y, ((,,, (x(A) = A) + (+ + (1 A) (1-11) #(X.))
If at least one of jú or it correctly specified, (A) has expectation (over the random treatment assignment) equal to y((A), which allows us to use it as proxy for the unobserved outcomes in evaluating counterfactual evaluation policies.
3.3 EXPLORING TREATMENT EFFECT HETEROGENEITY
Our first analysis tests one potential rationale for risk-based targeting strategies: the hypothesis that individuals with greater baseline risk will also tend to have greater treatment effects. We frame this as estimating E. y y [)) a conditional average treatment effect just with respect to value of the risk score b. We follow the doubly-robust approach to estimating CATES, where the pseudo-outcome difference (1) - xx) is regressed on the covariates of interest (Kennedy) 2023a). Because our covariate of interest, b, is one-dimensional, we use kernel regression method to estimate the CATE as a generic smooth function (see Appendix A.3).
In later analyses, compare the welfare gains of potential targeting policies. Here, we also use a doubly-robon estimator. Consider a hypothetical policy that assigns treatments A(X) E {0, 1} as a function of individuals' features X. The mean outcome under policy A, (A(X)) can be decomposed
] = { + Pr(A(X) = I) (A() = 1].
The term (1) Y (0) [A(X) = 1] represents the treatment effect on the treated population and captures how much the policy improves over no treatment. For policies with the same budget (equal Pr(A(X) = 1)). only this term varies and so we assess allocation policies by their ex- pected -eatment-on-tte-trreeee treatment effects Kennedy (2023b). we empirically estimate this quantity on a separate test set as
Following standard doubly-robust estimators for (group) average
A(X,)(x(()-xxx0000
-
EJ=1A(X3
3.4 INTRODUCING CONFOUNDING
Our next analysis aims to simulate conditions where we do not have access to perfectly conducted randomized controlled trials for our problem, in order to compare risk-based targeting to a plausi- ble alternative in real world settings: targeting according to observational, and potentially biased, estimates of the CATE.
We introduce varying levels of confounding to the RCTs that we study. We do this by simulat- ing adverse selection into treatment, where units are more likely be observed if the estimated individual-leved treatment effects deviate from the mean. Specifically, we generate the biased "ob- servational" dataset by removing data a systematic manner. This process, inspired by (Kallus & Zhou 2021), is controlled by a parameter l giving the fraction of data removed, with higher A corresponding to more biased estimates. From the treated units, we remove the examples that lie in the top 5% percent when ordered in descending order of (xx(1) (0)) (assuming treatment effect is positive) while for the untreated units, we remove the examples that lie in the bottom 1% of examples when ordered in descending order of (xx(1) - (0)). In simpler terms, for treated units, we remove examples for which the treatment 'went well' (most positive), while for untreated units, remove examples for which the lack of treatment did not go well(least positive).
5

---

Published as a conference paper at ICLR 2025
3.5 FAMILIES OF WELFARE FUNCTIONS
In order to simulate policymakers with varying preferences for who to treat, we compare risk-based targeting to treatment-effeet based targeting with respect to two paradigmatic classes of social wel- fare functions. A social welfare function maps vector of individual utilities to the policymaker's overall utility for the allocation. Both functions we study are commonly used instances of the class of weighted power mean functions, which contains all social welfare functions satisfying desirable ax- iomatic properties (Pardeshi et al. 2024). The two functions we investigate here are Weighted Util- itarian Welfare, given by (( w) Σ === Wipl and Nash Welfare, given by M(u) = [ Wi-
We consider utilitarian welfare with two sets of weights. First, the uniform weights wy 1Vi E [m]- Second, Wy E"=r""/
(XX, where a is a hyperparameter and 6' (X) is represents the percentile
score of the baseline risk for the 7th example, with the example with highest baseline risk having score and the example with lowest baseline risk having score 0. This assigns greater weight to individuals with high values of baseline risk for high values of a, thereby simulating a scenario where a policymaker might value treating treating these "high risk" individuals. a can be interpreted
as 2 log
where W75 is the weight given to the 75th percentile example by baseline risk and W25 is the weight given to the t5th percentile example by baseline risk.
MITER
The Nash social welfare function has commonly been used to achieve a balance between maxi- mizing total welfare (utilitarian) and ensuring equitable distribution (egalitarian) Caragiannis et al.) (2019): Charkhgard et al.] (2020). We consider unweighted Nash welfare (w; 14i). In order to avoid the complications of utility when using an unweighted Nash welfare function, we scale up the estimated utilities for each example to a minimum value of 1. Note that the Nash welfare can be equivalently formulated in log space Caragiannis et al. 2019 as 1 E log u4 When each individual's 's utility under an allocation policy corresponds to their realized outcome y y-(A(Xi) (e.g.. their income after the interventional period), we compare policies exactly as outlined in Equation but with (A(X)) replaced by log (A(X)), estimated by replicating the same procedure after taking logs of all outcome variables.
RESULTS
Figure [ how treatment effect varies as a function of baseline risk for each of the 5 datasets we study, with 95% confidence intervals shaded around the estimated treatment effects. These intervals are pointwise Wald-type confidence intervals (Kennedy et al., 2019). The estimated relationship between baseline risk and treatment effect variable across domains. In most domains, the point estimate shows a general upward trend, indicating that individuals at greater risk benefit more (on average) from treatment. However, in the NSW domain, the point estimate is essentially flat. In addition, the confidence intervals are wide for all domains and there is very little statistically signifi- cant significant evidence in favor of high-risk individuals benefiting more. Wide confidence intervals reflect that there is significant variance in the pseudo-outcomes estimated for different individuals the same level of baseline risk. That is, there is a great deal of variance in our estimated treatment ef- fects that is not explained by baseline risk. However, we do expect that risk-based targeting should, in most domains, perform better than a random allocation, since the point estimates generally show larger average effects at higher baseline risk.
Next, Figure [ shows the the average utility of the three targeting policies risk-based, treatment effect-based, and random on each of the datasets. The x axis on each plot represents the degree of confounding synthetically introduced, as discussed above. At the h = 0 point (æ 0) on the axis, no confounding is present, representing the ideal scenario as in an RCT. For risk- targeting, we observe a tendency towards improvement over random targeting, but with confidence intervals overlapping in all domains. For treatment-effect based targeting, we observe at most mixed success. On the STAR dataset, we see statistically significant evidence that targeting on treatment effects outperforms Disk-based targeting when no confounding is present, with the point estimate trending downwards and eventually drawing even with risk-based targeting as confounding increases. On the NSW dataset, we see a similar trend in the point estimates, but with overlapping confidence intervals. One the remaining datasets, treatment effect targeting has strongly overlapping confidence

---

Published as a conference paper at ICLR 2025
intervals with risk-based targeting and shows effectively flat trajectory with respect to the degree of confounding. We conclude that in these settings, either limited data or insufficiently rich features prevent the model from learning complex treatment effect heterogeneity patterns. On the one hand, we conclude that practitioners should be wary about whether the kind of data generated by many RCTs in practice suffices to learn strong predictors of heterogeneous treatment effects. On the other hand, we find no evidence that the presence of confounding makes matters worse unless there is meaningful amount of signal present start with. Appendix A contains results for alternate welfare functions, with similar results.
Ultrr Foor Dataset
NSW Dataset
8
Smootred
Smoothed
95% Confidence Interval
Teatment Effect
Confidance interval
2000
Teatment
1000
8th
Baseline Risk Percentlle
Baseline Percentila
Postoperative
Acupuncture Dataset
2.0
Smoothed DR Prediction
3.5-
95% Confidence Interval
Treatrent Effect
95% Confideare Interval
Treatment
0.5
0.0
-25
-5
06h
25th
50tn
79th
100th
STAR Dataset
Treament Effect
75
95%
Confidence Interval
50-
25-
0-
-25
-50
0th
25th
50th Beseline Risk Percentile
roth
100th
Figure 1:
Observing treatment effect heterogeneity across different settings by plotting treatment effect against baseline risk for each of our 5 datasets. We observe unique trend for each dataset, indicating a lack of a consistent well-defined relation between the two quantities
In order to isolate the impact of confounding in a setting where heterogeneous treatment effects would otherwise be predictable, we conduct semi-synthetic analysis that simulates targeting model with stronger signal. In particular, we simulate treatment effect-based targeting policy that ranks individuals by their pseudo-outcome directly, instead of the predicted pseudo-outcome using the covariates. This incorporates information on the realized outcome of the individual and so is not attainable in practice. However, mimics setting where the second-stage regression model has strong performance in the no-confounding setting. Differences between the two settings are detailed An The results, shown in Figure demonstrate that when treatment effects can accurately es- timated (A- = 0, no confounding), targeting based on treatment effects always produces significantly higher utilitarian welfare. This indicates that when policymaker seeks only to maximize aggregate benefit and can credibly estimate treatment effects, the gains from causal targeting are substantial.
As the level of confounding bias in treatment effect targeting increases (increased k),, its effec- tiveness decreases. However, when the policymaker has utilitarian preferences, targeting based on biased treatment effect estimates still performs at least as well as risk-based targeting (and typi- cally better) across all datasets, using even relatively biased observational data to learn treatment rules likely superior when the policymaker's goal is just maximize aggregate gain.
for relatively severe levels of confounding. This indicates that

---

Published as a conference paper at ICLR 2025
Ultra Poor Dataset
NSW
Postoperative Pain Dataset
00
Treatment Effect Basod
6000
Risk-Based
Risk Based Random
Usilty
4200
Rondom
1.5
2000
3.0
20
a.5
Teatment Effect Based
Risk Eased
Random
-05
0.1
0.2
0.3
0.4
0.3
0.4
0.2 Higher confounding
Higher confounding
Higher confounding
(a) Utilitarian Welfare on the UI- tra Poor RCT
(b) Utilitarian Welfare on the
(c) Utilitarian Welfare on the Postoperative Pain RCT
NSW RCT
Acupuncture Dataset
STAR
120
100
Utiltty
10
Umility
60
40-
Teatment Effect Based
Teeement Effect Based
Risk-Based
Disk-aaaed
Random
0.1
02
0.3
04
0.1
022
0.3
0.4
Higher confounding
Higher confounding
(d) Utilitarian Welfare on the Acupuncture RCT
(e) Utilitarian Welfare STAR RCT
the
Figure 2: Comparison of risk-based targeting to biased treatment effect-based targeting by plotting the benefit offered by each policy against the amount of data systematically removed from the RCT to introduce confounding
The second column of Figure shows alternative set of preferences, where the policymaker has a utilitarian welfare function with higher welfare weights on individuals with higher baseline risk (for these plots, a is 2 log 2, the ratio of the weight placed on individuals in the 75th percentile of risk to the 25th percentile is 2). This welfare function decreases the gap between treatment effect and risk based targeting as individuals with higher risk now have more weight associated with them. However, targeting on (biased) treatment effects is still preferable to risk-based targeting across all datasets in Figure 3 Similarly, under the Nash social welfare function (third column of Figure 33 we observe some differences across our chosen settings but the general trend remains the same: as we increase confounding (increase the percentage of data we systematically remove), the benefit we accrue by following treatment assignment policy based on biased treatment effect values decreases. At high levels of confounding, risk-based targeting accrues higher utility for a policymakers with a Nash social welfare function in the Ultra Poor setting in However, across all other datasets, the policymaker prefers to target based on treatment effects even at high levels of confounding.
This motivates us ask: how much greater must the policymaker weight high-risk individuals in order to prefer risk-based targeting? In Table | we give the minimum value of a at which risk- based targeting finally outperformed treatment effect based-targeting at each level of systematic data removal for the real and synthetic settings we consider respectively. We limit a such that the ratio of the th percentile weight the 25th percentile weight when sorted in descending order is less than 100; otherwise, only a few individuals have non-zero weights and estimating welfare gains becomes impossible. "na" indicates there no value (up to our upper bound) at which risk based targeting outperforms targeting based on treatment effects.
In general, we note that the required a values tend to be lower when we increase k. which follows directly from the fact that the treatment effect estimates become more biased at higher k. We note a dichotomous tendency in the results. In some cases, the value of alpha is O, where risk-based targeting performs at least as well as treatment-effect based targeting even at k = 0). When a takes a non-zero value, it is almost always at least 3 (and typically higher), indicating that the policymaker would need to place approximately 4.5 times more weight the welfare of individual at the 75th percentile of baseline risk than an individual at the 25th percentile. We conclude that when treatment effect-based targeting is effective all, relatively extreme welfare weights are needed to
8

---

Published as a conference paper at ICLR 2025
Ultra Poor Dataset
Ultra Poor Detaset
Uitra
boutreet eit oflind Baxed
6.55
Risk Based
Fandem
60
Ukility
40
00-
20
Treamment Effect Bacad
40-
6.90
Risk- Based
Random
26-
Ro dom
Hit- Sasied
Testment tfter Based
Higher confounding
2.4
0.5
0.10
0.13
0.20
3.35
0.31
3.40
u. B.y Mudo 0.e0 tub
(a) Utilitarian Welfare on the Ul-
(b) Weighted Utilitarian Welfare
(c) Nash Welfare on the Ultra Poor RCT
tra Poor RCT
the Ultra Poor RCT
NSW Dataset
NSW Dataset
Dataset
12500
Teatment Effect Based
12590
Treatrient Effect Based
11.35
Tecammnt Effect Rased
Risk Bared
10000
Based
11.30
Risk: Based
Random
Uniley
7500
Ubilit
11.0b
5000
11.00
2500
2250
-2300
0.2
0.2
0.3
0.4
0.1
0.2
0.3
D.4
0.S Higher confounding +
0.4
Higher confounding
Higher confounding
(d) Utilitarian Welfare on the NSW RCT
(e) Weighted Utilitarian Welfare on the NSW RCT
(l) Nash Welfare on the NSW
RCT
Postoperative Pain Dataset
Postoperative Pain Dataset
Postoperative Pain Dataset
Risk Based
Treatment Effect Based
Theatment Effect Based
Based
2.4
Rist-Bee
Treamment Effect Based
Random
Random
Random
Ukilty
Uailtyy
2.1
2.0
0.2
0.9
Higher confounding
Higher confounding
Higher confounding
(g) Utilitarian Welfare on the
(h) Weighted Utilitarian Welfare on the Postoperative Pain RCT
(i) Nash Welfare on the PostOp- erative Pain RCT
Postoperative Pain RCT
Acupuncture Dataset
Acupuncture Dataset
Acupuncture Dataset
Tretmmnt Effeci Baved
30
Rick Based
4.0
Randors
30
4.7
20
Teatmmett Effect Based
Risk: Based
4.5
Random
10
4.5
4.4
4.5
0.2
0.4
0.0
0.25
0.20
0.23
0.30
0.35
040
0.1
0.2
0.3
0.4
Higher confounding
Higher confourding
Higher confounding
(i Utilitarian Welfare on the
(k) Weighted Utilitarian Welfare on the Acupuncture RCT
(l) Nash Welfare on the Acupunc- ture RCT
Acupuncture RCT
STAR Dataset
STAR Dataset
STAR Dataset
Treatment Effect Blased
Treatment Effect Based
7.00
250
Risk-Besed
250
Risk Based
200
Rancom
200
Random
6.55
Usiltt
6.50
Treatment Effect Based
150
150
Fisk-Blssed
Random
100
100
0.50
50
6.15
6,70
3.4
0.2 Higher confounding
0.2 Higher confounding
Higher confounding
(m) Utilitarian Welfare on the STAR RCT
(n) Weighted Utilitarian Welfare on the STAR RCT
(o) Nash Welfare on the STAR RCT
Figure 3: Comparison of risk- based targeting to biased treatment effect-based targeting by plotting the benefit offered by each policy against the amount of data systematically removed from the RCT to introduce confounding but using observed pecudo-outcome data
9

---

Published as a conference paper at ICLR 2025
rationalize the adoption of risk-based targeting instead. Table 2 shows the required values for the setting when we do not use observed pseudo outcome values for treatment effect estimation.
Table 1: Values a for different at which risk-based targeting outperforms treatment effect based targeting. 'na' indicates such was found (using observed pseudo-outcomme data)
Dataset
5 5%
10%
15%
20% %
25 %
30%
35 %
40 %
Ultra Poor
8.5
5.5
4.5
3.5
2
NSW
6.5
8.25
4.25
3.5
Postoperative Pain
na
8
na
6.5
na
5.5
6.5
Acupuncture
7.5
6.75
6
3.75
4.5
5.75
STAR
Table 2: Values of for different at which risk-based targeting outperforms treatment effect based targeting. 'na" indicates such was found
Ultra Poor 5%
Dataset
10%
15%
20% %
25 %
30%
35%
40%
0
NSW
na
na
na
ma
0
0
na
na
Postoperative Pain
0
0
0
0
0
0
Acupuncture
3,5
3
6
7
7
3.5
STAR
3
3.5
3.5
2
0.5
1.5
Collectively, our results reveal two key insights. First, in many real-world settings, limited data makes it difficult to learn reliable mappings from features to treatment effects, which severely limit the effectiveness of treatment effect based targeting in practice. However, our analysis using directly observed pseudo outcomes, well real-data results in the settings with stronger sig- nal, both demonstrate that simple reliability of learning heterogeneous treatment effects is the key bottleneck: treatment effects could be estimated more reliably (through larger datasets or bet- ter modeling approaches), then policy maker is almost always better of using them for targeting, regardless of both the potential for confounding and inequality- averse preferences.
CONCLUSION
This paper presents a systematic comparison between two of the most popular treatment assignment policies in use by policymakers today: risk-based targeting and treatment effect based targeting. We find that risk-based targeting tends to produce higher welfare than uniformly random allocation, confirming some of the intuition behind its widespread use by practitioners. However, our analy- sis reveals both important practical limitations and significant potential benefits of treatment effect based targeting. The key practical challenge is that real-world datasets contain insufficient data to learn reliable mappings from features to treatment effects, making it difficult to accurately predict who will benefit most from treatment. This limitation helps explain why risk-based targeting, which requires learning simpler relationships, remains popular in practice.
At the time, analysis which simulates higher-accuracy treatment effect estimator shows the substantial potential benefits of treatment effect based targeting if these estimation challenges could be overcome. Even when accounting for confounding in treatment effect estimates and egal- itarian preferences for assisting high-risk individuals, treatment effect based targeting consistently produces better outcomes unless policymakers have extremely strong preferences for helping higher- risk individuals regardless of benefit. One limitation is that our investigation assumes essentially consequentialist perspective, where the policymaker's goal is to improve individuals' welfare as de- fined by their outcome. If policymakers have non-consequentialiss preferences, targeting directly a measure of vulnerability may be more appropriate.
Looking forward, we nevertheless find that there would be substantial gains to investing in data and modeling approaches that enabled effective causal targeting (at least with respect to the normative goals assess), Our results also suggest that enabling strong outcome prediction is worth accepting some risk of bias, potentially arguing in favor of increased use of observational administrative data.
10

---

Published as a conference paper at ICLR 2025
Acknowledgement: (Grant G-22-64474).
This work supported in part by the A12050 program at Schmidt Futures
The supplementary material includes code including data prepro- cessing and experimentation for each of the datasets. We also detail our procedures in the Appendix (dataset details) and in Section 4 (step by step experimental procedure).
Reproducibility Statement:
REFERENCES
C.M. Achilles, Helen Pate Bain, Fred Bellott, Jayne Boyd-Zaharias, Jeremy Finn, John Folger, John Johnston, and Elizabeth Word. Tennessee's s Student Teacher Achievement Ratio (STAR) project, 2008. URL
(ttp://...991///1100 7910/DVN/ SIMH9F
Emily Aiken, Suzanne Bellue, Dean Karlan, Christopher R Udry, and Joshua Blumenstock. Machine learning and mobile phone data can improve the targeting of humanitarian assistance. Working Paper 29070, National Bureau of Economic Research, July 2021. URL http: www. www.nber. org
org/papers/wa pwpppsppapssaaaa 90000
Eva Ascarza. Retention futility: Targeting high-risk customers might be ineffective. Journal of Marketing Research, 55(1):80-9) 2018. doi: 10.1509/jm:.660066.. URL https://doii
org/10.1505 jmr 0163
Susan Athey, Niall Keleher, and Jann Spiess. Machine learning who nudge: Causal predictive targeting in a field experiment on student financial aid renewal, 2024. URL httts::/aaxxi:
org/abs bog/abs//a 2310. 08672
Abhijit Banerjee, Esther Duflo, and Garima Sharma. Long-tern effects of the targeting the ul- tra poor program. American Economic Review: Insights, 3(4):471-86 December 2021. doi: 10. 10. 10.120777 20200667. aeri 2e2i.202
URL https: /wtp://wwwww...../
1257/
Ioannis Caragiannis, David Kurokawa, Hervé Moulin, Ariel D. Procaccia, Nisarg Shah, and Junxing Wang. The unreasonable fairness of maximum nash welfare. ACM Trans. Econ. Comput., 7(3), September 2019. ISSN 2167-8375. doi: 10.1345/335590. URL huttps://dof otq/10.
1145/3355902
Hadi Charkhgard, Kimia Keshanian, Rasul Esmaeilbeigi. and Parisa Charkhgard. The magic of nash social welfare in optimization: Do not sum, just multiply!, 03 2020.
Wendy Coster. Making the best match: Selecting outcome measures for clinical trials and outcome studies. The American journal of occupational therapy official publication of the American Occupational Therapy Association, 67:162-70, 03 2013. doi: 10.5014/ajt..2013.o00 20.000005.
Rajeev Dehejia and Sadek Wahba. Propensity mathh-ng methods for nonexperimental causal studies. The Review of Economics and Statistics, 84(1):151-611, 2002. URL https://
repec.
Rajeev H. Dehejia and Sadek Wahba. Causal effeets in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Staristical Association, 10044 1062, 1999. ISSN 01621459. URL
Jared C. Foster, J. M. Taylor, and Stephen Ruberg. idbettiaadeettiiicaaaiiino from rawhoommmnndmieedee clin- ical trial data. Statistics in Medicine, 30, 2011. URL http:s//app...mmmm......eeeee
brq/CozpusID: : 24046082
Frederic Gerdon, Ruben L Bach, Christoph Kern, and Frauke Kreuter. Social impacts of algo- rithmic decision-making:: A research agenda for the social sciences. Big Data & Society, 9(1):
205395172211089300, 2022, doi: 10. 10.17/2055951/21199930 URL httpt:://///// org/
10. T0.111772033551722109533

---

Published as a conference paper at ICLR 2025
Donald P. Green and Holger L. Kern Modeling Heterogeneous Treatment Effects in Survey Ex- periments with Bayesian Additive Regression Trees. Public Opinion Quarterly, 76())491-511, 09 2012. ISSN 0033-362X. doi: 10. 00.1093/po//ffs036. URL httppss///....or/1.1..ooo poq/nfs003
Vishal Gupta, Brian Rongqing Han, Song-Hee Kim, and Hyung Pack. Maximizing intervention
effectiveness. Management Science. (6611)):556--599 2020. doi: 10.1287/mIxc.2219.33377
URL https:://ai.oog//10 1tt/:/00.1221...11 www.m//cc.mmmmmmm.m..oomm wtw.:aassccc......m 2019.
Johannes Haushofer, Paul Niehaus, Carlos Paramo, Edward Miguel, and Michael W Walker. Tar- geting impact versus deprivation. Working Paper 30138, National Bureau of Economic Research, June 2022. URL http://
nber. wwtpppappp///aa/../ /33033
Jennifer Hill and Yu-Sung Su. Assessing lack of common support in causal inference using Bayesian nonparametrics: Implications for evaluating the effect of breastfeeding on children's cognitive outcomes. The Annals of Applied Statistiies, 1386 1420, 2013. doi: 10.1214/13-AAS6300
URL pttpps://dol wttp://.........../00.. 101.://1001111113-105555533
Jennifer L. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Gruphical Statisticx, 20(1):117-240, 2011. doi: 10.1198/jes: URL https:
1198/ jegs wwt.e.2ss.0.... 08162
Keisuke Hirano, Guido W Imbens, and Geert Ridder. Efficient estimation of average treatment effects using the estimated propensity score. Working Paper 251, National Bureau of Economic Research, March 2000. URL httt::/t://wee www.. http://wwwwwbbbrorrrarr/s///////
Nathan Kallus and Angela Zhou. Monimax-Optimma Policy Learning Under Unobserved Confound- ing. Management Science, (7(5))2870-2890 May 2021. doi: 12. 2020.369099999999 URL
https://idees org.iideaa.ee..ggg//ge./... http://a...aaaaa...eooeooooooooa
2890 .html
Edward H. Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects,
2023a. UBL_fttss.//axxx..orr///.s
14497
Edward H. Kennedy. Semiparametrric doubly robust targeted double machine learning: review
2023b. /tlLitttpp::/axxiooooog/ http:///gggabba////////a 06469
Edward H Kennedy, Shreya Kangovi, and Nandita Mitra. Estimating scaled treatment effects with multiple outcomes. Statistical Methods in Medical Research, 28(4); 1094-1144 2019. doi: 10.117/09220011777111 URL PMID: 29254442.
Amanda Kube, Sanmay Das, Patrick J. Fowler, and Yevgeniy Vorobeychik. Just resource allocation? how algorithmic predictions and human notions of justice interact. In Proceedings of the 23rd ACM Conference on Economics and Computation, EC '22, pp. 1 1184-124 1242, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 1781490395504. 3538305. URL http::ttp:/// http:///oooorgo//////// /
doi: 10.1145/3490486.
11411111444441 30 3538305
Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. Metalearners for estimating heteroge- neous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10):456-41653 2019.
Sören R. Künzel, Jasject S. Sekhon, Peter J. Bickel, and Bin Yu. Metaleamers for estimating het- erogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 1 116(10)41166-1165 2019. doi: 10. 1804597116. URL https://wwwt
pnas. wwtp:o/g//ooooooooooooooo/ooo hatoo//abb//00./ 1B0 1804597116
Robert LaLonde. Evaluating the econometric evaluations of training programs with experimen- tal data. American Economic Review, 76(4):604-2) 20, 1986. URL https::tt///e..... /ttp:////coPPPp..s/.
repec.org/Rer
186::: ::: 604-20
Fiona Mchardy and Frances F Chung. Postoperative throat: cause, prevention and treat- ment. Anaesthesia, 54, 1999. /ttp:: http:///e.......eaaaaaeaaaaaaara
31552
12

---

Published as a conference paper at ICLR 2025
Smitha Milli, Luca Belli, and Moritz Hardt. Causal inference struggles with agency
line platforms. In Proceedings of the 2022 ACM Conference on Faimess, Accountability, and Transparency, FAccT '22, pp. 357-365, New York, NY. USA, 2022. Association for Com- puting Machinery. ISBN 978 9781450393522. doi:
URL https:
1T5..//0//1.11115551111.1 45333100
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 3666646 2019. doi: 00.126/science.aax2222 science aax2342
URL http:: wttp://ceeecee..gg.///aa////////
Kanad Shrikar Pardeshi, Itai Shapira, Ariel Procaccia, and Aarti Singh, Learning social welfare functions, 2024. URL httpss://axxi..ooo wrt.orgg/aaa....... 17700
Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009.
Juan Perdomo, Tolani Britton, Moritz Hardt, and Rediet Abebe. Difficult lessons on social predic- tion from wisconsin public schools, 2023. URL https::ps..... //ar: org h//p//b///304/ 06205
Paul R. Rosenbaum and Donald B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983. ISSN 00063444, 14643510. URL
http::/wwtt www. jstor. wrw..rrg//sss.. (trrg/stable/233552
Donald B. Rubin.
Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66:688-701, 1974. URL https://app.
org Sttp://ooooossp0030323200
Marybeth Shinn and Molly K. Richard. Allocating homeless services after the withdrawal of the vulnerability index-service prioritization decision assistance tool. American Journal of Public Health, 1 112(3)3778-) 382, 2022. doi: 10.2105/AJPH.2221.3000
URL (Rttlstts://d orq
10. 2...100/ AJPH 2021 306628
PMID: 35196047.
Andrea Skelly, Joseph Dettori, and Erika Brodt. Assessing bias: the importance of considering con- founding. Evidence-based spine-care journal, 3:9-12, 02 2012. doi: 10. 105500000000111 1298595.
Jerzy Splawa-Neyman, D. M. Dabrowska, and T. P. Speed. On the application of probability theory to agricultural experiments. essay on principles. section 9. Statistical Science, 5(4):465-472, 1990. ISSN 08834237, 21688745, URL/http: //ww::://www... jstor. strtor.org/stablee22533333
Lu Tian, Ash A. Alizadeh, Andrew J. Gentles, and Robert Tibshirani. A simple method for timating interactions between a treatment and a large number of covariates. Journal of the American Statistical Association, 109(508)15517-13333 2014. ISSN 01621459. URL http:
wwww
Ravi Varadhan and John Seeger. Estimation and reporting of heterogeneity of treatment effects. 2013. URL httpsttss://app semant www.mmnnnnccchoooomooo :D 9040373
Andrew Vickers, Rebecca Rees, Catherine Zollman, Rob McCarney, Claire Smith, Nadia Ellis, Peter Fisher, and Robbert Haselen. Acupuncture for chronic headache in primary care: Large, pragmatic, randomised trial. BMJ (online), 328, 04 2004, doi: 10. 1313///m..3029242222
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523): 1224-11221 2018.
Bryan Wilder and Pim Welle. Learning treatment effects while treating those in need, 2024. URL http:////axxi..oo//a////////////
Steve Yadlowsky, Scott D. Fleming, Nigam Haresh Shah, Emma Brunskill, and Stefan Wager. Eval- uating treatment prioritization rules via rank-weighted average treatment effects. 2021. URL
https: //api. wwwant wew.a//ccccooaa.roooooocosaaaaaa
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 EXPERIMENT DETAILS
Real Setting:
We divide the RCT data into two splits such that one split used for training nuisance functions and the other split is used entirely for evaluation.
Following the DR estimator framework in (Kennedy 2023a), we divide the first split into two halves and and train set of outcome and propensity nuisance functions on each half. We then perform cross-fitting where the nuisance functions trained on one half are used to estimate pseudo outcomes for the other half. Finally, we train a random forest regressor strictly between the estimated pseudo outcome differences and a unit's features. This random forest model is applied to units in the eval- uation set to obtain treatment effect estimates for the evaluation set. The ground truth treatment effects for the evaluation set are obtained by using information about the observed pseudo outcome and taking the mean of the estimate provided by each set of nuisance functions.
Semi-synthetic Setting:
We divide the RCT into two splits such that we use each split to obtain treatment effect estimates for the other split and make maximal use of available data. Again, we use the DR estimator from (Kennedy 2023a) and use information about the observed pseudo outcome in estimating treatment effects (both ground truth and subsequent biased treatment effect estimates from confounded data).
A.2 DATASETS
Targeting the Ultra Poor (TUP) in India ((Banerjee et al. 2021 ): This RCT was conducted to study the long-term effects of providing large one-time capital grants to low income- families and observing how family income and overall consumption changed over a period 7 years. We consider family's total expenditure as the outcome, which is positively affected by treatment. We filter the dataset before use by removing null values and per- forming feature selection limit the number of covariates. The dataset consists of 796 ex- amples post filtering We quantify baseline risk b(X) as an estimate of baseline expenditure (O) X from machine learning model, with low values of F Y ( corresponding to high baseline risk and vice versa. This follows the hypothesis that households with low expenditure at baseline will benefit most from the treatment.
While constructing a doubly robust estimator to estimate pseudo outcomes for this dataset, we found the estimated propensity scores to be very high/low for certain examples, which would consequently scale pseudo outcome estimates to unusually large values. Therefore, we manually set propensity scores uniformly according to the treated:untreated ratio in the RCT.
NSW (National Supported Work demonstration) Dataset (Dehejja & Wahba 1999 2002
LaLonde 1986): This is a popular causal inference dataset that was used to estimate the impact of the National Supported Work Demonstration, a job training program, on ben- eficiaries' income in 1978. The covariates include demographic variables like age, race, marital status and academic background, along with the henficiary's 's income in 1975 prior to the experiment 425 control). Here too, we use an estimate of an individuals baseline income as a measure of risk, following the hypothesis that low income individuals will benefit more from the treatment.
baseline. The dataset consists of 722 examples (297 treated and
Tennessee's Student Teacher Achievement Ratio (STAR) project ((Achilles et al. 2008)):
The Tennessee State Department of Education conducted a comprehensive four-year study called the Student/Teacher Achievement Ratio (STAR) to examine the effects of class size on student performance. This research, backed by the Tennessee General Assembly, in- volved 11601 students across 79 schools. The study design included three different class- room configurations:
Small classes with 13-17 students per teacher
Regular classes with 22-25 students per teacher
14

---

Published as a conference paper at ICLR 2025
Regular classes with 22-25 students plus full-time teacher's aide
To ensure unbiased results, both students and teachers were randomly assigned these dif- ferent classroom types. The experiment began when the participants entered kindergarten and continued through their third-grade year, allowing for longitudinal analysis of the impact of class size on educational outcomes. In this paper, we only focus on the first two types of classes mentioned above, so as to stay consistent with treatment value being binary in other RCTss, This large-scale research project aimed to provide empirical evidence on the relationship between class size and student achievement. Again, we filter the dataset before use by removing null values and performing feature selection to limit the number of covariates. We focus on students kindergarten and a cumulative measure their scores various tests the outcome under consideration. The filtered dataset consists of 3712 students examples. Since we do not have the students' test scores at baseline, we train a random forest model rows corresponding students who did not receive the treatment with their test scores at endline being the outcome variable. The prediction offered by this model for every student is then used as a proxy for their baseline test scores and the nega- tive of this value is used as baseline risk. This follows the general hypothesis that students with low test scores need the treatment more.
Postoperative Pain Dataset: Patients undergoing operations like tracheal intubations often experience throat pain following treatment( & Chung] 1999p. This RCT was con- ducted to test the efficacy of gargling with licorice solution prior to endotracheal intubation reducing postoperative sore throat, which is common side-effect of the procedure. The investigation involved 236 adult participants scheduled for elective thoracic surgeries endotracheal tubes. The outcome we focus on is patient's 's throat pain 4 hours after surgery, measured on a discrete Likert scale from 0 to 7. Additional covariates include a patient's gender, BMI, age, Mallampati score, preoperative pain, surgery size and smoking status. Here, the effect of the treatment is to reduce the amount of throat pain, hence the treatment effect negative. In order to maintain consis- tency with other plots, we present results with the sign for treatment effect reversed. Since we do not have measured value for throat pain at baseline, we again train a random for- est model on rows corresponding to patients who did not receive the treatment with their throat pain at endline being the outcome variable. The prediction offered by this model for every patient is then used as a proxy for their baseline throat pain and consequently as the baseline risk. This follows the intuition that patients with more severe throat pain require the treatment more than their co-patients.
necessitating the use of double-
Acupuncture Dataset: This RCT aimed determine the effect of acupuncture therapy on headache severity in patients with chronic headaches. These measures were assessed at randomization, 3 months post-randomization and year post-randomization. We focus on headache severity year post-randomization. Headache severity is measured on a discrete Likert scale from to 5. The dataset consists of data from 401 patients with covariates including patient age, sex, chronicity of years of headache severity) and whether the headaches were diagnosed as migraines or not. Here again, the effect of the treatment is to reduce the severity of headaches, hence the treatment effect is negative. In order to maintain consistency with other plots, we present results with the sign for treatment effect reversed. We estimate headache severity baseline E(Y (0) using machine learning model and use it as a proxy for baseline risk, following the intuition that patients with more severe headaches need the treatment more.
A.3 KERNEL SMOOTHING
Given the kernel function K(u)= = exp(p the CATE estimate at b(X,) is given by:
+(((,.)) = Di=1 K( MX MX,)) )f,
(3)
ΣØ= K (XX)-8(X.)
where řj is the estimated difference in pseudo outcomes, for unit j. (1) - x,()). as determined by the doubly robust estimator, and a; is the adaptive bandwidth calculated as:
15

---

Published as a conference paper at ICLR 2025
0,
2 (6((41000 - -
(4)
for a window of 200 data points centered at i. The confidence intervals are computed using
weighted variance estimate:
Σj=1 K( b(X,, -(X))-((X))
[f) (b((.)))
CI f(b(X,)) + 1.96,
(5)
C K
6(X, (X,))((X,)) ((Xx) ))2
This approach allows us to capture the heterogeneity treatment effects across different levels of baseline risk while accounting for the varying density of data points.
A.4
ADDITIONAL PLOTS
A.4.1
EVALUATING BASED ON WEIGHTED UTILITARIAN WELFARE AND NASH SOCIAL WELFARE
STAR Dataset
STAR Dataset
120
Treatment Effect Based
6.825
Risk-Based
100
Random
6,800
80
6.775
Utility
Utility
6.750
40
6.725
Treatment Effect Based
20
6.700
Risk Based Random
0
6.675
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
Higher confounding
Higher confounding +
(a) Weighted Utilitarian Welfare
(b) Nash Welfare
Figure 4:
Comparing risk-based targeting to biased treatment effect-based targeting on weighted utilitarian welfare and Nash social welfare for the STAR RCT.
Ultra Poor Dataset
Ultra Poor Dataset
6.54
60
6.52
40
Utility
Utiliity
6.50
6.48
20
Treatment Effect Based
6.46
Treatment Effect Based
0
Risk-Based
Risk-Based
Random
6.44
Random
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
Higher confounding
Higher confounding ~
(a) Weighted Utilitarian Welfare
(b) Nash Welfare
Figure 5: Comparing risk-based targeting to biased treatment effect-based targeting on weighted utilitarian welfare and Nash social welfare for the Ultra Poor RCT.
16

---

Published as a conference paper at ICLR 2025
NSW Dataset
NSW Dataset
6000
Treatment Effect Based
10.6
Risk-Based
1000
Random
10.5
Utility
Utiliyy
10.4
2000
10.3
Treatment Effect Based
10.2
Risk-Based
-2000
Random
0.1
0.2
0.3
0.1
0.4
Higher confounding
Higher confounding
(a) Weighted Utilitarian Welfare
(b) Nash Welfare
Figure 6: Comparing risk-based targeting to biased treatment effect-based targeting on weighted utilitarian welfare and Nash social welfare for the NSW RCT.
Postoperative Pain Dataset
Postoperative Pain Dataset
Treatment Effect Based
Risk-Based
2.2
Random
2.1
Utility
Utility
2.0
1.9
Treatment Effect Based
Risk Based
1.8
Random
0.1
0.2
0.3
0.4
0.1
0.2
0.4
Higher confounding
Higher confounding
(a) Weighted Utilitarian Welfare
(b) Nash Welfare
Figure 7: Comparing risk-based targeting to biased treatment effect-based targeting on weighted utilitarian welfare and Nash social welfare for the Postoperative Pain RCT.
Acupuncture Dataset
Acupuncture Dataset
4.9
30
4.8
4.7
Utility
4.6
10
4.5
Treatment Effect Based
4.4
Treatment Effect Based
-10
Risk-Based
Risk Based
Random
Random
4.2
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
Higher confounding -
Higher confounding +
(a) Weighted Utilitarian Welfare
(b) Nash Welfare
Figure 8: Comparing risk-based targeting to biased treatment effect-based targeting on weighted utilitarian welfare and Nash social welfare for the Acupuncture RCT.
17

---

Published as a conference paper at ICLR 2025
EVALUATING BASED ON UTILITARIAN WELFARE WITH BUDGET 30% AND 40% OF A.4.2
THE POPULATION BUT USING OBSERVED PSEUDO OUTCOME INFORMATION
STAR Dataset
STAR Dataset
250
Treatment Effect Based
Treatment Effect Based
Risk-Based
200
Risk-Based
200
Random
Random
150
Utility
Utility
50
0
0.1
0.2
0.3
0.1
0.2
0.3
0.4
Higher confounding
Higher confounding
(a) Budget 30%
(b) Budget 40%
Figure 9: Comparison of risk-based assignment to biased treatment effect based assignment for the STAR dataset, with fixed budget of 30% and 40% the population respectively.
Ultra Poor Dataset
Ultra Poor Dataset
80
60
60
Utility
Utility
40
40
20
Treatment Effect Based
20
Treatment Effect Based
Risk-Based
Risk-Based
0
Random
Random
0.1
0.2
0.3
0.4
0.1
0.2 Higher confounding +
0.3
0.4
Higher confounding +
(a) Budget 30%
(b) Budget = 40%
Figure 10: Comparison risk- based assignment to biased treatment effect based assignment for the Ultra Poor dataset, with fixed budget of 30% and 40% of the population respectively.
18

---

Published as a conference paper at ICLR 2025
Postoperative Pain Dataset
Postoperative Pain Dataset
Treatment Effect Based
3.0
Treatment Effect Based
3.5
Risk-Based
Risk-Based
3.0
Random
2,5
Random
2.5
2.0
2.0
1.5
1.5
1.0
1.0
0.5
0.5
0.2 Higher confounding
0.2
Higher confounding
(a) Budget = 30%
(b) Budget = 40%
Figure 11: Comparison risk- assignment to biased treatment effect based assignment for the Postoperative Pain dataset, with fixed budget of 30% and 40% of the population respectively.
Acupuncture Dataset
Acupuncture Dataset
30
25
25
20
20
15
15
10
10
Treatment Effect Based
Treatment Effect Based
Risk-Based
Risk-Based
Random
Random
0.1
0.3 Higher confounding +
0.2
0.4
0.1
0.2
0.3
0.4
Higher confounding +
(a) Budget 30%
(b) Budget = 40%
Figure 12: Comparison risk- riske assignment to biased treatment effect based assignment for the Acupuncture dataset, with fixed budget of 30% and 40% of the population respectively.
NSW Dataset
NSW Dataset
Treatment Effect Based
10000
Treatment Effect Based
10000
Bisk-BBased
Risk-Based
Random
8000
Random
Utility
6000
Utility
6000
4000
4000
2000
2000
0
0.1
0.2
0.3
0.4
0.1
0.2
0.3 +
0.4
Higher confounding
+
Higher confounding
(a) Budget 30%
(b) Budget = 40%
Figure 13:
Comparison risk-based assignment to biased treatment effect based assignment for the NSW dataset, with fixed budget of 30% and 40% of the population respectively.
19

---

Published as a conference paper at ICLR 2025
A.4.3 EVALUATING BASED ON UTILITARIAN WELFARE WITH BUDGET 30% AND 40% OF THE POPULATION
STAR Dataset
STAR Dataset
Treatment Effect Based
100
80
Risk-Based
80
Random
60
Utility
Utility
Treatment Effect Based
20
Risk-Based
Random
0
0.1
0.2
0.3 +
0.4
0.1
0.2
0.3
0.4
Higher confounding
Higher confounding -
(a) Budget 30%
(b) Budget = 40%
Figure 14:
assignment to biased treatment effect based assignment for the STAR dataset, with fixed budget of 30% and 40% of the population respectively.
Comparison
Ultra Poor Dataset
Ultra Poor Dataset
Utility
Utility
20
10
Treatment Effect Based
10
Treatment Effect Based
Risk-Based Random
Risk-Based
Random
-10
0.1
0.2
0.3
0.4
0.1
0.2 Higher confounding +
0.3
0.4
Higher confounding -
(a) Budget = 30%
(b) Budget = 40%
Figure 15: Comparison bassed assignment to biased treatment effect based assignment for the Ultra Poor dataset, with fixed budget of 30% and 40% of the population respectively.
20

---

Published as a conference paper at ICLR 2025
Postoperative Pain Dataset
Postoperative Pain Dataset
2.5
Treatment Effect Based
2.5
Treatment Effect Based
2.0
Risk-Based
Risk-Based
Random
2.0
Random
Utility
Utility
1.0
1.0
0.5
0.5
0.0
0.0
-0.5
-0.5
0.2
0.3
0.2
0.3
Higher confounding
Higher confounding
(a) Budget 30%
(b) Budget 40%
Figure 16: Comparison of risk-based assignment to biased treatment effect based assignment for the Postoperative Pain dataset, with fixed budget of 30% and 40% of the population respectively.
Acupuncture Dataset
Acupuncture Dataset
Treatment Effect Based
15
Treatment Effect Based
Risk-Based
Risk-Based
Random
10
Random
Utility
Utility
5
5
0
-5
0.1
Higher confounding
Higher confounding
(a) Budget = 30%
(b) Budget = 40%
Figure 17: Comparison risk-based assignment to biased treatment effect based assignment for the Acupuncture dataset, with fixed budget of 30% and 40% of the population respectively.
NSW Dataset
NSW Dataset
6000
Treatment Effect Based
4000
Treatment Effect Based
Risk-Based
Risk-Based
4000
Random
3000
Random
2000
Utility
Utility
-1000
-2000
-2000
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
Higher confounding
Higher confounding +
(a) Budget = 30%
(b) Budget = 40%
Comparison fisk-based assignment to biased treatment effect based assignment for the NSW dataset, with fixed budget of 30% and 40% of the population respectively.
Figure 18:
21


Reviews:
[{"review_id": "Cbv8nLwP1O", "reviewer": "Reviewer_4SA7", "summary": "The authors compare the utility of targeting interventions based on estimated treatment effects with the utility of targeting based on predicted risk. The former is the method of choice from a utilitarian perspective. But practitioners and policy makers often choose the latter due to its simplicity and in cases where the normative goal is to assist those in greatest need.", "questions": "Visually, it seems difficult to reconcile the large difference between, say, Figure 2 top left panel and Figure 1 top left panel. It seems that on the TUP dataset, high risk is close to maximizing treatment effect, the highest values being around percentile 80. However, Figure 2 shows utility 15000 vs 5000 for treatment effect based vs risk vased. Can you explain?\n\nSimilarly, treatment effects for NSW seem to be largely constant around 1000, but Figure 2, second row left panel shows a massive advantage for treatment effect based targeting. This seems to be about a factor 7 (1.75 vs 0.25). Where do these large effect differences come from given that the treatment effect curve is essentially constant? None of the treatment effects seem to differ by more than a factor 2.\n\nI tried to figure this out by looking through the code, but I couldn't find code that generated Figure 2. So, I don't actually know where it came from and what it shows. Maybe I missed it. On that note, it would be very helpful to clean up and document the code for the final version. As is, it's very hard to follow.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1)\n\nThe comparison of these two targeting strategies is an important problem. I'm glad to see that the authors study this question. Despite its significance, there hasn't been much reliable insight so far. I'd love to see more work in this direction!  I'm weighing this strongly in my evaluation.\n\n(2)\n\nThe paper is very clearly written. The authors narrate a compelling story about the advantages of targeting based on treatment effect estimates, even if they are biased.", "weaknesses": "(1)\n\nAs compelling as the story is, I find the results less than conclusive. Looking at Figure 1, it looks like the confidence intervals around treatment effects are generally strongly overlapping across the entire x-axis of baseline risk. The one exception is the STAR dataset, where the lowest and highest point estimate have barely non-overlapping confidence intervals. As a result, another story consistent with the data is that effects are generally not so heterogeneous as conventional wisdom from the econometrics literature has it.\n\n\n(2)\n\nThe results in Figure 1 are actually a fair bit more favorable towards risk-based targeting than the introduction of the paper had me believed. Treatment effects generally increase with risk. Targeting the 80th to 90th percentile of risk generally seems to capture high treatment effects across all datasets. So, another story could be that we should exclude the most extreme values of risk from targeting, but other than that risk-based targeting sort of works.\n\n\n(3)\n\nI found it rather confusing to have unnormalized utility values on the y-axis. After all the sample size is rather arbitrary and does not correspond to the population-level utility obtained if the policy maker were to implement the given approach. Along the same lines, I found it difficult to reconcile Figure 1 and Figure 2. See question below.\n\n\n(4)\n\nIt would've been great to include datasets with real world confounding rather than the simulated confounding. My experience is that existing CATE estimation methods don't do very well in non-RCT settings. Might there be an advantage to risk-based targeting in non-RCT data?\n\n\nSuggestion:\n\nIt seems to me that the story is much less certain than the abstract and introduction make it sound. I would therefore appreciate it if you could indicate a greater level of epistemic uncertainty in the writing throughout. I don't think this would hurt the paper at all. As is, though, I'd worry that your writing suggests the question is essentially closed conclusively which would actually discourage additional work in this direction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SwDKZFHrOI", "reviewer": "Reviewer_1WAp", "summary": "The paper explores targeting strategies for allocating limited resources to maximize social welfare in areas like\nhuman services, healthcare, and education. Specifically, it compares risk-based targeting, which prioritizes\nhigh-risk individuals, with treatment-effect-based targeting, which uses machine learning models to estimate who\nmight benefit the most from interventions. Using five real-world randomized controlled trials (RCTs) across diverse\ndomains, the paper concludes that even biased estimates of treatment effects generally outperform risk-based\ntargeting. This finding suggests in addition to the widespread reliance on risk-based approaches, policymakers\ncould incorporate treatment effect estimation when feasible.", "questions": "Modeling Real-World Confounding: Could the authors expand on how their confounding\napproach aligns with real-world biases encountered in observational data? A discussion on potential limitations in modeling confounding factors might aid readers in interpreting results for specific applications.\n\nEthical Implications: How might the authors' conclusions address ethical concerns,\nparticularly in terms of balancing fairness (people at the most rish) with effectiveness\nwhen treatment effect targeting benefits some groups more than others? Because the paper concludes that a treatment-effect-based method is preferable to a risk-based method, it raises\nthe natural question of whether individuals at the highest risk should be prioritized, or if those with the greatest\npotential treatment outcome should be targeted instead.\n\nData Quality: Could the authors provide more information on the reliability and source of the\nAcupuncture Dataset?\n\nConfidence Interval Estimation: In Equation (5), a biased estimator is used. Could the authors\njustify why this choice was made? Was it primarily for computational efficiency, or are there\nother reasons for using a biased estimator in this context?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Scope: The work’s target is evident in its cross-domain empirical evaluation, making it applicable to multiple policy\nareas. The paper introduces innovative use of biased treatment effect estimates to assess targeting efficacy.\nQuality: Methodologically, the paper is solid, employing credible RCT data and a robust approach to measuring\ntreatment effect heterogeneity. The use of doubly robust estimation and varying social welfare functions adds\ndepth to the analysis.", "weaknesses": "Grammatical errors: there are notable grammatical errors in spelling and following the formatting of ICLR\ninstructions.\n\nScope of Treatment Effect Estimates: The reliance on simulated confounding could be more thoroughly justified;\nreal-world application requires consideration of domain-specific biases that could vary across different policy areas\nPotential Ethical Implications: Since the work could influence resource allocation in sensitive domains, further\ndiscussion on ethical considerations regarding inequality and bias would strengthen its impact and application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hk60eZl38S", "reviewer": "Reviewer_XFvr", "summary": "The paper compares ''risk-based targeting'' and ''treatment effect-based targeting'' across five datasets from RCTs in different domains. In risk-based targeting, the planner targets individuals based on their baseline risk, which is the expected outcome in the absence of treatment. This approach, commonly used by practitioners, does not account for causal effects. In contrast, treatment effect-based targeting involves first estimating the Conditional Average Treatment Effect (CATE) and then prioritizing individuals accordingly, often with the help of machine learning. A key concern is that these methods may introduce bias in the presence of unobserved confounding. The paper aims to provide empirically grounded guidance for navigating this tradeoff.\n\nFirst, the paper investigates the extent to which baseline risk serves as an effective proxy for targeting. To do this, treatment effects are estimated at different levels of baseline risk, revealing a surprisingly complex and not necessarily monotonic relationship between treatment effect and baseline risk. Second, the paper examines the potential additional gains from targeting based on treatment effect. Both utilitarian and Nash welfare are used to compare these mechanisms, showing that targeting based on estimated treatment effect can be up to three times more effective in nearly all cases. This remains true even when confounding is introduced into the CATE estimation and when the policymaker's preference for risk-based targeting is encoded in the social welfare function.", "questions": "Feel free to further discuss the first two weaknesses mentioned above.\n\nI have seen papers, such as [1], that discuss the complexities of estimating allocation welfare under budget constraints. The issue is that these constraints introduce dependencies between individual estimations, complicating the evaluation process. How is this relevant to your setting?\n\nShouldn't the results also depend on how many individuals are treated in each scenario? My intuition (supported by studies like [2]) suggests that the relative budget plays a significant role in determining which type of allocation mechanism is optimal.\n\nSince you studied a wide range of settings, did you find any recommendations that vary across these contexts? For instance, are there conditions under which risk-based targeting performs better?\n\nCould you elaborate on the choice of outcome of interest in the TUP dataset? It seems there’s an implicit assumption that individuals who show a larger increase in expenditure are more deserving of intervention. Why should this be the case? In contrast, a risk-based approach might assume that those who would have lower expenditures in the absence of interventions are more deserving. Could this actually be a more appropriate assumption?\n\n[1] Improved Policy Evaluation for Randomized Trials of Algorithmic Resource Allocation\n[2] Allocation Requires Prediction Only if Inequality Is Low", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This is an important issue, and I’ve personally never found a clear guideline on how best to approach it. I found the study original and of good quality.\n\nThe paper does a commendable job of incorporating datasets from diverse domains. \n\nIt also simulates the impact of unobserved confounders and considers the possibility of policymaker bias toward risk-based targeting.", "weaknesses": "My main concern is the reliability of comparing the estimated welfare from risk-based versus treatment effect-based targeting. The issue with the latter approach, particularly in resource-scarce settings, is that it first estimates CATE, then selects individuals with the highest estimated CATE given the available budget, and finally estimates the welfare of treating these individuals using the same estimated CATE. Even if the CATE is unbiased, this procedure can lead to severely biased welfare estimates in resource-constrained settings.\n\nIn Figure 1, the paper provides treatment effect estimates for each baseline risk percentile, along with confidence intervals. What exactly are we looking for here? If the goal is to rule out a monotonic relationship between baseline risk and treatment effect, none of the figures offer statistically significant evidence to support this.\n\nI also think the results should intuitively depend on the budget but this is not discussed in the paper. For instance, if the budget is very small, treatment effect-based allocation can be even more effective in exploiting heterogeneity. \n\nOverall, the writing is strong. I noticed the following minor typos: page 2 (potentially-based -> potentially-biased), page 4 (policy -> policy), page 5, second equation, page 7, Equation 4.5 is referenced which does not exist", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pZxGufgDgl", "reviewer": "Reviewer_WHnh", "summary": "This paper studies a common risk-based approach to treatment allocation applied to a diverse range of treatments with randomized controlled trial data available. The authors find that allocating treatment according to risk (ie, bad outcomes under the no-treat baseline) produces worse outcomes than allocating according to conditional average treatment effect estimates, even when CATEs are biased or the decision maker has a preference for treating high-risk individuals.", "questions": "Fig. 2 is great, almost telling the story of the paper without the audience needing to read anything else. What would make it even better is replacing \"Percentage of data removed\" with something like \"Higher values mean more confounding in CATE estimates\". Currently readers have to read the methodology to understand the x axis, but a better label would mean they could understand the x axis without knowing exactly how the confounding was introduced, and read the methodology if they wanted more details.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a simple, well-executed paper making a very important point. As the authors note, most algorithmic decision making approaches are based on risk or some manipulation of risk in pursuit of fairness goals. This paper demonstrates the problem with that approach across a wide range of interventions.\n\nThe authors anticipate the key criticisms of the policy based approach (that RCT data is hard to find and so CATEs may be biased, and that equity preferences might provide a non-utilitarian reason to prefer risk-based targeting), and provide compelling evidence that even under significant confounding or equity preferences one should still prefer allocations based on CATEs.", "weaknesses": "The weakest part of this paper is probably the discussion of Fig 1. The authors cite a \"unique trend for each dataset\" (in CATE conditional on risk score) as evidence that the risk-based policy is flawed. Visually, to me, the trends look pretty similar: they look like there is basically no relationship between CATE and risk for most RCTs studied. This is still evidence for the risk-based policy being flawed (so it doesn't invalidate the claim being made), but I think it's a more accurate way to describe the results. I think Fig 2 is more compelling than Fig 1 anyway, so it probably make sense to lead with that result for the most impactful paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)