Comparing Targeting Strategies for Maximizing Social Welfare with Limited Resources

## Abstract

Abstract Machine learning is increasingly used to select which individuals receive limited-resource interventions in domains such as human services, education, development, and more. However, it is often not apparent what the right quantity is for models to predict. In particular, policymakers rarely have access to data from a randomized controlled trial (RCT) that would enable accurate estimates of treatment effects – which individuals would benefit more from the intervention. Observational data is more likely to be available, creating a substantial risk of bias in treatment effect estimates. Practitioners instead commonly use a technique termed "risk-based targeting" where the model is just used to predict each individual’s status quo outcome (an easier, non-causal task). Those with higher predicted risk are offered treatment. There is currently almost no empirical evidence to inform which choices lead to the most effect machine learning-informed targeting strategies in social domains. In this work, we use data from 5 real-world RCTs in a variety of domains to empirically assess such choices. We find that risk-based targeting is almost always inferior to targeting based on even biased estimates of treatment effects. Moreover, these results hold even when the policymaker has strong normative preferences for assisting higher-risk individuals. Our results imply that, despite the widespread use of risk prediction models in applied settings, practitioners may be better off incorporating even weak evidence about heterogeneous causal effects to inform targeting.

## 1 Introduction

Policymakers often face the difficulty of allocating a resource-limited intervention with the goal of targeting the intervention towards those who will benefit most from it. Indeed, the causal inference literature documents that any given treatment may not have the same effect on every individual that receives it [, , ]. When there are observable features that correlated with greater benefit from the treatment, such variation can be used for targeting. Heterogeneity of treatment effect (HTE) refers to this nonrandom, explainable variability in the direction and magnitude of treatment effects for individuals within a population. Given this variability, policymakers often face the problem of selecting who to treat when having to assign a particular treatment to a group of people under a fixed budget. Machine learning methods seem to offer the promise of discovering richer forms of heterogeneity, allowing more effective targeting of interventions in practice.

The main challenge is that heterogeneous treatment effects are difficult to learn: doing so requires a potentially large amount of data that is unconfounded, i.e., where treatment is assigned in a manner (conditionally) independent of each individual’s outcomes. In an idealized setting, one could conduct a randomized controlled trial (RCT) and experimentally find the subpopulations that benefit most from a treatment. However, this is not always possible for two reasons: 1) Conducting an RCT takes time (potentially years) and resources that the policymaker may not be willing to spend. 2) In some domains, there are ethical objections to experimentation. For example, while the RCT is being conducted, people in genuine need of the treatment could be assigned to the control group and suffer negative outcomes. Policymakers may prefer to prioritize access to treatment via an already-available proxy metric that is believed to align with need if experimenting to gather additional evidence would be seen as unethical.

One particularly common proxy is to target according to the baseline risk each individual faces, i.e., their expected outcome in the absence of treatment (as opposed to the treatment effect, which is the difference in outcomes induced by treatment). This strategy has been referred to as risk-based targeting []. Individuals with poor predicted baseline outcomes may be seen as needing assistance the most. Policymakers may also believe that these individuals also stand the most to benefit since their status-quo prognosis is the worst. Importantly, baseline risks can often be learned using existing administrative data (from before a treatment was introduced) instead of requiring a new experiment, making this strategy easily implementable in many practical settings. For all of these reasons, risk-based targeting has seen widespread use by policymakers in a wide range of domains, including targeting humanitarian assistance [], allocating homelessness services via vulnerability scores [], and the use of “early warning systems" in education [].

Despite this widespread usage, there is only a limited amount of work which empirically assesses the effectiveness of risk-based targeting: do individuals with the greatest baseline risk actually tend to benefit the most from intervention? The few existing studies speak only to a single, specific application domain each. [] study an RCT where students in a university program were provided a nudge (treatment) as a reminder to renew their financial aid application, concluding that students with intermediate non-renewal risk saw the largest treatment effect. Students with greatest risk of non-renewal, who would be targeted under a risk-based strategy, saw less benefit. [] study a marketing domain and use two field experiments to show that targeting high-risk customers, or customers likely to churn as predicted by a machine learning model, can be ineffective, encouraging practitioners to use RCTs to better inform their decision. However, as discussed above, running a RCT may be infeasible in many settings. The alternative more likely to be available to practitioners is to simply estimate treatment effects using observational data which likely suffers from confounding, potentially leading to biased estimates of treatment effects.

How should practitioners navigate this tradeoff between a more easily-learnable label that may not always correlate with benefit from an intervention (baseline risk) and a difficult-to-learn quantity (heterogeneous treatment effects) that captures the impact of the intervention? This corresponds to the choice of the appropriate target for prediction, as opposed to the specific model used to make the prediction. The choice of outcome variable has been observed to exert a disproportionate influence on the impacts of machine learning systems in many settings [, , ]. In the setting of targeting interventions for causal impact, practitioners have little empirically-grounded guidance. Our goal in this work is to inform the selection of an objective function for machine-learning based targeting of scarce interventions. We make three contributions towards this goal:

First, we assess the efficacy of risk-based targeting on a wider variety of real RCT datasets encompassing settings in economics, healthcare and education in contrast to prior studies that generally focus on one dataset. We demonstrate how high values of baseline risk do not always correspond to high treatment effects, rendering risk-based targeting suboptimal if the policymaker adopts a typical utilitarian goal of maximizing the expected improvement in outcomes from the intervention.

Second, we compare risk-based targeting to targeting policies based on biased estimates of treatment effect obtained from confounded data. Such biased estimates are likely when conducting a full-fledged RCT is infeasible and policymakers have to rely on available observational data alone. Accordingly, potentially-based causal estimates represent the likely alternative to risk-based targeting in many domains of practical interest. To our knowledge, these two strategies have not been explicitly compared. We find that across even relatively severe levels of confounding, a utilitarian policymaker often prefers targeting according to biased estimates of the treatment effect rather than baseline risk.

Finally, we analyze the setting where a policymaker has inequality-averse preferences: oftentimes, policymakers may prefer interventions which benefit those who are worse-off to begin with even if they produce less aggregate impact. Such normative goals are one possible justification for risk-based targeting, even if risk-based targeting is less attractive in standard utilitarian terms. We compare the two targeting strategies under popular classes of social welfare functions which capture inequality-averse preferences. We find that treatment effect based targeting is almost always favorable to risk-based targeting, even in scenarios where policy makers are inequality-averse and the only available data is confounded to some degree.

## 2 Related Work

Measuring heterogeneity in treatment effect and choosing which subpopulations to assign a treatment to has long been an active avenue of research in causal inference literature with a variety of methods proposed to solve this problem. [, , , , ] use forest-based algorithms to identify groups that show heterogeneity in treatment effect with other identified groups. [] proposed to measure the interaction between treatment and covariates by numerically binarizing the treatment and including the products of this variable with each covariate in a regression model. [] uses meta-learners that decompose estimating the CATE into several sub-regression problems that can be solved with any regression or supervised learning method. The problem of choosing who to treat is closely related to identifying the heterogeneity in treatment effects. This often involves balancing policies based solely on estimates of conditional average treatment effect (CATE) with additional prioritization rules set by the policymaker. [] proposes rank-weighted average treatment effect metrics for testing the quality of treatment prioritization rules, providing an example involving optimal targeting of aspirin to stroke patients.

## 3 Preliminaries

Consider a setting where there is a population of individuals who are candidates for a treatment or intervention. Each individual has a feature vector $X\in\mathbb{R}^{d}$. Here we are concerned with binary treatments. Following Neyman ([]) and Ruben’s ([]) potential outcomes framework, we use $Y^{(1)}$ to denote the outcome that an individual would experience under treatment and $Y^{(0)}$ to denote the outcome they would experience if not treated. Their individual treatment effect, quantifying their benefit from receiving treatment, is $Y^{(1)}-Y^{(0)}$. We assume that $(X,Y^{(0)},Y^{(1)})$ are drawn i.i.d. for each individual from some joint distribution. In order to identify individuals who are likely to benefit, a common strategy is to use individuals’ observed covariates to predict the expected treatment effect. The conditional average treatment effect (CATE) at $X=x$ is defined as:

$$ $\tau(x)=\mathbb{E}\left[Y^{(1)}-Y^{(0)}\middle|X=x\right].$ (1) $$

Estimating the CATE in order to target based on treatment effects is a difficult statistical problem. Suppose we have access to data corresponding to $n$ people, labeled $i=1,...,n$, consisting of features $X_{i}$, a treatment assignment $W_{i}\in\{0,1\}$, and the observed outcome $Y_{i}=Y_{i}^{(W_{i})}$. Importantly, for each individual, we can observe only the outcome corresponding to the treatment they were actually assigned. Accordingly, identifying treatment effects typically requires a a no-unobserved-confounders assumption []:

$$ $\{Y^{(0)},Y^{(1)}\}\perp\!\!\!\perp W\mid X.$ (2) $$

This assumption is most credible in the context of a randomized controlled trial (RCT). In an RCT, the assignment of treatment, represented by $W_{i}$, is assigned independently of the potential outcomes $Y_{i}$ (potentially after stratification on covariates $X_{i}$). When data is purely observational, practitioners typically try to select a sufficiently rich set of covariates $X$ such that all potential confounders between outcomes and treatment assignment are measured. However, ensuring that all confounders are completely captured is notoriously difficult in practice, creating the likelihood that some bias in the estimated CATE remains [, , , ].

As an alternative to targeting on treatment effects, policymakers often decide to treat people who are more vulnerable or worse-off at present, without attempting to quantify the benefit these individuals receive from treatment. This is quantified via a ’baseline risk’; we refer to the resulting allocation strategy as ’risk-based targeting’[]. Baseline risk may sometimes be directly measured quantity (one of the covariates in $X$, for example baseline test scores in an educational context). In many settings though, it is estimated using a predictive model that uses the covariates as input. Let $b$ be a function that maps a set of covariates to a baseline risk measurement such that $b(X)$ represents the baseline risk and $b(X_{i})$ denotes the baseline risk associated with $i$. Then this method involves selecting individuals with the highest values of $b$ for treatment, implying that these individuals have the highest ’risk’ associated with them, which needs immediate resolution. It is important to note that this strategy requires only data on baseline outcomes prior to program implementation, with no information about the treatment’s effect being incorporated in the policymaker’s decision.

The example from [] makes the distinction between these two methods clearer. In an experiment where the objective was to "nudge" or remind students in a college program to renew their financial aid applications, targeting based on baseline risk assumes that students predicted to be least likely to renew their aid applications (as determined by a machine learning model) should be prioritized. Meanwhile, the results of the RCT conducted during this experiment show that high values of treatment effect correspond to students with intermediate likelihoods of renewing their aid applications prior to treatment, demonstrating in a disparity in the two methods of targeting.

## 4 Methods

### 4.1   Overview

Our goal is to compare risk-based targeting to treatment effect-based targeting on possibly confounded datasets, with varying degrees of confounding and under different social welfare functions for the policymakers making the treatment assignment polciy. To enable this comparison, we use a range of datasets from real-world randomized controlled trials (RCTs). Using RCTs enables us to credibly estimate heterogeneous treatment effects since the no-unobserved-confounders assumption is guaranteed to be satisfied. Then, we simulate each of the targeting policies of interest and compare their effectiveness with respect to the randomization-enabled treatment effect estimates, under varying utility functions for the policymaker. We now detail the methodology used in each step of this process, starting with the datasets used.

### 4.2   Datasets

We conduct experiments on a variety of RCTs across different domains as detailed below:

- •
Targeting the Ultra Poor (TUP) in India ([]): This RCT was conducted to study the long-term effects of providing large one-time capital grants to low income-families and observing how family income and overall consumption changed over a period of 7 years. We consider a family’s total expenditure as the outcome, which is positively affected by treatment.
- •
NSW (National Supported Work demonstration) Dataset ([, , ]: This study estimated the impact of the National Supported Work Demonstration, a job training program, on beneficiaries’ income in 1978. We consider an individual’s income in 1978 as the outcome, which is positively affected by treatment.
- •
Postoperative Pain Dataset:
Patients undergoing operations like tracheal intubations often experience throat pain following treatment []. This RCT was conducted to test the efficacy of a licorice solution at reducing postoperative sore throat. The outcome we focus on is a patient’s throat pain 4 hours after surgery. Here, the effect of the treatment is to reduce the amount of throat pain, hence the treatment effect is negative. In order to maintain consistency with other plots, we present results with the sign for treatment effect reversed.
- •
Acupuncture Dataset: This RCT aimed to determine the effect of acupuncture therapy on headache severity in patients with chronic headaches. Our outcome variable is headache severity 1 year post-randomization. Here again, the effect of the treatment is to reduce the severity of headaches, hence the treatment effect is negative. In order to maintain consistency with other plots, we present results with the sign for treatment effect reversed.
- •
Tennessee’s Student Teacher Achievement Ratio (STAR) project []: The Tennessee State Department of Education conducted a comprehensive four-year study called the Student/Teacher Achievement Ratio (STAR) to examine the effects of class size on student performance.
The study design included three different classroom configurations:
1) Small classes with 13-17 students per teacher, 2) Regular classes with 22-25 students per teacher, 3) Regular classes with 22-25 students plus a full-time teacher’s aide.
In this paper, we only focus on the first two types of classes mentioned above, so as to maintain consistency with treatment value being binary in other RCTs.
We focus on students in kindergarten and a cumulative measure of their scores on various tests as the outcome under consideration.

For each of these datasets, we estimate $E[Y^{(0)}|X]$ using a machine learning model applied to the RCT’s control group and set $b(X)=E[Y^{(0)}|X]$ or $b(X)=-E[Y^{(0)}|X]$ as appropriate (i.e., depending on whether larger outcome values are better or worse). For instance, children with lower baseline test scores in the STAR dataset and patients with high baseline headache severity in the Acupuncture Dataset are considered to be high risk.
Additional details about preprocessing for all datasets are included in A.

### 4.3   Estimating heterogeneous treatment effects

We estimate heterogeneous treatment effects on each dataset using a doubly-robust estimator []. The DR estimator splits the data into separate folds. For each fold, we estimate models for both the expected outcome and the treatment variable (estimating the latter even when the propensity scores are known can increase statistical efficiency []). Let $\hat{\mu}(X,A)$ be the estimated mean outcome for an individual with covariates $X$ and treatment assignment $A$, and $\hat{\pi}(X)$ be the estimated propensity score. For each individual in the held-out data for the fold, we estimate their pseudo-outcomes, defined as

$$ $\chi_{i}(A)=\hat{\mu}(X_{i},A)+\frac{1[W_{i}=A](Y_{i}-\hat{\mu}(X_{i},A))}{A \hat{\pi}(X_{i})+(1-A)(1-\hat{\pi}(X_{i}))}.$ $$

If at least one of $\hat{\mu}$ or $\hat{\pi}$ is correctly specified, $\chi_{i}(A)$ has expectation (over the random treatment assignment) equal to $Y_{i}^{(A)}$, which allows us to use it as a proxy for the unobserved outcomes in evaluating counterfactual evaluation policies.

### 4.4   Exploring treatment effect heterogeneity with respect to baseline risk

Our first analysis tests one potential rationale for risk-based targeting strategies: the hypothesis that individuals with greater baseline risk will also tend to have greater treatment effects. We frame this as estimating $E[Y^{(1)}-Y^{(0)}|b(X)]$, a conditional average treatment effect just with respect to value of the risk score $b$.

We follow the doubly-robust approach to estimating CATEs, where the pseudo-outcome difference $\chi_{i}(1)-\chi_{i}(0)$ is regressed on the covariates of interest []. Because our covariate of interest, $b$, is one-dimensional, we use a kernel regression method to estimate the CATE as a generic smooth function.
Specifically, we employ a Gaussian kernel smoothing method. We sort every individual/household by their baseline risk and center an adaptive Gaussian kernel about each data point.
The bandwidth of each kernel is determined adaptively based on the local density of the data, defined as half the range of baseline risk values within a fixed window of 200 data points. This ensures that we are able to estimate greater variation in data-rich regions of the space, while imposing greater smoothness at the extremes where less data is present.

Given the kernel function $K(u)=\exp(-\frac{1}{2}u^{2})$, the CATE estimate at $b(X_{i})$ is given by:

$$ $\hat{\tau}(b(X_{i}))=\frac{\sum_{j=1}^{n}K(\frac{b(X_{j})-b(X_{i})}{\sigma_{i} })\hat{\tau}_{j}}{\sum_{j=1}^{n}K(\frac{b(X_{j})-b(X_{i})}{\sigma_{i}})}$ (3) $$

where $\hat{\tau}_{j}$ is the estimated difference in pseudo outcomes, for unit $j$, $\chi_{j}(1)-\chi_{j}(0)$, as determined by the doubly robust estimator, and $\sigma_{i}$ is the adaptive bandwidth calculated as:

$$ $\sigma_{i}=\frac{1}{2}(b(X_{i+100})-b(X_{i-99}))$ (4) $$

for a window of 200 data points centered at $i$. The confidence intervals are computed using a weighted variance estimate:

$$ $\text{CI}=\hat{\tau}(b(X_{i}))\pm 1.96\sqrt{\frac{\sum_{j=1}^{n}K(\frac{b(X_{j })-b(X_{i})}{\sigma_{i}})^{2}(\hat{\tau}_{j}-\hat{\tau}(b(X_{i})))^{2}}{(\sum_ {j=1}^{n}K(\frac{b(X_{j})-b(X_{i})}{\sigma_{i}}))^{2}}}$ (5) $$

This approach allows us to capture the heterogeneity in treatment effects across different levels of baseline risk while accounting for the varying density of data points.

### 4.5   Introducing Confounding

Our next analysis aims to simulate conditions where we do not have access to perfectly conducted randomized controlled trials for our problem, in order to compare risk-based targeting to a plausible alternative in real world settings: targeting according to observational, and potentially biased, estimates of the CATE.

We introduce varying levels of confounding to the RCTs that we study. We do this by simulating adverse selection into treatment, where units are more likely to be observed if the estimated individual-level treatment effects deviate from the mean. Specifically, we generate the biased “observational" dataset by removing data in a systematic manner. This process, inspired by [], is controlled by a parameter $k$ giving the fraction of data removed, with higher $k$ corresponding to more biased estimates.

From the treated units, we remove the examples that lie in the top $k\%$ percent when ordered in descending order of $(\chi_{i}(1)-\chi_{i}(0))$ (assuming treatment effect is positive) while for the untreated units, we remove the examples that lie in the bottom $k\%$ of examples when ordered in descending order of $(\chi_{i}(1)-\chi_{i}(0))$. In simpler terms, for treated units, we remove examples for which the treatment ’went well’(most positive), while for untreated units, we remove examples for which the lack of treatment did not go well(least positive). The new dataset we obtain now has $k\%$ of the original examples systematically removed to increase confounding and can only provide biased estimates of treatment effect.

Our goal is to compare policies learned using this biased data to risk-based targeting. This evaluation is enabled by access to the original, randomized data. Consider a hypothetical policy that assigns treatments $A(X)\in\{0,1\}$ as a function of individuals’ features $X$. The mean outcome under policy $A$, $E[Y^{(A(X))}]$, can be decomposed as

$$ $\displaystyle E[Y^{(A(X))]}=E[Y^{(0)}]+Pr(A(X)=1)E[Y^{(1)}-Y^{(0)}|A(X)=1].$ $$

The term $E[Y^{(1)}-Y^{(0)}|A(X)=1]$ represents the treatment effect on the treated population and captures how much the policy improves over no treatment. For policies with the same budget (equal $\Pr(A(X)=1)$), only this term varies and so we assess allocation policies by their expected treatment-on-the-treated. Following standard doubly-robust estimators for (group) average treatment effects [], we empirically estimate this quantity as

$$ $\frac{1}{\sum_{j=1}^{n}A(X_{j})}\sum_{j=1}^{n}A(X_{j})(\chi_{j}(1)-\chi_{j}(0)).$ $$

### 4.6   Families of Welfare Functions

In order to simulate policymakers with varying preferences for who to treat, we compare risk-based targeting to treatment-effect based targeting on
some general utility/social welfare functions that fall under the category of ’weighted power mean functions’ as described in [].
The weighted power mean $M(\mathbf{u};\mathbf{w},p)$ for $\mathbf{u}\in\mathbb{R}^{d}_{+}$, $\mathbf{w}\in\Delta_{d-1}$, and $p\in\mathbb{R}\cup\{\pm\infty\}$ is defined as:

$$ $M(\mathbf{u};\mathbf{w},p)=\begin{cases}\left(\sum_{i=1}^{d}w_{i}u_{i}^{p} \right)^{1/p}&p\neq 0\\ \prod_{i=1}^{d}u_{i}^{w_{i}}&p=0\end{cases}$ $$

We choose this family of welfare functions because functions in this family satisfies a standard set of axiomatic properties [].
The parameter $p$ determines the specific type of welfare function:

- •
Utilitarian welfare ($p=1$):
$M(\mathbf{u};\mathbf{w},p=1)=\sum_{i=1}^{d}w_{i}u_{i}$
- •
Nash welfare ($p=0$):
$M(\mathbf{u};\mathbf{w},p=0)=\prod_{i=1}^{d}u_{i}^{w_{i}}$

We consider utilitarian welfare with two sets of weights. First, the uniform weights $w_{i}=1\forall i\in[n]$. Second, $w_{i}=\frac{n\cdot e^{\alpha b^{\prime}(X_{i})}}{\sum_{j=1}^{n}e^{\alpha b^{
\prime}(X_{j})}}$ where $\alpha$ is a hyperparameter and $b^{\prime}(X_{i})$ is represents the percentile score of the baseline risk for the $i^{\text{th}}$ example, with the example with highest baseline risk having score 1 and the example with lowest baseline risk having score 0. This assigns greater weight to individuals with high values of baseline risk for high values of $\alpha$, thereby simulating a scenario where a policymaker might value treating treating these "high risk" individuals for other reasons. $\alpha$ can be interpreted as $2\log\left(\frac{w_{75}}{w_{25}}\right)$ where $w_{75}$ is the weight given to the $75^{\text{th}}$ percentile example by baseline risk and $w_{25}$ is the weight given to the $25^{\text{th}}$ percentile example by baseline risk.

The Nash social welfare function has commonly been used to achieve a balance between maximizing total welfare(utilitarian) and ensuring equitable distribution(egalitarian) [, ]. Egalitarian welfare can sometimes leads to inefficiencies while utilitarian welfare can lead to unjust outcomes: Nash welfare often strikes a useful compromise between these two ends. We consider unweighted Nash welfare ($w_{i}=1\forall i$). In order to avoid the complications of utility when using an unweighted Nash welfare function, we scale up the estimated utilities for each example to a minimum value of 1.

Note that the Nash welfare can be equivalently formulated in log space [] as

$$ $\displaystyle\frac{1}{n}\sum_{i=1}^{n}\log u_{i}.$ $$

When each individual’s utility under an allocation policy corresponds to their realized outcome $Y_{i}^{(A(X_{i}))}$ (e.g., their income after the interventional period), we compare policies exactly as outlined in Equation 4.5, but with $Y^{(A(X))}$ replaced by $\log Y^{(A(X))}$, estimated by replicating the same procedure after taking logs of all outcome variables.

## 5 Results

Figure 1 shows how treatment effect varies as a function of baseline risk for each of the 5 datasets we study, with 95$\%$ confidence intervals shaded around the estimated treatment effects. These intervals are pointwise Wald-type confidence intervals [] and provide a measure of uncertainty for our smoothed estimates.
Treatment effects often increase with baseline risk, confirming some of the intuition behind risk-based targeting. However, the relationship is typically significantly more complex than a strictly monotone curve. Predicted gains often level off at a level of baseline risk which varies based on the dataset. In more detail:

- •
The Ultra Poor Dataset exhibits highest treatment effects at high levels of baseline risk: treatment effect first decreases then increases consistently before slightly tapering off.
- •
The NSW Dataset does not show any apparent trends in treatment effect as a function of baseline risk with the curve being mostly flat.
- •
The Postoperative Pain Dataset shows a sigmoid-like curve, with treatment effects increasing more rapidly at moderate levels of baseline risk.
- •
For the Acupuncture Dataset, we see treatment effects peaking at moderate levels of baseline risk after an initial dip and then slightly decreasing again before taking on mostly similar values for examples with high baseline risk.
- •
The Tennessee STAR Dataset displays a generally increasing trend in treatment effects with baseline risk, but with notable fluctuations.

These diverse patterns underscore the complexity of the relationship between baseline risk and treatment effectiveness. Rather than assuming a particular relationship, it is crucial to empirically investigate the heterogeneity of treatment effects for each specific intervention and population.

Figure: Figure 1: Observing Treatment Effect Heterogeneity across Different Settings by plotting Treatment Effect against Baseline Risk for each of our 5 datasets. We observe a unique trend for each dataset, indicating a lack of a consistent well-defined relation between the two quantities
Refer to caption: extracted/5993193/new_iclr/ultrapoor_perc.png

Figure: (a)
Refer to caption: extracted/5993193/new_iclr/ultrapoor_n1.png

Figure 2 shows the comparison between risk-based targeting and treatment effect based targeting for the 5 datasets we study, with varying degrees of confounding and under different social welfare functions used to represent policymaker preferences. For reference, we also show the performance of a random targeting strategy which allocates the same budget but targets uniformly random individuals. Both targeting methods seem to perform better than random targeting for almost all combinations of dataset/welfare functions we consider. The improved performance of risk-based targeting over random reflects the generally increasing relationship that Figure 1 shows between baseline risk and treatment effects. However, when treatment effects can be accurately estimated ($k=0$, no confounding), targeting based on treatment effects always produces significantly higher utilitarian welfare (often producing a treatment-on-the-treated effect of three times or more greater than risk-based targeting). This indicates that when a policymaker seeks only to maximize aggregate benefit and can credibly estimate treatment effects, the gains from causal targeting are substantial.

As the level of confounding bias in treatment effect targeting increases (increased $k$), its effectiveness decreases. However, when the policymaker has utilitarian preferences, targeting based on biased treatment effect estimates still performs at least as well as risk-based targeting (and typically better) across all datasets, even for relatively severe levels of confounding. This indicates that using even relatively biased observational data to learn treatment rules is likely superior when the policymaker’s goal is just to maximize aggregate gain.

The second column of Figure 2 shows an alternative set of preferences, where the policymaker has a weighted utilitarian welfare function (for these plots, $\alpha$ is $2\log 2$, the value at which the ratio of the 75th percentile weight to the 25th percentile weight is 2) which places greater weight on individuals with higher baseline risk, attaching a higher importance to the welfare of more vulnerable individuals. This welfare function decreases the gap between treatment effect and risk based targeting as individuals with higher risk now have more weight associated with them. However, targeting on (biased) treatment effects is still preferable to risk-based targeting across all datasets.

This motivates us to ask: how much greater must the policymaker weight high-risk individuals in order to prefer risk-based targeting? In Table I we give the minimum value of $\alpha$ at which risk-based targeting finally outperformed treatment effect based-targeting at each level of systematic data removal. We limit $\alpha$ such that the ratio of the $75^{\text{th}}$ percentile weight to the $25^{\text{th}}$ percentile weight when sorted in descending order is less than $100$; otherwise, only a few individuals have non-zero weights and estimating welfare gains becomes impossible. In general, we note that the required $\alpha$ values tend to be lower when we increase $k$, which follows directly from the fact that the treatment effect estimates become more biased at higher $k$. We note that most values in the table are at least $\alpha=4$ at which the policymaker places over 7 times more weight on the welfare of an individual at the $75\text{th}$ percentile of baseline risk than an individual at the $25\text{th}$ percentile. High values of $\alpha$ act as an indicator that a policymaker prefers to target "higher risk" individuals more than others. Interestingly, for the STAR dataset, there is no $\alpha$ value (up to our upper bound) at which risk based targeting outperforms targeting based on treatment effects. We conclude that relatively extreme welfare weights are needed to rationalize risk-based targeting.

**Table I: Values of $\alpha$ for different k at which risk-based targeting outperforms treatment effect based targeting. ’na’ indicates no such $\alpha$ was found**
| Dataset | 5% | 10% | 15% | 20% | 25% | 30% | 35% | 40% |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ultra Poor | na | 8 | 6.5 | 4.75 | 3.5 | 3 | 2 | 0.25 |
| NSW | na | 6.5 | 8.25 | 4.5 | 4.25 | 3.5 | 1.5 | 1 |
| Postoperative Pain | 7.5 | 7.5 | 5 | 8.75 | 8 | 6.5 | 6.5 | 5.5 |
| Acupuncture | na | 7.5 | na | 6.75 | 6 | 3.75 | 4.5 | 5.75 |
| STAR | na | na | na | na | na | na | na | na |

Under the Nash social welfare function (third column of Figure 2), we observe some differences across our chosen settings but the general trend remains the same: as we increase confounding (increase the percentage of data we systematically remove), the benefit we accrue by following a treatment assignment policy based on biased treatment effect values decreases. At high levels of confounding, risk-based targeting accrues higher utility for a policymakers with a Nash social welfare function in the Ultra Poor setting in 2. However, across all other datasets, the policymaker prefers to target based on treatment effects even at high levels of confounding.

Collectively, these results indicate that policymakers are almost always better off targeting interventions based on treatment effect estimates rather than baseline risk values from a machine learning model, unless they are extremely inequality-averse.

## 6 Conclusion

This paper presents a systematic comparison between two of the most popular treatment assignment policies in use by policymakers today in risk-based targeting and treatment effect based targeting. Using data collected in 5 randomized controlled trials in different settings, we propose a systematic method of introducing confounding in our data and show that policies that rely on biased estimates of treatment effect from confounded data gradually perform worse as we further confound the data. We also provide empirical evidence of non-monotonic heterogeneity in treatment effect as a function of the baseline risk, implying that risk-based targeting is often suboptimal from the perspective of maximizing the overall effect of treatment assignment. Importantly, our analysis extends beyond a simple unweighted utilitarian approach, encompassing weighted utilitarian and Nash welfare functions that respectively prioritize high-risk individuals and fairness. We argue that while risk-based targeting can be effective in certain contexts, especially when RCTs are infeasible and the observational data available is heavily confounded, incorporating even biased information about treatment effects is likely to result in improved welfare.

## Appendix A Appendix

### A.1   Datasets

- •
Targeting the Ultra Poor (TUP) in India ([]): This RCT was conducted to study the long-term effects of providing large one-time capital grants to low income-families and observing how family income and overall consumption changed over a period of 7 years. We consider a family’s total expenditure as the outcome, which is positively affected by treatment. We filter the dataset before use by removing null values and performing feature selection to limit the number of covariates. The dataset consists of 796 examples post filtering. We quantify baseline risk $b(X)$ as an estimate of baseline expenditure $E[Y(0)|X]$ from a machine learning model, with low values of $E[Y(0)|X]$ corresponding to high baseline risk and vice versa. This follows the hypothesis that households with low expenditure at baseline will benefit most from the treatment.
While constructing a doubly robust estimator to estimate pseudo outcomes for this dataset, we found the estimated propensity scores to be very high/low for certain examples, which would consequently scale pseudo outcome estimates to unusually large values. Therefore, we manually set propensity scores uniformly according to the treated:untreated ratio in the RCT.
- •
NSW (National Supported Work demonstration) Dataset ([, , ]: This is a popular causal inference dataset that was used to estimate the impact of the National Supported Work Demonstration, a job training program, on beneficiaries’ income in 1978.
The covariates include demographic variables like age, race, marital status and academic background, along with the benficiary’s income in 1975 prior to the experiment as a baseline. The dataset consists of 722 examples (297 treated and 425 control). Here too, we use an estimate of an individuals baseline income as a measure of risk, following the hypothesis that low income individuals will benefit more from the treatment.
- •
Tennessee’s Student Teacher Achievement Ratio (STAR) project ([]): The Tennessee State Department of Education conducted a comprehensive four-year study called the Student/Teacher Achievement Ratio (STAR) to examine the effects of class size on student performance. This research, backed by the Tennessee General Assembly, involved 11601 students across 79 schools.
The study design included three different classroom configurations:
–
Small classes with 13-17 students per teacher
–
Regular classes with 22-25 students per teacher
–
Regular classes with 22-25 students plus a full-time teacher’s aide
To ensure unbiased results, both students and teachers were randomly assigned to these different classroom types. The experiment began when the participants entered kindergarten and continued through their third-grade year, allowing for a longitudinal analysis of the impact of class size on educational outcomes. In this paper, we only focus on the first two types of classes mentioned above, so as to stay consistent with treatment value being binary in other RCTs.
This large-scale research project aimed to provide empirical evidence on the relationship between class size and student achievement. Again, we filter the dataset before use by removing null values and performing feature selection to limit the number of covariates. We focus on students in kindergarten and a cumulative measure of their scores on various tests as the outcome under consideration. The filtered dataset consists of 3712 students examples. Since we do not have the students’ test scores at baseline, we train a random forest model on rows corresponding to students who did not receive the treatment with their test scores at endline being the outcome variable. The prediction offered by this model for every student is then used as a proxy for their baseline test scores and the negative of this value is used as baseline risk. This follows the general hypothesis that students with low test scores need the treatment more.
- •
Postoperative Pain Dataset:
Patients undergoing operations like tracheal intubations often experience throat pain following treatment[]. This RCT was conducted to test the efficacy of gargling with licorice solution prior to endotracheal intubation on reducing postoperative sore throat, which is a common side-effect of the procedure. The investigation involved 236 adult participants scheduled for elective thoracic surgeries necessitating the use of double-lumen endotracheal tubes. The outcome we focus on is a patient’s throat pain 4 hours after surgery, measured on a discrete Likert scale from 0 to 7. Additional covariates include a patient’s gender, BMI, age, Mallampati score, preoperative pain, surgery size and smoking status. Here, the effect of the treatment is to reduce the amount of throat pain, hence the treatment effect is negative. In order to maintain consistency with other plots, we present results with the sign for treatment effect reversed. Since we do not have a measured value for throat pain at baseline, we again train a random forest model on rows corresponding to patients who did not receive the treatment with their throat pain at endline being the outcome variable. The prediction offered by this model for every patient is then used as a proxy for their baseline throat pain and consequently as the baseline risk. This follows the intuition that patients with more severe throat pain require the treatment more than their co-patients.
- •
Acupuncture Dataset: This RCT aimed to determine the effect of acupuncture therapy on headache severity in patients with chronic headaches. These measures were assessed at randomization, 3 months post-randomization, and 1 year post-randomization. We focus on headache severity 1 year post-randomization. Headache severity is measured on a discrete Likert scale from 0 to 5. The dataset consists of data from 401 patients with covariates including patient age, sex, chronicity(number of years of headache severity) and whether the headaches were diagnosed as migraines or not. Here again, the effect of the treatment is to reduce the severity of headaches, hence the treatment effect is negative. In order to maintain consistency with other plots, we present results with the sign for treatment effect reversed. We estimate headache severity at baseline $E(Y(0)|X]$ using a machine learning model and use it as a proxy for baseline risk, following the intuition that patients with more severe headaches need the treatment more.

### A.2   Additional Plots

Figure: (a)
Refer to caption: extracted/5993193/new_iclr/star_n2.png

Figure: (a)
Refer to caption: extracted/5993193/new_iclr/ultrapoor_n2.png

Figure: (a)
Refer to caption: extracted/5993193/new_iclr/licorice_n2.png

Figure: (a)
Refer to caption: extracted/5993193/new_iclr/acupuncture_n2.png

Figure: (a)
Refer to caption: extracted/5993193/new_iclr/lalonde_n2.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Visually, it seems difficult to reconcile the large difference between, say, Figure 2 top left panel and Figure 1 top left panel. It seems that on the TUP dataset, high risk is close to maximizing treatment effect, the highest values being around percentile 80. However, Figure 2 shows utility 15000 vs 5000 for treatment effect based vs risk vased. Can you explain?

Similarly, treatment effects for NSW seem to be largely constant around 1000, but Figure 2, second row left panel shows a massive advantage for treatment effect based targeting. This seems to be about a factor 7 (1.75 vs 0.25). Where do these large effect differences come from given that the treatment effect curve is essentially constant? None of the treatment effects seem to differ by more than a factor 2.

I tried to figure this out by looking through the code, but I couldn't find code that generated Figure 2. So, I don't actually know where it came from and what it shows. Maybe I missed it. On that note, it would be very helpful to clean up and document the code for the final version. As is, it's very hard to follow.
soundness: 3
strengths: (1)

The comparison of these two targeting strategies is an important problem. I'm glad to see that the authors study this question. Despite its significance, there hasn't been much reliable insight so far. I'd love to see more work in this direction!  I'm weighing this strongly in my evaluation.

(2)

The paper is very clearly written. The authors narrate a compelling story about the advantages of targeting based on treatment effect estimates, even if they are biased.
summary: The authors compare the utility of targeting interventions based on estimated treatment effects with the utility of targeting based on predicted risk. The former is the method of choice from a utilitarian perspective. But practitioners and policy makers often choose the latter due to its simplicity and in cases where the normative goal is to assist those in greatest need.
weaknesses: (1)

As compelling as the story is, I find the results less than conclusive. Looking at Figure 1, it looks like the confidence intervals around treatment effects are generally strongly overlapping across the entire x-axis of baseline risk. The one exception is the STAR dataset, where the lowest and highest point estimate have barely non-overlapping confidence intervals. As a result, another story consistent with the data is that effects are generally not so heterogeneous as conventional wisdom from the econometrics literature has it.


(2)

The results in Figure 1 are actually a fair bit more favorable towards risk-based targeting than the introduction of the paper had me believed. Treatment effects generally increase with risk. Targeting the 80th to 90th percentile of risk generally seems to capture high treatment effects across all datasets. So, another story could be that we should exclude the most extreme values of risk from targeting, but other than that risk-based targeting sort of works.


(3)

I found it rather confusing to have unnormalized utility values on the y-axis. After all the sample size is rather arbitrary and does not correspond to the population-level utility obtained if the policy maker were to implement the given approach. Along the same lines, I found it difficult to reconcile Figure 1 and Figure 2. See question below.


(4)

It would've been great to include datasets with real world confounding rather than the simulated confounding. My experience is that existing CATE estimation methods don't do very well in non-RCT settings. Might there be an advantage to risk-based targeting in non-RCT data?


Suggestion:

It seems to me that the story is much less certain than the abstract and introduction make it sound. I would therefore appreciate it if you could indicate a greater level of epistemic uncertainty in the writing throughout. I don't think this would hurt the paper at all. As is, though, I'd worry that your writing suggests the question is essentially closed conclusively which would actually discourage additional work in this direction.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Modeling Real-World Confounding: Could the authors expand on how their confounding
approach aligns with real-world biases encountered in observational data? A discussion on potential limitations in modeling confounding factors might aid readers in interpreting results for specific applications.

Ethical Implications: How might the authors' conclusions address ethical concerns,
particularly in terms of balancing fairness (people at the most rish) with effectiveness
when treatment effect targeting benefits some groups more than others? Because the paper concludes that a treatment-effect-based method is preferable to a risk-based method, it raises
the natural question of whether individuals at the highest risk should be prioritized, or if those with the greatest
potential treatment outcome should be targeted instead.

Data Quality: Could the authors provide more information on the reliability and source of the
Acupuncture Dataset?

Confidence Interval Estimation: In Equation (5), a biased estimator is used. Could the authors
justify why this choice was made? Was it primarily for computational efficiency, or are there
other reasons for using a biased estimator in this context?
soundness: 3
strengths: Scope: The work’s target is evident in its cross-domain empirical evaluation, making it applicable to multiple policy
areas. The paper introduces innovative use of biased treatment effect estimates to assess targeting efficacy.
Quality: Methodologically, the paper is solid, employing credible RCT data and a robust approach to measuring
treatment effect heterogeneity. The use of doubly robust estimation and varying social welfare functions adds
depth to the analysis.
summary: The paper explores targeting strategies for allocating limited resources to maximize social welfare in areas like
human services, healthcare, and education. Specifically, it compares risk-based targeting, which prioritizes
high-risk individuals, with treatment-effect-based targeting, which uses machine learning models to estimate who
might benefit the most from interventions. Using five real-world randomized controlled trials (RCTs) across diverse
domains, the paper concludes that even biased estimates of treatment effects generally outperform risk-based
targeting. This finding suggests in addition to the widespread reliance on risk-based approaches, policymakers
could incorporate treatment effect estimation when feasible.
weaknesses: Grammatical errors: there are notable grammatical errors in spelling and following the formatting of ICLR
instructions.

Scope of Treatment Effect Estimates: The reliance on simulated confounding could be more thoroughly justified;
real-world application requires consideration of domain-specific biases that could vary across different policy areas
Potential Ethical Implications: Since the work could influence resource allocation in sensitive domains, further
discussion on ethical considerations regarding inequality and bias would strengthen its impact and application.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Feel free to further discuss the first two weaknesses mentioned above.

I have seen papers, such as [1], that discuss the complexities of estimating allocation welfare under budget constraints. The issue is that these constraints introduce dependencies between individual estimations, complicating the evaluation process. How is this relevant to your setting?

Shouldn't the results also depend on how many individuals are treated in each scenario? My intuition (supported by studies like [2]) suggests that the relative budget plays a significant role in determining which type of allocation mechanism is optimal.

Since you studied a wide range of settings, did you find any recommendations that vary across these contexts? For instance, are there conditions under which risk-based targeting performs better?

Could you elaborate on the choice of outcome of interest in the TUP dataset? It seems there’s an implicit assumption that individuals who show a larger increase in expenditure are more deserving of intervention. Why should this be the case? In contrast, a risk-based approach might assume that those who would have lower expenditures in the absence of interventions are more deserving. Could this actually be a more appropriate assumption?

[1] Improved Policy Evaluation for Randomized Trials of Algorithmic Resource Allocation
[2] Allocation Requires Prediction Only if Inequality Is Low
soundness: 2
strengths: This is an important issue, and I’ve personally never found a clear guideline on how best to approach it. I found the study original and of good quality.

The paper does a commendable job of incorporating datasets from diverse domains. 

It also simulates the impact of unobserved confounders and considers the possibility of policymaker bias toward risk-based targeting.
summary: The paper compares ''risk-based targeting'' and ''treatment effect-based targeting'' across five datasets from RCTs in different domains. In risk-based targeting, the planner targets individuals based on their baseline risk, which is the expected outcome in the absence of treatment. This approach, commonly used by practitioners, does not account for causal effects. In contrast, treatment effect-based targeting involves first estimating the Conditional Average Treatment Effect (CATE) and then prioritizing individuals accordingly, often with the help of machine learning. A key concern is that these methods may introduce bias in the presence of unobserved confounding. The paper aims to provide empirically grounded guidance for navigating this tradeoff.

First, the paper investigates the extent to which baseline risk serves as an effective proxy for targeting. To do this, treatment effects are estimated at different levels of baseline risk, revealing a surprisingly complex and not necessarily monotonic relationship between treatment effect and baseline risk. Second, the paper examines the potential additional gains from targeting based on treatment effect. Both utilitarian and Nash welfare are used to compare these mechanisms, showing that targeting based on estimated treatment effect can be up to three times more effective in nearly all cases. This remains true even when confounding is introduced into the CATE estimation and when the policymaker's preference for risk-based targeting is encoded in the social welfare function.
weaknesses: My main concern is the reliability of comparing the estimated welfare from risk-based versus treatment effect-based targeting. The issue with the latter approach, particularly in resource-scarce settings, is that it first estimates CATE, then selects individuals with the highest estimated CATE given the available budget, and finally estimates the welfare of treating these individuals using the same estimated CATE. Even if the CATE is unbiased, this procedure can lead to severely biased welfare estimates in resource-constrained settings.

In Figure 1, the paper provides treatment effect estimates for each baseline risk percentile, along with confidence intervals. What exactly are we looking for here? If the goal is to rule out a monotonic relationship between baseline risk and treatment effect, none of the figures offer statistically significant evidence to support this.

I also think the results should intuitively depend on the budget but this is not discussed in the paper. For instance, if the budget is very small, treatment effect-based allocation can be even more effective in exploiting heterogeneity. 

Overall, the writing is strong. I noticed the following minor typos: page 2 (potentially-based -> potentially-biased), page 4 (policy -> policy), page 5, second equation, page 7, Equation 4.5 is referenced which does not exist

---

confidence: 4
contribution: 4
initial_rating: 10
presentation: 4
questions: Fig. 2 is great, almost telling the story of the paper without the audience needing to read anything else. What would make it even better is replacing "Percentage of data removed" with something like "Higher values mean more confounding in CATE estimates". Currently readers have to read the methodology to understand the x axis, but a better label would mean they could understand the x axis without knowing exactly how the confounding was introduced, and read the methodology if they wanted more details.
soundness: 4
strengths: This is a simple, well-executed paper making a very important point. As the authors note, most algorithmic decision making approaches are based on risk or some manipulation of risk in pursuit of fairness goals. This paper demonstrates the problem with that approach across a wide range of interventions.

The authors anticipate the key criticisms of the policy based approach (that RCT data is hard to find and so CATEs may be biased, and that equity preferences might provide a non-utilitarian reason to prefer risk-based targeting), and provide compelling evidence that even under significant confounding or equity preferences one should still prefer allocations based on CATEs.
summary: This paper studies a common risk-based approach to treatment allocation applied to a diverse range of treatments with randomized controlled trial data available. The authors find that allocating treatment according to risk (ie, bad outcomes under the no-treat baseline) produces worse outcomes than allocating according to conditional average treatment effect estimates, even when CATEs are biased or the decision maker has a preference for treating high-risk individuals.
weaknesses: The weakest part of this paper is probably the discussion of Fig 1. The authors cite a "unique trend for each dataset" (in CATE conditional on risk score) as evidence that the risk-based policy is flawed. Visually, to me, the trends look pretty similar: they look like there is basically no relationship between CATE and risk for most RCTs studied. This is still evidence for the risk-based policy being flawed (so it doesn't invalidate the claim being made), but I think it's a more accurate way to describe the results. I think Fig 2 is more compelling than Fig 1 anyway, so it probably make sense to lead with that result for the most impactful paper.

Meta-review: The paper compares two targeting strategies for allocating limited resources in social welfare applications: risk-based targeting and treatment-effect-based targeting. Using data from five real-world randomized controlled trials (RCTs), the authors find that targeting based on treatment effect estimates can outperform risk-based targeting, even when treatment effect estimates are biased or when policymakers prioritize assisting high-risk individuals. The study highlights the potential benefits of applying causal inference methods in policymaking, emphasizing that treatment-effect-based approaches yield greater social welfare gains across various scenarios.

Overall, the reviewers acknowledge that the paper addresses an important research question, employs a rigorous approach, and has significant potential impact. The main concerns revolve around the discussion of the results (e.g., Figure 1 and the new results added in rebuttal, as highlighted by XFvr). Considering the potential impact of the research and its generally strong execution, I am inclined to recommend accepting the paper. If accepted, please carefully address the reviewer comments when preparing the final version.

Decision: Accept (Poster)