The Value of Prediction in Identifying the Worst-Off
Authors: Unai Fischer-Abaigar
Christoph Kern
Juan Carlos Perdomo
Venue: ICML 2025 oral (2025)
Keywords: algorithmic decision making
resource allocation
machine learning and public policy
Primary area: social_aspects->fairness

Abstract: Machine learning is increasingly used in government programs to identify and support the most vulnerable individuals, prioritizing assistance for those at greatest risk over optimizing aggregate outcomes. This paper examines the welfare impacts of prediction in equity-driven contexts, and how they compare to other policy levers, such as expanding bureaucratic capacity. Through mathematical models and a real-world case study on long-term unemployment amongst German residents, we develop a comprehensive understanding of the relative effectiveness of prediction in surfacing the worst-off. Our findings provide clear analytical frameworks and practical, data-driven tools that empower policymakers to make principled decisions when designing these systems.

The Value of Prediction in Identifying the Worst-Off
Unai Fischer- Christoph Kern Juan Carlos Perdomo
without reference to the broader social context. The value Machine learning increasingly used in govern- ment programs to identify and support the most vulnerable individuals, prioritizing assistance for those at greatest risk over optimizing aggregate outcomes. This paper examines the welfare im- pacts of prediction in equity-driven contexts, and how they compare to other policy levers, such as expanding bureaucratic capacity. Through math- ematical models and real-world case study on long-term unemployment amongst German resi- dents, we develop comprehensive understanding of the relative effectiveness of prediction in sur- facing the worst-off. Our findings provide clear analytical frameworks and practical, data-driven tools that empower policymakers make princi- pled decisions when designing these systems.
Abstract
of risk predictor is ultimately determined by its impact on bottom-line welfare (e.g., graduation rates) and how these welfare impacts compare to those of other bureau- cratic alternatives (Johnson & Zhang, 2022). For example, to understand whether investments in prediction are truly valuable in Wisconsin, we need to assess how much better the risk predictor is identifying at-risk students relative to existing policies. We also need understand whether sophisticated prediction systems yield higher graduation rates amongst the underserved than structural investments in teacher training or better facilities.
Equity-driven programs are pervasive in applications like social housing, poverty targeting, and unemployment as- sistance. In these contexts, many government agencies are exploring how algorithmic prediction systems may be an improvement over their current profiling processes (Körtner & Bonoli, 2023). Yet, due to the absence of an overarching framework that allows the systematic assessment of the rela- tive impacts of different design decisions, efforts to improve predictive accuracy are rarely studied in concert with other policy levers such as expanding screening capacity.
1. Introduction
Faced with pressure to modernize, large bureaucracies are increasingly adopting risk prediction tools to improve ef- ficiency and better serve their populations. Beyond opti- mizing aggregate outcomes, investments these programs often aim to address historical inequities and prioritize the needs of the worst-off. For instance, in 2012, Wiscon- sin launched risk prediction system to explicitly address deep racial disparities in academic achievement and im- prove high school graduation rates amongst underserved students. More broadly, such systems are particularly rel- evant in settings where normative considerations demand prioritizing those at the greatest risk of adverse outcomes, and where well-established downstream interventions can meaningfully benefit these vulnerable individuals.
Building on recent work in budding area of learning in resource allocation contexts, we develop tools to evaluate the design and broader impact of prediction systems that aim to identify the worst-off members of a population. We develop a holistic understanding of the value of statistical prediction in these contexts through theoretical insights into foundational statistical models and real-world case study identifying long-term unemployment. Our results estab- lish clear theoretical and empirical criteria characterizing the relative value of core design decisions within these prob- lems. Specifically, we identify when improving prediction provides higher marginal benefit in helping an institution identify the worst-off. This is compared to alternative strate- gies, such as keeping prediction accuracy fixed, expanding bureaucratic capacity and sereening larger population.
From design perspective, these risk predictors are chal- lenging to evaluate because their value cannot be assessed
'Department of Statistics, University of Munich (LMU), Mu- nich. Germany MMunich Center for Machine Learning Germany 2 'Harvard University, Boston, MA, US. Correspondence to: Unai Fischer Fischer-Abaigar Fisheriiscchrraaaaaaaee stat.uni-man muenchen.de>.
Interestingly, we show that prediction is a first and last-mile effort. The impacts of improving prediction are always outweighed by those of expanded screening capacity, except for when the system explains either none or almost all of the variance in outcomes. While this relationship is moderated by costs, still largely holds when prediction improvements
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Value of Prediction in Identifying the Worst-Off
are more cost-efficient than measures that expand access.
tom 5%, they might have to screen 10% of the population.
These results are counternarrative to current efforts in em- pirical public policy where agencies focus on incremental improvements within complex prediction systems, start- ing from the solid baseline performances of their current processes (Desiere et al., 2019; Desiere & Struyven, 2021). Furthermore, implementing more complex profiling systems at scale comes with operational costs (such as staff training and data collection) which need to be contextualized by the cost-benefit ratio of expanding access. Our empirical case study explicates how to systematically assess the relative gains of these design components in a real-world application setting, translating formal insights into critical guidance for designers of these systems.
Given predictor f. a screening budget of a, and target parameter B. the value of a prediction system is equal to the fraction of the at-risk population that it identifies,
V (a, f: - PrD(f(>) & t(a) Y < ((3)),
where again t(a),,(3) are chosen to respect the design constraints. We focus on this notion of value since our driving motivation is to analyze domains like unemployment assistance, or poverty prediction, where there is no harm in the prediction system raising a false positive (π(x;) 1, > t(3).. By and large, the true value of the system is equal to the extent that it helps an institution efficiently identify the needy amongst large, diverse population.
Our results provide theoretically principled and empirically grounded tools for policymakers to make informed decisions when designing prediction systems to identify the worst-off. They also offer a practical framework to help determine how much should be invested in prediction relative to other interventions and how to decide when prediction systems are "good enough" for deployment.
The focus of our work is to build a holistic understanding of prediction in these contexts by evaluating the relative impacts of different design parameter, such as expanding screening capacity or improving prediction, on this notion of lote value (a, f). We develop these insights through theoretical investigations as well as in-depth empir- ical case study.
Mathematical Results. Following Perdomo (2024), we formalize the relative value of prediction for the worst-off by studying the prediction-acee access ratio or PAR. Intuitively, the PAR measures the relative change in value achieved by optimizing different policy levers.
1.1. Overview of Framework and Contributions
Setup. We consider a scenario where a decke seeks to identify worst-off members of a population, as de- termined by a real-valued welfare metric Y P. with the goal of prioritizing them for further screening and support. The population is represented by distribution D over fea- tures X and outcomes Y. The planner aims to identify all individuals whose outcomes Y fall below some threshold (()), Y & t(3). Here, 3 E [o, 1] is a parameter (quantile) that determines the size of the population that is at risk, Pr[y & [[3)] = B. For instance, in poverty prediction, Y is income, and the goal is to identify everyone whose income is below some value.
Marginal Value of Expanding Access
PAR = Marginal Value of Better Prediction
We formally define this quantity in Equation 3. While ini- tially developed to specifically study the value of prediction in allocation problems where allocating goods to individu- als had heterogeneous effects, here we extend this concept to analyze the value of prediction in a related, but distinct, setting where we aim to identify the worst-off. To solve this problem, the social planner has access to data (X.Y) YY) ~ 'D and builds a screening policy क : X' + (0, 1) that determines whether an individual with features æ is screened from the broader population to see if they belong to the worst-off group. Learning plays a fundamental role since the optimal policy is predict each person's 's expected outcome, f(x)= = Y x E [Y| X = x| and screen those in the bottom fraction, गy (x) = (x) < t(a)}.
Small values of the PAR (i.c. PAR < 1) indicate that small improvements in prediction yield much larger (relative) impact in the ability to target the worst-off than small expansion in screening capacity. The opposite is true if the PAR is greater than 1. Calculating this quantity is a funda- mental step in deciding which policy lever makes economic sense.
Costs. A full cost-benefit analysis requires combining the prediction-access ratio with the (marginal) costs of improve- ments in capacity CAsess and prediction Cpred- Once we factor in costs, it is easy to decide what to focus on. A social planner should expand access whenever
Unpacking this further, a E [0, 1]. is a design parameter that determines how many people the planner can screen, Pr(f(x) & t(a)]= = a. The amount of resources a need not be equal to the size of the target population B3. For instance, an organization might have normative goal of identifying the poorest 5% of individuals, but only have the resource to screen 1% of the population. Conversely, they might realize that predictions are not perfect, and that to identify the bot-
CACOSS Cred
PAR
and invest in better prediction otherwise. The (marginal) costs that competing policy levers carry are inherently
2

---

The Value of Prediction in Identifying the Worst-Off
context-dependent and will vary across application domains. In many applied settings the cost ratio is comparatively well understood, for example, the salary of an additional case- worker or the cost of a household survey. By presenting PAR separately from costs, we isolate the welfare side of the equation; domain experts can then plug in their own cost estimates to reach a policy decision. In particular, the PAR tells us how much we should be willing to pay for improvements in prediction versus expanding access.
ratio at least Ma -1(1-R()) Furthermore, if &
R² & 0.85 and & 3 or 0.2 & R & 0.5, B t 0.15, and & 0.5 then the local prediction-access ratio is at least 1.
Empirical Results. We complement our theoretical dis- cussion by presenting a methodology for policymakers to evaluate the prediction- ratio in practice. Using a real-world administrative dataset on hundreds of thousands of jobseekers in Germany, we show that our theoretical find- ings generalize to more complex, real-world context that closely resembles algorithmic profiling systems widely im- plemented in many countries. Notably, our results reveal that when considering non-local improvements, expanding screening capacity has an even greater impact compared to enhancing prediction accuracy.
We encourage future work to explore scenarios with more complex or less clearly defined cost structures. For instance, many practical applications involve recurring costs, such as ongoing staff salaries or periodic data collection, and fixed costs, such as initial investments in infrastructure or predictive model development. Analyzing how these cost structures affect welfare decisions over time, including amortization of fixed investments or identifying the point at which specific improvements become cost-effective, would significantly enhance our understanding of the relative value of prediction.
1.2. Related Work
Machine learning is increasingly used in the public sec- tor to allocate support by predicting individuals at risk of adverse outcomes Frischer-Abaight et al., 2024), with ap- plications spanning wide range of problem domains (De- siere et al., 2019; Blumenstock, 2016; Perdomo et al., 2023; Chan et al., 2012; Potash et al., 2015; Chouldechova et al., 2018). A large methodological literature draws on deci- sion theory, operations research, economics, and machine learning learn allocation rules from data (Elmachtoub & Grigas, 2022; Kitagawa & Tetenov, 2018; Manski, 2004; Fernández-Loria & Provost, 2022), with recent work in causal inference focusing on learning treatment policies from observational data (Athey & Wager, 2021; Kallus, 2021). However, many decision-makert rely on separately trained predictive risk scoring-systemt to solve "prediction policy problems" (Kleinberg et al., 2015). Recently, this work has been extended using causal inference to train and evaluate these systems (Coston et al., 2023; Guerdan et al., 2023; Boehmer et al., 2024).
To build intuition for the value of prediction in identifying the worst-off, we examine the prediction access ratio in one of the most basic statistical models. The outcomes Y are Gaussian, and the learner has access to predictor f(z) = y such that the errors Y - Y are also Gaussian and independent of Ý While extremely simple, the model yields surprisingly precise numerical insights that exactly match up in our real-world case study, where, of course, none of these assumptions hold. In this setting the quality of Y is fully summarized by the coefficient of determination R2 = corr(X (e, y).
Our first result identifies when local improvements in pre- diction have the highest impact:
Theorem 1.1 (Informal, see Theorem 3.2). a is at least constant, the local improvements in V with respect to R2
diverge in two regimes: (1) R2 + 1 and a = B. or (2) R² + O. In both cases, the prediction-accen ratio satisfies PAR(a, = O.
The widespread use of risk-scoring systems has raised con- cerns regarding their tradeoffs, pitfalls, and validity (Wang et al., 2024; Coston et al., 2023; Fischer-Abaigar et al., 2024). These concerns include not only questions of em- pirical performance but also of fairness and equity in how predictive systems shape access to public services (Baro- cas et al., 2023). Recent work explores alternative de- sign choices-such as employing aggregate rather than individual-Ievel predictions (Shirali et al., 2024), balanc- ing immediate needs with information gathering (Wilder & Welle. 2024), and introducing randomization (Jain et al., 2024) to improve downstream outcomes.
Predictions have the highest marginal impact at low and high -Vauuu, making them first- and last-mile effort. Our second result characterizes when the opposite is true. We prove that whenever screening capacities are severely limited relative to the size of the population one aims to iden- tify a < 3, the benefits of increasing a are overwhelming. Furthermore, it shows that the impacts of improving access are still relatively larger exactly in the regime where most current systems operate: / explains s 20% of the variance and a is equal to, or even slightly larger, than B.
Perdomo (2024) studies the prediction-accest ratio under both linear and probit models, with the latter closely related to our work. While they focus on binary welfare outcomes, we adopt a continuous welfare metric and a distinct policy objective: rather than evaluating changes in overall expected
Theorem 1.2 (Informal, see Theorem 3.1, Proposition 3.3). If the predictor f explains an R² fraction of the variance, where R2 is at least a constant, then the prediction access
3

---

The Value of Prediction in Identifying the Worst-Off
welfare, we measure the fraction of truly worst-off individ- uals who are identified. This captures a mathematically and conceptually distinct setting frequently encountered in the public sector. For instance, employment agencies often prioritize identifying and assisting individuals in greatest need, rather than optimizing average employment outcomes across all jobseekers. In addition, we introduce a set of empirical tools to analyze these tradeoff's in practice, while the work of Perdomo (2024) is purely theoretical.
In this model, the goodness of the predictions Y are en- tirely captured by the coefficient of determination R.,, which equals the squared correlation p² between Y and Y.
Our analysis extends to the log-normal distribution log Y ~ (1,, n7") under a a multiplicative error model Y = Y u with log u ~ (O 7²). Taking logarithms, leads to log Y log Y + log u. Since the logarithm is strictly in- creasing, the ordering of Y and Y is preserved under trans- formation. This allows us apply the same framework to the oogttransformed variables log Y and log Y. This exten- sion is particularly useful because many welfare outcomes, such as income distributions (Clementi & Gallegati, 2005), can be approximated by a log-normal distribution.
2. Formal Framework
We start by formally defining our screening problem.
Definition 2.1 (Screening Problem). The screening problem seeks to identify decision rule æ: R - {0, } that fraction of the worst-off population that is identified while adhering to resource constraints E (0, 1) that bound the percentage of the population that can be screened by the social planner:
Visualization. For a given screening capacity a and H²² value, we can illustrate the corresponding screening policy by plotting the probability Pr(Y < F (a) Y y that an individual with welfare outcome Y = y is screened. As shown in Figure I, lower values of Y correspond to higher probabilities of being screened. We focus on evaluating how effectively the screening policy identifies individuals in the worst-off segment of the population (i.e., on the left side of the B cutoff).
E (P) = 1 < PT s.t. E (+() < a x: (0.1)
max
The quantile Fr (B) denotes the welfare cutoff that identi- fies the worst-off / E (0. 1) fraction of the population.
Given perfect knowledge of the welfare outcomes y=Y; Y, the optimal decision policy is simple: rank individuals based on their outcomes Y and intervene in the bottom a- a-fraction of the population. In the general case, we have:
3. Theoretical Results
The decision-maker has (at least) two pathways to raise the policy value, which we refer to as policy levers:
Proposition 2.2. The optimal policy x²: R + +00,11 1} to
Expanding Access Increasing the screeming threshold from a a + Aa.- If full screening were possible
solve the screeming problem (Definition 2.1) is equal to
r" r²())) 1 (Y.) > F-1(1 - -a}} where F-1(1 o)) is
(a = 1), the 8-fraction would be fully identified, as V(T*) = 4 ( (a)"(()))--0())))
the (1- - a)) -quantile of sYY) Pr(y < Fy (B) (B) Y.
*(*-
88 1.
8
Policy Value in Gaussian Setting. For the theoretical in- vestigation, we assume independent, identically distributed errors = Y y sid AT ?2) that are independent of Ý. In this setting, the screening problem can be solved by ranking individuals in ascending order of their predicted outcomes Y and screening the bottom o-fraction (see Proposition C.I), achieving the policy value:
Improving Predictions Investing in better predic- tive models, modeled as increasing R² to R + ARe. Perfect predictions (R² 1) leads to op- timal allocation of available capacities: V(T*)
1 (min(+-1 (a), 0-1 (3))
Figure showcases improvements in access and prediction. Increasing capacity expands the fraction of the population screened, while improving R2 shifts probability mass across the 3 threshold, enhancing targeting accuracy.
V(**) = Pr(y < (a) Y & FF
In addition, assume welfare outcomes Y ~ N(H, n²) Because & is independent of y, this implies that Y and Y follow a bivariate normal distribution.
Following Perdomo (2024), a key quantity of interest is the prediction-access ratio (PAR), which quantifies the relative improvements in policy value from enhancing predictions versus improving access to screening. Specifically, the PAR is defined as:
Proposition 2.3. (Policy Value in Gaussian Setting) Let Y Y- & sid ~2 and Y (nn, then the value ((x*) of the optimal screening policy '² is given by
PAR = Vlo+tA..A.R) -V V(a.B,1+ ((.
(3)
22(0 (0)) -a)))-1(8))))
v(x'== V(.
(2)
3
In other words, the PAR can inform a social planner how much more they should be willing to pay for improvements in screening capacity relative to prediction. For example, a PAR > 2 implies that expanding the screening capacity by
where z2 (-) denotes the bivariate standard normal CDF with correlation p= V= - and Φ (-) is the quantile function of the standard normal distribution.
4

---

The Value of Prediction in Identifying the Worst-Off p =F =y)
1.00
B=0.2
1.00
B=0.2
1.00
[ =022
8:400
0.75
0.75
0.75
(())
P (())
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
2
Y
Figure 1, Screening Policy in Gaussian Setting, (Left) Probability of being screened for an individual with specific welfare outcome Y, given R² 0.25, a 0.2, and B = 0.2. The dashed line represents the unconstrained oracle policy, which perfectly sereens those in need. (Middle) Policy with expanded screening capacity, where a increases by Aa 0.2. (Right) Policy under an improved prediction model with R²² + Aga. where Ag² 0.2. The shaded area under Pr(y & F Pr[Y- & Fy (5)]. corresponds to the policy value.
+ (a) Y y]. weighted by fr(y) and normalized by
An yields at least twice the increase policy value com- pared to investing in improved predictions by AR.. Conse- quently, the social planner should prioritize investments in screening capacity, provided the costs of doing are not more than double those of improving predictions.
in policy value V with respect to improvements in R²:
(a., R2:)
lim (a. (0.0) A-+0
arg max
(l..3II++)
The local improvements in V diverge-ane and are maximized- in two regimes: (1) R2 - 1, a, = -,, and (2) R² + O. For both regimes, setting Ara = Aa = A, the local prediction- access ratio satisfies limA- PAR(a, ,, A) O.
3.1. Theoretical Bounds for the Prediction-Acces Ratio
In our setting, direct calculation of the PAR is challenging due to the policy value being analytically intractable and the problem featuring strong non- linearities. We derive bounds for specific cases and regimes that we consider particularly insightful, with a focus on marginal local improvements. In our empirical investigation, we find that the main results generalize well to a more complex, real-world setting,
According to Theorem 3.2, marginal improvements in pre- diction are most impactful in two distinct regimes. First, when predictive capacity is very low, even a small initial in- vestment can lead to disproportionately large improvements, provided that a minimal baseline of screening capacity is present. Second, R approaches one, further marginal improvements can also have a significant relative impact, specifically around the point where the screening capacity matches the requirements for screening the entire 3-segment of the population. See Figure 2.
What should priorities be if screening is very limited?
Theorem 3.1 (PAR for Small Screening Capacities). For any 0- < <R² < 1, A A A. 0 and 0- < 3 & 0.5 there exists threshold (3, R2, Are) such fhrtfor any a A. < t, PAR(a, R2, Da, Are) is at least
When are small inereases in screening capacity more impactful than improving predictions?
(2 R2) (5.1.a aa--aa-111000 (aaa--(110)) (1) +o(1)
Proposition 3.3 (PAR for Local Improvements). Let R?, B, and satisfy either R E (0.15, 0.85), 3 E (0.03, 0.5). and < 3. or R2 E (0.2, 0.5), B > 0.15, and a & 0.5. If A = Aa = A, then lim PAR(a, B, > 1.
where o(1) goes zero as approaches zero.
Suppose the available screening capacity a + A. is very small (a Aa {B), and assume there is baseline level of predictability (i.e., R²2 is bounded away from 0). Then Theorem 3.1 implies that the PAR can become very large. Specifically, for small a, 1-1 1 (1 a)) grows asymptotically like / /og (1/a). Consequently, the polynomial growth of a -1(1-n2²) drives the PAR to increase rapidly de- creases. It follows that in the scarce capacity regime, ex- panding the screening capacity has a far greater impact than improvements in prediction accuracy.
We find that the PAR remains above one as long as a < and R is not too extreme. For larger 3 values (i.e., 8 > 0. the PAR stays above one even for large a provided R² remains in a moderate range. Crucially, this represents the standard parameter regime in which most allocation programs operate, characterized by a moderate baseline of predictions and resource levels comparable to 3..
Numerical Simulations. We complement our theoretical investigation with numerical simulations of the PAR for different a, ,3 and R² values (see Figure 2). Consistent with our theoretical results, the PAR becomes large for small screening capacities (a & (3) and remains above one for a & B, provided a small baseline level of predictive perfor- mance has been established. The bounds in Proposition 3.3
When does prediction have the highest impact?
Theorem 3.2 (Maximally Effective (Local) Prediction Im- provements). Let < 3 < 1 be fixed and < a < 1. Consider the points that maximize the local rate of change
5

---

The Value of Prediction in Identifying the Worst-Off
are conservative, with PAR > 1 observed for broad range of R2 values. Prediction improvements are particularly im- pactful when Rt is small. Although the PAR falls below one in the high-R² and high-a regime, allocation is nearly perfect, making further improvements a "last mile" effort.
a' are required to reach specified fraction p of poor house- holds, i.e. a' = -inf fo: V(a,8) > p).
Improving Predictions. A decision-maker maker can improve a model's predictions through various pathways:
a) Data Collection Collect additional samples and in- crease the frequency of data collection. Social pre- diction systems are often vulnerable to distribution shifts over time in dynamic and evolving environments (Fischer-Abaigar et al., 2024; Aiken et al., 2023).
Discussion. We found several insights relevant to policy- makers aiming to iteratively improve screening system. First, establishing a baseline level of predictive performance is usually good starting point. Once this is achieved, ex- panding the screening capacity becomes the next priority. For very small capacities, Theorem 3.1 tell us that the PAR can increase significantly, making investments in screening capacity highly impactful.
b)
Data Quality Improve data quality (i.e., reduce errors and missing data) by means such as standardizing data collection processes, implementing centralized data management systems, and offering targeted training programs for staff.
Generally, expanding capacity to least the level where everyone in need could hypothetically be screened (a > 3) is likely cost-efficient. Once both screening capacity and predictive accuracy are high and the allocation system is close to optimal, improvements in prediction become relatively more valuable again for perfecting the system. However, this regime may rarely be reached in practice.
Collect Additional Features In government, this may involve integrating separate data sources across institu- tions (Sun & Medaglia, 2019; Wirtz et al., 2019).
d) Advanced Modeling Techniques Utilize more sophis- ticated modeling techniques, which might capture more complex patterns in the data but are often more costly to operationalize.
In Figure 2, we display the PAR for a cost ratio of 1/4. As expected, the regions where investing in R2 is more efficient expand, and some of the earlier nonquantitative bounds no longer apply. Nevertheless, the key insights remain consistent: when screening capacities are small, investments in expanding them are very effective, while improvements in 1²2 are more important when predictive accuracy is low.
In ressurce-constrainee settings, planners often focus on incremental improvements rather than rebuilding entire sys- tems. For instance, collecting small amount of additional data may boost 1² by a few points, uniformly reducing errors. To simulate such minor gains, we scale the model's 's residuals y = Ý + 8(Y - Y). choosing E (0, 1) so that R² increases by a target Ra (see Appendix B.3). This pre- serves the overall error structure, allowing to gauge how "similar but slightly better" model affects policy outcomes.
4. Empirically Evaluating the PAR
This approach can be extended in several ways. For exam- ple, residuals could be adjusted for specific subgroups to account for uneven prediction improvements (e.g., targeted data collection for rural or underrepresented populations). Alternatively, planners could retrain models under different conditions such as sample size, feature set, or architec- ture-and compare the resulting policy value.
While our theory offers broad intuition when expanding screening capacity or improving predictions is most effec- tive, policymakers need practical tools for their own systems. To support this, we develop a methodology to compute and ratio using empirical data,
interpret the prediction-aceees helping social planners identify the most efficient policy levers for their unique problem context.
Policy Value. As before, we define the allocation policy's 's value as the probability that the offst-off individuals are successfully identified:, i.c. V(a, B) = Priy < Fy (a)
5. Case Study: Identifying Long-Term Unemployment in Germany
Y 1y7(1)), In practice, this can be measured using a recall-like metric, capturing the proportion of truly at-risk individuals screened by the policy.
Public employment services (PES) across the globe make use of profiling approaches to identify jobseekers at risk of long-term unemployment to target preventative measures (Loxha & Morgandi, 2014). Starting from traditional rule- based approaches, many PES either test or already deploy algorithmic profiling to identify jobseekers in need of sup- port (Desiere et al., 2019; Körtner & Bonoli, 2023). While these profiling tools assist in allocating programs that ac- making design choices critical (Kern et al., 2024) - systematic assess- ments of their relative value compared to other measures for
Eis 1(8 (M EE, ((TEP7)
V(a, B)
Increasing Screening Capacity. Given chosen An the policy improvement can be directly computed V(a + An. ((, 3) by recalculating the empirical policy value at the new threshold. For example, in cash transfer programs (Blumenstock, 2016), a key question how many resources
count for large shares of PES spending
6

---

The Value of Prediction in Identifying the Worst-Off
1.00
1.00
Predictions: R,
ViGa Rgl 0.
0.75
Vio. B. &
Predictions:
1.5
0.50
0.50
1.0
0.25
0.25
0.00
0=
0.00
0.5
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
Screening Capacities
Screening Capacity:
(a) PAR
(b) 1/4 x PAR PAR
Figure 2. Numerical Simulation of the Prediction-Acces Ratio (PAR), Equation 3, for Ag² = A.. = 0.01 and 3 = 0.2. (Left) The PAR values. (Right) 1/4 PAR, representing a cost ratio 1/44. Each point represents a screening capacity a (x-axis) and R value (y-axis), with the color bar showing the PAR clipped to the range [0.5, 2.0]. Dotted black lines represent PAR 1, where improvements a and R² are equally effective. The purple line marks region in (ax, R²²) space where the policy value V(a, B, R²²) exceeds 0.9.
salle
3.5
Suntring Canadyy
Suasing
Suctring Cazediy
(a) R 0
(b) R 0.15
(c) R 0.9
Figure Prediction-Acces Ratio for Ax² A.. 0.1 across three regimes: (left) constant prediction (It² 0), (middle) trained model (R² = 0.15), and (right) near-perfect prediction (R² 0.9). As expected from our theoretical intuition, the PAR is large for small and in the middle plot, which represents the typical regime allocation systems.
improving jobseekers' outcomes remain absent.
2015, resulting in data on 274,515 different jobseekers and 553,980 unemployment spells. We refer to Appendix B. for additional information on the experimental setup and data.
We secured access to dataset' on German jobseekers de- rived from German administrative labor market records that cover a large portion of the German labor force. It merges multiple administrative data sources, containing a wide spec- trum of individual labor market information - including records on employment histories, received benefits, unem- ployment periods, participation in job training programs and demographic information. Such administrative records are the primary data source used by PES to build algorithmic profiling models (Bach et al., 2023).
Our focus is the 3-fraction of jobseekers with the longest expected unemployment durations, representing those most at risk. Germany, being unemployed for over one year (about 15% of cases in our data; Figure meets the legal definition of long-term unemployment (Bach al., 2023), but some countries adopt different cutoffs (Desiere et al., 2019).
Experimental Setup. We train a model to predict how long a newly registered jobseeker remains unemployed, defining the target Y as unemployment duration in days (capped at 24 months). Following Bach et al. (2023), we use set of covariates capturing demographic information, labor market history, and most recent job details. To ensure full 24-month observations and mimic realistic deployment scenario, we focus on unemployment spells beginning between 2010 and
5.1. Results
We train a CatBoost model (see Appendix B.2 for de- tails), achieving an Ra of 0.15 on the test set. This level of predictive power aligns well with what is typically observed in social prediction tasks (Salganik et al., 2020) and similar applied settings (Desiere et al., 2019).
How much does the screening capacity need to increase to target significant fraction of high-risk jobseek- ers? As expected, larger screening capacities increase both the policy value and the number of high-risk jobseekers screened (see Figure 4(a)). Focusing on the (German) LTU
'The dataset is provided via Scientific Use File by the Re- search Data Centre (FDZ) of the German Federal Employment Agency (BA) the Institute for Employment Research (IAB) (Schmucker & vom Berge, 2023a;b).

---

The Value of Prediction in Identifying the Worst-Off
cutoff (B R 0.15), our policy value aligns well with findings of previous studies? (Bach al., 2023).
despite the non-local improvements and more complex resid- ual structure. Notably, the theory's s focus on local improve- ments offers conservative perspective on capacity invest- ments: even under random allocation (R² = 0), securing a modest screening capacity (5 10%) is often the first prior- ity, while at very high R², gains in policy value diminish so rapidly once a 2 3 that the relative advantage of further prediction investments becomes negligible.
A planner might begin by setting a - B, ensuring that, in theory, enough capacity is provided to screen and support ev- ery high-risk jobseeker. A natural question then arises: how much additional capacity A. would be required to screen at least a specified percentage of high-risk individuals? This additional capacity represents the overhead that must be invested to account for imperfect predictions. We observe that the A. required to ensure at least 75% of high-risk jobseekers are screened remains consistently around 0.25 across different/ & values. While the policy value increases as a = 3 rises, the marginal improvements gained from increasing access decrease for higher a, resulting in some- what stable Aa across 3. In practice, this means we need to screen 25% more of the population to ensure adequate coverage.
When do small improvements in prediction error have the largest impact? From theory (Theorem 3.2), we expect local policy value improvements from better predictions to diverge as R² -> 0 and R² - when a = 3. This aligns with our results in Figure 4: for small A2,, the rate of local improvements in ((R²) with respect to R diverges. The location of the maximum in a also follows from the theory: as R + 1, the rate only diverges for a B, while for small R the maximum is at ar x 0.5.
What is the impact of improving screening capacity ver- sus prediction errors? We simulate small improvements in the R2 value by uniformly scaling the residuals by a multiplicative factor. To ensure that this approach approx- imates a realistic pathway of (marginally) improving the model, we train various models at different sample sizes. We then verify that as R² increases with the amount of training data, the variance of the residuals decreases, while the distribution remains largely unchanged in shape (see ratio for AR= = Aa = 0.1 in three scenarios (1) the trained BotBo model with R² 0.15, (2) cear-perfect pre- dictions with R²
What are the relative benefits and tradeoff's of using a simpler vs more complex prediction model? We compare a shallow 4-depth decision tree with the CotBoost model. As expected, the simpler tree shows a small drop in pre- dictive power (5% decrease in R²) which translates into a 1 1-8% reduction in policy value (see Figure 15). Compared to a uniform 5% increase in R² achieved by scaling the residuals (see Figure 14), the differences in policy value are only partially similar across a. The Ca Boost model does not provide a uniform improvement over the decision tree; for instance, it performs better at distinguishing longer unemployment spells.
Figure 12). We then evaluate the prediction-aceee
Ag and (3) constant predictions (It2 = 0), effectively randomizing screening decisions.
Despite this performance gap, the simpler model offers potential advantages: it fits on single sheet of paper, de- mands minimal computational infrastructure, can be easily explained to frontline case workers and resembles the cat- egorical prioritization rules common in public institutions. (Johnson & Zhang. 2022). Because more complex models incur higher costs, a planner might instead increase screen- ing capacity. Formally, we define
We observe rise in the PAR for small screening capaci- ties a (see Figure 3), consistent with Theorem 3.1. Under random allocation (12² = 0), the PAR stays below one for >> 0.1. This result aligns somewhat with Theorem 3.2, where we found that the (local) PAR approaches zero as R² O. Because consider A= 0.1 (rather than infinitesimal improvement, see Figure 13 for A= = 0.01), the PAR remains large at small a. For the Cat Boost model (R² 0.155, capacity improvements stay relatively more effective (i.e., PAR > 1) for larger a, matching Proposi- tion 3.3, where we found that for moderate Rd and a & 8,, the local PAR remains above one. Meanwhile, near-perfect predictions (R²² = 0.9) make capacity investments highly efficient, causing the PAR to diverge for a < B, then drop sharply near a 3 because the allocation becomes nearly optimal. When a > 8, the PAR stabilizes at one as numera- tor and denominator both approach zero.
A; inf { { A.
Ving((++A..00 Vinaa (a.0) }
Vexr(a,8) (Ccr(a,,)-Virz(((( (a,,8)
the smallest A that matches the policy-value gains of the CatBoost model. Empirically. A; mostly rises with (see Figure 15), consistent with our finding that the PAR decreases with a. By framing the difference between mod- els in terms of additional screenings, planners can directly compare the cost of increased capacity to that of deploying more complex model.
6. Conclusion
These observations broadly match our theoretical findings,
Foor the percentage of correctly identified LTU episodes, they report values of 0.29 at a as 0.1 and 0.58 at a ss 0.25, compared to our observed values of 0.28 and 0.56, respectively.
This paper develops a framework for quantifying the relative value of prediction in identifying the worst-off. We formal- ize tradeoffs between expanding screening capacity and
8

---

The Value of Prediction in Identifying the Worst-Off
0.2-
0:15 (x
0.55 (oil soms)
9--
044
0.2
0/7
Serwsing Capart
000
905
100
105
Caporty
(a) Policy Value
(b) R²
(c) R
R2
Figure (a) Policy value across different screening capacities (ax and worst-off fractions evaluated on the test set using the CatBoost regression model. A value of 0.15 corresponds to the 12-month cutoff used to define long-term unemployment in Germany. (b, c) The rate of local improvements V(R²) with respect to small changes R². Panel (b) shows that the local improvements become increasingly large as A Ø2 approaches zero. Panel (c) illustrates that improvements in prediction have the greatest impact when the capacity precisely matches the targeted fraction of the population (a = 9). Note that these are on a logarithmic scale.
insightful comments, as well as Frauke Kreuter, Patrick Schenk and Moritz Hardt for their valuable feedback.
vos.) 4.07
values
0:15
Sunzing Capelly
0.25 mxa))
Impact Statement
Our work offers principled framework for evaluating the relative benefits of using predictive models to target the most vulnerable populations, helping public agencies allo- cate limited resources more effectively. However, formal- izing complex institutional processes inevitably omits im- portant worldw details, risking biases misalignments if assumptions are not carefully examined. We encourage policymakers and researchers to incorporate fairness, trans- parency, and accountability measures when implementing these methods, particularly in where small design changes can disproportionately affect marginalized communities.
41004
0.0
455
101
Screnning Capacity:
Figure 5. The minimum additional screeming capacity that would need to be invested for the decision tree achieve policy value comparable to that of the CatBoost model.
improving predictive models, and show through both mathe- matical analysis and a real-world case study that prediction not always the most important piece of the puzzle so- cial allocation systems. Future work could examine more specific application settings and cost structures, including distinctions between fixed and recurring costs, and explore policy levers that improve prediction unevenly, for example, by reducing errors in high-risk subgroups or by increasing robustness to distributional shifts. More broadly, we see a need for clearer theoretical foundations understand the role of prediction in public-sector allocation, particularly in relation the institutional and administrative systems in which it is embedded.
contexts
References
Aiken, E., Ohlenburg, T., and Blumenstock, J. Mov- ing targets: When does poverty prediction model need to be updated? In Proceedings of the 6th ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies, COMPASS '23, pp. 117, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701498. doi: 10.1145/ 3588001.3609366 URI. https://doi.org//// org/10. 1145 1145/3588001- 3609369.
Acknowledgements
This work is supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, spon- sored by the Federal Ministry of Education and Research and by the Volkswagen Foundation, grant "Consequences of Artificial Intelligence for Urban Societies (CAIUS)". Juan Carlos Perdomo is supported by the Center for Research on Computation and Society (CRCS) at Harvard University and by the Alfred P. Sloan Foundation grant G-2020-11 13941. We would like to thank the anonymous reviewers for their
Athey, S. and Wager. S. Policy Learning with Observational Data. Econometrica,
2021.
Bach, R. L., Kern, C., Mautner, H., and Kreuter, F. The
impact of modeling decisions in statistical profiling, Data & Policy, 5:e32, 2023. doi: 10.10177dap.2223.22
Barocas, S., Hardt, M., and Narayanan, A. Fairness and
Machine Learning: Limitations and Opportumities. MIT Press, 2023.

---

The Value of Prediction in Identifying the Worst-Off
Blumenstock, J. E. Fighting Poverty with Data. Science, 2016.
Fischer-Abaigar, U., Kern, C., Barda, N., and Kreuter, F, Bridging the Gap: Towards Expanded Toolkit for Ai- driven Decision-making in the Public Sector. Government Information Quarterly, 41(4):10197, 2024. ISSN 0740- 624X. doi: http://dbbiang//000.oo11e...2222
Bochmer, N., Nair, Y., Shah, S., Janson, L., Taneja, A., and Tambe, M. Evaluating the Effectiveness of Treatment Allocation. arXiv preprint arXiv:2402.11771 2024.
URL httpps//www.ccenneddeeee... /tteecc//rrtcceee///////76662222
Chan, C. W., Farias, V. F., Bambos, N., and Escobar, G. J. Optimizing Intensive Care Unit Discharge Decisions with Patient Readmissions. Operations Research, 60(6):1323- 1341, 2012. doi: 10. 10.1227/0pre.112000 100, URL ht tps: //doi. ///ooi.og///0 http://..11111/.... 1tt...117//pp// 1120. 1105.
Guerdan, L.., Coston, A., Holstein, K., and Wu, Z. Coun- terfactual Prediction Under Outcome Measurement Er- ror. In Proceedings of the 2023 ACM Conference Fairness, Accountability, and Transparency, FAccT '23, pp. 1584-1598, New York, NY, USA, 2023. Associa- tion for Computing Machinery. ISBN 9798400701924. doi: 10. 10.1144/3593033335555 org org/10. 1ee.1111553999 3594101.
Chouldechova, A., Benavides-Prado, D., Fialko, O., and Vaithianathan, R. case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference Fairness, Accountability and Transparency, pp. 138-148. PMLR, 2018.
URL https: //doi.
Jain, S., Creel, K., and Wilson, A. C. Position: Scarce Resource Allocations That Rely On Machine Learning Should Be Randomized. In Forty-first International Conference on Machine Learning, 2024. URL https: / //openreview.o net forum?ide 4gxX6Ty6F.
Clementi, F. and Gallegati, M. Pareto's 's law of income distri- bution: Evidence for Germany, the United Kingdom, and the United States. Econophysics of wealth distributions: Hannophys-Kolkatte pp. 3-14, 2005.
Johnson, R. and Zhang, S. What is the Bureau- cratic Counterfactual? Categorical versus Algorithmic Prioritization in U.S. Social Policy. In Proceedings of the 2022 ACM Conference Fairness, Account- ability, and Transparency, FAceT '22, pp. 1671-1682, New York, NY, USA, 2022. Association for Comput- ing Machinery. ISBN 9781450393522.
Coston, A., Kawakami, A., Zhu, H., Holstein, K., and Hei- dari, H. A Validity Perspective on Evaluating the Jus- tified Use of Data-driven pacision-making Algorithms. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 690-704, 2023. doi: 10.1 10.11095.TMML545552200
doi: 10.1145/ 3531146.333322.. URL https: http///i.ogg/.....//////// 1145/353114 1145/333114.353222 3533223.
Desiere, S. and Struyven, L. Using Artificial Intelligence to classify Jobseekers: The Accuracy-Equity Trade- off. Journal of Social Policy, 50(2):367-385, April 2021. ISSN 0047-2794, 1469-7823. 50047279420000203.
Kallus, N. More Efficient Policy Learning via Opti- mal Retargeting. Journal of the American Statistical Association, 116(534):646-66 6616668,6,8,,, 2021. doi: 10.1080/
doi: 10.1017/
01621459.2020. 20..222......88. 1788948. URL https: / 10. 011000000621455 2020. 1788948.
Desiere, S., Langenbucher, K., and Struyven, L. Statistical Profiling Public Employment Services: An Interna- tional Comparison. Technical Report 224, OECD Pub- lishing, 2019. URL https://doi. org /trg/10..77. b5e5f16e- eenn.
Kern, C., Bach, R., Mautner, H., and Kreuter, F. When Small Decisions Have Big Impact: Fairness Implica- tions of Algorithmic Profiling Schemes. ACM Jour- nal on Responsible Computing, 1(4), November 2024. doi: 10. 10.1145/3689485. URL https: /ooi org/10. 1 1145/3689485
Drezner, Z. and Wesolowsky, G. O. On the Computation of the Bivariate Normal Integral. Journal of Statistical Computation and Simulation, 1990.
Kitagawa, T. and Tetenov, A. Who Should Be Treated? Empirical Welfare Maximization Meth- ods for Treatment Choice. Econometrica, 86 (2):591-616. 2018. doi: https://do..or//10.3999/ ECTA13288 URL https: / on inelibrary.. wi wiley. htmm///oaabs//...333
Elmachtoub, A. N. and Grigas, P. Smart "predict, then optimize". Management Science, 68(1):9-22 2022.
Fernández-Loría, C. and Provost, F. Causal Decision Mak- ing and Causal Effect Estimation Are Not the Same... and Why It Matters. INFORMS Journal on Data Science, 1(1): 4-16, 2022. doi: 10. 10.1287/jjds.2221..0o //doi. www. 1287 jids 2021.0006.
Kleinberg, J., Ludwig, J., Mullainathan, S., and Obermeyer, Z. Prediction Policy Problems. American Economic Review, 105(5):491-495 2015.
URL ht tpsp:
10

---

The Value of Prediction in Identifying the Worst-Off
Körtner, and Bonoli, G. Predictive Algorithms in the Delivery of Public Employment Services. In Handbook of Labour Market Policy in Advanced Democracies, pp. 387-398. Edward Elgar Publishing, 2023.
Tsurkov, E., van Loon, A., Varol, O., Wang, X., Wang. Z.,
Wang, J., Wang, F., Weissman, S., Whitaker, K., Wolters, M. K., Woon, W. L., Wu, J., Wu, C., Yang, K., Yin, J., Zhao, B., Zhu, C., Brooks Gunn, J., Engelhardt, B. E., Hardt, M., Knox, D., Levy, K., Narayanan, A., Stewart, B. M., Watts, D. J., and McLanahan, S. Measuring the Predictability of Life Outcomes with a Scientific Mass Collaboration. Proceedings of the National Academy of Sciences, 117(15):8398 8403, 2020. doi: 10. 10.1073/pnas. 1915006117. URL https: www. ppws. ong wwg//do/ /oii/ a abs / /
Loxha, A. and Morgandi, M. Profiling the unemployed: a review of OFCD experiences and implications for emerg- ing economies. Social protection discussion papers and notes, (91051), 2014.
Manski, C. Statistical Treatment Rules for Heterogeneous Populations. Econometrica, 12(4):1211-1246, 2004. doi: URL httpss//onniinellbraary wiley. com/doi/ 3bs/10..111/j..1 026144668-22222222 201.6.23000...
191 5006117.
https///ooioog//0.11111144442222
Schmucker, A. and vom Berge, P. Faktisch anonymisierte Version der Stichprohe der Integrierten Arbeitsmark- tbiografien (SIAB- (LABB-eegionalile) Forschungsdlatenzenttun der Bundesagentur für Arbeit (BA) im Institut für Arbeitsmarkt- und Berufsforschung (IAB), 2023a. 10.5164/IAB. 1..164/AAB.SIA--RTS.. .de.en.v1.
Version 7521 vl.
Perdomo. J. C. The Relative Value of Prediction in Algo- rithmic Decision Making. In Proceedings of the 41 In- ternational Conference on Machine Learning, ICML'24. Jrg.. 2024.
Schmucker, A. and vom Berge, P. Sample of Integrated Labour Market Biographies Regional File (SIAB-R) 1975
Perdomo, J. C., Britton, T., Hardt, M., and Abebe, R. Diffi cult Lessons on Social Prediction from Wisconsin Public Schools. arXiv preprint arXiv:2304.06205, 2023.
2021. FDZ-Datenreport 07/2023 (en), Research Data Centre of the Federal Employment Agency (BA) at the In- stitute for Employment Research (IAB), Nürnberg, 2023b. 10.5164/IAB.FDZD WWW.D............. 20..2.37...... en.vl.
Potash, E., Brew, J., Loewi, A., Majumdar, S., Reace, A., Walsh, J., Rozier, E., Jorgenson, E., Mansour, R., and Ghani, R. Predictive Modeling for Public Health: Preventing Childhood Lead Poisoning. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15, pp. 2039-2047, New York, NY, USA, 2015. Associa- tion for Computing Machinery. ISBN 9781450336642. doi: 10. 1111145/7732588278888888 org/100 11.0.11455/ 077.53222828888888 2788629.
Shirali, A., Abebe, R., and Hardt, M. Allocation Re- quires Prediction Only if Inequality Is Low. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: bettps://openree wpw.revieeeeeeeeeeeeeeeeeeeeeeee net forum? i id=Wii CA0hOF9.
URL https: //doi.
Sun, T. Q. and Medaglia, R. Mapping the challenges of Artificial Intelligence in the public sector: Evidence from public healthcare. Government Information Quarterly, 36(2):368-383 2019.
Salganik, M. Lundberg. I., Kindel, A. T., Ahearn, C. Al-Ghoneim, K., Almaatoug. A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., Datta, D., David- son, T., Filippova, A., Gilroy, C., Goode, B. J., Jahani, E., Kashyap, R., Kirchner, A., McKay, S., Morgan, A. C., Pentland, A., Polimis, K., Raes, L., Rigobon. D. E., Roberts, C. V., Stanescu, D. M., Suhara, Y., Usmani, A., Wang, E. H., Adem, M., Alhajri, A., AIShebli, B., Amin, R., Amos, R. B., Argyle, L. P., Baer-Bositis L., Büchi, M., Chung. B.-R., Eggert, W., Faletto, G., Fan, Z., Freese, J., Gadgil, T., Gagné, J., Gao, Y., Halpernn-annnet A., Hashim, S. P., Hausen, S., He, G., Higuera, K., Hogan, B., Horwitz, M., Hummel. L. M., Jain, N., Jin, K., Jurgens, D., Kaminski, P., Karapetyan, A., Kim, E. H., Leizman, B., Liu, N., Möser, M., Mack, A. E., Mahajan, M., Man- dell, N., Marahrens, H., Mercado-Garcia. D., Moez, V., Mueller-Gastele K., Musse, A., Niu, Q. Nowak, W., Omidvar, H., Or, A., Ouyang, K., Pinto, K. M., Porter, E., Porter, K. E., Qian, C., Rauf, T., Sargsyan, A., Schaffner, T., Schnabel, L.,, Schonfeld, B., Sender, B., Tang, D.,
Wang. A., Kapoor, S., Barocas, S., and Narayanan, A. Against Predictive Optimization: On the Legitimacy of Decision-making Algorithms That Optimize Predictive Accuracy. ACM J. Responsib. Comput., 1(1), March 2024. doi: 10. 10.1145/3636509, URL https: // org/ 10. 1145 3665099
Wilder, B. and Welle. P. Learning treatment effects while treating those in need. arXiv preprint arXiv:2407.0759% 2024.
Wirtz, B. W., Weyerer, C., and Geyer, C. Artificial Intel- ligence and the Public Sector Applicattoo and Chal- lenges. International Journal of Public Administration. 42(7):596-66 201.

---

The Value of Prediction in Identifying the Worst-Off
A. Theoretical Investigation
0.5
B =0.2
0.4
Density
0.3
0.2
0.1
(a) Normal Welfare Distribution
Figure 6. Normal welfare distribution, with vertical lines marking the quantile cutoff' (3 0.2). The shaded region the left of the vertical line represents the worst-off segment the population.
1.00
1.00
2.0
Predicions: R.
0.75
0.755
1.5
Prediction:
Vio. R. R.R. x09
1.5
Vio. 2.9
0.50
1.0
1.0
0.25
0.25
0.00
0000
0.5
0.00
8.0.0
0.5
0.0
0.2
0.3
0.4
0.1
0.2 Screening Capacity:
Screening Capacities:
(a) PAR
(b) 1/1 x PAR
Figure 7. Numerical Simulation of the Prediction-Access Ratio (PAR), Equation 3, for Ag² A. 0.01 and $ = 0.05. Each point represents screening capacity a (x-axis) and R² value (y-axis), with color har showing the PAR clipped to the range [0.5,2.0] 2.0]. The vertical black line marks 3, indicating the threshold above which sufficient resources prediction. Dotted black lines represent PAR 1, where improvements in a and R² are equally effective. The purple line marks the region in the (a, R²) space where the policy value V(a, 8, R?) exceeds 0.9. Values above 0.9 are located in the upper-right region beyond the purple line.
available to screen everyone under perfect
B. Experiments
B.1. Experimental Setup and Labor Market Data
The dataset is provided via a Scientific Use File by the Research Data Centre (FDZ) of the German Federal Employment Agency (BA) at the Institute for Employment Research (IAB) (Schmucker & vom Berge, 2023a;b). a 2% weakly anonymized random sample of the complete German labor market records from 1975 to 2017 and contains information on 1,827.903 individuals across 62.340,521 observations (Schmucker & vom Berge, 2023b).
We follow the same set of covariates and aggregation procedure for individual unemployment spells as described in Bach et al. (2023), incorporating demographic characteristics, labor market histories, and information about the most recent job. This results in 56 numerical variables and 24 categorical variables, which are one-hot encoded for model training, Figure shows histogram of individual unemployment durations, which we use as the basis for constructing the outcome variables. The distribution is right-skewed, with a concentration on short durations near zero and a long tail. Such a pattern commonly observed other welfare-related outcomes, such as health income metrics. We define as prediction target the duration of the unemployment period in days Y. capped 24 months3, Differentiating tail values is less important for
In practice, for a fixed B, the problem can also framed as classification task (see Appendix B.5).
12

---

The Value of Prediction in Identifying the Worst-Off
decision-makinn maniin. and capping also allows training across years with varying observation windows.
100000
75000
Count
50000
25000
0
500
1000
1500
Unemployment duration (days)
Figure Unemployment duration The red line marks 12 month threshold used classify a jobseeking episode long-ten unemployment (LTU).
To avoid the impact of significant labor market reforms in Germany and to ensure full observation of unemployment durations up 24 months, restrict our analysis unemployment episodes that began between 2010 and 2015. We use records from 2010 and 2011 to build the training dataset, records from 2012 for validation, and evaluate test performance on data from 2015 (see Figure 9). We left a gap between the training and test data periods to allow enough time for the outcomes in the training data to have been fully observed at test time, in order to mimic realistic deployment scenario starting at the beginning of 2015.
Training
Validation
Test
2010
2011
2012
2013
2014
2015
2016
2017
2018
Figure Stacked timeline diagram illustrating training (2010-2013), validation (2012-2014), and (2015-2017) data periods. Red dashed boundaries within each colored box indicate the possible start dates of unemployment episodes, while the full colored boxes represent entire observation phases each dataset.
B.2, Training Details
We use Booso (https//ccatboos..ai) for model training. The model was trained for maximum 5,000 iterations with an early stopping criterion shallow Decision Tree (max depth = 4) using the sci k weittleee ea rn package. All hyperparameters are kept at their default settings unless otherwise specified.
= 20) based on validation performance. Additionally, we train
B.3. Prediction Improvements
To simulate an increase in predictive power by a specified amount Aga, we adjust the model's predictions ý using the residuals Y Y. Starting with the original predictions Ý and true outcomes Y, we define the adjusted predictions as
- + 8)
13

---

The Value of Prediction in Identifying the Worst-Off
We then determine the d corresponding increase of Ag in the model's nt:
2i=1 (Y, 8)² 2 (Y, 81)²
8 1
12a
For specified 8, the new residuals
(1- 8)
Consequently, the variance decreases by multiplicative factor: Var(Y- -++)=
= -1 - Var(Y - -Y)
0.11
34.81
30.72
Density
Density
0.08
0.06
0.06
0.04
0.04
0.02
0.02
0..00
0.00
10
Residuals (Y Y)
Residuals (1 8)(Y Y)
(a) A 0
(b) A R² = 0.1
Figure Residual Distribution Before After Adjustment Figure (a) shows residual distribution for the original predictions (A R2 0), while Figure (b) shows the residual distribution after increasing the -alue (Aga 0.1) for the CatBoost model. The adjustment preserves the overall structure the residuals.
0.150
0.125
0.100
0.050
0.025
0.000
2500
Training Set Size (# Samples)
Figure 11. The R2 value on the test set for varying training set size (Cat Boost Regression).
14

---

The Value of Prediction in Identifying the Worst-Off
0.11
0.14
0.12
0.12
0.10
0 = 37.19
0.10
/ 35.31
Density
Density
0.08
0.02-
0.02
0.00
8.00
-15
-10
-5
10
15
20
-15
10
10
15
Residuals (Y Y)
Residuals (Y Y)
(a) 500 Training Samples
(b) 10, 000 Training Samples
Figure 12, Residual distributions the for models trained with varying training sizes.
B.4. Additional Figures
values
3.5
0.05 (+t 24 mos.
3.0
0.15 (ai mos.)
0.25 (+e 8 mos
2.5
PAR
1.5
1.0
0.00
0.05
0.30
0.15
0.20
0.25
0.30
0.:35
0.40
Screening Capacity:
Figure 13. Prediction-Aceess Ratio for R² 0 and n2 A. 0.01.
0.08
values
0.07
mos.)
V(p + Ap) V/(
mos.)
0.00
0.06
0.03
0.02
Screening Capacity: a
Figure 14. V(R² + Age) - V(R") for Cat model and Ape = 0.05.
15

---

The Value of Prediction in Identifying the Worst-Off
0.08
9 values
0.05 (+= 24 mos.)
Vearla, 8) Vynssfa.
0.07
0.15 (+e 12 mos.)
(+s mos.)
0.13
0.03
0.12
Screening Capacity:
Figure 15. The difference in policy value between a 4-depth decision tree and Cat:Boos model.
B.5. Binary Classification
Instead of predicting the exact duration of unemployment, the problem can be reframed binary classification task. For fixed 8, we can define a binary outcome: Y = 1fY > (1 3)). This approach more directly encodes the target of interest: identifying individuals who may require further screening or assistance. the chosen classifier provides estimates of class probabilities p(x), it can be used to formulate a decision policy 1(f(z) > FT - a.). However, this forces us to specify 3 and the resulting decision threshold prior to model training. This requirement reduces flexibility compared to continuous prediction setup, making classification more appropriate when the model is not intended for in other tasks and when 3 remains constant across the deployment context. Additionally, directly converting durations labels discards information on the precise unemployment durations that could be valuable for the modeling process.
As can be seen in Figure 16, the resulting policy values and true positive counts remain very similar compared to the regression case.
1.0
3 values
15000
values
0.8-
0.05 mos.)
(s: mos.)
Policy Value V [(x, 9)
0.25 (#r mos.)
mos.)
True Positive Count
12500
0.15 (+e 12 mos)
0.25 (+ mos.
0.6
10000
0.4
0.2
2500
0.0
0.2
0.1
0.0
0.1
0.2
0.3
0.1
0.5
Screening Capacity:
Screening Capacity:
(a) Policy Value
(b) True Positive Count
Figure 16. Policy Value and True Positive Count Test Set (Classification).
16

---

The Value of Prediction in Identifying the Worst-Off
C. Additional Propositions
Proposition C.1. (Optimal Policy with Gaussian Error) If Y ~ ~N(0, y²), then the optimal policy x²: - {0,13 to solve the screening problem (Definition 2.1) is equal to:
x" = Fy (a))
where F (a) is the a-quantile of Y. The value of the policy is V(T*) = Pr|Y < F. (a) Y Fy (8)
Proof. Since = + where ~ 20) it follows for the conditional distribution Y Y ~ >N ,²). Since Y Y is Gaussian, we can express the conditional probability from Proposition 2.2 in terms of the CDF of the standard normal distribution,
Pr(Y & (B) Y = + Fy (8)
(
To reproduce the ranking induced by Pr|Y & FF y. individuals can be ranked in ascending order of Ý. Thus, we can express the optimal policy (Proposition 2.2) in terms of ranking of Y,
x ( = 16 & (a))
where (
(a) is the a-quantile of Y. The value V(=*) that can by achieved by the optimal screening policy can then be expressed as:
V(T)) = E B - RF
E [ <F)'(a) Y (F)
= =Pr( < (a) Y < (3)
D. Proofs
D.I. Optimal Policy for Screening Problem: Proof of Proposition 2.2
Proof. We rewrite the policy value,
Y (B) ( x E
E
Pify (8)
E
E 11 < (8)) Y.]
E
(()))PT| (9) Y.]
To maximize the objective, individuals V; with the largest scores s s()) = Pr.y < (3) V.) should be prioritized. Thus,
the optimal policy is to intervene (T(Y.) 1) for the top a-fraction of the population ranked by Pr(Y < 2 Y].
D.2. Optimal Policy Value in Gaussian Setting: Proof of Proposition 2.3
Following Proposition C.1, the value of the optimal screening policy n¹ can then be expressed as:
V(=*) = Pr(r < (a) Y (9)
We can rewrite the conditional probability in terms of the joint distribution of Y and Ý, and note that Pr(y < Fy (()) =
Fy (Fy (B)) 3,
prjý < FT | < (3) = PrD < &
FF'(81)
17

---

The Value of Prediction in Identifying the Worst-Off
We then standardize Y 1²) and Y ~ 1f - 22) and make use that for normal random variable with mean j and variance o² the quantile function is (p) 14 1 1 (p).
Pr Z1 &
F7 (a)-F
Zz
Fy (8)- #3
Priy & F & F
12
"
3
Pr & x-1 (a) 2 10 (B)}
8
71 and Z2 are standard Gaussian with Cov(Z1 22) = TE 71]
1 1VP-39
Cov Y +8) Cov(V,
CovlY,8) v Thus,
T/
they are distributed according to a standard bivariate normal distribution with correlation Cov( E2) 1 Thus,
1 2 ( ( + 1--66000 (8): V(**)= = = (1)) (8)
where
(a) x-)
$2 ( (a), Φ 1- (8)) (8) =
se (2; dæi dæ2
-20
and
1
62 (2) 2TT 2 p2
D.3. Prediction-Access Ratio for Small Screening Capacities: Proof of Theorem 3.1 Using Taylor's 's theorem,
B,, + +R)) - V(a,8,R) = ARZ OR V(a, 3, R² + RR)
where PR= E (0, 1). We know from Lemma D.3,
h-1 (a) v R (3)
V (a, B, R²) B,
< 88 R2)
OR
R2
where R2= R²= pr= A Ra. For < 0.5 and B && 0.5, know x-1 (a) 0 and x-1 (3) < 0. It follows, that for any
E1 0, < R² and 0 < B, there exists threshold value t > 0, such that for all & t1, we have
(1 11) Φ (a)
Φ (a) v - (3)
Φ (a)
< <1 E1
V1 R
R
v1 R
If a < 3 we find (a) - / R2 Φ- -1 (33 < 0. Since (0x) < ((x') for < x <0.
-- (a) v R (8)
Φ-1 (a)
(1 (1-e1) /1 R
8t R2
R2
V1 R2
B BV 8t R2(1 (R?)
B 8t Rē(1 -R?) $ (1 (a))
B/8TRR(1 RR(1- 1 (xo0 -1 (1 - -a))
18

---

The Value of Prediction in Identifying the Worst-Off
where R. (1-61)
For any E2 0, there exists threshold t2 > such that for a t2, we can apply Lemma B.5. from Perdomo (2024) to arrive at the following incquality:
VI-R R
Φ (nD (1 - -i)) < 2T
(11 (1++2)V2TaQ)) 2Ta (1 (1- a)) n²
)a)
Thus,
e2) 2Ta (1 (1-a) a)) n2
V(a,3, +Ar) V,a,3, & AR
B4 RE R2)
We can use Taylor's theorem again and from Lemma D.I we know that
Q
V(a 3, (a, Do
V(a
Po Pa,
da
1 ② (3)
v g2 (a + PaDa)
a B
v1 R
where Pa (0, 1). Since 0 < 3 and 0 < R² there will always be small enough a + Aa such that
Φ (B) - R p.
Since d (x) > 1/2 for x > 0, it follows
Aa
V(a 3a,3,R9 V(a, 3, R²) 23
follows for the prediction-access ratio,
Au 2T R22(1 R2) (1+t2) (1 aa))
-(1-2)) I-R2
V(a + 3, R²) V(a, B, R²²) V(a, 3, R2 AR) V(a, B, R?))
Ra
For small a, Φ -1 (1-a) a) grows asymptotically like /log (1/a). Consequently, the polynomial growth of - 11(11-11 R²) drives the PAR to increase rapidly decreases. Since I- increases with R. and R² & R2, we can lower bound the PAR by inserting R² instead of R.:
Da 2T R2(1 T - R22)
-(1-E1)² I-
Aa, V(a + A., 8, R²) - V(a, B, R2)
2 (1 (1-a)) a))
A
V(a, 3, R AR²) V(a, ,,, R?)
We can simplify the lower-bound by noting that 0 < and 0 < can be made arbitrarily small by selecting sufficiently small threshold for + A.. Specifically, E2 < holds for & 0.05 (see Lemma A.6 in Perdomo (2024)).
+o(1 to(1)
V(a A. 8, R2) V(a, 3, R2)
R RR) (5.1 at- (1 ai) I-
Ag
V(a, 3, R2 + Aga) - V(a, 3, R2)
D.4. Maximally Effective (Local) Prediction Improvements: Proof of Theorem 3.2
We know from Lemma D.2,
V(a, R R - V(a, ((,,8,R%)
② V(a, 3, R²)
lim
OR
02 (w-1 (a), -1
(a),0-(()):)) (8):p)
23
19

---

The Value of Prediction in Identifying the Worst-Off
We insert the (.) and arrive
1
8, B. R²)
0R
4T R R
71
x exp
(a)² + $ (3) 2v R24 (a))0aa()000)))))
(a) $ (5))
2(1 R2)
T2
The prefactor 7i diverges R2 + R² + 0.
If R2 + 1, the exponential term will generally suppress the polynomial growth of the prefactor. However for B, find for the exponent
1
1 1 v R
(a)² + -- (3)² 2V -0---000--000--0----000000000000
1 (8)
2(1 - R22)
1 R2
4 (a)
(1 + v R
1b -1 (3)
2
which is finite. Therefore, 3K 3a, R²) becomes unboundedly large if 3 and R² 1.
If R + O. the prefector Ti diverges again too. The exponent then simplifies to
2(1 - R2) (0-1 (a)² + x (B) 2V R2 (a) x- (3)) = - (a) - (8)
(3)
If and 3 are not set arbitrarily small or large
1 UR
((,, 33 R22) will diverge. The local PAR (Lemma D)11)
lim V(a A, B, R2) (a, B, 12) A
2 (a, 12)
Ba
V(a,
3, R + A V (a, R2)
J
V(a, R2)
BR2
4 (8) / R (()
+
VI-R² R2
RV 22 (
(a), 4- (3)
approaches zero in both regimes.
D.5. Prediction-Access Ratio for Local Improvements: Proof Proposition 3.3
We know
lim V,a+A,8,17* (a, A V(a, R2 A) V(a, R2)
0 ((a 3, R²)
8a
V( B, R?)
Using Lemma D.1 and Lemma D.3 we find a lower bound for the PAR:
4 (3) / - (a)
V(a - A., R2) V R2) V(a, B, R (A)) - V(ax, B, R?)
VB R2)
R
④ 1(8) VR 1(a)
T1
VI-R²
T2
We then denote == + 4 (1) that we need to find the smallest possible 2 to find a lower bound for Tg. Generally, decreases with a and increases with We treat both cases separately:
T: ((a) and T2
((2) We know from Lemma D.4 that increases with z. It follows
(=))
/1 R?
)(z)
+(3)
20

---

The Value of Prediction in Identifying the Worst-Off
1. For & B we find + (88( R V1-R²
< 2. Since 1-v decreases with R and 3 & 0.5 we can lower bound the vI-R²
expression by setting R² 0.15 and B 0.03. Thus, 15. & Z and 0.59 & T2. Since 0.15 & R 0.85 we can
lower bound the prefactor 1.79 < Tı.
2. For 0.5, it follows (-) VI-R²
by setting क (a 0.5) 0. Since 0.15 3 and 0.2 R & 0.5, it follows
0.52 < 72 and < Ti
In both cases, we combine the lower bounds of Ti and T2 to find
< V(a + A 3, R²)
3,a,3,R² R²) V(a, B, R2 + V(a, B, 12
D.6. Technical Lemmas
Lemma D.1 (Derivative w.r.t. a.).
8 V(a, B, R²)
1
Φ-1 1 (3) - VReΦ-1
(4)
da
B
v1 R2
Proof. In the Gaussian setting we find for the policy value (Proposition 2.3),
D2 @ (a), 10 (B):
V(a, (00,3,13 3,
B
We first apply Leibniz integral rule,
22 ( (a), 10.0-((0 :B):pp)
V(a 3, R²) =
da
da
B
4 (3)
Φ 12
(a) dæ da
+ (a)
3 000
(8) (B)
2 (Φ -1 (a), p) dæ2
36 (Ø-1 [a))
We insert the bivariate density de (+) and substitute Z2 pp -1 (a) = uV 1-p 2²
(3) S (0-1 (a), 22; dæ2
1
Bd (0-1 (a))
-00
4 (3)
1
c -1/2(x3- "(a)+ 1(a)3)/(1-33 d
3d 16
(a])
2T
p2 5-00
1
(4
(a))/ I-P
du
2m30 (0-1 (a))
(0-1118) (0-1(18-pp)) -(a)\/ (h03//1-pp 1-p"
1
(a)2
E. -1/2u² du
3r ( (a))
-00
1
(5)
p -1 (a)
-11 (3) R2 -1 (a)
+b
B
1 p²
B
v1 - R2
Lemma D.2 (Derivative w.r.t. R²).
a
1
v(a, (,a,B,15 3, R2)
02 (4 (+1 (a), ② (8))
(5)
dRe
2, n2
where the (.) is the standard bivariate density.
21

---

The Value of Prediction in Identifying the Worst-Off
Proof.
0 $2 (
(a), $ (B)
(,
OR.
0R
8
1 Op 0
12 ① (a) 1 10 (3):p))
aR Op
D2 ④ 1 (a), 4 10,00013)11 (3):0)
2Bv R Op
Φ2 (0-1 (a). 4 (3):
23v R
where Φ2 (-) is the standard bivariate density. We utilized R p², and in the final step applied the partial derivative of the standard bivariate cumulative distribution with respect to its correlation p (Drezner & Wesolowsky, 1990).
Lemma D.3 (Upper bound of Ri2 derivative). Let 0 & VR2 & 1. Then,
0 V(a, 3, R²)*) <
1
Φ-1 1 (3) (3) - V RR (a)
(6)
0R
B B R2)
v1 R²
0 V(, B, R²) <
1
VR (3) -1- (a))
B BT - 122) ④
(7)
0
vI - 12
Proof. We know from Lemma D.2,
0
1
V(a, R, =
22 ( ( 2- (a),
23 VR
R
4T R2(1- R?)
1
④h 1 (a) 2
+ -
2V (a) (8))
exp
2 R2)
Since 0 < VTF & 1,
$-1 (ai) + Φ (9)² 2V R2 (0)0-(a))~1100 (B)
(0 1003-22V5000 R$ (a) 5-
R
= (0- (0) R
0
Similarly,
Φ (a)² Φ (B)² 2 2 (a)
Φ (3) > R 1 -
Thus,
1
V Va, (a,,,,13 RR) <
(x-1 (8) --0--VR----000 (a))-
exp
OR?
Bv Re - R2)
2(1 R2)
$ (3) v R T (a)
B 8T (1 ①
VI- R
and
0
RR (3) --1 (a)
V(a, B, Vaa,B,RF) B,RR) <
B B -R²) $
dR3
-1- R
22

---

The Value of Prediction in Identifying the Worst-Off
Lemma D.4. The ratio
(z)
(8)
Φ (z)
is increasing in
+(=) Proof. We compute the derivative of the ratio
a(=}²
+
(2) 2σ (2)
(2)
1+
Oz 0 (z)
(2)
Φ (z)
For the derivative is clearly positive. For we start by rewriting.
4 (z)
④ (z)
+
& (()
Φ (z)
(z)
2
Since 02
[(z) + +0))))
0(z)
o()) M1x)
+ + (()
b(a)
< and d(z)
+ $ (2) $
0, follows for = < that
(()) + + (z) < 0. Thus for any 2,
0 4 (2)
> 0
Oz (2)
23


Reviews:
[{"review_id": "LvnLHMwpWa", "reviewer": "Reviewer_V6gy", "summary": "The authors study the question when and whether policymakers should invest into improving their predictions or expanding screening, with respect to bottom-line welfare. They set up a simple theoretical model, and validate the resulting hypothesis in semi-synthetic experiments on the German labor market.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for engaging with our work and for your constructive feedback. We very much appreciate it!\n\nBeta Values. 0.15 reflects the official government threshold used in Germany. However, we included additional values to account for differing long-term unemployment thresholds used in other countries. To improve clarity, we will reorganize the figures and expand the captions in the revision.\n\nComparison to Perdomo 2024. Thank you for the question. Our work conceptually builds on Perdomo (2024), who introduced the prediction-access ratio (PAR) to analyze trade-offs between prediction quality and resource availability. The main difference between our work and his is that we develop a new theoretical analysis of the PAR focused specifically on the welfare of the *worst-off* population rather than *average* welfare, as in Perdomo 2024. These two settings are conceptually and mathematically distinct.  Furthermore, relative to Perdomo 2024, we introduce a set of empirical tools to analyze these tradeoffs in practice (the previous work was purely theoretical). We will clarify and expand the comparison to Perdomo (2024) in the revised related work section.\n\nSection 4. We are happy to improve the transition into Section 4 and connect it more explicitly to the empirical case study in the revised manuscript.\n\nFigure 4(c). Figure 4(c) shows the rate of improvement in policy value with respect to small changes in prediction quality in a high $R^2$ regime (predictions close to perfect), as a function of available resources $\\alpha$. Incremental prediction improvements have the greatest impact when resources precisely match the targeted fraction of the population ($\\alpha = \\beta$). For both insufficient ($\\alpha < \\beta$) and surplus ($\\beta > \\alpha$) resources, further prediction improvements are less impactful. We will expand and improve the caption for Figure 4 in the final version. Thanks for raising this.", "claims_and_evidence": "They claim to identify when improving predictions provides a higher benefit in helping identify the worst-off individuals, relative to expanding screening of the population, namely when the prediction system explains either none or almost all of the variance in the outcome (measures in $R^2$). They support this claim via theoretical findings on a simplistic model (Gaussian outcome & residuals), and in experiments on real-world data.", "methods_and_evaluation_criteria": "They define welfare/value of a prediction system as the fraction of the at-risk population identified. This seems like a plausible definition for applications that focus on the worst-off individuals.", "theoretical_claims": "I checked the correctness of Proposition 2.2. and 2.3. I merely skimmed over the others, it seemed plausible though.", "experimental_designs_or_analyses": "The experimental setup is clear. However, the figures always show the results for different values of $\\beta$, although it was argued that $\\beta$ is 0.15 at the beginning. This confuses me and made them hard to read, especially Figure 4.", "relation_to_broader_scientific_literature": "The work extend existing considerations from Perdomo, 2024. The main difference seems to be the definition of the welfare, here fraction of the at-risk population vs linear/probit model. It would be helpful if the authors are more clear with ideas are truly their and which are adapted to their setting.\n\nPerdomo, J. C. (2023). The relative value of prediction in algorithmic decision making. arXiv preprint arXiv:2312.08511.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper is well-written, and addresses an important question for policymakers.", "other_comments_or_suggestions": "The considerations for incremental improvements of the prediction model in Section 4 seems out of context. The authors say that it allows them to gauge how similar but slightly better models affect policy outcomes, but the actual follow-up is only in the empirical section.", "questions_for_authors": "1. Could you please differentiate which ideas are truly new and which are adapted from Perdomo, 2024?\n2. Could you please explain Figure 4(c)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AcnAtN2Z0U", "reviewer": "Reviewer_gDhp", "summary": "This paper investigates the value of ML prediction when the goal is to identify the worst candidates as a screening problem. Formulating the problem as a screening problem with threshold, this paper derives 2 main theorems that characterizes the behavior of screening policies in general, and complement the studies with empirical evaluation whose outcome largely matches the theoretical derivation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback. We're especially glad you found the paper clear and relevant, and appreciate the time you took to engage with our work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A. Not familiar with field - intuitively sensible.", "experimental_designs_or_analyses": "Yes, the design is consistent with the theoretical setup.", "relation_to_broader_scientific_literature": "This paper investigates the value of ML-based policies in the context of screening.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: the paper is well-written, with the motivation of the problem as well as intuition of solution clearly explained. The problem investigated is of high relevance in real-time application of machine learning to social problems.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3rpAu186Qi", "reviewer": "Reviewer_aH4L", "summary": "In this paper, the authors consider a scenario where, for example, a government agency wants to identify the x% worst off in a population, based on a predicted metric $Y$. Once we have the predicted $Y$ for the population, we can sort everyone and have a follow-up check on those that scored worst. However, there is a budget $\\alpha$ for how many follow-up checks we can do. The authors argue that we should mostly care about the true positive rate in this case: how many of the people in the x% worst off have we sent to the follow-up check? We have thus two levers to affect the result: we can improve the predictions for $Y$ or we can have more follow-up checks. The paper asks the question when you should do which and makes some recommendations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed, thoughtful questions and suggestions for improving the paper! We appreciate your encouraging feedback on our conceptual framework and empirical analysis.\n\nCost Ratio and PAR = 1. Thank you for raising this. We fully agree that $\\Delta_{\\alpha}$ and $\\Delta_{R^2}$ represent changes with different costs.  For clarity and generality, we deliberately chose not to explicitly model these costs, as they naturally vary across application domains. We believe that examining the PAR independently remains valuable to disentangle benefits from costs. Policymakers typically find it easier to estimate relative costs of practical policy levers (e.g., hiring additional staff or conducting more frequent surveys to improve predictive accuracy) compared to evaluating how these changes will impact the policy value. We therefore focused on shedding insight on the welfare ratio with the understanding that domain experts have knowledge of the cost ratio and can easily factor these into the cost-benefit analysis (as in your expression). \n\nWhile our results hold for general $\\Delta_\\alpha$ and $\\Delta_{R^2}$, we chose PAR =1 with these fixed ($\\Delta_\\alpha = \\Delta_{R^2}$) purely for the sake of discussion. We are happy to point out how this choice is made for simplicity and will add further discussion on how the true threshold is at (cost of prediction) / (cost of access) as you point out. In the final version, we will move up and expand the discussion on marginal costs to the introduction, and we will aim to provide further context on the PAR = 1 threshold to avoid any confusion. Your perspective and comments on this issue are very helpful. Thank you.\n\nWhy differ from Perdomo 2024. This previous work focuses on the impact of different policy levers on *average* social welfare. By contrast, we focus on identifying the impacts of various policy levers on the welfare of the *worst-off* individuals. This captures a (mathematically and conceptually) distinct as well as a practically important setting often seen in the public sector. For instance, employment agencies commonly aim to identify and support those in greatest need, rather than maximizing average employment outcomes across all job seekers. We will clarify this distinction from Perdomo (2024) in the related work section of the revised manuscript.\n\nChoice of $R^2$. We will clarify the choice of $R^2$ and discuss other measures that could also be applicable in the final version of the manuscript. The short answer is that $R^2$ tends to exactly characterize predictability in our theoretical models. \n\nPresentation. Thank you for your suggestions on improving the presentation—your comments are very helpful in this regard. We will streamline the introduction to reduce redundancy and clarify the presentation to ensure that key concepts are formally defined before they are used. Given the additional space that comes with the revision, we will happily include a discussion at the end of the paper and will include subcaptions for the figures in the revised version of the manuscript.", "claims_and_evidence": "The core of the paper is a mathematical model of how to determine whether there is more value in improving predictions or more value in increasing the follow-up testing budget.\n\nI think the framework they define makes sense and is mostly well justified. This is mostly quite a good paper.\n\nThere is just one place where I think the claims are at the very least misleading; maybe even wrong: the fact that $\\Delta_{R^2}$ and $\\Delta_\\alpha$ are treated as having the same \"unit\". There is a paragraph on page 5, column 2, lines 251-264, which admits that this is wrong to do; it admits that one should consider the costs of $\\Delta_{R^2}$ and $\\Delta_\\alpha$ instead, but then the rest of the paper treats $\\Delta_{R^2}$ and $\\Delta_\\alpha$ as comparable anyway – for example, in Figure 3, you draw a line at PAR=1 as if that were a meaningful threshold in any way!\n\nI would think PAR should be defined like this:\n\n$$\n\\mathrm{PAR} = \\frac{\\frac{\\partial V}{\\partial\\alpha}\\frac{\\mathrm{d}\\alpha}{\\mathrm{d} c}}{\\frac{\\partial V}{\\partial R^2}\\frac{\\mathrm{d}R^2}{\\mathrm{d} c}}\n$$\n\nWhere $c$ is the total cost. This definition is robust to change of variables. For example if you were to go from $R^2$ to $2R^2$.\n\nI understand why the authors didn't want to do it like this – it means having to model the total cost as well – but right now it just seems wrong to me.\n\n---\n\nI would also have wished for more of a direct contrasting with Perdomo (2024). Why make different choices than that paper?", "methods_and_evaluation_criteria": "There isn't really a *method* proposed in the paper, so I can't really judge that. One thing that is maybe a bit odd is that when $R^2$ (the coefficient of determination) is picked as the variable that tracks how good the prediction model is, there is no discussion of any alternative variables one could choose and $R^2$ is then used even in the theorems.", "theoretical_claims": "I did not check the correctness of the proofs, but the theorems all look very reasonable.", "experimental_designs_or_analyses": "The numerical simulations and the case study are very well done. Here the paper really shines.", "relation_to_broader_scientific_literature": "The paper seems to build on Perdomo (2024) and seems to be a reasonable extension of that work, but I'm not really an expert in that field.", "essential_references_not_discussed": "None that I'm aware of.", "other_strengths_and_weaknesses": "## Weaknesses in the presentation\n\n- To me, section 1 is too long and too repetitive. The authors also made the choice of first explaining many things informally in section 1 and then explaining them formally in section 2. This, to me, was very confusing and I would have preferred reading the formal explanations directly. Reading the informal explanations first did not help me.\n  * For example, on page 2, PAR is defined informally, but all I kept wondering was how you want to compute it, which is then finally revealed on page 4. Maybe at least reference the later definition in the first informal definition.\n  * Informal Theorem 1.1 uses $R^2$ without any previous mention of it. Just please at least briefly define it before the theorem (or better yet, scrap the whole informal theorem stuff)\n  * Even when $R^2$ is finally defined on page 4, it happens *after* Proposition 2.3, which also references it\n- There is no Discussion/Conclusion at the end of the paper. I think even just a paragraph would be nice here. You can easily get enough space for this by trimming section 1.", "other_comments_or_suggestions": "- I would like it if the figures with multiple plots had subcaptions so that they can be better read at a glance.", "questions_for_authors": "1. Is there a justification for why PAR=1 is treated as a significant threshold even though it seems to me to be a pretty arbitrary one?\n2. You write “rather than evaluating changes in overall expected welfare \\[like Perdomo (2024) does\\], we measure the fraction of truly worst-off individuals who are identified”; can you expand on why you deviate from Perdomo (2024) here?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SV1HulJrTS", "reviewer": "Reviewer_FwzC", "summary": "The paper investigates the welfare impacts of using machine learning prediction systems in equity-driven government programs, particularly in identifying and supporting the most vulnerable individuals. The authors develop a framework to evaluate the relative effectiveness of prediction systems compared to other policy levers, such as expanding bureaucratic capacity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to carefully read our manuscript and provide comments. It is very much appreciated.\n\nOther Domains. Yes, performing similar evaluations in other empirical domains would be valuable, but beyond the scope of this paper due to the substantial effort required to access administrative, government datasets. While each context should be analyzed individually, the close alignment between our theoretical results and empirical findings might suggest that the PAR could behave similarly across different settings. Our paper exemplifies how to do this for a specific real-world, high-stakes setting and we hope it encourages future work in other domains. \n\nFairness Evaluation. Thank you for raising this point. We fully agree that fairness concerns, such as differential performance across protected subgroups (e.g., gender), must be considered in the design of allocation systems. We view our analysis of policy levers as a necessary—but not sufficient—step toward understanding when investments in predictive systems are justified. We believe these subgroup concerns are an important direction for follow up work.\n\nWe appreciate your suggestion of relevant references and will more explicitly connect our work to the literature on fairness in algorithmic decision-making (including Hardt et al., 2016 and Chouldechova et al., 2018) in the revised related work section. We will also include additional figures in the revision that show model performance across demographic subgroups.\n\nAdditional Models. We selected CatBoost as it is widely regarded as a strong, out-of-the-box model for tabular data. Government agencies often rely on standard, readily available ML models, and thus our primary aim was not to optimize predictive accuracy but rather to illustrate trade-offs that arise with commonly used predictive methods. Our findings remain robust when using alternative models, such as random forests.\n\nCosts. Thanks for bringing this up. The costs of improving prediction accuracy versus expanding screening capacity will typically differ and be domain-specific. A complete cost-benefit analysis would multiply the PAR by the ratio of marginal costs of both policy levers. Policymakers often have clearer estimates of these direct costs, while the PAR helps them determine how much more they should be willing to pay for one lever relative to the other. We will expand these cost considerations in the introduction of our revised manuscript.\n\nPresentation. We will streamline the introduction and link the informally stated main results to their formal counterparts to ensure technical clarity.", "claims_and_evidence": "The claims made in the paper are generally well-supported by both theoretical and empirical evidence. The authors provide clear mathematical formulations and proofs to support their theoretical claims, particularly regarding the prediction-access ratio (PAR) and the conditions under which prediction improvements are most effective. The empirical case study on long-term unemployment in Germany is well-designed and provides convincing evidence that expanding screening capacity often has a greater impact than improving prediction accuracy, especially in resource-constrained settings.\n\nHowever, one potential limitation is that the empirical results are based on a single case study (long-term unemployment in Germany), which may limit the generalizability of the findings to other contexts. While the authors provide a theoretical framework that is broadly applicable, additional empirical validation across different domains would strengthen the paper's claims.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem at hand. The authors use a combination of theoretical modeling and empirical analysis to evaluate the effectiveness of prediction systems in identifying the worst-off. The prediction-access ratio (PAR) is a novel and useful metric for comparing the relative benefits of improving prediction accuracy versus expanding screening capacity. The empirical evaluation is based on a large, real-world dataset, and the authors use standard machine learning techniques (e.g., CatBoost) to predict unemployment durations, which is appropriate for the task.\n\nOne potential improvement could be the inclusion of additional evaluation metrics, such as fairness or equity considerations, given that the paper focuses on equity-driven programs. While the authors discuss the importance of identifying the worst-off, they do not explicitly evaluate whether their prediction systems disproportionately affect certain demographic groups.", "theoretical_claims": "The theoretical claims in the paper are well-supported by mathematical proofs and derivations. The authors provide detailed proofs for their key results, including the conditions under which the prediction-access ratio (PAR) is maximized and the impact of expanding screening capacity versus improving prediction accuracy. The proofs are sound and correctly derived, and the authors provide intuitive explanations for their results.\n\nOne minor issue is that some of the theoretical results are presented informally (e.g., Theorem 1.1 and Theorem 1.2), which could make it difficult for readers to fully understand the technical details.", "experimental_designs_or_analyses": "The experimental design and analysis are sound. The authors use a large, real-world dataset on German jobseekers to evaluate their framework, and they carefully control for potential confounding factors by focusing on unemployment spells that began between 2010 and 2015. The use of CatBoost for predicting unemployment durations is appropriate, and the authors provide a clear methodology for simulating improvements in prediction accuracy by scaling residuals.\n\nOne potential limitation is that the authors do not explore the impact of different model architectures or feature sets on the prediction accuracy. While they compare a shallow decision tree to CatBoost, a more comprehensive comparison of different models could provide additional insights into the trade-offs between model complexity and prediction accuracy.", "relation_to_broader_scientific_literature": "The paper is well-situated within the broader scientific literature on machine learning for public policy and resource allocation. The authors draw on prior work in decision theory, operations research, and causal inference to develop their framework, and they cite relevant studies on the use of risk prediction systems in government programs. The paper extends prior work by focusing on the relative value of prediction in identifying the worst-off, rather than optimizing aggregate outcomes.\n\nThe authors also discuss recent work on the trade-offs and pitfalls of risk-scoring systems, which is an important consideration in the design of equity-driven programs. However, they could have more explicitly connected their work to the growing literature on fairness and bias in machine learning, particularly in the context of public policy.", "essential_references_not_discussed": "Relevant works that could be cited include:\n\nFairness in Machine Learning: The paper could discuss recent work on fairness-aware machine learning, such as Hardt et al., 2016, which provides a framework for evaluating and mitigating bias in predictive models.\n\nAlgorithmic Fairness in Public Policy: The authors could also cite Chouldechova et al., 2018, which discusses the trade-offs between accuracy and fairness in algorithmic decision-making, particularly in the context of child welfare screening.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The empirical results are based on a single case study. Could the authors comment on the generalizability of their findings to other domains, such as poverty targeting or social housing? How might the PAR differ in these contexts?\n\nThe paper focuses on identifying the worst-off, but does not explicitly address potential biases or disparities in the prediction systems. Have the authors considered the fairness implications of their framework, particularly in terms of how different demographic groups might be affected by the screening policies?\n\nThe authors compare a shallow decision tree to CatBoost, but do not explore other model architectures or feature sets. Could the authors provide additional insights into how different models might impact the prediction-access ratio (PAR) and policy value?\n\nThe paper discusses the relative value of improving prediction accuracy versus expanding screening capacity, but does not provide a detailed cost-benefit analysis. Could the authors provide more information on the costs associated with improving prediction accuracy (e.g., data collection, model training) versus expanding screening capacity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a stylized model of policy problems where the goal is to identify applicants to a program who are the worst-off. It studies the marginal value of investing resources in improving a predictive model for this task compared to expanding the capacity to screen more applicants. The focus of the paper is on both theoretically characterizing this tradeoff as well as an empirical case study of an unemployment program. Reviewers found the paper to be an important contribution to the emerging literature on the value and tradeoffs associated with the use of prediction and social settings. I also believe it is important for the field to pay more attention to these issues. 

I do agree with the suggestion made by Reviewer aH4L, that the "units" of changes in prediction vs access are not the same. While the authors are correct that costs are both application-specific and typically easier to determine, the right interpretation of the tradeoff might vary substantially depending on both much we expect them to differ (eg if in some cases improving prediction can be viewed as a fixed cost, vs a marginal cost for each additional applicant screened). I encourage the authors to expand their discussion of this point in the revised paper.
Decision: Accept (oral)