Performative Risk Control: Calibrating Models for Reliable Deployment under Performativity
Authors: Victor Li
Baiting Chen
Yuzhen Mao
Qi Lei
Zhun Deng
Venue: NeurIPS 2025 poster ()
Keywords: risk control
performative prediction
statistical learning theory
Primary area: theory

Abstract: Calibrating blackbox machine learning models to achieve risk control is crucial to ensure reliable decision-making. A rich line of literature has been studying how to calibrate a model so that its predictions satisfy explicit finite-sample statistical guarantees under a _fixed_, _static_, and unknown data-generating distribution. However, prediction-supported decisions may influence the outcome they aim to predict, a phenomenon named _performativity_ of predictions, which is commonly seen in social science and economics. In this paper, we introduce _Performative Risk Control_, a framework to calibrate models to achieve risk control under performativity with provable theoretical guarantees. Specifically, we provide an iteratively refined calibration process, where we ensure the predictions are improved and risk-controlled throughout the process. We also study different types of risk measures and choices of tail bounds. Lastly, we demonstrate the effectiveness of our framework by numerical experiments on the task of predicting credit default risk. To the best of our knowledge, this work is the first one to study statistically rigorous risk control under performativity, which will serve as an important safeguard against a wide range of strategic manipulation in decision-making processes.

Performative Risk Control: Calibrating Models for Reliable Deployment under Performativity
Victor Li", Baiting Chen', Yuzhen Mao², Qi Lei', and Zhun Deng*
New York University
2 University of California, Los Angeles
Sttanford University
University of North Carolina at Chapel Hill
Abstract
Calibrating blackbox machine learning models to achieve risk control is crucial ensure reliable maaiii..mmaiin..... A rich line of literature has been studying how calibrate model so that its predictions satisfy explicit finite-sample statistical guarantees under a fixed, static, and unknown data-generating distribution. How- ever, prediction-supportt decisions may influence the outcome they aim predict, a phenomenon named performativity of predictions, which is commonly seen in so- cial science and economics. In this paper, introduce Performative Risk Control, a framework to calibrate models to achieve risk control under performativity with provable theoretical guarantees. Specifically, we provide an iteratively refined cali- bration process, where we ensure the predictions are improved and rist-controlled throughout the process. We also study different types of risk measures and choices of tail bounds. Lastly, we demonstrate the effectiveness of our framework by numerical experiments on the task of predicting credit default risk. To the best of our knowledge, this work is the first one to study statistically rigorous risk control under performativity, which will serve as an important safeguard against a wide range of strategic manipulation in decision-making processes
Introduction
We have entered era characterized by the ubiquitous deployment of increasingly complex models, such as large language models (LLMs) with billions of parameters [133 These models play important roles in our daily lives, informing us, shaping our opinions, and deciding allocation of societal resources. However, in most scenarios, these models can only be treated as a blackbox, either because they are too complex to understand simply because their details are kept private by companies. Given their influence in today's 's society and their blackbox nature, it is urgent to derive new tools ensure reliable deployment.
recent line of work 4 has been investigating methods to calibrate the predictions blackbox machine learning models, generally known (conformal) risk control. The goal is to ensure reliable deployment by satisfying explicit, finite-sample statistical guarantees (either with high probability marginally) for controlling risk. The introduced frameworks are lightweight, agnostic to the data-gent data-generating distribution, and do not require model refitting- Specifically, (conformal) risk control, blackbox model f is given us, and need post-process it using calibration data in order to make our final predictions. The calibration process is governed by low-dimensional parameter À (e.g..& a threshold). For example, and study false negative rate (FNR) control in
'Code for experiments here: http:://githaab.cc///vt/rrrrrg
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

tumor segmentation, where f: x' + [0, 1]dxd takes an image as imput and outputs scores for all
the d pixels in image z, For given A, one can produce (x := {i f(x); 2 }} to include
all the pixels with "high" scores. The aim is to control the expected loss:
1
y n Tx())
B( D
y
where label y is the set of pixels truly containing polyp segmentations, f(z); is the i-th coordinate of fx)) and 'D is the underlying data-generating distribution.
In contrast to tumor segmentation, other applications- paciaittiooosppprluuarraaa those in sociotechnical settings must account for performativity, in which predictions of machine learning models impact the outcome to predict 77. For instance, bank may adjust decision threshold parameter that governs the bank's loan policies. Such policies then impact the purchasing patterns of the population, which in turn change the features used by the bank in predicting default risk. Similarly, in content moderation, stricter content content-filering policies can affect how users compose their posts. A generation-come parameter in a power plant, which controls unmet demand by setting supply based on demand forecasts, can alter future demand patterns. What unifies these settings is the cyclical relationship between the decision- makiin parameter and the distribution is intended to control risk for. So far, this important setting of risk control on blackbox models under performativity eludes the literature. In light of this, we initate the study of Performative Risk Control (PRC).
Formalizing our goal. We post-process the outputs of the pre-trained model ((x) in Y to generate predictions (x) E y" indexed by a scalar threshold A. Here, Tx could either be standard predictions, i.e., y" = y. or prediction sets, i.e., y = 2 In this paper, we mainly study the expected risk. However. since we further consider the performativity of predictions, the measure of our interest is mainly the performative expected risk defined as
R(A)= := ((E,))/DIA) [C(y, (x))]
where D(A) is the distribution induced by the decision threshold À and is typically unknown. In Sec. we also discuss a broader class of measures beyond expected risk. By carefully setting the parameter X, control user-chosen error rate, regardless of the quality of f. Specifically, our goal is to calibrate f on a calibration dataset Zeal (specified later in Sec. 3.1) and achieve the following:
Definition 1.1. Let À E A be the threshold obtained by calibrating Teal. We say that risk R()) is (ar o----rffrrmative-ris--oomme
if
P(R(A) < 2 6-
where the randomness is taken on Teal. Here, 2 0 and 6 E 10, are both specified by the user.
Our contribution. Our central contribution to introduce an iteratively refined procedure to select risk-controlling decision thresholds in performative environment, as illustrated informally in Fig. Our framework greatly generalizes the previous line of work on risk control for a static distribution to dynamic and model-dependent distribution. In particular, our work can handle strategic manipulation of input distributions in decision- processes. decision-processes. Our framework satisfies explicit finite-sample statistical guarantees, and we present experimental results that highlight its practical utility. To the best of our knowledge, we are the first to study risk control under performativity, and we believe our work will serve an important safeguard in a wide range of applications in social science and economics.
1.1 Related Work
Performative prediction. Performative prediction generalizes the concept of strategic classification the idea that individuals adjust their features to game a classifier-by formalizing the prediction- influenced data D()) function of the model parameters 0 (7]. Follow-up works on performative prediction have extended the framework to various settings. Miller et al. [9] provides conditions for
which performatively optimal model parameters can be found, while Brown al. [IO considers the history of model deployments when modeling performative distribution shifts. Jagadeesan et [ and Chen et al. [12] cast performativity into a bandit problem, with the assumption of additional knowledge through access to the samples. In these settings, the goal is to find setting of parameters that some way minimizes an expected loss and obtain performative optimal point ppo = arg ming EEnD(O) ((2; 0). However, due to the unknown D(0), this task

---

Decision maker (us)
Empirical risk of 7. on shaped data
Performativity guard
shaped
Safe
data
deployment
Confidence
bands
Society
7.
min
[-1
2
>saee
Figure I:
An overview of the Performative Risk Control (PRC) framework. Left: We, the decision maker, use samples from past deployments to determine X₂ that is safe to deploy in the performative environment. Right:
We do so by defensively choosing a λx careful to consider generalization error due to sampling and performative error due to deploying in the new distribution induced by dς as further explained Sec.
intractable, so much focus has been devoted to studying performative stability, i.e., finding Ops arg ming Eg>D((es) ((z;)) In our setting, our objective is instead to control the expected loss, ie., find the most aggressive setting of the threshold Á while guarantceing R(()) is risk controlled. To the best of our knowledge, risk control in the performative setting has yet to be explored. For an overview on performative prediction, Hardt and Mendler-Dünner
13 serves as a solid reference.
Distribution-free uncertainty quantification. Conformal prediction aims to endow black-box models with rigorous, sample statistical guarantees [4] Conformal risk control is an extension to this framework that handles more general class of loss functions [ These works are concerned with controlling the expected risk at some threshold ar; further work by Snell al. 17 extends this framework to more general quantile risks.
Risk control under distribution shift. The canonical conformal prediction procedure relies on the assumption that the calibration and test data are i.i.d., or more generally, exchangeable. Because this assumption often does not hold in real-world environments, statistical extensions of conformal prediction have been developed to incorporate distribution shift. Tibshirani et al. [18] deals with covariate shift, while Gibbs and Candes [9] handles distribution shift in the online setting by continuously re-estimating single parameter to achieve exact coverage. Barber et al. (20) generalizes conformal prediction to account for nonexchangeability through a procedure that weighs points in the calibration data higher if they are closer in distribution to the test data. Other works focus on making coverage robust to set of possible test distributions, e.g., those in an f-divergence ball around the training distribution [1 and those due to feature alterations by strategic agents after model deployment [22].
2 Setup
This section introduces our setting and goals and their significance more formally.
2.1
Setting and Notation
For E No, we use [] to denote (1, For simplicity, we denote 2 = x y and (I. y). Recall that we have prediction function indexed by threshold X, where for fixed À E A. TA is a function mapping X' to )" Here, y" could either y or 2 We further use {(2, A) short for fyy, (x)).
Monotonicity. Throughout the paper, we consider {(2, X) be non-increasing with respect to À for any Z. This is mainly following and motivated by previous work in standard risk control [5). On
3

---

one hand, one can think of the case that Tx is a set-valued function as in [5] 6 that
X1 A2 = Tx, (x) - TAa
A larger set includes more candidates and provides larger tolerance region that will decrease the loss values of loss functions such as the classical one corresponding to tolerance region, i.c. S) 1 fy & S}. which satisfies S C 8" + () 2 S"y On the other hand, when Tx (2x) a standard prediction, larger À indicates a more conservative but safer decision. For the example of credit loans, Tx
}} may indicate whether an applicant gets credit (1 means they got it). As À decreases, the credit firm relies more on automatic decision decision-making making, accepting a trade-off where it gains efficiency in processing applications but faces increased firm risk as measured by the loss function {(2 A).
< 1[)(x)
Setup for A. We consider the case where there exists safe threshold Aute such that f(2, Amfee) = 0, and we consider A [mmin Asafe] for Amin E R For the example of Tx(x) 1 [f)(x) S 1 }} above, we can choose Aate = 1 and in the example of detecting tumor segmentation in [5, we can
choose Asafe large enough to make Sxate include all possible labels, so that fly, Saac) = 1 & Stue 0.
2.2 Our Desiderata and Significance
As mentioned in the introduction, our main goal is to select an appropriate threshold Å by using the calibration dataset Teal, such that
P(R(A) aa)
However, there might be multiple choices for and how to define "appropriate" under performativity nuanced. We further illustrate our extra desiderata here.
Achieve user-specified conservativeness. Our setting prescribes that a larger leads to being more conservative. As an extreme case, choosing À = Asafe can guarantee zero loss, thus satisfying (a, pperformative-ris-ccootte
for all a 2 0. But this safe-guard choice of À is not useful in practice since it sacrifices too much utility in providing useful class information: for credit scoring, this choice is equivalent to manually reviewing all applications. This illustration demonstrates that a user of this framework may further desire choose À that is less conservative. To formalize this, we wish the final choice of À also satisfies
((((() a Aa) 2
for a small user-specified Aa. In particular, we show that our framework allows users to choose Aa such that Aa + 0 as n + co. This means that our framework can provide a also satisfying a tight lower bound guarantee for the performative risk, i.e., À the optimal and least conservative in an asymptotic sense.
Safe at anytime. Similar to the literature of performative prediction 77, our algorithm an iterative algorithm (see 3 for details). On a high-level summary. our algorithm guarantees that the Å updated in each iteration is improving, i.e., Å, < Â until we find our final À that satisfies the user-specified conservativeness. However, we don't 't want any of the X, 's to be chosen too aggressively. resulting a threshold too small to be risk-controlled. This is especially important policy-making, where a policy's 's risk must be closely monitored as its usage is incrementally increased. Thus, we further require that the trajectory of the X 's is safe for all / E [T]:
P(R(A.) S a) 2 1-8, 8, where T' is the iteration of return in our algorithm.
3 Performative Risk Control
In this section, we describe PRC's main results. We provide an iterative procedure compute a series of safe thresholds. Specifically, we demonstrate that our framework provides statistical guarantees on the ability to maintain risk control user-specified level a throughout the iterative process, well as obtain À achieving user-specified tightness at Aa.
*Our setting follows (5), in which the loss value becomes trivially too large at Amia and too small at Aute- This holds generally in risk control literature.

---

3.1 Our Procedure
To give a high-level summary, we provide an iterative procedure to obtain less conservative thresholds at every iteration. Specifically, until the process is terminated, at time t 2 1, we make progress by choosing Å such that X < X With probability at least 1 - 8, our selections remain safe at for all [T] and upon termination, the final selection achieves Åa. PRC starts with setting Åo Asaee and then
any thme-that is, R()))
user-specified tight risk control with R(AT) does the following process:
Performative Risk Control (PRC)
For 2 1, we sample Z = f(11-1,5, Ut 11-1. E X x M i.i.d. drawn from D(4+-1) (here, do An) and calculate
de min Wib
where V(Ar-1, 6) is calculated based on the calibration set Taa 2₂ < A - AA for progress measure Aλ, which means A makes enough progress over A then continue with + 1. Otherwise, return d
We specify details of V and AA later. Notice that the calibration dataset Zeai in Def. I.] is the union of calibration sets throughout the process, i.e., Teal = U.Teal- The formal and complete algorithm is listed in Alg. [
3.2 Technical Details
In this subsection, we spell out details regarding various parameters and functions in our procedure. Before that, we need state our key assumption. Intuitively, if the distribution shift of D(A) between different X's is too dramatic, it will be hard to achieve performative risk control. Thus, we impose our key distributional assumptions below, which is similar to the one proposed in classical performative prediction (7] but more tailored to the risk control setting.
Definition 3.1 ((7. p. g) ssensitivviyy Let g Z R be deterministic function, and denote Dj (A) as the induced distribution g(z) where ~ D(A). distribution map D(.) is (Y, p, P.g) -sensiiivee for all \1, X2 E A.
W, (D (D, (A,)) S 7/A1 - A2
where Wp denotes the Wasserstein- distance.
We remark here on the similarities and differences between this definition and the definition of c-sensitivity in [7]. First, while e-sensitivity parameterizes distribution shifts through the model parameters 8, (7: p. -sensiivvvvy does so through the risk-control parameter X. Both shifts modify the distribution of samples of the shift in the distribution of the samples themselves. In (T.P. -eseiiiityyyyy, p left to be arbitrary, and measures how much the distribution of the transformed samples g(z) changes as result of the shift z.
(I. y). In e-sensitivity, p is set to 1, and quantifies the magnitude
Intuitively, we view À as affecting the distribution of samples which in turn impacts the loss distribution. The following assumption applies (7. P, (...P)))-oo g)sensiitivity our setting: the loss distribution arising from samples drawn from similar thresholds are also similar.
Assumption 3.2 (Lipschitz. Distribution Mapping). We assume that D(.) is (7,2, - sensitive for all X e A.
We
ready to state further technical details. Without loss of generality, can consider N. n for all 2 0 since we can denote n as the minimum of } presented in our procedure, and all the results below still hold.
(A)( X') to be the empirical mean on the loss of D(A) when evaluated against the threshold X' With this notation, I could also be denoted (1x)) We further denote X') = (2,
Choice of V. We denote iid. samples (A)) ~
5

---

To construct V. we require knowledge of a confidence width c(n, 8') defining pointwise confidence
bounds R (A, X', 8 ;= R., (A, =c(,, 8') that satisfy the following property for d and X 2 A for all 2 encountered the procedure:
IP (2 R(( R (A6-1, 21
(I)
where 8' = and 7 is upper bound the number of iterations T. We discuss choices for c(n, 0")_"cludiingn derivations from Hoeffding's inequality, Bernstein's 's inequality, Hoeffding- Bentkus's inequality, and the central limit theorem-in App. A. We are further often interested in the risk profile of R()e,)) -), whose difference from R R(A can be bounded via sensitivity. We call this difference the performative enror, or the error that arises from evaluating on two different distributions. In nutshell, we hope to update Å -> Åe by accounting for increases in the risk due to generalization and performative error, which leads to the following form:
V(r-11, ). Ru 1,
d/,
-(d--12))
cmpirical loss
confidence width
performativity guard
The user-chosen parameter 7 > guards against performative effects; we require 7 2 While y is typically unknown, one estimate its potential range magnitude based domain knowledge of the problem at hand to then choose a high enough т.
Choice of AA. We choose AA
Aa 2c(n, oe(n,8/T)) Once the update Ar- - de falls short
of the progress measure AA, PRC terminates and outputs A4.
Number iterations T vs. guaranteed convergence Т. This distinction is necessary given that per-iteration eтrоr rates 8' compound and T is initially unknown. After T iterations. the trajectory error rate is 8',. so defining 8' = 6/T allows us to bound this error rate at o'T = 8(T/T) S 8. While the definition of 7 increases the confidence width and increases the complexity of PRC, the benefit is that PRC can inform the user whether theoretical guarantees are possible for given (ax, Au, 8) prior to any samples being collected.
Obtaining T and AA. Since we require Aλ progress on each iteration on the interval [Amin- Asafe PRC terminates in at most [Asate - Amm) / AA] iterations. Hence, choosing T 2 [(Asate Amin) AA] or AA 2 (Auate \/in)/ guarantees convergence, i.e. T < T, always. Further, because AA is also function of 7", we need to jointly solve (T) AX) E N x R. If there are solutions, PRC cannot guarantee that its sequentially produced solutions are safe; this case, we just return Asafe-
Details about Aa. Suppose let T, n oo with T Cn" (for some constant C' and 1/2)
and AA (Asate Amm)/TT Then, Aa (In(n)/ Vn). We leave this derivation App. A.2
We summarize all the discussions above with the following detailed and complete algorithm in Alg.
Algorithm Performative Risk Control
Require: Ao,, 5, loss f, distribution mapping D(.), Amin: Asate, safety parameter
Ensure: Output À
1: Initialize Ao <- Asale
2: Jointly solve AA) E s.t. 27AA Aa 2c(n, 5/T) and AA > (Asafe Amin) T/T.
Break ties by choosing the solution with the minimum T. If there are no solutions, return Asate-
3: for 1 7 do
4: Receive samples (4-11)7 iid D(A--)) where Åo do
5: Set Set + min inf(X A V(A.-1, o0)dr-1)
6: if 7 2 Â--1-1 - AA then
7: return &
8: end
9:
end for

---

The analogy to 3.3 Theoretical Guarantees
We now state our main theoretical result. By the following theorem, we demonstrate how our procedure can realize the promise of achieving user-specified conservativeness and being safe and reliable at anytime.
Theorem 3.3 (Risk control of PRC). Suppose Assumption holds, so that the distribution mapping D(-) is (7, 1, f(. -ee-sssitieee for all E A. Assume further that the loss function {(z, X) is continuous in for all z, and that the safety parameter T satisfies T 2 7, If the initial joint solve of (T, AA) produces at least one value, Algorithum guarantees that with probability 1 8, the following three conditions are met simultaneously:
(i) Safety in the iterative process:
R(A--1,) (1) Sa, 15tsT,
(ii) Safety at any time:
R(A)) Sa. 051ST,
(iii) Tightness of Xx
R(Ar) 2 a Aa, T 1.
Remark 3.4. In Thm. 3.3 we rely on the key distributional assumption 3.2 Because the distribution
shift is measured on the loss samples themselves, Alg. uccommodutes model finetuning (e.g. via supervised finetuning or reinforcement learning) between iterations, as long as assumption satisfied. Consequently, PRC allows data and model to co-evolve, as long finetuning maintains relatively stable loss distribution.
4 Extension
In this section, we discuss how PRC can be extended to handle quantile-based risk measures, which commonly used in quantify tail risk and widely used in mathematical finance. In particular, both conditional variance-at-risk (CVaR) and expected risk belong to quantile-based risk measures. Denote the cumulative distribution function (CDF) of losses for z ~ P(A) and evaluation threshold X' as F(w: A, X) == P(E(z, X') < w). Further, recall that the inverse of CDF F is defined as F F-1 (p) == inf(x: F(x) 2 p}, which leads to the following definition.
Definition 4.1 (Quantile-basee risk measure). Let (p) be weighting function such that vp(p) 2 and fo (p)dpp = 1. The quantile-based risk measure defined by e is
(F) = S'' ((D)F
From now on, we refer the quantile-based risk R, (A, the expected risk case is made clear when (p)) = 1. Indeed,
Ro, S. Fr F xx):; ESND(A)((, R(A,X)
Other values of (p) allow to control the 8-VaR (e.g., the 90th percentile of losses) and 3-CVaR (e.g., the average of the worst 10% of losses). To extend Thm. 3.3 we need to construct a confidence width c(n, 8'). To do so, note that have knowledge of the empirical loss CDF of the samples from iteration evaluated against any threshold X/:
fa A X) = n Σ 1(((2(((-1)))))) s w).
X, We follow the technique in 17 and 23 to create LCB Pr.m( d N, 8') and UCB Fr.m
A 88))) based on the empirical CDF, where
IP [L.n(W: 2, )))+ < F(w; Pe-1 X' S (w, dr-1 X', 88)) 2 - Vi
7

---

We are Next, similar to how (A) is constructed from F(- X, X,), we construct R (21-1, ,')), Pep. (x--1,)), X), and R (a (,, X', 8*) from -,-,, Fal.: (,,1,, and Fi.n(: (-; A X,8"), respectively. Note that the risk measure's lower bound Ry 11=11;1 N, 8') corresponds to the CDF upper bound Ft >,,,, and vice versa (see App. A.4 The bounds on risk measure Rip (A satisfy the following:
vin P 1 1,, S Rop (Ae S fet (Ae (2r-1. 2 -
Let c(n, 8, X²) max R Finally, we compute the confidence width c(n, 8') using the technique described in App. A.I now ready to extend Thm. 3.3 to quantile-based risk measures.
R w.n (2- Ry,, (2,-1,1)))) Rup.n (de-1, x²) R (dp-1, 1,, 8')).
Theorem 4.2 (Quantile risk control of PRC). Alg. D replace the confidence width with the one derived above. Further, replace R,, (At-1, X) with Ry (x(11,)) A) in the definition of V in line 5 Let u,v E [1, oo] with 1/u + 1/U 1. If the following four conditions hold: (i) the distribution mapping (7,, u, E(,, (() -))-snsiitive for all À E A, (û) the loss function E(z, X) is continuous in X for all z, (iii 1/v
T 227 ? [oo dp))
and (iv) the initial joint solve of (T, ΔX) produces at least one value: then, this modified version of Alg. guarantees that with probubility 1 6, the quantile risk Ry of the
iterates À satisfies the same three guarantees as those Thm. 3.3
5 Experiment
This section considers the application of our framework to credit scoring in both the expected and quantile risk settings. We present the most important results here and leave details to App
Type II Error Control for Credit Scoring. We first consider type II error control for the binary classification task. Labeling the positive class y 1, we form the standard prediction Tx,c(z)
1 where À E [Amin Asate] = [0. 1]. Here, f(x) estimates the probability that x 0 and abstain from making a prediction otherwise. We consider the type II error fyy TT,((r)) = y(1 Tx,c(z)). accumulating loss for each positive instance assigned to the negative class. We apply this formulation automatic credit approval. Credit applicants submit their applications with features x. The model f(x) predicts the probability of serious delinquency, and y labels whether one occurred. When (x,) O. the application is automatically accepted; otherwise, it is flagged and subjected to further human review. The threshold [o, 1] trades off the balance of these two, with lower À corresponding more automatic acceptances. We focus solely on the Type II errors made by the model.
clipp -------------- f(x) 0,
2
is positive. We assign to the negative class when (x,((z)
However, À is performative. We are not deploying À into static distribution; applicants respond by maximizing their chances of receiving credit approval. Denote x zAA) as an applicant's 's features in response the deployment of X. We simulate this shift in the features by impacting (=x(A)) as follows:
ma
s) if fxx - - otherwise
f(((A)) = ((())
for all À and choice of where s-= 0.3) is value we choose. This shift is (() we explore the validity of this claim in App. B.I
We demonstrate results on a class-balanced subset of a Kaggle credit scoring dataset 24), following 77. Our simulation experiment is reported in Figure 2 We control risk at a 0.3 with Aa 0.082. We use n = 2000, the CLT bound, and = 0.1. We experimentally verify that y < 1.38 (see App. B. As expected from Thm. 3.3 PRC iterates safely as it converges upon final that is both risk controlldl and not too conservative for large enough т.
We use c 10 This standard prediction approximates the indicator function (x)()) S 1- as as For simplicity, our theory assumes continuous loss function and does not deal with losses with bounded jump discontinuities. hence the approximation here. However, our theory can easily be extended to handle these cases well, as done in Angelopoulos al. (6).
8

---

1.0
10a
10
10
10
0.4
02
10a3
0-
8
10
12
16
10
12
14
16
Iteration
Iteration
0.30
Upp Bound
use Bound
0.25
0.20
0.3
Risk
Risk
0.10
0.1
0.05
Uppe bound Bound
0.00
0.1
0.5 t
0.8
Iteration
Figure 2: Type II error control in credit scoring. We use the same coloring scheme to label trajectories with different settings of across the subplots. Top left: 5 0 shows algorithm that does not account for performativity. As increases, the incremental updates to Xx become grained and conservative. This safety comes at the trade-off of longer trajectories. Top right: An illustration of our stopping criteria for different values of т. To guarantee tightness with higher T, AA-displayed as the colored, horizontal dotted lines-needs be lower. Once Å - Å, falls below Aλ, PRC ends exploration and returns Åς = A Bottom left: 1, 000 trajectories with r 1, each trajectory corresponding to different cut between calibration and validation sets. The iterates Åe were chosen based the calibration set, and the displayed risk is the validation set. Let (Zr,i M= denote the validation samples from iteration L. For each iteration 4, the plot displays 1 En= E(2,,,, Åτ) followed by n. E(2,,,, 1c. most noticeable when = 0. Risk controlled for both quantities. Bottom right: scatterplot of R(Ar) for each т. For 2 % PRC achieves its goal of being (o., peepffformtiiee-iin -isk--rro----- Even for some values of (e.g.. 0.5, 0.8, 1.0), the final risk still controlled (also see App. B.T
fine-
Quantile Risk Control in Credit Scoring. a highly imbalanced dataset where the negative class much common than the positive class, risk control small expected type II error will not be very meaningful. Quantile-based risk measures target scenarios like this one, in which we seek guarantees the tail end of negative outcomes. Given that measures, we adjust our setup to more accurately reflect a setting in which they would be useful. For each positive delinquency on each iteration, simulate realized cost by drawing from U(0, 1], which can be thought of as the amount the institution loses on an individual, via being late on payments or defaulting, scaled between and 1. We also assume knowledge of the base rate p of credited individual experiencing serious delinquency; in this experiment, p 0.0640.
investigating quantile-based risk
We focus the 3-CVaR risk measure, with 8 0.9. We that depends 5, T. B, and p (see App. B.2 We increase many samples because 1 of the population has a loss of 0 and is not informative for us in setting
quantile-based based-CLT confidence width 10. 000. Note that require this
X. This setting follows Thm 4.2 with
1, cc, and 7 [c' (p))
1/e
7/
<
We experimentally verify < 0.205, so Thm. 4.2 applies when 2 2.05. Fig. 3 shows our results for a 0.25 and Aa 0.12. We observe that the confidence bounds are somewhat generous; each trajectory ends with a final R(Â)) inside the bounds. However, this width is necessary given that this dataset is highly skewed and that we controlling the most variable 10% of losses.

---

0.35
Upor Bound
Bound
0.30
0.25
Risk
0.20
0.10
0.10
0.05
0.00
0.00
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
0.2
0.4
1.6
Iteration
Figure 3: 90%-CVal type emor control in credit scoring. These plots correspond to the bottom two plots in 2 2 but for the 90%-CVaR risk measure. The left plot shows instead of
Conclusion
We have introduced theoretically sound and empirically effective framework for risk control under performativity, a previously unexplored topic in the risk control literature. To mitigate performativity. our framework iteratively refines series of decision thresholds, careful to control risk. As concrete example. we examined the deployment of credit scoring system for loan approvals and found that PRC successfully controls risk, even as credit applicants seek strategically manipulate their outcome. One limitation of PRC is sample efficiency, which stems from the iterative nature of performative algorithms and the necessity form statistical bounds under distribution shift. While we feel that our desiderata of being safety at anytime is important in the performative setting, one could achieve higher sample efficiency by relaxing the desiderata to being safe on average over time. Nevertheless, given the growing role of blackbox models in shaping policies of societal significance, PRC offers an effective and proactive strategy for their safe deployment.
Broader Impacts This paper proposes a framework to safely and reliably deploy models whose predictions influence society. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted.
Acknowledgments
We would like to thank Juan Perdomo and Lihua Lei for helpful discussions and feedback. We also thank the anonymous reviewers and area chairs for their insightful comments. QL acknowledges support of NSF DMS-2523382.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rXiv:2303.08774 2023.
[2] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Saan-Bapiiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
[3] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
[4] Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms achieve risk control, 2022. URL
https: aariiiiii...//... org/ abs//bb0....a///a/a/////bb/ 01052
10

---

[5] Stephen Bates, Anastasios Angelopoulos. Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, rist-controlling prediction sets, Journal of the ACM (JACM), 68(6))1-34, 2021.
[6] Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Confor- mal risk control. arXiv preprint arXiv:2208.022: 02814, 2022.
[7] Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning, pages 7599-7609. PMLR, 2020.
[8] Moritz Hardt, Nimrod Megiddo. Christos Papadimitriou, and Mary Wootters. Strategic classi- fication. Proceedings of the 2016 ACM conference on innovations in theoretical computer science, pages 111-122, 2016.
[9] John P Miller, Juan C Perdomo, and Tijana Zmnic Outside the echo chamber: Optimizing the performative risk. In International Conference on Machine Learning, pages 7710-7720. PMLR, 2021.
[10] Gavin Brown, Shlomi Hod, and Iden Kalemaj. Performative prediction in a stateful world. In International conference on artificial intelligence and statistics, pages 6045-6061. PMLR, 2022.
[11] Meena Jagadeesan, Tijana Zrnic and Celestine Mendler-Dünner. Regret minimization with performative feedback. In International Conference on Machine Learning, pages 9760-9785. PMLR, 2022.
[12) Yatong Chen, Wei Tang, Chien-Ju Ho, and Yang Liu. Performative prediction with bandit feedback: Learning through reparameterization. arXiv preprint arXiv:2305.01094 2023.
[13] Moritz Hardt and Celestine Hendler-Dünner. Performative prediction: Past and future. arXiv preprint arXiv:2310. 16608, 2023.
[14] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in Random World. Springer-Verlag, Berlin, Heidelberg, 2005. ISBN 0387001522.
[15] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(12):371-421, 2008. http:///: org/ http://rrrrgpppprrreeeeeeee v swaffrafrBra
html
[16] Anastasios Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 2021.
[17] Jake C Snell, Thomas P Zollo, Zhun Deng, Toniann Pitassi, and Richard Zemel. Quantile risk control: A flexible framework for bounding the probability of high-loss predictions. arXiv preprint arXiv:2212. 13629, 2022.
[18] Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.
[19] Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34: 14:66661167, 2021.
[20] Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, (1(2):816-845, 2023.
[21] Maxime Cauchois, Suyash Gupta, Alnur Ali, and John Duchi. Robust validation: Confident predictions even when distributions shift. Journal of the American Statistical Association, pages
1 1-66, 2024.
[22] Daniel Csillag, Claudio José Struchiner, and Guilherme Tegoni Goedert. Strategic conformal prediction, 2024. URLL/ttpp:://arrii.oor///b/b///
[23] Thomas P Zollo, Todd Morrill, Zhun Deng, Jake C Snell, Toniann Pitassi, and Richard Zemel. Prompt risk control: rigorous framework for responsible deployment of large language models, arXiv preprint arXiv:2311.13628 2023.

---

[24] Credit Fusion and Will Cukierski. Give me some credit.
https: ftapss://kggle..om com/
2011. Kaggle.
[25] Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization, 2009,
.7440
[26] A. W. van der Vaart. Asymptotic Statisties, Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998.
12

---

0.16
0.10
2
0.08
0.04
Hoertding
Bemstein bound
0.02
Hoef
CIT bound
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Figure 4: A comparison of Hoeffding and the risk level-dependent bounds. Plots 2c(n, 8*) vs. for n 2000, T' = 100, d 0.1, and 8' = 8/TT. 2c(n, 8') represents the lower limit of achievable tightness Aa. Lower confidence widths are better.
A Proofs
A.1
Choice of bounds
The purpose of the confidence width c(n, is to guard against generalization error. Because need access to c(n, 8') before the initial deployment and c(n, 6') must guard against worst-case distributions that could arise from performative shift, the bounds presented here are looser than those found in the literature, yet still fairly tight. In this section, we detail how we can derive confidence widths from variety of concentration inequalities, a comparison of which is shown in Fig. 4 for bounded loss scaled in the range [0, l].
Hoeffding's inequality For any A, X' A, i D(A), then ((2 E [0, 1]. Because these bounded random variables, Hoeffding's inequality applies, which says that for any E 10,
P (( R(A,)) s <c(n, 8') 2 21-8
where 5'n, V ln(2/8'.. While present Hoeffding's inequality for illustration,
sharper bounds in practice. The rest the bounds we consider in this section are sample-dependent. We use knowledge of to calculate their confidence widths, as detailed next.
Risk level-dependent bounds Sample-dependent concentration bounds require some care. Our algorithm calculates the confidence width c(n, 8') before the first deployment without access to the distributions of future deployments. This task of using sample-depender bounds without initial access to them is further complicated by performative shift, which impedes from more specific characterizations of these future distributions, and hence from obtaining sharper bounds. However, have knowledge of the risk control level and can that information minimize c(n, 8') while making it wide enough to control risk throughout, and at the end of, the iterative process.
Specifically, iteration and can calculate R. (3; X) for any X. Rr (S,-1, A) the sample mean loss when evaluating with X; from it, we calculate the maximum sample variance, which occurs when the loss distribution is Bernoulii with probability R, (5-1,, A). Since concentration inequalities use these two quantities to construct their bounds, the exact distribution we draw from does not matter, and the derived confidence widths will work for the distribution at any round, regardless of performativity.
receive samples from the previous round {2-11)) lid D(Ae-1)
Next, note that our algorithm is functionally equivalent we replace Line of Alg. with the following update rule:
13

---

Xmiid inf(X E A (,) + max(0, d A) <a).
Since max(0, A-- - X) 2 always, the candidates for Agll are a subset S. == {A E A let (de-1, A, 8¹) < ar. Assume for all À we have access to a pointwise confidence width c(n,, 8', A) that bounds A.) by knowledge of the sample statistics calculated from Ra X) above. Let R 1,,-1,,,, X, 8²) (Re-1: A. X, 8')
2 c(n, 8', X'). Define the update that does not consider performativity as Apid == inf(Sa] We set the confidence width as
c(n, 8') max(c(n, 8)) X Amiid.
This setting ensures that throughout the entire trajectory of Å for which we bound A. 1e)=1 T(A.11 Xe) S c(n2, 8'), )e) S a also. Next, we use this general result obtain confidence widths from Bernstein's 's inequality, the Hoeffding-Benne Benkus inequality, and the central limit theorem.
guarantee with high probability that R(A--1, Åt) + + Tdi-11
Bernstein's 's inequality
Proposition A.1 (Empirical Bernstein Bounds 23 Theorem 11D. Let x= (X1,..,, X,) be vector of independent random variables with values in [0, 11. Let ō > 0. Then with probability at least 1 we have
2V (X) 1n28
7/8 3(n 6(n-1) 1)
[[ +
where Pa (X) denotes the sample mean and Vn (X) denotes the sample variance.
We observe the sample mean (A- and can upper bound the sample variance with Rr, ()-11, A)(1 (St-,,,,))))) A)), which is obtained when the losses Bernoulli. We further convert the one-sided bound into a two-sided bound, giving us the following:
cn, 8, A) = 2R, (d--1. A)(1 (dv-1, A)) In(88')
7ln(4/8)
3(n-1) 1)
We then use the technique described above in App. A.I to obtain the confidence width.
The Hoeffding- Benttus inequality
Proposition A.2 (Hoeffding-Benill Benttus inequality p-values 4 Proposition 1]). The following is valid p-value for H R(A)= > 3 for n samples
HB (n, (A). min (exp ( (nhh (Rn(A) eP (Bin(n, 9) s In [n)n(())))) (2) p
where R(X) is bounded risk, (x) is the sample risk, hr (1 a alog(a/b) + (1 a) log((1 a)/(1- and A denotes the minimum.
R() X) falls in the interval
(ft, (Sc-1, X, 8'), Pet
(
the following two hypotheses are rejected:
UBB R(AA-1,,)) > gUB = fet (Ar- 1,,11, with p-value p BB Re, (de BUB),
HLB - - > 3LB = 1- (1-1, with value p (n, JLBB(n,
(n -
We simply choose the narrowest confidence width that provides us with error control:
+p c(n,8' 8',))== = arg minp UB (n, Rr Rh, (A. BUB + p (n, 1 Pr, (1 - 7 - 8'.
c20
14

---

Central limit theorem
Given Rx (A A), the maximum sample variance is
(n,
(( 2) - Rr 1)))
Denote the cumulative distribution function (CDF) of the standard normal as t. With probability 1 8', [ - (1 A) I c(n,8 8', X) where c(n, , A) = + 1-1(1 -2))
A.2 Asymptotic results for the lower bound
Recall that Aa = 2T + 2c(n, 8/T) where / [(Asate Ammm)/A]; First, we need to ensure that there exist solutions (7, ΔX) in this asymptotic regime. For a given AA, we can choose the minimum π subject to its constraint and let n - oc. Doing so allows c(n, 8/T) + O. Since AA can be chosen arbitrarily small, can make Aa go zero.
Suppose we let A - and scale the number samples n as AA = On (for some constant C and sufficiently high r 2 1/2). We remark that the confidence width for the bounds consider can be
expressed asymptotically as c(n, 8/T) = O(V [o(T//6) = O(V /m(1/AX)) = O (l(()) Hence, asymptotically
= 0 lnn
vTT
Aa 27AA + 2c(n,
In n
= =( + +
n/
Inn = =0(-)
Vn
for 2 1/2. PRC allows users to choose a vanishingly tight lower bound Aa - 0.
A.3 Main proofs
In this section, we build towards final goal of proving Thm. 3.3 the performative error, arising from risk measures with the same evaluation threshold but whose samples are drawn from different distributions, can be bounded. We then demonstrate the validity of UCB calibration to bound the generalization error. Taken together, we can then prove Thm. 3.3
We do by first showing that
The following lemma the definition of quantile- based risk measure from Def. 4.1
Notably,
(p)) is a weighting function; (p)) = 1 when we consider expected risk. is important to remark
that when (p) 1. [o dp) 1/u
for E [o)
Lemma A.. Let u, v (o) with 1/u 1/u 1. If [ (o))) 1/u < M and D(.) is (7, ((, seeeensiiiee for all X A, then for any X2, A2, A E A:
Rs()1, A) - (A2, A)I S 7MIii A21-
15

---

Proof.
[R, (22, A)1 S (D))
/// ((() (.: (2; A)dp
((::11:)) (pp)
/ (()) [F (:: A1, (.; 22, 2)] dp
! (()) | (.; - (-; 12 :222 dp
1/u (Lo (F-IF'(6:
1/0
{1,)) (F;
F (p= )2, )2, 2)1 dp)
M (L. (1: Fr; A2; 12,1111 dp)
1/0
(D..(((A)) S M - X2l
As a note on notation, from now on until the rest this proof section, we write the risk measure as Rt understanding that this refers to the more general form Rg (where the subscript v has been dropped).
In the following proposition, we restate Theorem A.1 in Bates et [3] in our performative setting. This theorem is key to turning the pointwise bound used in line 5 of Algorithm Tinto proof of the validity of each iterate 2.
Proposition A.4 (Validity of UCB Calibration). Assume we have deployed some X E A to obtain samples from the distribution D()). Let the true risk function R(A, :) -> and its 8' upper confidence bound (UCB) R (A,:; 8;)) A + R be continuous and non-nnonennn con-increasing functions. Suppose there exists some X E A such that both
R(A, X) S a and (2, Sa
Consider the smallest x whose UCB falls below the risk level:
x =nf E 1 <a
Then,
((R() 21
Proof. First, we claim that (A, 8') = (A,-) + c(n, 8') is non-increasing and continuous. By the property of the loss function, Rr, (A, is non-increasing and continuous. Further, c(n, 8') is constant by construction, proving the claim.
Assume R(X, xt) > a. Consider the smallest d, that controls the risk
X = inf{X' E A R(A, X) S a}
Because -(),-) is non-increasing and continuous, X A., Further, since x satisfies R (A, and Åt (2) is also non- con-increasing and continuous, Å (X, Xx, 8') a. However, since R(A, A.) = (by continuity), the pointwise confidence bounds in Equation ensures that this occurs with probability at most 8'.
Now that we have established how we can bound the performative and generalization error, we now prove the following important theorem, which can then be used to prove each point of Thm. 3.3
Theorem A.5 (Safety of Each Deployment). Take the setting of Theorem 3.3 With probability 1 - 8.
RiA, Ae) I a is ensured for all in 0 < T (here, denote Å 2o).
16

---

Proof. We establish this theorem by verifying the following inductive statement: on iteration t, R(X,) I a for all s in 0 I S t with probability 1 - öt/TT.
To establish the base case, recall the definition of Asaee from which we see ((Z, Ao) E(Z, Asafs)
0 < a, trivially satisfying R(Ao)= < a with probability 1. Afterwards, for 2 1, there are tWO cases:
Case 1: de = År-1. From induction, R().) < a for all in 0 S S - 1 with probability
1 8(t - 1)/T. Since Å = d R().) < for all in 0 S S t with probability - öt/T <
1 1 6(t 1)/T.
Case 2: Å < d This implies x, inf A V()--1, S at. The following chain of inequalities holds with probability 1 5/T::
R(a,) R() 22) + R(dr.1)) R()
S R( 2o) + 7M l7x- - -dc-1l
I R (dc-1, 8/T) + 7M 1ate 22-11
= T Thx=1 - Xx) + 7M 1ic d
(T- 1^x- -
where the second line follows from Lemma A.3 the third line from Proposition A.4, and the fourth line follows from the continuity of the loss function ((z, A) in for all 2.
Since by induction R(}x) < for all in 0 S < / with probability 1 - 1-8(1- -(1-8))))) 1)(- taking
the union bound verifies that risk is control for all in S t with probability 1 8t/T. Since T < 7, this completes the proof.
Having established Thm. Thm we can now prove each part of Thm. 3.3 which we do the following
[o' [p)p)
1/u
three proofs. Note that we denote M
as in Lemma A.3. and M 1 for the
case of expected risk.
Proof of part (i). We verify the following inductive statement: with probability at least öt/T, R(},-1, ),) S for all in <s S t. Once verified, since PRC takes most T iterations to finish, the theorem follows.
Base case: 1=
If Å Ao then with probability 1, R(Aa, {n) 0 a by the definition of Asate- Otherwise, Å = inf (A e A (Vo, S a) and with probability 8/7.
R(Do;S1) - Rtt (No, (1,571,5/7)
= a TM (Ao - 21))
The first inequality follows from Proposition
A.4 and the last follows because if X, =
inf {XEA) (((--1,,,0) s a) then Aι
then
Ao-
Inductive step: t> 1
Assume R( R(A 1, )x) S a for all s in < with probability - 8(t 1)/T. On iteration f,
either 2 = dx-1 or 2₂ < S f = A we appeal to Thm. A.5 to ensure that R(A,- 1, A₂) S a with probability - 5t St/T. The error rate of 5(t - T) from the inductive step does not accumulate with that of Theorem A.S because both rely on the same pointwise UCB R (At-1.)) per iteration in the same trajectory. Hence, their maximum becomes the new error rate öt ét/T.
17

---

Otherwise, d < Ae-1. In this case, we follow similar steps to the base case. With probability
1 - 666,
R()e-1;)1) dx) _ 1et 11,-1;22,8/77 TM (de-1 )c)
S a
Taking the union bound with the inductive assumption, we complete the proof.
Proof of part (ii). Appealing to Thm. A.5 completes this part of the proof.
Proof of part (iii). Algorithm I returns \r as the final risk controllng conservativeness parameter. From Equation with probability 1 - 8' = 118- 8/T,
Rn (rT-,,Ar) c(n, 8') = R (\) I Rm (AT-1, + c(n 8').
There are two cases to consider at the algorithm's end.
Case 1: bp A This implies dr-1 S inf E sa) = Aupid
R(AT) R(AT-1)
2 RT (DT 11
R (AT-1 -1
1. 8 8/T) 2c(n, 8/T)
2 Rt (ZT Aqiid 8/T) 20(n. 8/T)
a - T(AT-1 - )qidd) 20(n 8/T)
2 a - 2c(n, 5/T)
where the first inequality is with probability - 5/T and the second inequality follows from the continuity and monotonicity of (X, for all A.
Case 2: ÀT- AA d < Â--1- Recall that by Lemma A.3 bound the performative error:
R R(Ar,2x) (1 s MM/rr d--1l.
With probability 1- = 1 88/T
R(³r) R(AT- 2x) + R(AT, )r) B R(
2 R(ÂT-1, Ax) 7M A - -Arl
(,-1, 27, 8¹) 7M M åTl
(2r 2r, 8¹) - 7 årl 20(22 8¹)
T(Ax )x) - dvl 20(n2 8')
2 a (Ā - 2z) 2c(n188)
2 a 2T 20(n,
Aa
18

---

where we set ÁA such that Aa = 2TAA + 2c(n, 8'). Doing so allows us to achieve Aa tightness with probability
A.4 Quantile extension proofs
The risk measure's lower bound corresponds to the CDF upper bound. Assume we have access to lower and upper CDF confidence bounds such that for all À E (det and X A,
(FL.n X, 8)) 1 F(w:: X, & Fu,n(w (w: X.X 2 11-
Vw
We rewrite the above as PP [FL, 3 F Fu -1-88 8', dropping the and X for the rest of this section. The following property establishes that FL,n I F F Fu,n implies Füj 3 F = FI..- Proposition A.6. If F 3 G, then F-1 = G-1
Proof. Recall the generalized inverse of CDF H is
(p) = inf(# R H(x) p). pe (0,1).
Since F(x) S G(z) for all x, we have fa : G(x) 2 w} S (x: F(x) u), so
(u) = inf(x F(x) 2 of 2 inf(= G( 2 u) (w).
Hence F-1 --!
Applying weighting function does not change this relationship. Hence,
( (R X,,) & (o s ( 2
desired.
Proof of Theorem 4.2
This follows consequence of the proofs in App. A.3
B
Experiments
In this section, further detail experimental setup and include other supporting results. Our credit scoring simulations were conducted on a CPU and can be finished in a few hours on a standard laptop computer.
B.1 Type II Error Control for Credit Scoring
Set up and 7-sensitivity Our data is sourced from the training split of Kaggle credit scoring dataset
24 which contains roughly 150k data points, 10k of which ending up serious delinquency
1). We first sample 1.5k from each class to train a logistic classifier to function as our model (ar). Because the rest of this dataset is so highly imbalanced, the expected loss would be close 0. For illustration purposes, we sample 8.5k instances from the negative class so that the proportion of positive and negative classes are balanced. Then, for each trajectory. we sample points without replacement to serve as the calibration set, with the remaining serving as the validation set on which the risk is evaluated on. To generate multiple trajectories, we sample different calibration and validation sets from the balanced subset.
y
We now prove that simulated shift is (7, 1, ((,, -Se-siiie for all X. Recall that our shift is defined as
f(x(2)) maxx(0, f(x) s) if f(:) fx) otherwise.
(f(x)
19

---

2.5
density
2.0
1.5
0.5
0.0
0.2
0.4
0.6
0.8
1.0
f(x)
Figure 5: Histogram of predictions far) on the base distribution.
where s(= 0.3) is an arbitrary parameter we define.
Consider Å and A so that 1 - A < A2 - + Then, f(x((1) and fa(x(A2) are equivalent except in the narrow region [ + + + (here, we assume we start with the same and that the region inside [0, 1]). Let be the probability of positive case, and let C' be the maximum value of the PDF of predictions f f(x) on the base distribution given = 1. It is possible that the threshold À causes discrepancy of 1 in the loss for these data points in the population by flipping the prediction, e.g., À = A1. Hence, W1 (De(
D((.A)) S pCIA1 Aal for all
Now consider when λι + < Ag. If we use a threshold À so that )1 + + X < X2 the original loss distribution and that induced by are the same. However, for the distribution induced by A2 the original predictions falling in the region 1 - X, À + s] flip from 0 to 1. Similar to above, this implies that Wi (De(.A) De(..)) S pCs for all A.
Combining the two cases above, we have Wi (De( (A1), D((,A)) S pC min(s, [1 -22)) S PC|A A2 for all X, achieving 7-sensitivity with pC. We experimentally verify the values of C' and p to be 3.20 and 0.432, respectively. Importantly, we calculate C using the balanced subset rather than the training set. We use 20 bins for the histogram (see Fig. 5). Hence, =pC < 1.38.
Further experimental discussion Here, we include supporting plots Fig, and further discuss our experimental results. Fig. 6 the proportion of runs with a Aa s R())) S a, and Fig. displays similar plots to the bottom left plot of Fig. for different values of r.
Fig. ploot the failure probability for different values of т, defined as the proportion of trajectories 2 close to 7 < 1.38,
where the final AS does not satisfy a Aa S R(Ar) < a. For r 1 and the failure probability is essentially zero (much less than 6), which some may argue constitutes loose coverage. While is true that PRC's tightness guarantees are looser than those prevalent in the risk control literature, the unique setting of applying risk control under performative shift warrants their size, which is driven by three main factors. One, our algorithm needs to protect every iteration for unknown functions R()). The risk could be adversarial and necessitate many iterations, all of which PRC needs to protect. Two, our calculation of is uniform bound across the iterations, necessitating a larger safety parameter 7; for example, it is possible that the "true" is between = 0.2 and т 0.5, at which point the empirical failure rate crosses With the other parameters such as the number of samples held constant, a larger corresponds "insider" access to the statistics of the first or future distributions. We plan for the worst-case in terms of sample mean and sample variance in computing the confidence widths. These factors contribute to the size of the bounds.
larger Aa. Finally, we do not assume
B.2 Quantile Risk Control in Credit Scoring
Set up, 7-sensitivity, and the confidence width For this experiment, we adopt the same setting as in expected risk and use the same shift technique. However, after training the logistic classifier,
20

---

Relative Frequency
Falare ty
Drurde
0.8
0.6
0.2
0.1
0.2
0.5
0.8
2
Figure 6: The relative frequency of runs that end with by with a - -a S <R(AT) S a for risk control in credit scoring.
0.4
0.25
0.3
0.20
Risk
Risk
0.2
0.15
0.1
0.05
0.0
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Iteration
Iteration
0.30
0.30
0.25
0.25
0.20
0.20
Risk
Risk
0.15
0:15
0.10
0.10
0.05
0.5
0.05
Upser bound
Upper bound Boand
Bound
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5 20.0
Iteration
Iteration
Figure 7: Analog of the bottom left plot in Fig. for 00,0.2,0.5,2) 2}.
avoid any subsampling, using the rest of the dataset. Hence, while C' remains the same, drops from 0.432 to 0.064, reducing 7 to 7 = pC < 0.205.
To calculate the confidence width, we follow a recipe based on a version of CLT that is applicable to quantile-bazt based risk measures. The recipe calls for knowledge of the loss CDF. Recall that our loss is given by {y, Tx())) = (0,0, 1](1 - (x)=(x));; we make a minor approximation to this loss function by taking 0, that the loss is binary. Denote by p' the proportion of the population that has 1. Because at most of the population has = 1, we know that p' S p. which holds for any given distribution-induee induciig À and threshold X'. Knowing p' allows us derive the CDF and its inverse, which are piecewise linear because of the uniform distribution.
v(1 - Tx)((=)
when w when w S 1 when w > 1
F(w)
p' + p'w
0
when 0 <<<<s-1p when 1 1
(-1(r)=
1-r
P
21

---

By Theorem 22.3 in Vaart (26, we have
(F() - (F)(F) to N(0,
where
( ( (
dr dr.
Since the 3-CVaR metric bounded everywhere and has one discontinuous point, we can apply this theorem (appealing to dominated convergence). We split the calculation of (1b,,, F)) into two cases.
Case 1: 3 pl
0² F
A(r (Frd)
(1 B)
JO
1
F(r - (Fr)
(1 B3²
Using F(r)= - p/ + p/r gives us
1
F) (1 - B)2 12
3p
Case II: 3> > 1 p-p We require (11)=1--/+p' function to be nonzero. Similarly to above, the upper bound can be changed from co to 1 because F(r) = 1 for r 2 1. The lower bound is then changed to a = 1-8
B, orr 2 (3-1+ p\/p for the weighting
d²
F(r F(r)
(1 B)2
S.
F(r) (F())
(1 3)-
F(r) -F())drdi
(1 333
(1- +NDGGN -p'r + 1 )
(1 B)²
8
33 + 129 12
We examine 8 = 0.9 (90%-CVaR) and when the range of p' is [0, p). where 3 < 1-12 p. Taking the maximum (v), F),, which occurs precisely when p' p, we obtain the variance, which allows to construct the following confidence width that applies for any choice of shift-inducing A and threshold X/:
(4 3p)p
c(n,8'== v- 52/2)
12n
Further experimental results Here, we include supporting plots to Fig. Fig. shows the proportion of runs with a - Aa < R(Ar) < a,, Fig. shows sample trajectories for different values of analogous to the top two plots of Fig. and Fig. 10 displays the risk trajectory for different selections T.
B.3 Risk Control in Demand Forecasting and Generation
In this synthetic experiment, we model power plant that determines electricity generation at each timestep based on demand forecasts and risk-control policy. The objective to minimize overproduction while limiting the amount of unmer demand. Performativity arises through demand elasticity: surplus electricity can be stored batteries and later used to offset shortages, linking one timestep's is risk control decisions with future demand.
22

---

Relative Frequency
Falume Prsbablitt
Vitturdd
0.8
Oct ofBunds
0.4
0.2
0.2
0.4
1.6
4
Figure 8: The relative frequency of runs that end with a dp with Da S R(xr) S a for quantile risk control in credit scoring.
10
10a4
0.8
10
10-1
10-4
0.2
10:
20 Iteration
10
12
Iteration
Figure 9: Analog of the bottom left plot in Fig. for r control.
E {0,0.2 0.4, 1, 1.6, 2, 4} in quantile risk
0.30
0.25
0.25
Risk
Risk
0.15
0.10
0.05
bound bount
Waser boad
0.00
0.00
Lover Bourd
0.75
1.25
1.50
1.75
2.00
Iteration
Iteration
0.20
0.20
Risk
0.15
Risk
0.15
0.10
0.10
0.05
0.05
Wasel
Upor
0.00
0.00
4
10
20
30
40
Iteration
Iteration
Figure 10: Analog of the bottom left plot in Fig. 2 for the 90%-CVaR risk measure and (0, 1,4).
23

---

Data-ganergting process. We simulate electricity demand at 10-minute resolution from 2005-202 using a smooth, periodic model with additive Gaussian noise. Letting denote the timestamp, we define the normalized daily phase as
8 daystart(s)
(a(s)
24h
where (systart(s) is 00:00 of the same calendar day. We simulate the demand process as follows:
+ A sin 2 (T((()))))))))))))))))())) 0.5)) + Est Eg ~((00
with μ 30,000, 15,000, a 1, 1.000; the phase shift aligns peak demand PM. We then train a random forest regressor fe on data up to the end of 2007, using sine-cosine encodings of the hour day and monthly phase as features Hs with the target as 1/a.
Setting At timestep 8, the regressor forecasts demand J. fo The power plant then generates (A( (1 xX)ğ, amount of electricity, where determined from a validation set (explained later) and E [0, 1]. We make the assumption that performativity can be modeled as follows:
R Us (A) Un Σ B" (U)-(AA)) -a-())-)-11 Us-r),
where ys is the demand of timestamp 8 in the absence of performative effects, R specifies the number of past timesteps influencing current demand, and controls the degree which excess insufficient generation in the past affects current demand. For the sake of illustration, we simplify the model by setting R- The power plant's goal is to control the amount of ummet demand while accounting for this performativity: the continuous loss function is given by f(x,, Moi A) = (y. (A) - (1 + KA)9.)+
Next, we use validation data from 2008 to determine appropriate settings for c(n, 8'), and Note that in these calculations, we assume knowledge of the distribution mapping D(A). While this knowledge can be exploited to directly calculate a threshold that achieves our desiderata from the first batch samples, we deliberately use PRC for the sake of illustration.
Evaluating K, c(n, 8'), and 7 from the Validation Set Next, we use validation data from 2008 to determine appropriate settings for k, c(n, 8'), and Note that in these calculations, we assume knowledge of the distribution mapping D(A). While this knowledge can be exploited to directly calculate threshold Á that achieves our desiderata from the first batch of samples, we deliberately use PRC for the sake of illustration.
First, Ai is chosen large enough so that risk is controlled at Asato = 1, i.e., (1+ n²) ü. yo + (U)-1( 30-1) 2
(N(aatte) yo (Aeate) 2 O, We select A such that at least 95% of the validation data satisfy this inequality. Next, variance for the CLT confidence width c(n, 8') is defined via
Var ((r., Vo; A)) Var Vur((u (1 kA)&.)
and by searching over a grid of possible À values, we obtain the maximum sample variance to use in the bound at confidence level 8'. Finally, the Lipschitz bound is derived from the loss
f(E,, Mo; A) = [v. (A) - (1 +1+ =A)0.]
which satisfies the local Lipschitz property
[((r,, Us: A1) - (xE, Usi A2) S [N. + (Kgs-1]|A - dzl- We therefore set J as the 95th percentile of kÿe + 3r0e-1 across the validation set.
PRC Results Finally, we run PRC on the rest of the data, calibrating on one month (more precisely, 28 days) and testing the next. We run one trajectory starting each quarter to simulate randomness. We use a = 300, Aa = 85, = 0.1, N 4032 (the number 10 minute intervals 28 days), and | 0.2. On the validation data, 0.059, the variance for the CLT bound is 373155, and
т 3204.43.
Fig. I demonstrates our results. While the non-performative algorithm (7 0) exceeds the risk threshold on the first iteration, the performative algorithm (r = -320444) carefully reduces X, as to maintain risk control.
24

---

10
3204.43
0.8
10-
0.4
10
0.2
10-1
0.0
2
10
12
14
16
18
20
2
10
12
14
16
18
20
lteration
Iteration
Upper
Lower Bound
200
300
Risk
Risk
150
100
== 3204.43
100
Upper
Lower Bound a Aa
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5 20.0
0.0
3204.43
Iteration
Figure 11: Unmet demand error control. We the same plotting scheme in Fig.
25

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We illustrate the setting of performative risk control in Secs. and provide a framework targeting this setting in Sec. Jalong with an extension in Sec. and demonstrate the framework's ss utility by experiments on predicting credit default risk in Sec. 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed the limitation of our framework in the conclusion, in which inherit the issue that risk control calibrates models with simple postprocessing and may not always be as effective finetuning the whole model. This simple postprocessing can lead to the procedure to only output Asafe-
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide the full set of assumptions the main context (Sec. and complete proofs in App.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We specify the elements needed to reproduce the experimental results in Sec. and App.
Guidelines:
The answer NA means that the paper does not include experiments. the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
27

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will provide open access to the experimental code supplementary
material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
soi.amimsssiooiiciioooooy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. wwtpccccpppcccpuc.ccoce htte://wiissseeeeeeeeee www.ooouwissioonnoo.oooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify the calibration and evaluation procedures and other calculations in Sec. and App.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Fig. displays all 1, 000 trajectories and clearly states that variation is due different
splits (our setting is not amenable to error bars, so we display information on statistical significance in this way). Further plots on experimental statistical significance are included in App.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We state that our experiments can be done on laptop computer in App.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have reviewed the Code of Ethics and find that this work follows the ethical
guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]

---

This paper proposes framework to safely and reliably deploy models whose predictions influence society, with potential positive societal impacts. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release any data models that have high degree for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit and cite the Kaggle credit scoring dataset, with a link the references. It is available publicly with public domain license.
Guidelines:
30

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This research theoretical in nature and does release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
31

---

This research does not involve crowdsourcing nor research with human subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not involve LLMs as any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
32


Reviews:
[{"review_id": "57DbNrUPjx", "reviewer": "Reviewer_gYio", "summary": "The paper studies risk control in the performative setting, e.g. where individuals/organizations/datapoints will strategically attempt to manipulate their features so as to influence risk control variables such as the size of the output set that the model assigns to them. Specifically, the authors adapt a standard iterative approach to performative prediction for risk control, which they refer to as Performative Risk Control. The algorithm follows a common iterative approach to performative prediction: in this case, starting with a conservative threshold, collecting new data from the distribution induced by the previous threshold, building an upper confidence bound on the new threshold (accounting for potential distribution shift, and assuming smoothness as is standard), and repeating. The proof of the main theorem arguing the convergence of the method (which is entirely deferred to the Appendix without even a sketch) implements the usual performative prediction proof approach by adapting UCB to select updated thresholds; the convergence rates and assumptions match exactly what one would expect from standard performative prediction results.", "questions": "Questions:\n- Addressing the weaknesses would be helpful in making me feel comfortable with a higher score.\n- Normally, performative prediction is motivated by individuals/organizations that want to bias their own prediction in a certain direction. However, risk control is normally applied to higher order statistics, most commonly uncertainty set size. The degree to which individuals/organizations are incentivized to manipulate features to influence such stats is less clear; I don't doubt that such a motivation exists, but what are some concrete examples?\n- What does poly mean on L25?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The authors argue for an interesting problem and while the paper could better motivate performative risk control, it's not hard to see why this may be of interest (e.g. why a suspect may hope that a judge forms a wider uncertainty set for guilt).\n- The actual theoretical result (which is the main contribution) largely follows a standard recipe for adapting a learning problem to the performative setting but its nice to see the details worked through; the form of the algorithm, assumption, and rates appear reasonable.\n\nWeaknesses:\n- The current discussion for motivating performative risk control leaves something to be desired; see questions. This is important because, while the theorem proof isn't very involved and mainly just follows a standard recipe, the paper could still have a meaningful contribution in arguing for the study of performative prediction + conformal provided the problem is better motivated.\n- Writing can be generally improved, especially in technical parts. For example, the statement of Theorem 3.3 is a hard to read: delta alpha isn't defined, you need to scan up to L163 to read it instead; same thing with Algorithm 1 (esp. Line 2). I don't believe that the lack of clarity is unavoidable (the algorithm is not too complicated, though the authors may find it cleaner to reparameterize $\\Delta \\alpha$ and $c(\\cdot)$ ?), just that the writing just needs more polishing.\n\nComments:\n- It may be worth editing Figure 2 so that tau = 0.001 is labeled as the non-performative algorithm (or more precisely to include one where you exactly ablate the performative aspect with tau = 0). Given the way the experiments are setup (distribution shift is synthetically added exactly so that these types of performative prediction algorithms work well), the result of the experiment seems somewhat expected, but it's still nice to have experimental validation that the paper's implementation for risk control does work.\n- While the existing related work discussion covers performative prediction fairly thoroughly, the authors may consider also including discussion on the connections to Stackelberg literature (though this is a point of broader friction between the two communities rather than a shortcoming of this paper).", "was_revised": true, "final_justification": "I maintain a bias towards acceptance.", "rebuttal": "Thank you for the thoughtful and detailed feedback. We address your questions below and hope that our answers could ease your concerns.\n\n> **W1/Q2**: Motivation and concrete examples\n\nPerformative risk control (PRC) is motivated by the phenomenon of **performativity** in decision-making. Since the goal of (conformal) risk control is to calibrate any black-box model by choosing low-dim calibration parameters like thresholds to ensure reliable decisions, the phenomenon of performativity will naturally happen---the target distribution will change according to the chosen parameters. This is particularly prevalent in applying risk control in many socio-technical systems. This finding is the motivation behind our paper. We provide credit scoring as an example in L27-32, where a bank's loan policies set by $\\lambda$ change the purchasing patterns of the population, and hence the features used in loan prediction itself.\n\nTo be more comprehensive, we here provide additional concrete motivating examples.\n\n**Resume screening example.** A company may set $\\lambda$ to determine the percentage of applicants to screen while controlling the risk of screening out solid applicants. Formally, we can interpret the calibrated prediction $\\mathcal{T}\\_\\lambda(x)$ in one of two ways. One, $\\mathcal{T}\\_\\lambda(x)$ is an uncertainty set that can take one of the following values: \n**{accept}**, **{reject}**, and **{accept, reject}**. Since the purpose of resume screening isn't to automatically accept applicants, **{accept}** doesn't make sense as an option. Hence, $\\mathcal{T}\\_\\lambda(x)$ is either **{reject}** or **{accept, reject}**: either the applicant is automatically rejected or the screener is uncertain which class the applicant belongs to, and the applicant passes onto the next round. Seen in this way, we can also view $\\mathcal{T}\\_\\lambda(x)$ as a point prediction, in which the applicant is either automatically rejected or passes onto the next round. In both interpretations, the decision made is the same, showing how the application of risk control to higher-order statistics affects real decisions and incentivizes individuals to manipulate their features to their advantage. Concretely, the performativity aspect in this example comes from screening rigor impacting how applicants apply, e.g., to what extent an applicant \"AI-proofs\" their resume or optimizes keywords.\n\n**Content modification example.** A social media platform may set $\\lambda$ to determine how much to censor. This, in turn, impacts what users post. For example, in response to more rigorous censoring, they may try to do more \"clean up\" on their post to ensure it is not censored. Hence, deployment of $\\lambda$ impacts the distribution of posts. At your suggestion, we can include these examples in our exposition.\n\nIn summary, we find performative risk control to be highly relevant to socio-technical systems, and our work provides an algorithm and theoretical framework for safely deploying models within these dynamic social environments.\n\n> **W1**: Technical contributions\n\nWe appreciate the reviewer pointing out that one important contribution of this work is **initiating** the study of performativity in the setting of risk control. Meanwhile, we want to point out that, on the technical side, our theorem proof is standard to keep our analysis as simple and realistic as possible. \n\nHowever, our setting has particular technical challenges. Specifically, we formally stated new desiderata and the significance of our framework in Sec. 2.2, which required us to carefully design an algorithm to satisfy them. Meanwhile, in extending risk control to the performative case, we found the need to reframe the definition of sensitivity (Def. 3.1) and deal with a confidence bound that grows with the number of iterations. In [7] (*Performative Prediction*), sensitivity was framed on the feature-labels $z$, while we framed sensitivity on the loss to alleviate a Lipschitz condition on the loss function. Further, our extension of risk control to an iterative setting---without presuming knowledge of $T$, the number of iterations---required resolving several technical challenges, which included the definition of $\\tilde T$ and various bound calculations in Appendix A.2.\n\n> **W2**: Writing clarity\n\n**On Thm 3.3**: We agree and we will plan to rearrange the statement of this theorem in a more readable format.\n\n**On $\\Delta\\alpha$**: We point out that we had $\\Delta\\alpha$ defined as a user-controlled tightness parameter in the technical setup sections (Sec 2.2 L97-99, Sec 3 Intro L107, L114). Let us know if you think a reader might prefer a more explicitly defined $\\Delta\\alpha$ in the theorem itself.\n\n**On Alg 1 L2**: Yes, we agree that Line 2 is quite complex. In a future revision, we will try to make it less verbose, perhaps splitting it into two steps. However, we do believe that the joint-solve of $(\\tilde T, \\Delta\\lambda)$ is crucial to our algorithm, and this complexity stems from a design choice in which we sacrificed simplicity for user utility.\n\nSpecifically, our goal is to provide a high probability guarantee for the entire trajectory at the beginning *before any samples are collected.* To do so, we require knowledge of a confidence width on the first iteration (and all others), which in turn requires knowledge of the number of iterations $T$ (unknown). This is because to prove high probability guarantees for the entire trajectory, we use high probability guarantees on each individual step and *somehow have to compound those error rates.* Since the error rate increases with $T$, we faced a design choice. Either we guarantee an error rate that increases with $T$, or we increase some complexity in Alg 1 (e.g. L2). We saw the first option as undesirable to a user; it would be better to know the maximum error rate $\\delta$ of the entire procedure before starting, so we took the second option. Indeed, doing so also loosens the bounds; if we had knowledge of $T$, we could use the confidence width $c(n,\\delta/T)$ rather than $c(n,\\delta/\\tilde T)$. However, we saw the trade-off as worthwhile, since for most bounds we considered, the difference would have been a small factor similar to $\\sqrt{ \\frac{\\ln(2\\tilde T/\\delta)}{\\ln(2T/\\delta)} }$ (computed from Hoeffding's). Hence, the complexity of step 2 stems from a design choice aimed at maximizing PRC's utility to the user: we believe the algorithm should inform the user whether theoretical guarantees are possible *before* data collection and believe this benefit warrants the complexity of the joint-solve in Alg. 1 L2.\n\n**On reparameterizing $\\Delta\\alpha$ and $c(\\cdot)$**: $\\Delta\\alpha$ is user-specified. We imagine a user typing in what constant they want for $\\Delta\\alpha$, similar to how $\\alpha$ is user-defined. $c(\\cdot)$ is just a generalized shorthand referring to the confidence width, which depends on the number of samples $n$, the error rate $\\delta'=\\delta / \\tilde T$, and the type of concentration bound used.\n\n> **C1**: editing Figure 2 so that tau = 0.001 is labeled as the non-performative algorithm\n\nAgree with both points. In the revision, we will make clear that tau = 0.001 corresponds to the algorithm that does not account for performativity.\n\n> **C2**: connections to Stackelberg literature\n\nThanks for the suggestion. This work, especially the credit scoring experiment (which follows [7] *Performative Prediction*), has strong ties with the strategic classification and more broadly the Stackelberg literature. We can illustrate this connection more in our related works section.\n\n> **Q3**: \"poly\" on L25\n\nSorry about the typo. We meant \"polyp segmentation\" as described in our reference [5]. We used this example to illustrate and motivate (conformal) risk control. Thank you for pointing this typo out."}, {"review_id": "cGAiZbhLW4", "reviewer": "Reviewer_hboG", "summary": "The paper introduces a procedure for risk control in performative settings. The basic idea is to combine the model and assumptions from performative prediction with conformal risk control to obtain a method that controls risk while allowing for performative distribution shifts. The procedure is shown to be valid and it is evaluated in a semi-synthetic credit scoring experiment.", "questions": "Suppose $D(\\lambda)$ comes from some known parametric model, or we have some other knowledge of the distribution shifts. How would your procedure change in that case? Can you give a more powerful version of the method?\n\nHow can you ensure $\\Delta\\alpha$ tightness for any value? I'm thinking of a setting where the risk function is discontinuous and has sharp jumps. In that case you shouldn't be able to get arbitrarily close to $\\alpha$?\n\nCan you comment more on the role of $\\tilde T$? What would happen if we didn't have the condition $\\Delta\\lambda \\geq (\\lambda_{\\mathrm{safe}} - \\lambda_{\\min}) / \\tilde T$?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Studying risk control under performativity is interesting and novel to the best of my knowledge. The paper gives well-motivated example setting, such as the lending problem. Given that this is a new problem setting, I wish there was more high-level discussion on what are the desiderata in this problem (e.g. we probably care about what the induced distribution looks like). In general, more conceptual discussions would help the exposition. The procedure (Alg. 1) makes sense, though it would be nice to give some instantiations where some steps simplify (e.g. step 2).\n\nLine 102: This should be with high probability.\nLine 118: You say: \"Notice that the calibration dataset ... in Def. 1.1 is the union of...\" This is not stated in Def. 1.1.", "was_revised": true, "final_justification": "Overall I am supportive of acceptance. I believe more high-level discussions would be help, and the method is not exactly simple, but this is a good problem and a good first stab at a solution.", "rebuttal": "Thank you for the thoughtful and positive feedback. We address your questions below and hope that our answers can ease your concerns.\n\n> **W1**: Desiderata in this problem\n\nThanks for pointing out the novelty of our setting. For desiderata,  we refer you to the high-level desiderata and significance of performative risk control (PRC) in Sec. 2.2. To be more concrete, in the example of credit scoring, we imagine an algorithmic model gradually being deployed to customers. This model is tasked with identifying low-risk opportunities and automatically accepting customers for credit loans. Initially, $\\lambda$ is set so that only the safest customers get automatically approved. We continue to monitor the model's risk as we continue lowering $\\lambda$, which expands automatic credit acceptance to more customers.\n\nOur desiderata are to \"achieve user-specified conservativeness\" (L97) and \"safe at any time\" (L101). As applied to credit scoring, the first statement means we utilize the model as much as we can, which is useful for both reducing manual work and expanding credit access. The second statement means that we hope to control the risk of unpaid loans, which is bad for both parties. Our work seeks to answer how we can maximally utilize the model in a performative setting while ensuring safety.\n\n**On \"we probably care about what the induced distribution looks like\"**: Ideally, we would like to make Alg. 1 applicable to all sorts of induced distributions. Sometimes, the induced distributions may actually lower risk, which would be the case if the threshold $\\lambda$ is polarizing, making the classification task easier. In most cases, the induced distributions may be antagonistic, which we explored in the credit scoring experiment where applicants attempt to strategically manipulate their features. This setting is reasonable for other applications besides credit scoring, e.g., automatic resume screening and content moderation. To keep our analysis tractable, we impose a smoothness constraint on the distribution shift as is common in the literature. We hope that this explanation adequately addresses your concern.\n\n> **W2**: \"more conceptual discussions would help the exposition\"\n\nThanks for the suggestion. In our future revision, we plan to illustrate our desiderata more with credit scoring as an example and highlight PRC's applicability in other settings like automatic resume screening and content moderation, as in our response to **W1**.\n\n> **W3**: \"The procedure (Alg 1) makes sense, though it would be nice to give some instantiations where some steps simplify (e.g. step 2)\"\n\nAlg. 1 was designed with two items in mind: simplicity and utility to the user. We do agree that step 2 is complex and will attempt to clarify why this level of complexity is needed. The answer ties into your questions **Q1** and **Q3**. The most relevant bits are in the next two paragraphs.\n\nAs mentioned in our response to **Q1**, having more information on $D(\\lambda)$, e.g. knowing its parameterization, dramatically simplifies the problem. In that case, we would not need Alg. 1 to be an iterative algorithm, can avoid the complexity of step 2, and can obtain $\\hat\\lambda$ on the first go.\n\nAs pointed out in **Q3**, the role of $\\tilde T$ is quite important. PRC requires knowledge of a confidence width on the first iteration (and all others), which in turn requires knowledge of the number of iterations $T$ (originally unknown). This is because to prove high probability guarantees for the entire trajectory, we use high probability guarantees on each individual step and *somehow have to compound those error rates.* Since the error rate increases with $T$, we faced a design choice (see **Q3** for more details). Either we guarantee an error rate that increases with $T$, or we increase some complexity in Alg 1. We saw the first option as undesirable to a user; it would be better to know the maximum error rate $\\delta$ of the entire procedure before starting, so we went with the second option. Indeed, doing so also loosens the bounds; if we had knowledge of $T$, we could use a confidence width $c(n,\\delta/T)$ rather than $c(n,\\delta/\\tilde T)$. However, we saw the trade-off as worthwhile, since for most bounds we considered, the difference would have been a small factor similar to $\\sqrt{ \\frac{\\ln(2\\tilde T/\\delta)}{\\ln(2T/\\delta)} }$ (computed from Hoeffding's). Hence, the complexity of step 2 stems from a design choice aimed at maximizing PRC's utility to the user: we believe the algorithm should inform the user whether theoretical guarantees are possible *before* data collection and believe this benefit warrants the complexity of the joint-solve in Alg. 1 L2.\n\n> **W4**: L102 should be with high probability\n\nYes, we will revise this line to be clearer. Thanks for the reminder.\n\n> **W5**: L118 \"Notice that the calibration dataset ... in Def. 1.1 is the union of...\"\n\nYes, $\\mathcal{I}\\_\\text{cal}=\\cup_t \\mathcal{I}\\_\\text{cal}^t$. We decided to leave out this detail in Def. 1.1 for exposition purposes. The goal of PRC is to find a desirable $\\hat\\lambda$, and Def. 1.1 is meant to illustrate our work's alignment with the broader risk control literature. For this reason, we chose to keep the exposition simple by just writing $\\mathcal{I}\\_\\text{cal}$.\n\n> **Q1**: Can we give a more powerful version of our method supposing that $D(\\lambda)$ comes from some known parametric model?\n\nYes, if $D(\\lambda)$ could be modeled completely, we would not need to make our algorithm iterative. We would then be able to calculate $\\hat R_n(\\lambda,\\lambda):=\\frac{1}{n}\\sum_{i=1}^n \\ell(z_i(\\lambda),\\lambda)$, the finite-sample approximation of $R(\\lambda)$, directly off of the initial samples $z_i(\\lambda_{\\text{safe}})$ for $i = 1,\\dots,n$ and calibrate directly based on the initial cohort. We point out that our distribution mapping in the credit scoring experiment is deterministic and amenable to this version of the method; however, we aimed to keep our analysis focusing on unknown distribution shifts. To keep the analysis tractable, we follow the literature in imposing a Lipschitz constraint on our distribution shifts. Here, the iterative method comes into play because we do not assume that we know $z\\_i(\\lambda)$ directly from $z\\_i(\\lambda\\_\\text{safe})$. \n\n> **Q2**: How can you ensure $\\Delta\\alpha$ tightness for any value? ... In that case you shouldn't be able to get arbitrarily close to $\\alpha$?\n\nThat's correct. This is where the assumption of loss **continuity** is critical. In Thm 3.3, we state that the loss function $\\ell(z,\\lambda)$ must be continuous in $\\lambda$ for all $z$. This assumption allows us to get our risk arbitrarily close to $\\alpha$. For the setting where the risk function is discontinuous and has sharp jumps, one can refer to Sec 2.2 of [6] (*Conformal Risk Control*).\n\n> **Q3**: The role of $\\tilde T$ and what happens if we didn't have the condition $\\Delta\\lambda \\geq (\\lambda_\\text{safe} - \\lambda_\\text{min})/\\tilde T$\n\nWe defined $\\tilde T$, an upper bound on the number of iterations $T$, so that we could calculate the confidence width $c(n,\\delta/\\tilde T)$ at the beginning. The advantage of this method is that the PRC algorithm can give the user feedback on whether their specified level of tightness $\\Delta\\alpha$ can be theoretically guaranteed *before any samples are collected,* as detailed in our response to **W3**.\n\nIf we didn't have the condition $\\Delta\\lambda \\geq (\\lambda_\\text{safe} - \\lambda_\\text{min})/\\tilde T$, or $\\tilde T \\geq (\\lambda_\\text{safe} - \\lambda_\\text{min})/\\Delta\\lambda$, we may not be able to achieve user-specified conservativeness, especially if the optimal $\\hat\\lambda$ is close to $\\lambda_\\text{min}$ and each iteration decreases $\\lambda$ by barely more than $\\Delta\\lambda$. In this case, $\\lambda$ may not \"make it in time\" to $\\hat\\lambda$. It is still possible that PRC terminates on a good $\\hat\\lambda$, but with no theoretical guarantees."}, {"review_id": "a59h8CeMcf", "reviewer": "Reviewer_5vJ9", "summary": "This paper introduces performative risk control, a framework that post processes model predictions to achieve provable risk control under performativity. The proposed procedure is an iterative calibration process, and is shown to be safe at anytime and tight when the performative distribution $D(\\lambda)$ is Lipschitz continuous wrt to $\\lambda$ and a valid confidence width $c(n,\\delta)$ (which can be obtained via e.g., Bernstein's inequality or CLT) is provided. Experiments on the Kaggle credit scoring dataset demonstrate the effectiveness of the proposed framework in predicting credit default risk.", "questions": "1. In the experiments on the Kaggle credit scoring dataset, the authors assume the performativity---dependent on $\\lambda$ ---affects the data distribution through impacting the prediction function $f(x(\\lambda))$ in a specific way given at the bottom of page 8. I wonder if the theoretical guarantees hold only when the Kaggle dataset satisfies this assumption, and what would happen if the assumption is violated.", "limitations": "1. The PRC method requires access to i.i.d. samples from $D(\\widehat{\\lambda}_t)$ for each $t \\in [T]$. In practice, collecting such data can be expensive and time-consuming, as it involves deploying $T$ models sequentially and gathering $n$ data points from each model before proceeding to the next. The paper would benefit from a discussion of this limitation and potential strategies for mitigation.\n\n2. As mentioned before, it would be beneficial for the authors to clarify the technical novelty of this work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-written. The assumptions and theorems are clearly stated and the consequences and limitation are discussed. This is the first work that proposes a provable risk control method when the data exhibit performativity. \n\nOn the other hand, the proposed method seems to be a straightforward generalization of that in \\[1] for i.i.d. data to the setting where performativity exists, since for each $\\widehat\\lambda_t$, the authors assume access to $n$ i.i.d. samples from $D(\\widehat\\lambda_t)$. The technical contribution is less clear.\n\n[1]. Bates, S., Angelopoulos, A., Lei, L., Malik, J., & Jordan, M. (2021). Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6), 1-34.", "was_revised": true, "final_justification": "I would recommend to accept this paper.", "rebuttal": "Thank you for providing thoughtful, helpful, and positive feedback. We address your questions below and hope that our answers can further ease your concerns.\n\n> **W1/L2** Technical novelty\n\nSince our paper is the first to study risk control in the performative setting to our knowledge, we opted for standard methods to keep our analysis simple and realistic. However, in the process, we did find the need to resolve certain technical challenges.\n\n- The main challenge lay in extending risk control to an iterative setting *in which we do not assume knowledge* of the required number of iterations $T$. We opted to resolve this challenge using simple techniques, such as creating a simple stopping rule parameterized by $\\Delta\\lambda$ and using a conservative confidence width $c(n, \\delta/\\tilde T)$ based on an upper bound $\\tilde T$ on $T$.\n- A second challenge was getting bounds like Bernstein's inequality and CLT to work *for an unknown distribution*, which we demonstrate in App A.2. We would expect these bounds to be quite wide because they do not assume much on future distributions, yet we demonstrate how we can achieve reasonable bounds, as demonstrated in the credit scoring experiment.\n- We also note that we did find the need to reframe the definition of sensitivity (Def 3.1) compared to [7] (*Performative Prediction*). There, sensitivity was framed on the feature-labels $z$, while we framed sensitivity on the loss to alleviate a Lipschitz condition on the loss function.\n\nIn summary, we agree that most of our methods are standard. However, we see that our main contribution is highlighting a critical area of study, in which the safe and gradual deployment of societal algorithms must account for performative effects.\n\n> **Q1**: \"In the experiments on the Kaggle credit scoring dataset, the authors assume the performativity---dependent on $\\lambda$---affects the data distribution through impacting the prediction function $f(x(\\lambda))$ in a specific way given at the bottom of page 8. I wonder if the theoretical guarantees hold only when the Kaggle dataset satisfies this assumption, and what would happen if the assumption is violated.\"\n\nOur distribution mapping $f(x(\\lambda))$ is meant to show one example of an adversarial distribution shift in which individuals will shift across the threshold $1-\\lambda$ if they are sufficiently close to it. In Appendix B.2, we show that this simulated shift is $(\\gamma,1,\\ell(\\cdot,\\lambda))$-sensitive for all $\\lambda$, where we resolve $\\gamma\\leq1.38$. Hence, $f(x(\\lambda))$ is just one instantiation of a class of distribution mappings that satisfy Assumption 3.2, which is a key assumption for achieving theoretical guarantees (Thm 3.3). The value $\\gamma$, when compared to the safety parameter $\\tau$, determines whether theoretical guarantees can be achieved. If $\\tau\\geq \\gamma$ is *violated*, we may or may not achieve tight risk control, as shown in the bottom-right plot of Fig. 2. For example, $\\tau\\in\\{0.2,0.5,0.8\\}$ work, but $\\tau\\in\\{0.001,0.1,0.2\\}$ don't.\n\n> **L1**: \"The PRC method requires access to i.i.d. samples from $D(\\hat\\lambda_t)$ for each $t\\in[T]$. In practice, collecting such data can be expensive and time-consuming, as it involves deploying $T$ models sequentially and gathering $n$ data points from each model before proceeding to the next. The paper would benefit from a discussion of this limitation and potential strategies for mitigation.\"\n\nWe agree with you that this is a limitation. However, this limitation mainly stems from inheriting the sampling requirements from conformal prediction and the iterative nature of performative algorithms.\n\nWe note that for the applications we had in mind---credit scoring, automatic resume screening, content moderation---we did not expect $T$ to be very large. Indeed, in our credit scoring experiment, we had $T \\approx 10$. With $N\\approx 2000$, we would require $20,000$ samples (and not all at once but over some period of time).\n\nFurther, knowledge of the distribution mapping $\\mathcal{D}(\\cdot)$ would help to mitigate the sample requirements of PRC. Indeed, if we could exactly parameterize $\\mathcal{D}(\\cdot)$, we could directly calculate $\\hat R_n(\\lambda,\\lambda):=\\frac{1}{n}\\sum_{i=1}^n \\ell(z_i(\\lambda),\\lambda)$, the finite-sample approximation of $R(\\lambda)$, from the initial samples $\\{z_i(\\lambda_\\text{safe})\\}_{i=1}^n$ and calibrate directly based on this initial cohort. We would not need to iterate and could keep $T=1$. However, our analysis focused on unknown distribution shifts, as is often the case in practice."}, {"review_id": "l1wAIb5CT4", "reviewer": "Reviewer_zDgW", "summary": "The paper proposes a new framework to calibrate blackbox prediction models in environments where predictions influence future data distributions—a setting known as performativity. Building on recent work in risk control and conformal prediction, the authors extend these methods to account for performativity. They develop an iterative algorithm that selects a sequence of decision thresholds $\\lambda_t$, guaranteeing both finite-sample statistical risk control and eventual tightness around a user-specified target risk level $\\ Delta\\alpha$, despite the distributional shifts induced by previous deployments. The approach is demonstrated through experiments in credit risk scoring, under both expected risk and quantile-based risk measures like CVaR.", "questions": "1. Monotonicity Assumption: The assumption that $\\ell(z;λ)$ is non-increasing in $\\lambda$ seems restrictive and significantly simplifies the problem. It effectively encodes a unidirectional notion of risk, implying that the decision-maker is only penalized for type II errors (e.g., false negatives), and that increasing conservativeness always reduces loss. This overlooks settings where there are different types of errors where non-monotonic losses arise, and make the search for an optimal threshold $\\lambda$ much easier. \n\n2. Motivation for loosening the threshold (Section 2.2): The rationale for choosing a less conservative (i.e., smaller) threshold $\\lambda$ is stated in terms of \"preserving utility\" and \"useful class information,\"  (line 97) but these terms are not clearly defined. It would be helpful to clarify what utility means in this context -- what constitutes \"useful class information\" (which seems to be not captured by the loss $\\ell(z; \\lambda)$?) More broadly, a more detailed discussion of the trade-off between safety (risk control) and utility would strengthen the motivation behind relaxing the threshold.\n\n3. Tumor segmentation example: I find this example difficult to interpret. My understanding is that $\\lambda$ represents a deployment parameter that influences both (a) the prediction set size (e.g., the number of pixels labeled as tumor), and (b) the data distribution in the next round via performative feedback. However, it is unclear how the induced distribution $D(\\lambda)$ should be interpreted in this context. Does a larger $\\lambda$ simply produce more inclusive prediction sets, or does it also affect how future data (e.g., future tumor scans or labels) are generated? A more detailed explanation of how performativity manifests in the tumor segmentation setting would help clarify this.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths:  \n\nThe paper addresses a crucial gap in the risk control literature—how to ensure safe deployment in settings where predictions causally influence future data. This is particularly relevant in real-world socio-technical systems like credit scoring or policy-making. The extension to quantile-based risk measures like CVaR is also valuable for applications that care about tail risk rather than average-case loss. \n\n\n## Weaknesses:\n\nThe major weakness of the paper is the monotonicity assumption which greatly simplify the complexity of the problem. The paper mentions a trade-off between “safety” and “utility” but doesn't clearly define what utility means or how to measure it in practice. The tumor segmentation example used to motivate the method is difficult to interpret in a performative setting. It’s unclear how prediction set size (controlled by $\\lambda$) influences future data distribution. In addition, it's a bit unclear how the method is specifically designed to account for \"performativity\" -- the proposed method seems to work for any case where the distribution is smooth and doesn't change dramatically? See more details in the question section.", "was_revised": true, "final_justification": "The paper is the first work to propose a provable risk control method under performativity, which is a valuable contribution to both communities.  A clarification of the additional technical contributions specific to the performative setting would be valuable.", "rebuttal": "Thank you for the detailed feedback. We address your questions below and hope that our answers can ease your concerns.\n\n> **W1/Q1**: On the monotonicity assumption. \"The assumption that $\\ell(z;\\lambda)$ is non-increasing in $\\lambda$ seems restrictive and significantly simplifies the problem. It effectively encodes a unidirectional notion of risk, implying that the decision-maker is only penalized for type II errors (e.g., false negatives), and that increasing conservativeness always reduces loss. This overlooks settings where there are different types of errors where non-monotonic losses arise, and make the search for an optimal threshold $\\lambda$ much easier.\"\n\nWe adopt the monotonicity assumption for two primary reasons. First, monotonicity is commonly used and adopted in the risk control literature (see [6] *Conformal risk control*). Second, and more importantly, for the performative socio-technical systems we considered, a monotonic loss has important semantic meanings and makes great sense.\n\nTo begin with, we would like to clarify the details regarding the monotonic assumption we make. Our goal is to control the risk $R(\\lambda):=R(\\lambda,\\lambda)$. *We do not assume this quantity to be monotonic.* We agree that assuming this quantity to be monotonic would greatly simplify the complexity of the problem. Instead, we assume $R(\\lambda,\\cdot)$ is monotonic due to $\\ell(z,\\cdot)$ being monotonic in the second term, as you highlighted. The difference is that the risk may increase or decrease when we change distributions. If we have $R(\\lambda,\\lambda')$ where the data distribution arises from $\\lambda$ and we use $\\lambda'$ to evaluate, then we have no monotonicity assumption when $\\lambda$ changes and assume monotonicity when $\\lambda'$ changes.\n\nFor example, in credit scoring, we imagined an institution gradually deploying an algorithmic model. This model is tasked with identifying low-risk opportunities and providing automatic credit approval to applicants. The threshold $\\lambda$ controls to what extent this model is used. We saw the objective as maximizing efficiency by leveraging the model as much as possible while maintaining safety guarantees on it as we try to find an optimal level to deploy it. *Because we viewed the model as inherently risky, we also viewed its increased usage as increased risk, leading us to frame our problem with the monotonicity assumption.*\n\nHowever, we do agree that studying the non-monotonic case has its use cases. In the credit scoring example, the threshold $\\lambda$ may define the balance between automatic acceptances by the credit model and the number of applications subjected to further human review. One might want to design a holistic loss function on this entire pipeline, which includes the errors made by the credit model, the errors made by human reviewers, and the time cost of human review. This loss function would not be monotonic.\n\nHow would one attempt this problem? To clarify, one key aspect in safe deployment is \"safe at any time.\" This aspect is both important for applications and makes the problem worthwhile to study (otherwise, one can just use binary search). Similar to the monotonic case that we study in the paper, one would have to (1) start from a \"safe\" $\\lambda$ and (2) iteratively adjust $\\lambda$ to optimize some objective while maintaining safety guarantees. (1) Without monotonicity, defining $\\lambda_\\text{safe}$ is not obvious. We would need other assumptions on the loss $\\ell(z,\\cdot)$ besides monotonicity or resort to domain expertise to define $\\lambda_\\text{safe}$. (2) However, afterwards, one can use a similar method to PRC. Let $U(\\lambda)$ be the utility function defined on the domain of $\\lambda$ (in our paper, we took this to be monotonically decreasing in $\\lambda$), and assume we have deployed $\\hat\\lambda_{t-1}$. We should choose $\\hat\\lambda_t$ in a way that maximizes the utility function $U(\\lambda)$ while controlling risk by reasoning about $R(\\hat\\lambda_t)$ from the sample risk $\\hat R(\\hat\\lambda_{t-1}, \\cdot)$. **So we think the procedure under non-monotonicity is much the same.**\n\nIn contrast to the non-monotonic setting, we do find a natural interpretation for $\\lambda_\\text{safe}$ in the monotonic case: an institution does not use their model, abstaining from predictions (e.g. the credit model makes no automatic acceptances). For adherence to the literature and interpretation reasons, we assumed monotonicity, although our algorithm should be amenable to adjustments to handle non-monotonicity.\n\n> **W2/Q2**: \"The paper mentions a trade-off between “safety” and “utility” but doesn't clearly define what utility means or how to measure it in practice.\" \"Motivation for loosening the threshold (Section 2.2): The rationale for choosing a less conservative (i.e., smaller) threshold $\\lambda$ is stated in terms of \"preserving utility\" and \"useful class information,\" (line 97) but these terms are not clearly defined. It would be helpful to clarify what utility means in this context -- what constitutes \"useful class information\" (which seems to be not captured by the loss $\\ell(z;\\lambda)$?) More broadly, a more detailed discussion of the trade-off between safety (risk control) and utility would strengthen the motivation behind relaxing the threshold.\"\n\nIn credit scoring, and other use cases like automated resume screening and content filtering, **\"utility\" means efficiency**. When we have \"useful class information,\" that information can help inform quicker decisions and increase efficiency. In credit scoring, the \"useful class information\" of classifying someone as creditworthy can allow a credit institution to automatically give them credit without the need for too much manual review. Hence, this \"useful class information\" helps preserves the utility of the model by helping us save time.\n\nWhat constitutes \"useful class information?\" In credit scoring, we might classify an applicant **{creditworthy}**, **{not creditworthy}**, or **{creditworthy, not creditworthy}** (e.g., we don't know). The first two constitute \"useful class information.\" The last one doesn't.\n\n**On the trade-off between safety and utility**: we imagine $\\lambda$ as a lever we pull in how much we utilize the model. As $\\lambda$ decreases (the model is used more), safety is decreased and utility is increased because we cede control to the model and try to save time. That is why we seek the minimum threshold $\\lambda$ so that we can maximize utility while maintaining safety guarantees (Sec 2.2). While we do not explicitly define how \"utility\" is *measured* since we believe that will vary on a case-by-case basis, we do motivate why we wish to minimize $\\lambda$ in our algorithm (L97). We can update the manuscript for better clarity regarding this point.\n\n> **W3/Q3**: \"The tumor segmentation example used to motivate the method is difficult to interpret in a performative setting. It’s unclear how prediction set size (controlled by $\\lambda$) influences future data distribution.\"\n\nWe apologize for the confusion. The point of the tumor segmentation example was to help slowly introduce performative risk control (PRC) by first introducing conformal risk control, which was the perspective of the authors when they began this work. For tumor segmentation, we agree that it is difficult to imagine how $\\lambda$ may influence the distribution of future tumor scans and labels. We used this example to lead into the following paragraph (L27-32) in which we introduce risk control in the performative setting (PRC). There, we highlight PRC via our flagship example, credit scoring, where performativity does come into play. A bank that uses the calibration parameter $\\lambda$ to inform its final algorithmic decision should expect to see some performative effects from using that parameter, e.g. due to individuals attempting to strategically update their features to their own advantage.\n\n>  **W4**: \"In addition, it's a bit unclear how the method is specifically designed to account for \"performativity\" -- the proposed method seems to work for any case where the distribution is smooth and doesn't change dramatically?\"\n\nOur analysis does indeed make the assumption that performative effects due to changes in $\\lambda$ are \"smooth,\" an assumption both key to our method and relied upon in the performative prediction literature (e.g. $\\epsilon$-sensitivity as found in Def 3.1 of [7] *Performative Prediction*). Where performativity comes in is highlighted in L32-33. Standard risk control literature examine the risk $R(\\lambda) := \\mathbb{E}\\_{(x, y) \\sim \\mathcal{D}} \\left[\\ \\ell(y, \\mathcal{T}\\_\\lambda(x))\\ \\right]$. We analyze the same expression with a small change $\\mathcal{D} \\rightarrow \\mathcal{D}(\\lambda)$. To control this quantity and account for performativity, our method specifically formulated Def. 3.1 and Assumption 3.2, which were instrumental to the proofs of our main results (Alg 1, Thm 3.3). In this manner, we specifically designed to account for performativity, and we did so by requiring that performative effects be smooth."}]

Author rebuttal:

Meta-review: This paper introduces Performative Risk Control, a framework that addresses risk control in performative settings, defined in the paper as a situation prediction-supported decisions may influence the outcome they aim to predict. The authors extend existing risk control and conformal prediction methods to account for these performative effects. The method is validated through experiments, including a credit risk scoring application.

Reviewers found the motivation for the paper - risk control in performative settings to be interesting. The paper was well-written with clear assumptions and theorems, though some reviewers note that the technical contribution may be a relatively straightforward adaptation of existing methods to the performative setting. Reviewers also pointed out specific areas where the motivation and assumptions, such as the monotonicity assumption, could be better explained and justified. Nevertheless, the reviewers found the approach promising and the technical details convincing. The authors are encouraged to revise the paper in light of reviewer feedback.
Decision: Accept (poster)