Distributionally Robust Performative Prediction
Authors: Songkai Xue
Yuekai Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: performative prediction
distributionally robust learning
misspecification
distribution shift
Primary area: safety_in_machine_learning
TL;DR: This work designs a robust learning framework to better approximate the true performative optimum in the presence of distribution map misspecification.

Abstract: Performative prediction aims to model scenarios where predictive outcomes subsequently influence the very systems they target. The pursuit of a performative optimum (PO)—minimizing performative risk—is generally reliant on modeling of the distribution map, which characterizes how a deployed ML model alters the data distribution. Unfortunately, inevitable misspecification of the distribution map can lead to a poor approximation of the true PO. To address this issue, we introduce a novel framework of distributionally robust performative prediction and study a new solution concept termed as distributionally robust performative optimum (DRPO). We show provable guarantees for DRPO as a robust approximation to the true PO when the nominal distribution map is different from the actual one. Moreover, distributionally robust performative prediction can be reformulated as an augmented performative prediction problem, enabling efficient optimization. The experimental results demonstrate that DRPO offers potential advantages over traditional PO approach when the distribution map is misspecified at either micro- or macro-level.

Distributionally Robust Performative Prediction
Songkai Xue Department of Statistics University of Michigan sxue@umich edu
Yuekai Sun Department Statistics University of Michigan yuekai @umi edu
Abstract
Performative prediction aims to model scenarios where predictive outcomes sub- sequently influence the very systems they target. The pursuit of performative optimum minimmizng performative risk generally reliant modeling of the distribution map, which characterizes how deployed ML model alters the data distribution. Unfortunately, inevitable misspecification of the distribution map lead to poor approximation of the true PO. To address this issue, we introduce novel framework of distributionally robust performative prediction and study a new solution concept termed as distributionally robust performative optimum (DRPO). We show provable guarantees for DRPO robust approximation the true PO when the nominal distribution map is different from the actual one. Moreover, distributionally robust performative prediction can be reformulated as augmented performative prediction problem, enabling efficient optimization. The experimental results demonstrate that DRPO offers potential advantages over traditional PO approach when the distribution map is misspecified at either micro- or macro-level.
Introduction
In numerous fields where predictive analyties play an important role, decisions made the basis of machine learning models do not just passively predict outcomes but actively influence future input data. Consider the domain
financial services, such as credit scoring and loan issuance, where model's 's decision to grant or deny application can affect the applicant's 's future financial behaviors and, consequently, the profile future applicants. Similarly, educational settings, the decision process for school admissions can shape the applicant pool, as those who are accepted often share their success strategies, indirectly influencing the preparation of future candidates. These examples highlight the study of performative prediction (29). a recent framework that facilitates formal examination of learning in the presence of performative distribution shift resulting from deployed ML models.
Delving deeper into the formulations of performative prediction, the concept of distribution map emerges as pivotal. This map characterizes the impact that a deployed ML model has on the underlying data distribution, which crucial element in navigating performative effects. The literature primarily revolves around the pursuit of performative stability (PS), a model which is optimal for the distribution it induces [29 25 [ 8) However, more ambitious target is the performative optimum (PO), which seeks the minimization of performative risk, the risk of the deployed model on the distribution induces. Efficiently achieving PO typically necessitates modeling of the distribution map [26 14 [5] 24). Practically, the precise influence of model on the data ecosystem is intricate and dynamic, making perfect specification an unattainable ideal.
In this work, we propose a distributionally robust performative prediction framework that aims enhance robustness against spectrum of distribution maps, thereby mitigating the issue of misspecification. Our contributions are summarized as follows: I) in Section 2 we formalize the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

DRPO concept, anchoring it within the performative prediction literature as a robust alternative; 2) in Section [] we provide theoretical insights into the efficacy of DRPO, demonstrating its resilience in the face of distribution map misspecification; 3) in Section we recast distributionally robust performative risk minimization as an augmented performative risk minimization problem, facilitating efficient optimization; 4) in Section we showcase DRPO's advatanges over conventional PO by empirical experiments. The paper concludes with a summary and discussion.
1.1 Related Work
Performative prediction. Performative prediction is an emerging framework for learning models that
influence the data they intend predict. The majority of research focuses on performative stability [29 [2
5
30
21
8 7. albeit with a few exceptions aiming at performative optimality. (26) propose
two-stage plug-in method for finding the PO. [14 propose to find the PO by a parametric model of
[18] solves the PO when the problem is outcome performative only. (24] argue that the PO with a misspecified nominal distribution map can still reasonably approximate the true PO, as long as the misspecification level is not significant. This claim is also supported by our theory and experimental findings. Unlike them, we demonstrate that the DRPO is comparable to the PO if the misspecification is small, whereas the DRPO can offer substantial advantages over the PO if the misspecification moderate large. Moreover, the DRPO ensures reasonable performance for all distribution maps in an uncertainty collection surrounding the nominal distribution map, rather than just a single distribution map.
the distribution map. This method is extended by 15 to stateful performative prediction.
Distributionally robust optimization. DRO solves a stochastic optimization problem by optimizing under the worst-case scenario over an uncertainty set of probability distributions. Most popular DRO frameworks are based on go-divergence [13 2 12 20 [ and Wasserstein distance [1] 27 6 31
10
The existing DRO literature pays no attention to performative prediction except for (28)
28 study the repeated distributionally robust optimization algorithm, which repeatedly minimizes the distributionally robust risk at the induced distribution. They show such repeated training algorithm yields distributionally robust performative stable (DRPS) solution, assuming conditions analogous finding the performative stable (PS) solution.
to the validity of repeated risk minimization 22 25
Nevertheless, the DRPO solution, which lies at the heart of the distributionally robust performative prediction problem, is not the subject of their study. Furthermore, they lacks theoretical guarantees regarding the proximity of the DRPS to either the PS or PO solution.
2 Methodology
2.1 Performative Prediction Essentials
Let θ denote the )inite-dimensoona) model parameter space, 2 denote the data sample space, and P(Z) denote the set of probability measures supported on 2. In performative prediction, aim to find a / E θ that achieves low performative risk
PRague (0) Ez- (Z)
(2.1)
where L: x θ + R is the (known) loss function, and D Dtrue : - - P(Z) is the true distribution map. The true performative optimum (true PO) @ro,true is known to minimize the (true) performative risk:
frr,true arg min (PRerue (0) = Ez-Da [[) Ag
(2.2)
is easy to see that if we know the true distribution map, then we can evaluate the true performative risk and find the true performative optimum well up some finite sample error which is negligible as the sample size goes to infinity. However, the true map Derue (-) is unknown in general, thus posing a significant obstacle in the pursuit of evaluating and optimizing the true performative risk.
To enable the optimization performative risk, it is necessary to have known nominal distribution map D()) that is believed to closely approximate the unknown true distribution map Dirue (-). Then one can find the performative optimumn (PO) by minimizing the nominal performative risk:
OPo E arg min (PR(0) (DO) (0())((ZZ000 [((20))]-
(2.3)
BEO
Because the distribution map is inevitably misspecified, D(.) * Dirue the true performative risk generally not minimized by Ppo. Therefore, we treat 2.3 as solution concept to approximately

---

solve the true performative risk minimization problem (2.2) and refer (2.3) to standard performative prediction. Now we provide several illustrative instances of potential sources that may lead the misspecification of distribution map, i.e., D()) * (-rue
Modeling error. The modeling of (()) be either deterministic model of explicit form statistical model with model parameters to be estimated. The misspecification of (()) may stem from modeling error.
Example 2.1 (Strategic classification). Let 2 = X x y where X' is the feature space and y is the label space, so that we are in the supervised learning regime. Strategic classification relies on working model of individual's 's data manipulation strategy:
(a(x)= arg max,' {ug ---ccll c(x,2')).
(2.4)
where De(.), u and .) are known as individual's 's best response function, utility function, and cost function, respectively. The best response function can be lifted to the measurable space of X x y so that we have the response map
x To
(===
[m8 11
A0(x)) y [
arg max (ug(=')) - c(xxx'))
y
The nominal distribution map D(-) is fully characterized by T and the sampling distribution of Drrue (see details in Appendix Ey. In this example, the distribution map D(.) could be misspecified because the individual's 's utility function and cost function could misspecified.
Example 2.2 (Location family). Location family postulates translation model:
Z D(0) = z _ Zo + A0 where Zu- ~ Derue(O).
where A distributions of D(00), ((01) D(0x), then A is partially identified up to linear subspace of Iddin( ( xe)
is unknown and therefore must be estimated. If we observe the sampling
A E {M M 0o) = pi - po for k E [K]},
where Ph is the mean of D(0)). In this example, the distribution map D(.) can be misspecified because the model parameter
only partially identifiabl.
Distribution shift. Consider the training and test environments have different distribution maps, Derain (-) and Drest (-), respectively. We specify the nominal distribution map as the training distribu- tion map (--) = P-rain(-). Then D(-) can be misspecified for the true distribution map Derue (-) Drest (-) due to the difference between the training and environments Dirsin (-) J D Prest(-). (-).
Example 2.3 (Disparate impacts and fairness). A population is comprised of majority and minority subpopulations (e.g., by race or gender). The population distribution map mixture of the subpopulation distribution maps: pppp(() (0) ~/Dmaj(0) (1-7)Duuin (f). In fair machine learning. a theme is to check whether an ML model has disparate impacts on different subpopulations or biased against the minority. Suppose that work with D(-) popp(.) and target the minority Deroe D m-in()). However, the population distribution map and the minority distribution map may differ. In this example, the distribution map D(-) ) is misspecified because subpopulation shift.
-
2.2 Distributionally Robust Performative Prediction
From an intuitive perspective, the PO solution (..3 has the potential to achieve low performative risk PR (Opo) when the nominal distribution map D(-) closely aligns with the true distribution map (-). However, D--) and (true()) may be quite different. such cases, the PO solution may incur high performative risk. To address this issue, propose distributionally robust formulation for performative prediction, where we explicitly incorporate into the learning phase the consideration that the true distribution map (-) (-) is different from the nominal one D(.).
Let D(-II-) the KI. divergence, i.e.,
dQ dP
dQ dP dQ,
D(QIP)=
dP = log
where (t() log/ / for any >> 0. Here dQ/dP the Radon- derivative, and we implicitly require the probability measure Q to be absolutely continuous with respect to P. With the
3

---

KL divergence, we can define family of distribution maps around the nominal distribution map. Specifically, the uncertainty collection around D with critical radius p is defined
U(D) - (
((D(O))D(()) & p. v E @).
The radius p reflects the the magnitude of shifts in distribution map we seek to be robust to. We remark that the value of P can be preseribed or be selected in data-driven ways. We postpone the discussion critical radius calibration to Section 4.3
Definition 2.4 (Distributionally robust performative risk). The distributionally robust performative risk with the uncertainty collection U(D) defined as
DRPR(0)=
sup D:DEu(D)
E B2- (((:0))
(2.5)
In other words, the distributionally robust performative risk (RPR(0) measures the worst possible performative risk incurred by the model parameterized by 0 among the collection of all alternatiee distribution maps that are p-close to the nominal distribution map DD With this intuition, it is natural define an alternative solution concept which minimizes (2.5).
Definition 2.5 (Distributionally robust performative optimum), The distributionally robust performa- tive optimum (DRPO) defined as
ORRO arg mig DRPR(0).
(2.6)
lee
We refer the method (2.6) distributionally robust performative prediction. When comparing (.6 and (2.3). we view the DRPO (2.6) as competing solution concept to the PO 22. because both of them aim for achieving low performance risk (2.1)
2.3 Generalization Principle of DRPO
Distributionally robust performative prediction asks to not only perform well on a fixed performative prediction problem (parameterized by the distribution map D), but simultaneously for range of performative prediction problems, each determined by a distribution map in an uncertainty collection u. This results in more robust solutions, that is, those DRPOs which are robust misspecification of distribution map. The uncertainty collection plays key role: it implicitly defines the induced notion of robustness. Moreover, distributionally robust performative prediction yields natural approach for certifying out-of- sample performance, which is summarized by the following principle.
Proposition 2.6 (Generalization principle of distributionally robust performative prediction). Suppose that the uncertainty collection U contains the true distribution map Dirue, then the true performative risk is bounded by the distribution robust performative risk: PRarue S DRPR(() for any 0 e e. In consequence, have PRarue PRarue(PDRPO) S DRPR
Essentially, 2f is chosen appropriately, the corresponding DR performative risk upper bounds the true performative risk, and thus DRPO enjoys provable guarantees on its incurred performative risk.
2.4 Benefits of DRPO: A Toy Example
toy example is employed to facilitate a conceptual understanding of the advantages DRPO over PO in relation to achieving improved control over the worst-case performative risk.
Let R. θ [-1, 1), and f(2 f) 8z. Let the nominal distribution map be D(B) =N(((()) for some f [] + R and o² > 0. Then the nominal performative risk is
PR(0)
2x) = (0)
Regularization effect. With the dual formula given in Section 3.1 one can derive the distributionally robust performative risk directly:
DRPR(() PR(() + h (enalty(())
where the penalty function Penalty (0) = V 2 0 penalizes the deviation of 0 from the origin 0. and the critical radius p tunes the level of regularization. That is to say, in this toy example, the distributionally robust performative risk minimization problem is essentially a L -e-regularized performative risk minimization problem.

---

Better worst-case control. To be more concrete, we let f(0) a1l + ao for some > 0. For any D E (D), let the performative risk of D be (0) Ez= [[( 0) If D is the true distribution map, then PR, (f) is the true performative risk that we incur. Through direct calculation (see details in Appendix one show that @DRPO is more robust than Ppo in the sense of worst- case performative risk control, that is, (EU())) (Rp(O 2 SuP)Eu(D) PRg(ODRPO)
ga?
2aj for any fixed p 252 This example shows that the worst-case performative risk of the PO can be arbitrarily larger than that of the DRPO, as d1 + O. The message that DRPO offers certain advantages over PO in terms of mitigating worst-case performative risk is also supported by the empirical results shown in Section
Theory
3.1 Strong Duality of DRPR
The evaluation of distributionally robust performative risk DRPR(0) given (2.5) involves an infinite dimensional maximization problem which is generally intractable. Fortunately, it is in fact equivalent to minimization problem over a single dual variable.
Proposition 3.1 (Strong duality of DRPR). For any 0 E O, we have
DRP)(() = inf log Ez-
[ (2.00)/
+ mp]
(3.1)
#20
The dual reformulation (3. will served as the cornerstone of developing algorithms for finding the DRPO in Section As byproduct of Proposition 3.1 characterization of the worst-case
distribution map which attains the supremum in (2.5) is given in Appendix B
3.2 Excess Risk Bound
For now, we are interested in bounding the excess risk: E(Ò) - PR (0) mingge PR (0) PRimue PRerue (ORerue((PO,rrue)) where 0 is an approximate solution to the true PO 0ro.tmue which could particularly be OPO and ÓDRPO The excess risk captures the performance of 0 relative to the oracle performative optimum tru, providing direct measurement of the suboptimality of in terms of performative risk. As follows, we show the excess risk bounds of the PO solution and the DRPO solution, (0ro) and E(@pro)), and compare them.
Proposition 3.2 (Excess risk bound of the PO). Suppose that we have bounded loss function [{(2;)) < for any E 2,0 ④ and some (D((tmn(0)[[((O))
0. Then we have
E(PPo) S V2B sup Bee
(3.2)
Proposition 3.3 (Excess risk bound of the DRPO). Suppose that D(Derue (are:ee@@Doooooll (ODnro)[D((ORro)) Then we have
S
E(fbaro) S Varz= | [( (ro,true]] + ovvP).
(3.3)
Comparing Proposition 3.2 and 3.3 we see that the excess risk bound of the DRPO can localized to the true PO while the excess risk bound of the PO is entangled with the full parametric space O. Although we are comparing two upper bounds which can be not tight enough, the comparison sheds lights the potential benefits of using DRPO over PO solution. Even if in the case of no significant improvement of using DRPO over PO solution, the excess risks of them are comparable, thus doing no harm. Our insight has been verified through toy example in Section 2.4 and well experimental results Section 5 In Appendix(C we generalize Proposition 3opposiiion333oo to the scenario when the uncertainty collection doesn 1 cover the true distribution map, i.e., D(Daruc ([brro) > p.
4 Algorithms
We recall a standard algorithm for performative risk minimization in Appendix In the following subsections, we will see that any off-the-shelf algorithms for finding the PO can be utilized as an intermediate algorithm for finding the DRPO by using our proposed algorithms. Moreover, we provide practical considerations for the selection of a critical radius in the last subsection.
F
5

---

4.1 Distributionally Robust Performative Risk Minimization
By the strong duality in Proposition 3.1 DR performative risk minimization is equivalent to the following optimization problem jointly over (6)
min (R)R(O) > min inf (0(0, p) - log Eew
[ + =uo (4.1)
aE #20
This suggests an alternating minimization, summarized in Algorithm 1 where we learn @DRO by fixing j and minimizing on 0 and then fixing 0 and minimizing on μ alternatively until convergence.
The step of minimizing on with fixed μ (Line 4 in Algorithm[]] [] is itself perfor- mative risk minimization problem, which
Algorithm 1 DR Performative Risk Minimization
1: Input: radius p. nominal distribution map D(0)
can be solved by any suitable performa- tive risk minimization algorithm. The step of minimizing on f with fixed (Line 5 in Algorithm I can be solved by the line search or the Newton- Newton-Raphson method since (f, μt) is convex in μ.. The total cost of Algorithm[]] iis therefore com- parable to that of the performative risk minimization algorithm used in the intermediate step. Lastly, the alternating minimization algorithm in common practice guarantees global convergence (to station- ary point) regardless of how the optimization parameters are initialized. With the strong convexity assumption, the alternating minimization algorithm guarantees convergence to the global minimum.
2: Initialize f
3: while je has not converged do
4: Update A arg mingge (Ez)D(()) (Z)00)
5: Update μ 4- arg minp2o (0, 1)} (bb in (4.1
6: end while
7: Return: 0
4.2 Tilted Performative Risk Minimization
Treating as hyperparameter which can be tuned by practitioner, the solution of the dual problem (4.1) can be denoted by (p), (p),x())))))))))))))))))))))(( (p)). One can show that 1² (p) is decreasing function of p. An
intuition is that μ oo, we have arg mingge (v)(0, })) arg mingge {Ez.D(o) [(( 0)]
which reduces to the original performative risk minimization problem, or the distributionally robust performative risk minimization problem with p 0 (see an formal explanation in Appendix F.
With this motivation, instcad of tuning p. we can tune μ (or the inverse of it, denoted by a = -1 and solve the a-tilted performative risk minimization problem:
θTPO E arg min (P)(()= = Ez-D(() [cl(2:0))]
(4.2)
OE
where TPR(O) stands for the tilted perfor-
Algorithm Tilted Performative Risk Minimization
mative risk and @rpo is the tilted performa-
tive optimum, that is, the performative opti- 1: Input: tilt a, nominal distribution map D(0) mum of the tilted problem. In order to have
2: Update 0 <- arg mingge (Ez=D:() [e00(2
stronger distributional robustness property, 3: Return: 0
we tune a to be larger. Given the correspondence 1 (p), should have θrro with equals @DRO Therefore, the tilted performative risk minimization implicitly solves a
with م (/ll)=i"(///aaaa corresponding DR performative risk minimization problem. Finally, we remark that exponential tilting is a statistical method that has been around at least since the exponential family [17] was first invented. More recently, has been applied operation research [ and machine learning 22 23
4.3 Calibration of Critical Radius
The performance of the DRPO is contingent on the uncertainty collection radius p, which is typically difficult to choose priori without additional information. The greater the value of p. the higher the level of distributional robustness, and thus the greater the tolerance for distribution map misspecifica- tion. Therefore, the selection of p reflects practitioner's risk-aversion preference. In this subsection, we present two simple, yet effective calibration techniques for selecting p.
Post-fitting calibration. Without additional information, we can only hope to be robust to a prescribed set of distribution maps, say Σ. The assumption of Proposition 3.3 requires only the uncertainty collection at the DRPO, which reduces to an uncertainty ball centered at D(ODRPO), encompassing Therefore, in order to provide a provable guarantee for all
the true distribution distribution maps in E, the radius p can be chosen based on the following criterion:

---

Peal arg min
max D(Derue IID) s
where
P20
Derin
D(D [D) = (Otoo
Here D is the estimated KL divergence, and (p) is indexed by the radius p to highlight its dependence on p as a tuning parameter to be calibrated. We implement the post-fitting calibration approach (with bisection search for p) in Section [5.1
Calibration set. With additional information, such as a small set of calibration data, we can pick o (or a if we use Algorithm 2) by evaluating the performance of (p) on the calibration set. Consider Example 2.3 where the training and test distribution maps may differ, we can conduct the following grid searching procedure to choose p: 1) for a candidate set C of p's, we compute (ODRPO(p) : o E C} under Dirain: 2) we obtain a few calibrating samples from Drest, on which we evaluate the performance of (p); 3) we select Peal E C with the best calibration set performance. The calibration set approach is implemented in Section 5.3
To conclude this subsection, we discuss the computational costs of the proposed calibration methods. For general problems, these calibration methods necessitate a grid search, which may be computa- tionally expensive. Fortunately, for specific problems (for example, experiments in Section [.1]. we can exploit the decreasing nature of the estimated KI. divergence as a function in p. As a result, we can use bisection search rather than grid search to significantly reduce computational costs.
5 Experiments
We revisit 2.11pless2.1 2.2 and 2.3 and compare the PO and the DRPO empirically. For a particular (true) distribution map, either the PO or the DRPO may have the potential to outperform the other in terms of the performative risk evaluated at this particular distribution map. In contrast to the PO, however, the DRPO aims guarantee reasonable performance for all distribution maps in an uncertainty collection around the nominal distribution map. To ensure the performance of the DRPO on a (set of) specific distribution map(s), the radius p (or the tilt a) must be calibrated. Lastly, each shaded region in figures below shows the curve's ' standard error of the mean from 30 trials.
5.1 Strategic Classification with Misspecified Cost Function
In reference to Example 2.1 we examine strategic classification involving cost function that is misspecified. The experimental setup resembles that in (29]. The task is credit scoring, specifically predicting debt default. Individuals strategically manipulate their features to obtain a favorable classification.
Consider an instantiation of the response map (2.4) such that ug(x) -T and e(z,z") 2e x ||aatrat - Fstras 113 + 0o ||ænon-strat X nton-strat 1.2 Without loss of generality, let the first m features be strategic features and the last d m features be non-strategic features. Let B diag(1.....
.,....... 0) E IRd where the first m diagonal elements are I's and the others are 0's. ... 1,
Then the best response function is Ax(x) x cBB.
For the base distribution D(O), we use a class- balanced subset of a Kaggle credit scoring dataset (16), individual's historical data, including their monthly income
CC-BY 4.0). Features encompass and credit line count. Labels are binary where the value of 1 represents a default on a debt and O otherwise. There a total of 3 strategic features and 6 non-strategic features. We use logistic model for the classifier and the cross-entropy loss with L-rreulaaizzatin for the loss function L.
Consider the cost function is misspecified by its performativity level e. We specify the cost function with the nominal performativity level = 0.5. However, the true performativity level Etree might not be 0.5, but in the range of [0.5 0.5n, 0.5 0.5n] for some 77 2 0.
The left plot of Figure ssoows performative risk incurred by the PO and the DRPO's with various radius p's. Note that the PO can be understood as the DRPO with p O. As p increases, the DRPO aims to achieve more uniform performance across a wider range of Éurue- The middle plot of Figure shows relative improvement in worst-case performative risk of the DRPO to the PO as the radius p
'The relative worst-case improvement of 0 to OpO is mAX PRizuc mMA)) (R
100%.
maXe PRarue (Spo)
7

---

R0
Sa4 Wore: impoveneet
0.35
2.579
D1PO 2.02
DSP0 3.00
8.0%
Extimaree Divegerce
2.30
1-
2.050
DMVO with pm 2305
6.0%
2.425
DSPO with pm 2.04
5,20-
2.000
2.0%
6.0%
6,500
6.525
2.00
5.0
0.2
5.4
0.6
0.0
1.0
3.50
0.01
0:22
0.03
0.04
0.50
0.01
0.02
Thue Pertommtt Level Cowr
Radiue
Sad
Figure 1: Results of Experiment 5.1 Left: performative risk incurred by the PO and the DRPO's with various radius p's. Middle: relative improvement in worst-case performative risk of the DRPO to the PO as the radius increases, for different range of misspecification 1j's. Right: radius versus estimated KI. divergence between PopROO and D(ODRro(p)), where vertical bands indicate the calibrated radius Peal's s.
190 wtth am
wh0 a- 5.2
23.0%
0.012
170
20.0%
0.002
13.0%
10.0%
0.005
5.6
0.002
3.2
0.5
0.3
True Parforaliity Level
intam Cot Dun Variatee
Figure 2: Results of Experiment 5.2 Left: performative risk incurred by the PO and the TPO's with
various tilt a's. Middle: relative improvement in worst-case performative risk of the TPO to the PO as the tilt a increases, for different range of misspecification 7j's. Right: the correspondence relationship between the radius p and the (inverse of) optimal dual variable μ
increases, for different range of misspecification 7/'s. Curves positioned above the horizontal dotted line indicate that the DRPO outperform the PO in terms of worst-case performative risk. When there is larger range of misspecification 1), the DRPO has greater potential to beat the PO. The right plot of Figure demoontreess the post-fitting calibration of radius Peal described in Section vertical bands indicate the calibrated radius Peal 's, which lead to the DRPO's with sound re ative worst-case improvement and avoid overconservative solutions, as depicted by the corresponding bands in the middle plot of Figure
The
5.2 Partially Identifiable Distribution Map
Recall Example 2.2 examine location model for distribution map where the mis- specification arises from the estimation eгтor of the model parameter. Let [0, 0. 02 0o ... Og fo] dx and U [m - no μ2 po ... ag - [o] E Bdxk where = (O). The unknown parameter can only partially indentified through the equation AV = U when K < d. A particular estimate of A is Ā = UVt = U(V -'VT, where Vt is the Penrose inverse of V. In fact. the parameter A is only identifiable up the subspace W = (UVT + E. span ET) C CNVT))), where W is the left null space of V. Precicely, we have AV U if and only if E W.
In this experiment, we still use the credit data. We observe sampling distribution of D(0). D(ei), and D(e)), where e, is the i-th canonical basis. For the true distribution map, the performativity of the first two features is 0.5, while the performativity of the other 7 features is étrue- In short, Atrue (0.5, 0.5, Strues Strue). We the range of étrue be [-0.5m, 0.5ng for 7 2 By using the estiamte A, we model the performativity of the first two features correctly, but wrongly model the other features non-strategic. This time fit TPO by Algorithm Saporithmminsteeaa of DRPO.
The left plot of Figure 2 shows performative risk incurred by the PO and the TPO's with various tilt a's. As increases, the TPO performs more uniformly across a wider range of Ctrue- The middle plot of Tigure[[show relative improvement as the tilt a increases, for different range of misspecification 1j's. Without misspeicification, 1) 0, the PO for sure better than the TPO. With moderate to large misspecification, 7) E (2/3, 4/3, 2}. the TPO demonstrates certain advantages over the PO. The right of Figure shows the relationship
worst-case performative risk of the TPO to PO

---

between the distributionally robust performative risk minimization and the tilted performative risk minimization: fitting DRPO with (which returns the optimal dual variable μ²) is equivalent to fitting TPO with 1/1²".
5.3 Fairness without Demographics
Referencing to Example 2.3 we examine the scenario where the population distribution map mixture two subpopulation distribution maps. We train classifier using the population distribution map pop. but target its perfor- mance on both the majority and minority, D Dajj and min- The distribution map is therefore misspecified due to subpopulation shift.
Population
0.8
Majority Minority
0.7
Performative
0.6
0.5
The experimental setup resembles that in (28] Note that the credit dataset used in the previ- ous experiments lacks demographic features. To enable oracle access to demographic informa- tion, synthetic data is generated for performa- tive classification task. The synthetic dataset exemplifies scenario in which linear decision boundary is not able to effectively classify both the majority and minority groups, necessitating trade-off between them.
0.4-
0
Tilt
Figure 3: Results of Experiment 5.3 Performa-
tive risk the population, the majority, and the minority, as the tilt increases. The vertical band indicates the calibrated tilt a ss.
Figure shows performative risk of the population, the majority, and the minority incurred by the TPO, as the tilt increases. The PO (which is TPO with = 0) exhibits the lowest performative risk the population, but the greatest disparity between its performance for the majority and minority groups. As a increases, the TPO reduces the performance gap between the two groups at the expense of an increased population performative risk. This suggests that the distributionally robust performative prediction framework has the potential mitigate unfairness towards the minority group, even the absence of demographic information. Using small calibration set with demographies, we can calibrate the tilt Oreal" 's via the calibration set approach described in Section 4.3 The goal is to calibrate the tilt to satisfy a four- fifth rule with minimal population performative risk. The vertical bands in Figure shows the calibrated tilt Clcal 's reasonably meet the goal.
6 Summary and Discussion
In this work, we present distributionally robust performative prediction framework that aims to improve robustness against a variety of distribution maps, thereby mitigating the problem of distribution map misspecification. We show provable guarantees for DRPO as a robust approximation to the true PO when the nominal distribution map differs from the actual one. We developed efficient algorithms for minimizing the distributionally robust performative risk. Empirical experiments are conducted to support our theoretical findings and validate the proposed algorithms.
The components of our approach are not new, but we are combining them in novel way to solve relevant problem. To be precise. the proposed approach novel in the context use the idea of distributional robustness to solve the practical problem of distribution map misspecification in performative prediction. In addition, it is novel study the solution concept of distributionally robust performative optimum (DRPO), both theoretically and algorithmically.
In Appendix [H] extend the KI. divergence distributionally robust performative prediction frame- work to a general şp-divergence distributionally robust performative prediction framework. Further- more, is possible go beyond general yp-divergence. An extension a Wasserstein DRO version is a natural direction for future research, calling for the development of new algorithms 5
Thh minority's 'ss performative risk is not 25% higher than that the majority, as motivated by four-fifth fifth nule documented Uniform Guidelines on Employment Selection Procedures, 29 C.F.R. $1607.4(D) (2015). Due space constraints, we provide additional materials for discussion (e.g.. limitations) in Appendix

---

Acknowledgements
This work was supported by the NSF under grants 21 2113364, 2113373, 2414918.
References
[1] Amir Ahmadi-Javid. Entropic value-at-risk: A new coherent risk measure. Journal of Optimiza- tion Theory and Applications, 155:1105 1055-103,,,,, 2012.
[2] Güizin Bayraksan and David Love. Data-driven stochastic programming using phi-divergences. In The operations research revolution, pages 1-19. INFORMS, 2015.
[3] Jose Blanchet and Yang Kang- Sample out-of-sample inference based on wasserstein distance. Operations Research, 69(3):985-1013, 2021.
[4] Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2))565-600) 2019.
[5] Gavin Brown, Shlomi Hod, and Iden Kalemaj. Performative prediction stateful world. In International Conference on Artificial Intelligence and Statistics, pages 6045-6061. PMLR, 2022.
[6] Ruidi Chen and Ioannis C Paschalidis. robust learning approach for regression models based on distributionally robust optimization. Journal of Machine Learning Research, 19(13), 2018.
[7] Roy Dong, Heling Zhang, and Lillian Ratliff. Approximate regions of attraction in learning with decisionnddeeedeen distributions. In International Conference on Artificial Intelligence and Statistics, pages 1112-1 184. PMLR, 2023.
[8] Dmitriy Drusvyatskiy and Lin Xiao. Stochastic optimization with decision-dependent distribu- tions. Mathematics of Operations Research, 48(2):954 998, 2023.
[9] John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 2021.
[10] Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research, 48(2):603 6565,0366566666666 2023.
[11] Jun-ya Gotoh, Michael Jong Kim, and Andrew EB Lim. Robust empirical optimization is almost the same mean-variance optimization. Operations research letters, 46(4):448-452, 2018.
[12] Vishal Gupta. Near-optimal bayesian ambiguity sets for distributionally robust optimization. Management Science, 65(9):2242 2019.
[13] Zhaolin Hu and Jeff Hong. Kullback-leible divergence constrained distributionally robust optimization. Available at Optimization Online, 1(2):9, 2013.
[14] Zachary Izzo, Lexing Ying, and James Zou. How to learn when data reacts your model: performative gradient descent. In International Conference on Machine Learning, pages 4641 4650. PMLR, 2021.
[15] Zachary Izzo, James Zou, and Lexing Ying. How to learn when data gradually reacts to your model. In International Conference on Artificial Intelligence and Statistics, pages 3998-4035, PMLR, 2022.
[16] Kaggle. Give me some credit. huteas//wwwa.aagggll.ccccc//c/es 2012.
[17] Robert Keener. Theoretical statistics: Topics for a core course. Springer Science & Business Media, 2010.
[18] Michael P Kim and Juan C Perdomo. Making decisions under outcome performativity. In 14th Innovations in Theoretical Computer Science Conference (ITCS 2023). Schloss-Dagstuhl- Leibniz Zentrum für Informatik, 2023.
10

---

[19] Anton Kleywegt, Alexander Shapiro, and Tito Homem-de de Mello. The sample average approximation method for stochastic discrete optimization. SIAM Journal on optimization, 12 (2):479-502, 2002.
[20] Henry Lam. Recovering best statistical guarantees via the empirical divergence-based distribu- tionally robust optimization. Operations Research, 67(4):109--1105 2019.
[21] Qiang and Hoi-To Wai. State dependent performative prediction with stochastic approxima- tion. In International Conference on Artificial Intelligence and Statistics, pages 3 3164-3 31663186. PMLR, 2022.
[22] Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization. arXiv preprint 20X0::2007.01662, 2020.
[23] Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. On tilted losses in machine learning: Theory and applications. Journal of Machine Learning Research, 24(142):1-79, 2023.
[24] Licong Lin and Tijana Zmic.
Plug-in performative optimization.
arXiv preprint
wrX1:22305.18788 2023.
[25] Celestine Mendler- Nendler-Düinner, Juan Perdomo, Tijana Zrnic and Moritz Hardt. Stochastic opti- mization for performative prediction. Advances in Neural Information Processing Systems, 33: 4929 499994999,000 2020.
[26] John P Miller, Juan C Perdomo, and Tijana Zrnic Outside the echo chamber: Optimizing the performative risk. In International Conference on Machine Learning, pages 7710-7720. PMLR. 2021.
[27] Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: performance guarantees and tractable reformulations. Mathemati- cal Programming. 171(1-2):115 155-):1116-16,, 2018.
[28] Liam Peet-Pare, Nidhi Hegde, and Alona Fyshe. Long term faireess for minority groups via performative distributionally robust optimization. arXiv preprint arXiv:2207.05777, 2022.
[29] Juan Perdomo, Tijana Zmnc, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learning, pages 7599-7609, PMLR, 2020.
[30] Mitas Ray, Lillian J Ratliff, Dmitriy Drusvyatskiy, and Maryam Fazel. Decision-dependen risk minimization in geometrically decaying dynamic environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8081-8088, 2022.
[31] Soroosh Sefteezzaee-Abaade Daniel Kuhn, and Peyman Mohajerin Esfahani. Regularization via mass transportation. Journal of Machine Learning Research, 20(103):1 1888 2019.
Supplementary Materials for Distributionally Robust Performative Prediction
This supplementary materials contain the omitted details, technical proofs, and additional results pertaining to the main article "Distributionally Robust Performative Prediction". Section A the missing deriving steps for the toy example in Section 2.4 are provided. In Section B we provide
characterization of the worst-case distribution map which attains the supremum (2.5). and show that the DR performative prediction regulates the right tail of the performative losses. In Section C we show a generalized excess risk bound for the DRPO. In Section [ all of the deferred proofs are presented. In Section EE] we recall a standard algorithm for performative risk minimization. In Section we explain the claim in Section 4.2 Section G we provide omitted experimental details and additional empirical results. In Section H. we generalize the KI. divergence DR performative prediction framework to general y-diivergence algorithm to find the associated DRPO. In Section provide additional materials for discussion.
performative prediction version, and propose an

---

A Toy Example in Section 2.4 (Continued)
Recall that R. + [-1, 1,1, 1). {(2; 0) 0z. and the nominal distribution map is D(0)
((f(()) for some f [-[1,11] + R and x² > 0. Then the nominal performative risk is
()(() (2==x0(( 0f())-
We firstly show that the distributionally robust performative risk is given by
(RPR()) PR(0) + Penaltyyy)))
where the penalty function Penalty (0) = v 22 10).
Proof. By Proposition 3.1 the strong duality of DRPR, and the well-established moment generating function of Gaussian distribution, we have
DRPR(0) sup D:DEU(D)
E Bz=-io,1((2;00
inf #20
{u E. D0)
+ uP]
(() 0202
inf
+ tHp
#20
2p²
020²
0f(0) + inf
+ pp
#20
2/
ff(0)
2p 10]
PR(() + vp Penalty(@).
Therefore, we derive alternative form of DRPR()), which is PR(0) with an Le-reullarizaionn term.
Now we are in a more concrete setup that ʃ(0) = 0g0 + og for some 01 40 > Recall that for any D E UDD, we denote the performative risk of D be
(0) = E ( ((z60)]
We secondly show that
sup (R5(0PO) 2 sup (Ro(ODRPO) + po? DEU(D)
DE((D)
20,
of
for any fixed p S
20
Proof. In fact, we have
-00 ODRPO
22pa - do and DRPR((DRPO)
-(ap
2pp222
OPo
2a
2a
4a
By Proposition 2.6 the generalization principle of distributionally robust performative prediction, we have
sup (Ro((ORRO)) DRPR((DRPP) -(ag v 22 DEU(D)
401
On the other hand, consider an adversarial distribution map Daav (0) = Naa + 0g One can show that DKL (Daiv (0), D()) = for all 0 E θ and therefore Dady UDD). Then
2po 2poo"o?? o²).
sup PRo(Opo) 2 PR (8po) = -(ao BEL(D)
22
pa²
4a1
201
Therefore complete the proof.
12

---

Multi-variate case. We extend the variiar example to multi-variate case. Let 2 = =R. θ = [-1, 1] and {(2; 8) A Let the distribution map be D(0) (f) Σ) for some f
[-1, 1] Fd and 2 is positive semi-definite. One can show that the performative risk is
0 = D [{(Z;0) = =Rz-warrre
and the distributionally robust performative risk is
DRPR(O) sup E 22-001) 0()) 2p0 E0 = PR(0) + Penalty (0), D.DEU(D)
(A.1)
where the penalty function Penalty(0) 220 20 penalizes the deviation of from the origin 0, and the critical radius p tunes the level of regularization. We also note that in this case the distributionally robust performative prediction problem can be transformed second-order cone constrained program (.I).
B Worst-Case Distribution Map
As byproduct of the proof of Proposition 3.1 here we provide characterization of the worst-case distribution map which attains the supremum 2.5).
Proposition B.1 (Worst-case distribution map). Suppose the problem (3.1) is solved by the unique fr" (0) > 0 for any O define distribution map satisfying the density ratio equation
dD((0) ((2:9)/w'(8)
4A e.
(B.1)
dD()) B22-iolee12227/0000
Then we have that DD is the unique worst-case distribution map, that is,
D(0)=
arg sup E [2- D(o):DEU(()
(B.2)
Proposition shows that the worst-case distribution map D is an exponentially tilted distribution map with respect the nominal distribution map D, where D puts more weights on the high ends.
Figure 4 shows histogram of performative loss for the PO, the DRPO with = 0.02, and the DRPO with م 0.04, under the setup of Experiment 5. with étrue 0.5. These plots displayed in left-to-right manner demonstrate that the DRPO regulates the right tail the performative losses. Moreover, as the radius p increases, there corresponding increase in the degree of regulation effects.
4000
1003
mea1
mean
med in suartle cuartii
media
3000
quartie
2500
quarte
quarti
2000
2500
2000
Figure 4: Histogram of performative loss under Experiment 5.1 with Ctrue 0.5. Left: histogram for the PO. Middle: histogram for the DRPO with p = 0.02. Right: hitogram for the DRPO with p 0.04.
C Generalized Excess Risk Bound
We upper bound E(PDRro) when the uncertainty collection doesn't cover the true distribution map. To be specific, we analyze the excess risk bound of the DRPO when ((Dtrru(ODDRP)) (C (((DRPO)) > p.
13

---

Proposition C.1 (Excess risk bound of the DRPO in general). Suppose that we have bounded loss function (2; 0)| for any Z, 8 E è and some B > 0. Then we have
E(pDuro) S v Var Za [e 0PO,true) + o(vp)
(C.1)
2B P=D(Pii((Osnoo))sp
inf
v D(Diime(Obrro))
Comparing (C.1 to 3.3 we have an additional term in the excess risk bound which accommodates and accounts for the misspecification of uncertainty set around (ODDRO)) which doesn' 't necessarily cover Druue (ODRP)). Furthermore, it is not difficult to see that DD ((DRPO))/D(()) then the last infimum term the upper bound of (C.I) vanishes and (C.1 reduces (3.)). Therefore, Proposition C.I provides a generalized excess risk bound for the DRPO than Proposition 3.3
S
D Deferred Proofs
For simplicity of notation, we denote the true PO by 0po, which is denoted by 0pO,true in the main article.
D.1 Proof of Proposition 3.1 and Proposition B.1
Proof of Proposition 3.7 We only have show that for any fixed 0 E O, the dual form (3.1 holds.
This follows Theorem presented [3]
We refer the readers to Proposition H.2 for a standard derivation for the dual form of a general (p-divergence distributionally robust performative risk.
Proof of Proposition B.1 This follows Proposition presented in 13 and the discussion paragraph right after the proposition.
D.2 Proof of Proposition 3.2
Proof. We have the following decomposition of the excess risk bound of the PO,
PRtrue (Opo)
min PRtrue (0) 9
PR (froo PRerue (8po)
= -PRaqee (Opo) PR(Opo) + - PR(O PR(0)0) PRtque (8
50 definition 6no
<usup 0E0
PR(() PRirie(())
sup 0-0
Ezz (((Z:)) 0)]
( ({(
<2B sup
Dry true (t) (())
Bee
SBBB sup (Pm)
The last two inequalities are due the asumption bounded loss, [[(2;0) S B, and Pinksker's inequality.
14

---

D.3 Proof of Proposition 3.3
Proof. We have the following decomposition of the excess risk bound of the DRPO,
P (fDRPO) min PRtrue (0)
HEE
P (ODDO Pi (PP)
PRtrue (fpuro) DRPR (ADRR) DRPR(ODRPO) DRPR(OPO) + DRPR((FF))
PRerue (ppo)
50 by generalization peinciple
50 by definition of ÚDrP
DRPR((Pp)) PRtrue
Varz~D(e;o)) (e; fo)lo o(VP).
The last equality is due to the sensitivity property of KL divergence-based DRO
9
D.4 Proof of Proposition C.1
Proof. For any distribution P such that (P)P((()))))))) S we have the following decompo- sition of the excess risk bound the DRPO,
PRerue(DDPP) min PRtrue 0EG
PRerue(ODRPO) PRerue (0%o)
PR (ODRP))
E Ez~p p((( [ (DRPO)] + <0 generalization principle
DRPR DRPPR(DRPOO
+ DRPR(ODRPOO DRPR((FO) DRPR( PRerue ((
50 by definition of 0DnPO
2 (DPFO) DRPR((F0) PRtrue
PR ((2 + ( Var o(v))
- De - 02000000000000222222222002010001 Varz-oion o(pp)
(PDrroPuue(OoRoo)111 + (2222 +
(Durue (PPRDDilP + (22 +
By the arbitrariness of in the divergence hall (pu(()[((OOPOO)) S p. we complete the proof.
E Performative Risk Minimization
We recall standard algorithm for performative risk minimization when the nominal distribution map
D(-) is modeled by a response map To 2 - 2 and samples from the base distribution Derue (O):
Z ~ D(0) = z To where Zo (O).
This is popular model for distribution map including strategic classification (see Example 2.1 and location family (see Example 2.2p as prominent examples.
It is not hard To and the base measure Durue because
that the nominal distribution map D(.) is fully characterized by the response map
D(0) Top (Pruuo for any 0 E O
that is, the measure D(0) is the pushforward measure of Dirue under the transport map To. The performative risk can then be reformulated
PR(0) E (2;0) (Pz (0) |8 (Z; 0)) Ez (TTo))
With the last formula, the performative risk minimization problem becomes standard (possibly nonconvex) stochastic optimization problem, which can be solved efficiently by any popular stochastic
15

---

optimization algorithm, e.g., sample average approximation (SAA, [1].. given the loss function {(2; 8) and the response map (2) are sufficiently differentiable with respect
We refer to Section 1.1 for a summary of existing performative risk minimization algorithms.
F An Explanation to the Claim in Section 4.2
In Section 4.2 claim that arg mingee (1²(0, n)) x arg mingee (E (((; as Jr oo. This is due to the fact of that e ~ x + for small x > 0 and the argmax/argmin theorem. Therefore, have
arg gee
{ log (o)
(e((2:0)//)
+ Hp}
[((2:))
-1 1 up up} 1
arg ge
p plog Bzw
arg millage (Ez~D(0) ((2;0))
as μ + 00.
G Experimental Details and More Results
For the loss function (xx, yı 0), we adopt the cross-entropy loss with L--reuularizztion, that is,
{(x, y: 0) = - log ho - (1 - -) log(1 ho)))) + 11011.
where ho(x) (1 + exp(-0- =
and À 0.001. Given model 6. predict ① ((ha(z) 2 0.5). In addition, each replicate of the experiments is run on a local machine with an Intel Xeon Gold 6154 CPU and 32GB of RAM less than an hour execution time.
G.1 Strategic Classification with Misspecified Cost Function in Section 5.1 (Continued)
The data preprocessing procedure for the credit dataset 16 follows the procedure documented in 29 After that procedure, the base distribution D Dirue (0) has n 14878 data points with equal probability mass. We treat the distribution map associated with Druee (0) as the underlying test distribution map which is unknown us. We generate IID samples from Dirue (0) to get training base distribution D(0), that is, ((0)= ~ Dirue (0) and then we have a training nominal distribution map induced by D(0). This training procedure is repeated for 10 trials, and the shaded region in each of the figures in this paper represents the standard error of the mean calculated from the 10 trials for the corresponding curve.
The left of Figure 5ishows performative balanced error rate, which refers the balanced error rate (BER) on the test model-induced distribution, incurred by the PO and the DRPO's 's with various radius p's. Because the cross-entropy loss serves as surrogate for classification error, we see patterns of these curves similar to those in the left of Figure [ increases, the DRPO achieves more uniform performance across wider range of étrue- On the other hand, because the performative classification error is not exactly the criterion we minimize, different patterns are also observed: the DRPO with relatively large radius p 0.04 outperforms the PO for all ctree E [0. 1]. As indicated by the right of Figure 5 by increasing the DRPO constantly improves the relative worst-case performance in terms of performative BER for 7 E (0.6, 0.8. 1.0). Here the relative worst-case improvement in performative BER of 0 to Ppo is defined by max. BER -max (FR x 100%.
max. BER
For implementing the post-fitting calibration procedure in Section 4.3 we have to estimate the KL divergence between (O (p)) and ((PDRPO(p)) for any fixed Here we index ODRPO by the radius p emphasize its dependence p. Recall that
Z ~ D(0) + _ 1 Zo - 0.5A0 and ~ (0)
2 d - where Zo ~ Derue (0)
Here AT = [BT Oa] (see Section 5.2 for the definition of B) and we have training base distribution D(0) from Derue (0). We use inexact parametric method to estimate the KL divergence. Assume
16

---

Performative Balanced Error Rate
PO
Relative Werst-Cade Improvement
17.5%
0.6
0.34
DRPO with p= 0.01
15.0%
0.8
p
n 1.0
0.33
p=
12.5%
0.32
P
10.0%
0.31
5.0%
2.5%
0.29
0.0%
0.0
0.2
0.4
0.6
1.0
0.00
0.01
0.02
0.03
0.04
True Performativity Level
Radius
Figure 5: Additional results of Experiment 5.1 Left: performative balanced error rate incurred by the PO and the DRPO's with various radius p's. Right: relative improvement in worst-case performative balanced error rate of the DRPO to the PO as the radius p increases, for different range of misspecification 1y's.
Dtrue (0) (m, E), then
(DDrruel (D) D( 0.5A0DRPO 2)] - Eure AU E))
0.5(Etrue 0.5) 2-1 A-Dro-
Let E be the sample covariance of D(0), plug-in method implies estimate of the KL divergence, which is given by
(0Doee(obxro())))0000reeeee) 0.5 - 0 T--1 A-DDOO.
G.2 Partially Identifiable Distribution Map in Section 5.2 (Continued)
The data generating procedure is the same that in Experiment 5.1 The left of Figure 6 shows performative balanced error rate (BER) incurred by the PO and the TPO's with various tilt a's. increases, the TPO achieves
Similar the left of Figure performance across a wider range of Grue' 2) the TPO with relatively large tilt = 0.5 outperforms the PO for all Écrue [[1,1] The right of Figure 6 shows relative improvement in worst-case performative balanced error rate of the TPO to the PO as the tilt a increases, for different range of misspecification the PO in terms of performative BER. With moderate to large misspecification, the TPO shows significant advantages over the PO terms of relative improvement worst-case performative BER.
observe that 1)
uniform
Without misspeicification, 1/ O, the TPO has comparable performance {2/3, 4/3, 2}.
Worst- improvementt
14.0%
Perfomattiv Balanced Broor Rate
0.42
n=
TPO with = 0.1
12.0%
2/3
0.40
TPO with a 0.2
TPO
10.0%
2
4/3
TPO with a= 0.4
8.0%
TPO with == 0.5
6.0%
0.36
0.34
2.0%
0.32
0.0%
-1.0
-0.5
0.0
0.5
1.0
Performativity Level Suve
Tilt
Figure 6: Additional results of Experiment 5.2 Left: performative balanced error rate incurred by the PO and the TPO's 's with various tilt a's. Right: relative improvement in worst-case performative balanced emorr rate of the TPO to the PO as the tilt a increases, for different range of misspecification s.
17

---

G.3 Fairness without Demographies in Section 5.3 (Continued)
We adopt the following data generating process similar to that in 28 Let X ~ Nu Ea) )+ (1 7)N (un, EB). Let 0.8 so that group A is the majority group and group B is the minority group. Let Aud
ld μa = 0.8 las and Σn = En 0.1 Ia. If X comes from group A, then label Y = 0f 1 S MA La If X comes from group B, then label Y 0 X I 1gg The distribution map follows:
X1:1d/2] X1:14/21 - +01:12/21
X (Ld/2]++1)d (Ld/21+1))d
so that the first [d/2] features are strategic features, and controls the strength of performativity. We choose = 10 and 0.5 wrap up the setup. Finally, we assume knowledge of the true performativity and observe IID samples of size 12500 from the population base distribution. In short, we eliminate the effect of population distribution map misspecification, and instead concentrate on the effect of subpopulation distribution map shift.
Figure 7 shows the performative accuracy, which refers the accuracy the model-induced distribution, of the population, the majority, and the minority incurred by the TPO, as the tilt a increases. Because the cross-entropy loss serves as a surrogate for classification error, we see patterns of the three curves similar to those in Figure 3 The PO (which is TPO with = 0) exhibits the highest performative accuracy the population, but the greatest disparity between its performance for the majority and minority groups As a increases, the TPO reduces the performance gap between the two groups the expense of an decreased population performative accuracy. This suggests that the distributionally robust performative prediction framework has the potential to mitigate unfairness towards the minority group. even in the absence of demographic information.
0.85
Performetive Accuracy
0.80
Population
0.70
Majority
Minority
0.65
0.60
0.55
7
Tilt
Figure 7: Additional results of Experiment 5.3 Performative accuracy of the population, the majority,
and the minority, the tilt a increases.
we have to obtain access to a few samples with the group membership information. Here we allow ourselves to observe IID samples of size Neal 125 with the known group membership from the population base distribution.
For implementing the calibration set procedure in Section 4.3
H Extension to General g-Divergence
The KL divergence distributionally robust performative prediction framework can be extended to general ip-divergence distributionally robust performative prediction framework. Recall that spdiivergence defined by
(Q11)) /
dP.
dP
where R, + R, and ((1) O. Here dQ/ is the Radon -ikkodmm derivative, and we implicitly require the probability measure Q to be absolutely continuous with respect to P. Note that if we choose (t))= log l, then we recover the framework presented in the main article. Now we keep ip as generic function and we will instantiate some popular families of e-divergence after the presentation of the general framework.
18

---

With the ip-divergence, we can define family of distribution maps around the nominal distribution map. Specifically, the uncertainty collection around D with critical radius p defined
u(D) = ( P(2)) D (,((())))((O) p. e).
Definition H.1 (e--iivereence distributionally robust performative risk), The y-divergence distribu- tionally robust performative risk with the uncertainty collection (DD) is defined
DRPR,(())
sup D:DEU(D)
Ez- [e 8).
(H.1)
The evaluation of şe-divergence distributionally robust performative risk DRPR,, given in involves an infinite dimensional maximization problem which is generally intractable. Fortunately, in fact equivalent to minimization problem over two dual variables. This is given by the following strong duality result.
H.
Proposition H.2 (Strong duality DRPR.). For any 0 E O, we have
EEwDion p (12:0)-))
-
+ up / }
DRPR = inf
(H.2)
#20,vER
where (s)(s) sup, {st - (tt)} is the convex conjugate of
Proof. We only have to show that for any fixed 0 E θ, the dual form (H.2) holds. We introduce
the likelihood ratio L(Z) = dD(()) (O)) By change of variable, we can rewrite the g-divergence distributionally robust performative risk (H.I)
sup ((0)
E 2N~ [[:
sup [BZ-D() (TZ) (3-D(0) ((()))))((((())))) S p, Bz>D(o) [T(Z)] =1], L20
where the supremum takes over measurable functions. This gives us a constrained optimization problem. Let y 2 be the Lagrange multiplier for (E)DD()) (Z))] I and IK be the Lagrange multiplier for Ez~r(0) = The corresponding Lagrangian is
((,, EZ~D(O) ( (10))(())(((((())) + pp
For regular yp- d-veeeeeeeee, we have
E 2z [[[Z;)]]
sup (D10I((())
inf sup ((I.
a20,ERR L20
inf sup -1Z Hp((L(Z) + +up+
"20,vER 1.20
((2:0) - L(Z) - v(((2)) + Hott
inf sup Ez-D(O) (
#20,eER L20
1
((Z:0) V 1000-()) } + up unttup o
inf #20.vER
ENzD(O)
usup
11
(((Z:0) v) up +1ppt t B
tu
inf a20,ER
EznD(0)
ju0"
Here the last equality holds according to the definition of the convex conjugate :p" (-)).
To develop an algorithm for finding the DRPO, we introduce the (n, v) -uuugmeeee performative risk.
Definition H.3 ((1, ((u.v))augmeeted performative risk). The (μ, v) auggeetedd performative risk is defined by
AugPR. (0, p, v) EEzDD(0) uu"
+ up +
(H.3)
19

---

Algorithm 3 Augmented Performative Risk Minimization
1: Input: nominal distribution map D(0)
2: Update (0, < arg mi0eE (EzNDIo) |n0* (112)0)=1) + up + u}}
3: Return:
From (H.2) and H.3), J. we see that the minimization problem of DRPR, (f) over 0 is equivalent to
the minimization problem of A, (0, μt, v)
jointly over (0, pr, v), which is itself performative
risk minimization problem. We summarize this procedure in Algorithm 3
For special choice of 4, the strong duality result (H.2) can be reduced to involving only a single dual variable. Now we instantiate --divergence as the Cressie-Ress family:
1h kt
(x(1)
k:A 1)
for k 1. Like KL divergence, a distributionally robust performative prediction problem induced by a divergence from the Cressie-Read family has a dual reformualtion with single dual variable:
DRPR (0) = inf { pk(k 1)) [max ((Z; μs. 03*+] 1/k- + 1 #20
(H.4)
where k2 is the conjugate number of & such that 1/k + 1/k, = 1. Therefore, an algorithm parallel to Algirthml I can be developed in similar fashion based on the single-variable dual form (H.4).
As final remark, all of the theoretical results regarding excess risk bounds (see Proposition 3.3
and Proposition C.I) are still valid for the general y-diiergencce distributionally robust performative prediction (which means the result statements won't change if one replaced the KL-divergence by any g-divergence).
On the other hand, it is possible extend Proposition 3.2 to general y-diivergence, and the result statement needs a slight modification. By generalized Pinsker's inequality, there is an increasing function F [0.2 2] + R+ such that (P( 2 F(TV(PIIQ)) Then for general yp-divergence. Proposition 3.2 can be modified to {(po)) I 2B supsge F (D (O) or E(fro) S 2BF (supace (Druue (e)(()))))(( due to monotonicity of F(-). For a concrete example, F(v)
2-v (u 1} when yz()= = (L 1)2.
I Additional Discussion
In this appendix, we provide additional materials for discussion.
Extension to Wasserstein distance. is possible to use Wasserstein distance define the uncertainty collection within framework. Our algorithms can be modified to compute Wasserstein DRPO. For example, one can establish the strong duality of Wasserstein DRPO and develop an alternating minimization algorithm similar to Algorithm[] However, the new algorithm involves an additional step of transport cost-regularized loss maximization due to the more complex dual form Wasserstein DRO. For the corresponding theory, we expect that the square root of variance in 3.3 would be replaced by the Lipschitz norm of É(,, (ro,true).
Conservativeness of Peal and trade-off in selecting p. The main text covers the discussion of the conservativeness of Peal in two ways. Firstly, we show the trade-off selecting p. For values of p ranging from small to moderate, DRPO outperforms PO in terms of performative risk (and similarly for worst-case performative risk). Conversely, for large values of p, PO is better than DRPO. There exists an "sweet spot" of p where DRPO yields maximal benefits over PO. This trade-off between DRPO and PO is demonstrated in any "vertical slices" of the left plot of Figure [ (and similarly in the lines of the middle plot of Figure | for worst-case performance). Secondly, show the performance of the calibrated radius Pcal in the middle plot of Figure T where the vertical bands indicate the calibrated radius Peal- Although Peal doesn't 't achieve the best possible worst-case improvement (which an impossible oracle), it achieves a comparable performance, especially when v) E (0.8, 1.0). This demonstrates the effectiveness of Peat chosen by calibration method.
Difference between TPO and TERM, and difference between DRPO and simple DRO. TPO (see Section 4.2 and Algorithm 2 differs from TERM [22] in that it takes into account performativity,
20

---

whereas TERM ignores. To be precise, TPO minimizes Ez~D(o) [e00 while TERM minimizes (cadt where D(0) is the distribution map and ( is the base distribution. An analogous explanation applies to the difference between DRPO and simple DRO. TERM (or implicitly equivalently simple DRO) is ineffective the context of this work's problem setup because it fails to account for performativity. Note that our methods are not doing DRO because the uncertainty set here depends on the model parameter A. We borrow the idea of distributional robustness, but we have a fundamentalyy different problem at hand.
Absence of small calibration set. The calibration set is not always available. Here clarify the calibration set method and briefly discuss a possible solution when there is no calibration set. In the experiment of Section 5.3 only need small set of calibration data, which aligns with the regime of "weak group information". is generally difficult to calibrate radius for uncertainty set and most distributional robustness related work only concerns the effect of increasing and p. We do more to demonstrate that there are some practical calibration methods that work. In the absence of any calibration data, can specify the radius by some prior belief. For example, consider Ppop (0) yD Dmajor (0) + (1 D)miinoo (f) as in Example 2.3 If one believes that V1 S S 72 for some 71, 72 then one can upper bound the divergence jor (6) PP po (0) S log(?)) and D(Dminor (0) pop (f)) < - llog(1 - 72). Further, one can choose = max mmaxx-log(71))) log(?1). - log(1 } when using DRPO. However, this choice of radius could be conservative.
Algorithmic convergence. The convergence guarantees of the proposed algorithms can be established on case-by-case basis, depending the specific "inner" algorithm solver employed for solving performative risk minimization. Existing algorithmic convergence results for model-based PO solvers 26 [4 be taped into our algorithms naturally. Moreover, empirical results in Section 5 validate the effectiveness of our proposed algorithms.
Broader impacts. This paper presents work whose goal is to advance the field of machine learning, especially the subfield of performative prediction. Despite the development of new algorithms, their direct socictal impact are limited to those outlined in the original paper performative prediction
29
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's ' contributions and scope. The structure of the work adheres to method, theory, algorithms, and experiments, which correspond to the contributions claimed in the abstract and introduction.
Justification:
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the work in Section and Appendix
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper provide the full of assumptions and a complete and correct proof following the guidelines. All proofs are deferred to the Appendix.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sufficient details are provided in Section 5 and Appendix G
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The data is modified from public dataset (16] generated sythetically (see
details in Section 5 and Appendix and the code is available upon request.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setting and details are provided in Section 5ndd Appendix G
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We report the standard error of the mean, as clarified in Section
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
22

---

Justification: The computer resources are provided in Appendix G
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The authors of the work have thoroughly reviewed the NeurlPS Code of Ethics.
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss broader impacts of the work in Appendix
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Licenses existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators of the existing dataset are properly credited in Section 5.1 and in the reference 16).
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
23


Reviews:
[{"review_id": "6HQD6scgtP", "reviewer": "Reviewer_bCrg", "summary": "Summary: The paper revisits Performative Prediction, a setting proposed in 2020 where the chosen model used to minimize a prediction loss also induces a distribution shift in the data through a distribution map. When the true distribution map is unknown, the learner must use a nominal map to approximate the change of distribution resulting from the chosen model, but this may result in large errors. Using a distributionally robust formulation of the problem, they propose a way to guarantee that the learnt performative optimum has bounded error (if the chosen radius is well calibrated). \n\nOverall, I believe the paper proposes good ideas and could be accepted. I have a few comments below, as well as some more minor ones about the writing. I will consider revising my score after the rebuttal.", "questions": "What are the pros and cons of such approach? How general is it when the family of distribution maps is more complex than a linear function? When should one use this approach?", "limitations": "Experiments are on semi-simulations and use very simple model. \n\nThere is little discussion of the challenges of this DR approach when distribution maps are more complex (non-linear), high-dimensional.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Sound theoretical setting and analysis. \n\nClassic but elegant algorithmic approach with practical extensions.\n\nEmpirical evaluation of all proposed methods.", "weaknesses": "Lack of discussions of the trade-offs / pros and cons. \n\n\"Toy\" experiments on simple models.\n\n--\n\n\nMajor remarks:\n\n* Overall, the paper is not too hard to follow despite a number of slightly awkward writing issues that I’ve listed further below. However I am missing a real discussion section that puts into perspective the chosen approach and the obtained result. I would like to see a conclusion with a take-home message. I think the problem of not knowing the true distribution map is important but the distributionally robust approach chosen here is only one way to resolve it that focuses on worst-case guarantees. I think this is best shown in the experiment section but there are not many comments on it. For now the “summary and discussion” is essentially a summary with some pointers to extra results in appendix. What are the pros and cons of such approach? How general is it when the family of distribution maps is more complex than a linear function? When should one use this approach? \n\n* Strategic classification example: Example 2.1 is a bit ill-defined because neither u nor c depend on \\theta. In Section 5, it is a bit clearer as it is instantiated on a simple example (perhaps overly simple, is that common in this literature?). In the end the Kaggle data you cite is only used to estimate a base distribution but then all data is simulated, right? Figure 1 (left) shows that for a large interval of values of \\epsilon_{true}, the performative risk obtained with the nominal map is actual lower than the distributional risk. I feel like the comment on this result l. 300-302 could be a bit expanded and insist on why this is typical for DR approaches. \n\n\n\nMinor remarks:\n\n* Your introduction clearly went through a beautiful GPT pass: “Delving deeper into the formulations of performative prediction, the concept of a distribution map emerges as pivotal.” While I generally don’t mind GPT assistance, I think this sentence does not carry much meaning. I think your introduction fails to really convey the motivation for distributionally robust guarantees, and how they differ from other existing guarantees. The sentences are complex and convoluted (e.g. “Practically, the precise influence of a model on the data ecosystem is intricate and dynamic, making perfect specification an unattainable ideal.”). I’d recommend making a pass sentence by sentence and making sure that everything is said efficiently and simply.\n\n* Related work: “[28] study the repeated distributionally robust optimization algorithm, which repeatedly minimizes the distributionally robust risk at the induced distribution.” -> repetition, use \\citet\n\n* “The radius ρ reflects the the magnitude of shifts”-> double the\n\n* “The dual reformulation (3.1) will be served as the cornerstone of developing algorithms” -> will serve? also “cornerstone of developing”? not sure about this sentence either\n\n* Section 5 did not go through the same scrutiny of spell checks / GPT: l 315: “indentified”, l.324: ”estiamte”\n\n* Please correctly use \\citet and \\citep everywhere, cf https://medium.com/@vincefort/phd-lessons-part2-ce830329c86f for guidelines, copied below for your convenience:\n“Once you have an entry in your .bib file for a paper that you want to cite, say the key is author2022paper, you have two major ways of citing it: within the text or as a parenthesis. You can set the within-text citation as “Following \\citet{author2022paper}, we do …” and it will render as “Following Author et al. (2022), we do …”. The parenthetical citation is set as “… this has been shown in prior work \\citep{author2022paper}” and renders as “… this has been shown in prior work (Author et al. 2022)”. Make sure to not confuse the two types, as it can be a bit confusing for the reader and impair the flow.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We are glad to know that the reviewer appreciated our work from multiple aspects. We address the questions and concerns below, and we look forward to interacting with the reviewer during the discussion period.\n\n> What are the pros and cons of such approach? How general is it when the family of distribution maps is more complex than a linear function? When should one use this approach?\n\nThe cons of our approach is additional computational cost. Because the true distribution map is unknown, the learner must rely on a nominal distribution map estimated from data, resulting in modeling or estimation errors. Due to this reason, as long as the additional computational cost is affordable, one should always use our approach, as the pros of our approach is that it is robust to distribution map misspecification. The linear distribution map is a standard setting that people consider in the research area of performative prediction (see simulations in Section 5.2 of [Perdomo et al. (2020)](https://arxiv.org/pdf/2002.06673)). However, our approach continues to work for more complex distribution maps.\n\n\n> Experiments are on semi-simulations and use very simple model. Is that common in this literature?\n\nYes, that is common in the literature. Our experiments align with the established convention in performative prediction research area, which utilizes synthetic/semi-synthetic data and simple model with almost no exceptions. A semi-synthetic dataset is often a real-world data simulator, which comprises real data as the base distribution and a synthetic performativity setup. The usage of semi-synthetic data is common in the literature because it is difficult to come by a dataset to learn the real distribution map.\n\n> There is little discussion of the challenges of this DR approach when distribution maps are more complex (non-linear), high-dimensional.\n\nThe theory and algorithms of our DR approach extend to the non-linear and high-dimensional cases. Conceptually, when dealing with non-linear and high-dimensional distribution maps, our approach can provide more benefits because distribution map estimation errors can be larger.\n\n> Example 2.1 is a bit ill-defined because neither u nor c depend on \\theta. In Section 5, it is a bit clearer as it is instantiated on a simple example.\n\nWe will change $u(x)$ to $u_{\\theta}(x)$ to emphasize that the utility function depends on the model parameter. As the reviewer pointed out, this is actually instantiated in an example in Section 5 (particularly Subsection 5.1).\n\n> In the end the Kaggle data you cite is only used to estimate a base distribution but then all data is simulated, right?\n\nThe reviewer is correct that the data is semi-synthetic: the base distribution comes from the Kaggle data, whereas the performativity setup is synthetic. \n\n> Figure 1 (left) shows that for a large interval of values of \\epsilon_{true}, the performative risk obtained with the nominal map is actual lower than the distributional risk. I feel like the comment on this result l.\n\nFigure 1 (left) shows the curves for some discrete values of $\\rho$'s. We didn't show the result of an \"infinitesimal\" $\\rho$ (i.e. a $\\rho$ extremely close to $0$), of which the performative risk curve should be \"almost surely\" slightly lower than that obtained with the nominal map, except for the point at $\\epsilon_{\\operatorname{true}} = 0.5$. Now let's consider a continuous spectrum of $\\rho$'s. For values of $\\rho$ ranging from small to moderate, DRPO outperforms PO in terms of performative risk (and similarly for worst-case performative risk). Conversely, for large values of $\\rho$, PO is better than DRPO. There exists an “sweet spot” of $\\rho$ where DRPO yields maximal benefits over PO. This trade-off between DRPO and PO is demonstrated in any “vertical slices” of the left plot of Figure 1 (and similarly in the lines of the middle plot of Figure 1 for worst-case performance).\n\n> 300-302 could be a bit expanded and insist on why this is typical for DR approaches.\n\nAs $\\rho$ increases, the DRPO aims to achieve low worst-case performative risk over a wider range of $\\epsilon_{\\operatorname{true}}$. This requires a trade-off between regions of $\\epsilon_{\\operatorname{true}}$ with low and high performative risk. Figure 1 (left) shows that as $\\rho$ increases, the DRPO achieves more uniform performance (\"flatter\" performative risk curve) across a wider range of $\\epsilon_{\\operatorname{true}}$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDaOUbhvPK", "reviewer": "Reviewer_Zy5f", "summary": "The paper proposes a framework that applies distributionally robust optimization to optimizing performative to obtain the distributionally robust performative optimum (DRPO). Specifically, the framework optimizes the worst case performative risk over the uncertainty set of distribution maps. The paper presents theoretical guarantees that bound the excess risk the approach as well as the benchmark approach. The paper also proposes tractable optimization algorithms for solving the distributionally robust problem and presents numerical simulations demonstrating the benefits of the proposed method.", "questions": "1. The paper proposes two algorithms for the distributionally robust framework. Which algorithm is used in each of numeric examples and are there benefits using one over the other? Or is one algorithm just a special case of the other? \n2. Can more insight be shed on the differences between the two excess risk bounds (prop 3.2 and prop 3.3)? In what settings are the two bounds and subsequently the excess risk significantly different?", "limitations": "The authors adequately addressed limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper outlines a clear approach for applying distributionally robust optimization to the performative prediction/performative risk optimization setting. The descriptions of how to implement the framework well written. They also demonstrate how to obtain bounds on the excess performative risk to show the distributionally robust approach can perform as well as existing performative optimum approaches. The examples that were followed up upon in the numerics provided motivating examples of when applying the distributionally robust approach would be sensible.", "weaknesses": "The paper presents the distributionally robust optimization approach in an idealized setting where the form and structure of the distribution map is well defined and only a few key parameters need to be estimated. It would be more helpful to see how to arrive at this idealized setting from raw data. Overall, the paper could include more details about performative prediction to help readers understand how their approach fits into the bigger picture.\n\nThe paper's numerics and theoretical guarantees also do not allow readers to get a clear idea on how and if the distributionally robust approach improves excess risk. The theoretical guarantees while useful do not seem to necessarily be tight, so the improvement on the rates of the bounds does guarantee better empirical performance. Additionally, the numerics do not seem to study the excess risk metric (at least in the main body) which makes it hard to verify tightness of the theoretical guarantees and understand the practical benefits of the distributionally robust approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We address the questions and concerns below, and we look forward to interacting with the reviewer during the discussion period.\n\n> Connection between numerics and theoretical guarantees.\n\nThe numerical experiments and theoretical findings support the benefits of using DRPO over PO from different aspects. The theoretical findings demonstrate DRPO’s potential to outperform PO in achieving lower performative risk at a single distribution map. The numerical experiments show DRPO’s capacity to attain a more favorable worst-case performative risk across a collection of distribution maps, in comparison to PO.\n\n> The paper proposes two algorithms for the distributionally robust framework. Which algorithm is used in each of numeric examples and are there benefits using one over the other? Or is one algorithm just a special case of the other?\n\nThe first experiment uses Algorithm 1 to find DRPO, while the second and third experiments use Algorithm 2 to find TPO. While both DRPO and TPO certify a certain level of robustness to misspecification of distribution maps, each has its own benefits over the other. On the one hand, DRPO exactly accounts for the uncertainty set size, whereas TPO does not. On the other hand, TPO is computationally more efficient, whereas DRPO requires more run time. The two algorithms are not special cases of each other. Their relationship should be understood as that the tilted performative risk minimization implicitly solves the corresponding DR performative risk minimization problem, and there is an implicit correspondence between the solution path $\\theta_{\\operatorname{DRPO}}(\\rho)$ and $\\theta_{\\operatorname{TPO}}(\\alpha)$.\n\n> Tightness of the theoretical guarantees? Can more insight be shed on the differences between the two excess risk bounds (prop 3.2 and prop 3.3)? In what settings are the two bounds and subsequently the excess risk significantly different?\n\nThe excess risk bound for DRPO (3.3) is sharp in the asymptotic regime $\\rho \\to 0$. This is a consequence of the sensitivity property of KL divergence-based DRO. The excess risk bound for PO (3.2) is sharp in its rate, but the constant is likely loose. The key insight from the two bounds is the excess risk for DRPO has a localization property (the leading term of the excess risk vanishes even for fixed $\\rho$) which is not shared by the excess risk of PO. This suggests that the DRPO bound is sharper than the PO bound.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NFVbIZl9AN", "reviewer": "Reviewer_UHwo", "summary": "The paper addresses the performative prediction problem wherein the deployment of a model leads to a shift in the true data distribution via a distribution-shift map. In standard performative prediction, the distribution map is unknown - it is either assumed to be of a simple form like a location-scale shifting map, or estimated via a few deployments of models to see how the data is reacting. This paper proposes a distributionally robust performative prediction setting, where the objective is to minimize the distributionally robust performative risk (DRPR) which is the worst-cases performative risk over a set of distribution maps that contains the true distribution map. The uncertainty set is defined using KL divergence, and standard results from distributionally robust optimization (DRO) are used to obtain a dual form for DRPR. The paper then proposes to optimize the dual form of DRPR using an alternative minimization approach. The paper also proposes a simple calibration procedure to set the radius of uncertainty appropriately. The paper also proposes \"tilted\" performative risk minimization akin to the recently proposed tilted ERM (TERM). The proposed approach is validated via simulation on simple settings such as strategic classification with misspecified cost function, perfirmative prediction where the true distribution map is a linear location shifting functional.", "questions": "- Can you try to extend the result on regularizing effect of DRPR beyond the toy example in section 2? It seems like this should be possible. \n- What is the effect of setting the parameter in tilted performative risk minimization to a negative value? It will be nice to compare this setting to the corresponding setting in TERM.", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The framework of distributionally robust performative prediction is sound, and deserves attention. \n- The presentation is clear for the most part. \n- The regularizing effect of DRPR is, although not surprising, good to see.\n- The connection made with TERM, and the proposed tilted performative risk minimization is interesting, and deserves further exploration. \n- The experiments make a good case for the usefulness of the proposed approach.", "weaknesses": "- The theoretical results follow straightforwardly from existing results on distributionally robust optimization (DRO). The novelty is simply in bringing together the DRO framework and performative prediction.\n- One of the steps in the proposed alternative minimization algorithm for DRPR itself is a performative risk minimization problem, which needs multiple deployments to solve. This means that the proposed DRPR framework needs even more deployments than standard performative risk minimization - it seems unrealistic for such an approach to work in a \"real-world\" setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We are glad to know that the reviewer appreciated our work from multiple aspects. We address the questions and concerns below, and we look forward to interacting with the reviewer during the discussion period.\n\n> The novelty is simply in bringing together the DRO framework and performative prediction.\n\nThe components of our approach are not new, but we are combining them in a novel way to solve a relevant problem. The novelty has been discussed in Section 6 between L371 and L375.\n\n> One of the steps in the proposed alternative minimization algorithm for DRPR itself is a performative risk minimization problem, which needs multiple deployments to solve.\n\nIt depends on how one models the distribution map. When the distribution map is properly modeled (see the pushforward in Appendix E between L540 and L552), DRPR minimization doesn't require more deployments than standard PR minimization. For the most general case of distribution map modeling, Algorithm 1 needs additional cost of deployments.\n\n> Can you try to extend the result on regularizing effect of DRPR beyond the toy example in section 2? It seems like this should be possible.\n\nWe extend the uni-variate example in Section 2 to its multi-variate case in Appendix A between L487 and L494. Regarding the general form of the loss function and distribution map, it is not expected that there will be a concise and elegant closed form formula, similar to the one found in the toy example.\n\n> What is the effect of setting the parameter in tilted performative risk minimization to a negative value? It will be nice to compare this setting to the corresponding setting in TERM.\n\nA negative tilt parameter in TERM suppresses the hard examples (the samples with high loss values) by assigning them less weight. When interpreting the hard examples as outliers, TERM with a negative tilt parameter is outlier-robust because it ignores the outliers to some degree. Similarly, tilted performative risk minimization with a negative tilt parameter can be viewed as an outlier-robust performative risk minimization approach. Unlike TERM, tilted performative risk minimization considers performativity. As a result, tilted performative risk minimization with a negative tilt parameter can be used when the distribution map (a family of distributions) is contaminated by a portion of outliers, but not just a single distribution. A negative tilt parameter, on the other hand, breaks the connection between the distributionally robust and tilted performative predictions. Because distributionally robust performative risk minimization focuses on hard examples but not ignoring them. In theory, the duality results necessitate a tilt parameter that is non-negative.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AucDRP4jJ6", "reviewer": "Reviewer_67u7", "summary": "In this paper, the authors propose a new framework and algorithm for finding performatively optimal solutions in the performative prediction framework. Performative optima are in general hard to find, and to do so, a number of approaches start by trying to estimate the underlying distribution map: a task which incurs fundamental statistical or modeling error. The main contribution of the paper is that the authors bring to bear tools from the robust optimization literature to find solutions which explicitly account for this error and are provably robust to it. The main technical results of the paper are structural theorems bounding the performance of the distributional robust performative optima, and guidelines around the selection of hyperparameters for their algorithm. \n\nDisclosure: I previously reviewed this paper for ICML and I was disappointed that it was not accepted to the conference then. I thought it was a very strong paper back then and I think the authors have done an even better job now.", "questions": "“L77: it is easy to see..” Aren’t there cases where the optimization problem is still computationally, if not statistically, intractable?\n\nAssume that PR(theta) = E_{z~ D(theta)} ell(z, theta) is a tractable optimization problem. I understand that the mapping x —> exp(x) is a monotonic transformation, but could you provide a formal statement showing why one would still be able to tractably solve this modified version of the problem, assuming that the first version is tractable?\n\nCould you spell out in more detail exactly what the grid search is over when you state that L270 “for general problems, these calibration methods necessitate a grid search….”", "limitations": "Yes, these have been discussed to the extend necessary.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The motivation and ideas behind the paper are excellent. I think it’s an extremely natural solution to think about robust optimization approaches for finding a performatively optimal solution. The author provide very clear and insightful examples showing the value of their ideas and I’m glad they’ve established a bridge between these previously disparate areas of ML research. I think this paper is a significant contribution to the growing literature on performative prediction and will lead others to build on their work. The experimental section of the paper was particularly well executed and nicely rounds out the paper.", "weaknesses": "The paper has very few weaknesses, but to be somewhat nitpicky, I think that the authors would do well to clarify the computational complexity behind some of their algorithms and make precise statements regarding when and why the robust optimization problem can be solved, assuming that the non-robust (nominal) performative optimization problem is tractable. Please see below for further comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We are glad to know that the reviewer appreciated our work from multiple aspects. We address the questions below, and we look forward to interacting with the reviewer during the discussion period.\n\n> “L77: it is easy to see..” Aren’t there cases where the optimization problem is still computationally, if not statistically, intractable?\n\nThe reviewer is correct that even when the optimization problem is statistically tractable, it can still be computationally intractable. The recipe laid out in L77-79 relies on knowledge of the distribution map and an argmin oracle. Without the argmin oracle, it is often still possible to obtain a local minimum (with say first-order methods). We will clarify this in a revised version of the manuscript. \n\n> Could you provide a formal statement showing why one would still be able to tractably solve this modified version of the problem, assuming that the first version is tractable?\n\nTo our understanding, the reviewer refers to PO as the first version of the problem, and DRPO as the modified version of the problem. As long as it is possible to at least solve the PO problem, then Algorithm 1 provides a way to reduce solving the DRPO problem to solving a sequence of PO problems.\n\n> Could you spell out in more detail exactly what the grid search is over when you state that L270 “for general problems, these calibration methods necessitate a grid search….”\n\nFor the second calibration method \"calibration set\", the grid search performs the three steps listed between L265 and L268. For the first calibration method \"post-fitting calibration\", the grid search executes the following procedure according to the equations between L258 and L259: for a sorted candidate set of $\\rho$'s, $\\rho_1 < \\rho_2 < \\ldots < \\rho_m$, we try $\\rho_i~(1\\leq i \\leq m)$ from small to large and return the first $\\rho_i$ such that $\\max\\_{\\mathcal{D}\\_{\\operatorname{true}} \\in \\Xi} \\widehat{D}(\\mathcal{D}\\_{\\operatorname{true}}(\\theta\\_{\\operatorname{DRPO}}(\\rho\\_i)) \\|\\| \\mathcal{D}(\\theta\\_{\\operatorname{DRPO}}(\\rho\\_i))) \\leq \\rho\\_i$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper revisits the performative learning setting in which the learner wishes to control the risk of a model over a test distribution that is sensitive to the choice of the model that is deployed. This is a novel niche topic but certainly an interesting and challenging setting that is connected to more well-known questions in ML such as covariate shift, lifelong ML or robust estimation. More precisely, the paper draws connections with distributionally robust optimization by studying methods to determine the worst-case performative risk over a set of distribution maps (that map parameters to test distributions) that contains the true distribution map.

While staying technically close both to ideas found in the original papers on performative prediction and to techniques originating from the distributionally robust optimization literature, this paper introduces novel ideas on a recent topic of interest which justify acceptance for the conference.
Decision: Accept (poster)