Anticipating Performativity by Predicting from Predictions
Authors: Celestine Mendler-Dünner
Frances Ding
Yixin Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: performative prediction
performativity
causal identifiability
supervised learning
domain adaptation
concept shift
social impact

Abstract: Predictions about people, such as their expected educational achievement or their credit risk, can be performative and shape the outcome that they are designed to predict. Understanding the causal effect of  predictions on the eventual outcomes is crucial for foreseeing the implications of future predictive models and selecting which models to deploy. However, this causal estimation task poses unique challenges: model predictions are usually deterministic functions of input features and highly correlated with outcomes, which can make the causal effects of predictions on outcomes impossible to disentangle from the direct effect of the covariates. We study this problem through the lens of causal identifiability. Despite the hardness of this problem in full generality, we highlight three natural scenarios where the causal effect of predictions can be identified from observational data: randomization in predictions, overparameterization of the predictive model deployed during data collection, and discrete prediction outputs. Empirically we show that given our identifiability conditions hold, standard variants of supervised learning that predict from predictions by treating the prediction as an input feature can find transferable functional relationships that allow for conclusions about newly deployed predictive models. These positive results fundamentally rely on model predictions being recorded during data collection, bringing forward the importance of rethinking standard data collection practices to enable progress towards a better understanding of social outcomes and performative feedback loops.

Anticipating Performativity by Predicting from Predictions
Celestine Mendler-Dünner Max Planck Institute for Intelligent Systems, Tübingen cmendler@@uubingen mpg.
Frances Ding
Yixin Wang University of Michigan yixinoQumi ch edu
Univerity of California, Berkeley frances@berkeley. eddu edu
Abstract
Predictions about people, such their expected educational achievement or their credit risk, can be performative and shape the outcome that they are designed to predict. Understanding the causal effect of predictions on the eventual outcomes is crucial for foreseeing the implications of future predictive models and selecting which models to deploy. However, this causal estimation task poses unique chal- lenges: model predictions are usually deterministic functions of input features and highly correlated with outcomes. This can make the causal effect of predictions on outcomes impossible to disentangle from the direct effect of the covariates. We study this problem through the lens of causal identifiability. Despite the hardness of this problem in full generality, highlight three natural scenarios where the causal effect of predictions can be identified from observational data: randomization in predictions, overparameterization of the predictive model deployed during data collection, and discrete prediction outputs. Empirically we show that given our identifiability conditions hold, standard variants of supervised learning that predict from predictions by treating the prediction as an input feature can find transferable functional relationships that allow for conclusions about newly deployed predictive models. These positive results fundamentally rely on model predictions being recorded during data collection, bringing forward the importance of rethinking standard data collection practices to enable progress towards better understanding social outcomes and performative feedback loops.
Introduction
Predictions can impact sentiments, alter expectations, inform actions, and thus change the course of events. Through their influence on people, predictions have the potential to change the regularitics in the population they seek to describe and understand. This insight underlies the theories of performativity [38] and reflexivity [62] that play an important role in modern economics and finance. Recently, Perdomo et al. [51] pointed out that the social theory of performativity has important implications for machine learning theory and practice. Prevailing approaches supervised learning assume that features X and labels Y are sampled jointly from a fixed underlying data distribution that unaffected by attempts to predict Y from X. Performativity questions this assumption and suggests that the deployment of a predictive model can disrupt the relationship between X and Y. Hence, changes to the predictive model can induce shifts in the data distribution. For example, consider lender with predictive model for risk of default performativity could arise individuals who are predicted as likely to default are given higher interest loans, which make default even more likely [41], akin to a self-fulfilling prophecy. In turn, a different predictive model that predicts smaller risk and suggests offering more low-interest loans could cause some individuals who previously looked risky
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

to be able to pay the loans back, which would appear as a shift in the relationship between features X and loan repayment outcomes Y. This performative nature of predictions poses challenge using historical data to predict the outcomes that will arise under the deployment of future models.
1.1 Our work
In this work, we aim to understand under what conditions observational data is sufficient to identify the performative effects of predictions. Only when causal identifiability is established can we rely on data-driven strategies to anticipate performativity and reason about the downstream consequences of deploying new models. Towards this goal, we focus on a subclass of performative prediction problems in this paper where performative effects of predictions solely surface as a shift in the outcome variable, and the distribution over covariates X is unaffected by the prediction Y. Our goal is to identify the expected counterfactual outcome
My (a. 0) é E| x. do = 0)]
Understanding the causal mechanism My is crucial for model evaluation, as well as model opti- mization. In particular, it allows for offline evaluation of the potential outcome Y of individual X subject predictive model fnew with the prediction Y fnew( before actually deploying it.
The need for observing predictions. We start by illustrating the hardness of performativity-agnoon learning by relating performative prediction to a concept shift problem. Using the specifics of the performative shift, we establish lower bound on the extrapolation error of predicting Y from X under the deployment of new model fnew that is different from the model fuain deployed during data collection. In particular, the extrapolation error grows with the distance between the prediction functions of the two models and the strength of performativity. This lower bound on the extrapolation error demonstrates the necessity to take performativity into account for reliably predicting Y.
Predicting from predictions. We then explore the feasibility of learning performative effects when the training data recorded the predictions and training data samples (X, Y) are available. As an identification strategy for learning My, we focus on building meta machine learning model that predicts Y for an individual with features X, subjected to prediction Y. We term this data-driven strategy predicting from predictions; it treats the predictions as an input to the meta machine learning model. The meta model seeks to answer "what would the outcome be if we were to deploy a different prediction model?" Crucially, this "what if" question is causal in nature; it aims to understand the potential outcome under an intervention which is different from merely estimating the outcome variable in previously seen data. Whether such transferable model is learnable depends on whether the training data provides causal identifiability [49] Only after causal identifiability is established can we rely on observational data to select and design optimal prediction models under performativity.
Establishing identifiability. For main technical results, first show that, in general, observing Y is not sufficient for identifying the causal effects of predictions. In partieular, if the training data was collected under the deployment of a deterministic prediction function, the mechanism MM can not be uniquely identified. The reason is a lack of coverage in the training data as X and Ý are deterministically bound. Next, we establish several conditions under which observing Ý is sufficient for identifying My- The first condition exploits the presence of randomness in the prediction. This randomness could be purposely built into the prediction for individual fairness, differential privacy, or other considerations. The second condition exploits the property that predictive models are often over-parametertie paramtterizedd which leads to incongruence in functional complexity between different causal paths, enabling the effects of predictions to be separated from other variables' effects. The third condition takes advantage of discreteness in predictions such that performative effects can be disentangled from the continuous relationship between covariates and outcomes. Together, these results reveal that particularities of the performative prediction problem can enable us recover the causal effect of predictions from observational data. In particular, we show that, under these conditions, standard supervised learning techniques can be used to find these transferable functional relationships by treating predictions as model inputs. Empirically, we demonstrate that supervised learning succeeds in finding My even in finite samples.
We conclude with discussion of limitations and extensions of our work, pointing out potential violations of the modeling assumptions underlying our causal analysis and proposing directions for future work.

---

1.2 Broader context and related work
The work by Perdomo al. [51], initiated the discourse of performativity in the context of supervised learning by pointing out that the deployment predictive model can impact the data distribution train models on. Existing scholarship on performative prediction [c.f., 51, 42, 12, 44, 24, 26, 68, 45, 52, 31] has predominantly focused on achieving a particular solution concept with a prediction function that maps X to Y in the presence of unknown performative effects. We are interested in understanding the underlying causal mechanism of the performative distribution shift. Our work is motivated by the seemingly natural approach of lifting the supervied-llearinng problem and incorporating the prediction as input feature when building meta machine learning model for explaining Y. By establishing a connection to causal identifiability, our goal is to understand when such data-driven strategy can help anticipate the down stream effects of predictions
This work focuses on the setting where predictions lead to changes in the relationship between covariates X and label Y. while the marginal distribution P(X) over covariates is assumed to be fixed. This setting where performativity only surfaces in the label describes an interesting subclass of problems falling under the umbrella of performative (aka. model-induced or -eeiiionndeeeedennt) distribution shifts [51, 37, 12]. Our assumptions are complementary to the strategic classification framework [8, 20J that focuses on a setting where performative effects concern P(X), while PX) is assumed to remain stable. Consequently, causal questions in strategic classification [e.g.. 22, 3, 59] are concerned with identifying stable causal relationships between X and Y. Since we assume P(Y (X) can change (i.e. the true underlying "concept" determining outcomes can change), conceptually different questions emerge in our work. Similar in spirit to strategic classification, the work on algorithmic recourse and counterfactual explanations [32, 28, 65] focuses on the causal link between features and predictions, whereas focus the down-stream effects of predictions.
There are interesting parallels between our work and related work on the offline evaluation of online policies [e.g., 35, 63, 36, 58]. In particular, [63] explicitly emphasize the importance of logging propensities of the deployed policy during data collection be able mitigate selection bias. In our work the deployed model can induce a concept shift. Thus, we find that additional information about the predictions of the deployed model needs to be recorded to be able to foresee the impact of new predictive model on the conditional distribution (Y) beyond enabling propensity weighting [55]. A notable work by [66] investigates how predictions at one time step impact predictions in future time steps. Complementary
these existing works we show that randomness in the predictive model is not the only way causal effects of predictions can be identified.
For our theoretical results, we build classical tools from causal inference [48, 57, 64]. In particular, we distill unique properties of the performative prediction problem to design assumptions for the identifiability of the causal effect of predictions.
The causal force of prediction
Predictions can be performative and impact the population of individuals they aim to predict. For- mulized in the language of causal inference [48]: the deployment of a predictive model represents an intervention on a causal diagram that describes the underlying data generation process of the population. We will expand on this causal perspective to study instance of ths performative prediction problem described below.
2.1 Prediction as a partial mediator
Consider a machine learning application relying on a predictive model f that maps features X to a predicted label γ. We assume the predictive model f is performative in that the prediction ý (X) has direct causal effect the outcome variable Y of the individual it concerns. Thereby the prediction impacts how the outcome variable Y is generated from the features X. The causal diagram illustrating this setting is visualized in Figure 1.
The features X E X Rdd are drawn i.i.d. from fixed underlying continuous distribution covariates Dx with support X. The outcome Y E ) C IR is a function of X, partially mediated by the prediction γ E y. The prediction Ý is determined by the deployed predictive model f X' - y. For given prediction function f. every individual is assumed to be sampled i.i.d. from the data generation process described by the causal graph in Figure 1. We assume the exogenous noise ε is zero mean, and y allows the prediction function to be randomized.
3

---

1
X = Ex
εx ~ Dx
(1)
y 5(x Y g(X, 8) EY
E, Dj
(2)
εy Dy
(3)
Figure 1: Performative effects mediated by predictions for a given f
Note that our model is not meant to describe performativity in its full generality (which includes other ways f may affect P(X, Y)). Rather, it describes an important and practically relevant class of performative feedback problems that are characterized by two properties: 1) performativity surfaces only in the label Y, and 2) performative effects are mediated by the prediction, such that Y 1L Y. rather than dependent on the specifics of the decision rule.
Application examples. Causal effects of predictions on outcomes have been documented in multiple contexts: A bank's prediction about the client (e.g., his or her creditworthiness applying for a loan) determines the interest rate assigned them, which in turn changes client's 's financial situation [41]. Mathematical models that predict stock prices inform the actions of traders and thus heavily shape financial markets and economic realities [38]. Zillow's 's housing price predictions directly impact sales prices [39]. Predictions about the severity of an illness play an important role in treatment decisions and hence the very chance of survival of the patient [34]. Another prominent example from psychology is the Pygmalion effect [56]. It refers the phenomenon that high expectations lead to improved performance, which is widely documented in the context of education [6], sports [61] J. and organizations [16]. Examples of such performativity abound, and we hope to have convinced the reader that the performative effects the label are important for algorithmic prediction.
2.2 Implications for performativity-agnostt learning
Begin with considering the classical supervised learning task where Y is unobserved. The goal is to learn a model h X' + y for predicting the label Y from the features X. To understand the inherent challenge of classical prediction under performativity, we investigate the relationship between X and more closely. Specifically, the data generation process (Figure 1) implies that
P(Y = / x)
(4)
This expression makes explicit how the relationship between and that we aim to learn depends on the predictive model governing (() As a consequence, when the deployed predictive model at test time differs from the model at training time, performative effects surface concept shift [17]. Such distribution shift problems are known to be intractable without structural knowledge about the shift, implying that we can not expect h to generalize to distributions induced by future model deployments. Let us inspect the resulting extrapolation gap in more detail and put existing positive results on performative prediction into perspective.
Extrapolation loss. We illustrate the effect of performativity a simple instantiation of the structural causal model from Figure 1. Therefore, assume a linear performative effect of strength > 0 and base function 91 : + +
predictive performance using
g(x,))= = (X) + at.
(5)
Now, assume we collect training data under the deployment of a predictive model fa and validate our model under the deployment of fg. We adopt the notion of a distribution map from Perdomo al. [51] and write D (f)) for the joint distribution over (X, Y) surfacing from the deployment of a model f. We assess the quality of our predictive model h X - y over distribution Dxy( induced by via the loss function É: y x y - I and write R, (h) == E Ex (nn (n)(())) y) for the risk of h on the distribution induced by f. We use hý for the risk minimizer h2 == argmin (h). and 7 for the hypothesis class optimize over. Proposition bounds the extrapolation loss and can be viewed as a concrete instantiation of the more general extrapolation bounds for performative prediction discussed in [37] within the feedback model from Figure 1.
Proposition 1 (Hardness of performativity-age aggnotii prediction). Consider the data generation pro- cess in Figure I with g given in (5) and fe, fo being deterministic functions. Take a loss function

---

- that is y-soooth and p-strongly convex in its second argument. Let hj, be the risk minimizer over the training distribution and assume the problem is realizable, i.e hio E H. Then, we can bound the extrapolation loss of h on the distribution induced by so as
Y n de (fo. fo) 2 A Aofo-fo ( 11 o (fg.
2
where d (So. [6) = Eiz (fo(z) - fa(x)) and A A (h) = R (h) - (h).
The extrapolation loss A (hj,) ) is zero if and only if either the strength of performativity tends to zero (a: 0). or the predictions of the two predictors fo and fe are identical over the support of Dx. If this is not the case, an extrapolation gap is inevitable. This elucidates the fundamental hardness of performative prediction from feature, label pairs (X when performative effects disrupt the causal relationship between X and Y.
The special case where a = 0 aligns with the assumption of classical supervised learning, in which there is no performativity. This may hold in practice if the predictive model is solely used for descriptive purposes, or if the agent making the prediction does not enjoy any economic power [21]. The second special case where the extrapolation error is small is when d (So, fo) - O. In which case Dxy (fo) and Dxy (f+) are equal in distribution and hence exhibit the same risk minimizer. Such scenario can happen, for example, if the model fo is obtained by retraining fo on observational data and fixpoint is reached (fe = h ). The convergence of policy optimization strategies to such fixpoints (perfromative stablity) has been studied in prior work [e.g.. 51, 42, 12] and enabled optimality results even in the presence of performative concept shifts, relying on the target model fo not being chosen arbitrarily, but based on a pre-specified update strategy.
3 Identifying the causal effect of prediction
Having illustrated the hardness performattvity-aagosii learning, we explore under what conditions incorporating the presence of performative predictions into the learning task enables to anticipate the perfromative effects of Y on Y. Towards this goal, assume that the mediator Y in Figure 1 observed- -the prediction takes on the role of the treatment our causal analysis and we can not possibly hope to estimate the treatment effect of treatment that is unobserved.
3.1 Problem setup
Assume we are given access to data points (z. U. v) generated i.i.d. from the structural causal model in Figure under the deployment of a prediction function fo. From this observational data, we wish to estimate the expected potential outcome of an individual under the deployment of an unseen (but known) predictive model fa- We note that given our causal graph, the implication of intervening on the function f can equivalently be explained by an intervention on the prediction Y. Thus, we are interested in identifying the causal mechanism:
My 9) := B| x, do = 9)]-
Unlike PX)) the mecahnism My is invariant to the changes in the predictive model governing X)) Thus, being able to identify My will allow us to make inferences about the potential outcome surfacing from planned model updates beyond explaining patterns in historical data. We can evaluate My to infer y for any at ÿ = fz for fo being the model of interest.
For simplicity of notation, will write D(fo) to denote the joint distribution over (X, (X,Ý, Y) of the observed data collected under the deployment of the predictive model fa. We say My can be identified, if it can uniquely be expressed as function of observed data. More formally:
Definition (identifiability). Given a predictive model f. the causal graph Figure 1, and a set of assumptions A. We say My is identifiable from D(f), for any function that complies with assumptions A and h(x,, 5) -M (2. ÿ) for pairs (x. 0) E (upp(Dxy(f)) it must also hold that h(x,, 9) = My (z, ÿ) for all pairs (x, 9) E x D.
Without causal identifiability, there might be models // AMy that explain the training distribution equally well but do not transfer to the distribution induced by the deployment of new model. Causal identifiability is crucial for enabling extrapolation. It quantifies the limits of what we can infer given access to the training data distribution, ignoring finite sample considerations.
5

---

Identification with supervised learning. Identifiability of My from samples of D(fo) implies that the historical data collected under the deployment of fe contains sufficient information to recover the invariant relationship (7). As a concrete identification strategy, consider the following standard variant of supervised learning that takes in samples (Ir, V/ y) and builds a meta-model that predicts Y from X, y by solving the following risk minimization problem
hSL == argmin (((x.0.))-D((.) (h(z, 1) - y)² 4Cи
(8)
where H denotes the hypothesis class. We consider the squared loss for risk minimization because it pairs well with the exogeneous noise εy (3) being additive and zero mean. The strategy (8) an instance of what we term predicting from predictions. Lemma 2 provides a sufficient condition for the supervised learning solution hsL to recover the invariant causal quantity My.
Lemma 2 (Identification strategy). Consider the data generation process in Figure I and set of assumptions A. Given a hypothesis class H such that every h E H complies with A and the problem is realizable, i.e., My E H. Then, if My is causally identifiable from D(fo) given A, the risk minimizer hss in (8) will coincide with My.
3.2 Challenges for identifiahility
The main challenge for identification of My from data is that in general, the prediction rule fo which produces Y is a deterministic function of the covariates X. This means that, for any realization of X, we only get access to one y = [x(X) in the training distribution, which makes it challenging to disentangle the direct and the indirect effects of X on Y. To illustrate this challenge, consider the function hx,,)) == My (x. fz)) that ignores the input parameter ý and only relies on æ for explaining the outcome. This function explains y equally well and can not be differentiated from My based on data collected under the deployment of deterministic prediction rule So. The problem is akin to fitting linear regression model to two perfectly correlated covariates. More broadly, this ambiguity is due what is known as lack of overlap (or lack of positivity) in the literature of causal inference [47, 23]. In the covariate shift literature, the lack of overlap surfaces when the covariate distribution violates the common support assumption and the propensity scores not well-defined (see e.g.. Pan and Yang [46]). This problem renders causal identification and thus data-driven learning of performative effects from deterministic predictions fundamentally challenging.
Proposition 3 (Nonidentifiabiiity from deterministic predictions). Consider the structural causal model in Figure 1. Assume Y non-trivially depends on Y, and the set y is not singleton. Then, given a deterministic prediction function f. the mechanism My is not identifiable from D(f).
The identifiability issue persists as long as the two variables X, Y are deterministically bound and there is incongruence hidden structure that can be exploited to disentangle the direct effect of X on Y from the indirect effect mediated by Y. In the following, we focus particularities of prediction problems and show how they allow us to identify My.
3.3 Identifiability from randomization
We start with the most natural setting that provides identifiability guarantees: randomness in the prediction function fo. Using standard arguments about overlap [47] we can identify My (ar, 0) for any pair x, ÿ with positive probability in the data distribution D(fa) from which the training data is sampled. To relate this our goal of identifying the outcome under the deployment of an unseen model fe we introduce the following definition:
Definition (output overlap). Given two predictive models fo, So, the model fo is said to satisfy output overlap with fo. if all E X and any subset y' y with positive measure, it holds that
Pffa(x) E 3"1 ([fo(z) E J
> 0.
(9)
In particular, output overlap requires the support of the new model's predictions fo( to be contained in the support of fx( for every potential x E X. The following proposition takes advantage of the fact that the joint distribution over (X, Y) is fully determined by the deployed model's predictions to relate output overlap to identification:

---

Proposition 4. Given the causal graph in Figure 1. the mechanism (z 9) is identifiable from D(fe) for any pair 2, & with ÿ = fo( as long as fo is a prediction funcciooo that satisfies outpur overlap with fa.
Proposition 4 allows us to pinpoint the models fo to which we can extrapolate to from data collected under fe. Furthermore, it makes explicit that for collecting data to learn about performative effects, it is ideal to deploy a predictor fo that is randomized so that the prediction output has full support over Y for any æ. Such a model would generate a dataset that guarantees global identification of My over X' x y and thus robust conclusions about any future deployable model fg- One interesting and relevant setting that satisfies this property is the differentially private release of predictions through an additive Laplace (or Gaussian) noise mechanism applied to the output of the prediction function [13].
While standard in the literature, a caveat of identification from randomization is that there are several reasons decision-maker may choose not to deploy a randomized prediction function in performative environments, including negative externalities and concerns about user welfare [29], but also business interests to preserve consumer value of the prediction based service offered. In the context of our credit scoring example, random predictions would imply that interest rates are randomly assigned to applicants in order to learn how the rates impact their probability of paying back. We can not presently observe this scenario, given regulatory requirements for lending institutions.
3.4 Identifiability through overparameterization
The following two sections consider situations where we achieve identification, without ran- domization, from data collected under deterministic fe. Our first result exploits incongruences in functional complexity arising from machine learning models that are overparameterized [e.g. 30j. By overparameteriiztion,, we refer to the fact that the representational complexity of the model is larger than the underlying concept it needs to describe.
Assumption 1 (overparamettrrizatonn)) over X if there no function g' E G and e E IK such that f(x) = c- g'(x) for all x E X.
We say function f is overparameterized with respect to G
challenge for identification is that for deterministic So the prediction can be reconstructed from X without relying on Y, and thus h(z 9) = My (z, fo(z)) can not be differentiated from My based on observational data. However, note that this ambiguity relies there being admissable h such that h(- ()) for fixed I can represent fa. If fo is overparameterized with respect to the hypothesis class H. this ambiguity is resolved. Let us make this intuition concrete with an example:
Example 3.1. Assume the structural equation for y in Figure is g(x, 9) a. + 3ÿ for some
unknown a. 3. Consider prediction functions fe of the following form fe(z) = =z²² + &x for some 7 2 O. Consider H be the class of linear functions. Then, any admissable estimate h E H takes the form h(s, 9) = a'x + (3'g. For h to be consistent with observations we need a' + f = a + BB and B'ny= = 37y This system of equations has a unique solution as long y > 0 which corresponds to the case where fe overparameterized with respect H. In contrast, for ? = 0 the function h(. ú) = (a + ( would explain the training data equally well.
The following result generalizes this argument to separable functions. Proposition 5. Consider the structural causal model in Figure I where fo is a deterministic
Assume that g can be decomposed g(X, Y) 9i (X) + aÿ for some a > 0 and 91 E G, where the function class G is closed under addition (i.e. 91.92 E 9 = a1 91 + a2 va--99 92 E G Yo,, 0g E R.. Let H contain functions that are separable in X and Y, linear in Y, and Vh E H. it holds that h((,0) ú) E G for a fixed ÿ. Then, if fo is overparameterized with respect to G over the support of Dx, My identifiable from D(fa).
3.5 Identifiability from classification
A second ubiquitous source of incongruence that can exploit for identification is the discrete nature of predictions in the context of classification. The resulting discontinuity in the relationship between X and Y enables us to disentangle My from the direct effect of X on Y. This identification strategy is akin to the popular regression discontinuity design [33] and relies on the assumption that all other variables in X are continuously related to Y around the discontinuities in Y.
In Appendix B we discuss two additional natural sources of randomness (randomized decisions and noisy measurements of covariates) that can potentially help identification with appropriate side-infomation.
7

---

10"
107
16"
RMSE
rrcuded artt
16-
10a
10
10'
10a
0.0
0.2
0.5
0.8
1.0
0.0
0.2
0.5
0.5
0.0
0.2
0.5
0.5
1.0
0.0
0.2
0.5
0.8
1.0
Performatisty strungth
Purforativily strengthie
Parformativity strength
Performaiiiity strungth
(a) identifialle setting
(b) mandomzzdd
overpannmeteerized fo
discrete fo
Figure 2: Extrapolation error of supervised learning with and without access Y. (a) In the non-identifiable setting, adding Y as a feature harms generalization performance. (b)-(d) Randomization, overparameterization, and discrete predictions are each sufficient for avoiding this failure mode.
Proposition 6. Consider the structural causal model in Figure I where fa is a deterministic duterministiifunncionn
Assume that the structural equation for Y is separable o(X,Y) 91 (1 ) + 92 (Y), AX, Y for some differentiable functions 91 and 92. Further, suppose X is continuous random variable and Y is discrete random variable that takes on at least two distinct values with non-zero probability. Then, My is identifiable from D(fo).
Similar to Proposition 5, the separability assumption together with incongruence provides way to disentangle the direct effect from the indirect effect of X on Y. Separability is necessary in order to achieve global identification guarantees without randomness, the identification of entangled components without overlap is fundamentally hard. Thus, under violations of the separability assumptions, we can only expect the separable components of be correctly identified. Similarly, a regression discontinuity design only enables the identification the causal effect locally around the discontinuity. Extrapolation away from the decision boundary to models fø that substantially different from so increasingly relies on separability to hold true.
Empirical evaluation
We investigate empirically how well the supervised learning solution lIsT. in (8) able to identify the causal mechanism My from observational data in practical settings with finite data.
Methodology. We generated semi-synthetic data for experiments, using Census income prediction dataset from folktables .org [11]. Using this dataset as a starting point, we simulate training dataset and test dataset with distribution shift as follows: First, we choose two different predictors fo and fe predict a target variable of interest (e.g. income) from covariates X (e.g. age, occupation, education, etc.). If not specified otherwise, fo is fit to the original dataset to minimize squared error, while fø is trained on randomly shuffled labels. Next, we posit a function g for simulating the performative effects. Then, we generate a training dataset of (X, Y, tuples from the causal model in Figure 1, using the covariates X from the original data, and fe to generate Ý and Y. Similarly, we generate a test dataset of (X, Y, Y) tuples, using X, g. fe. We assess how well supervised methods learn transferable functional relationships by fitting model hsi. the training dataset and then evaluating the root mean squared error (RMSE) for regression and the accuracy for classification on the test dataset. In figures, visualize the standard error from 10 replicates with different random seeds and we compare it to an in-distribution bascline trained and evaluated on samples of of.)) If not specified otherwise we use N 200, 000 samples.
4.1 Necessity of identification guarantees for supervised learning
We start by illustrating why our identification guarantees are crucial for supervised learning under performativity. Therefore, we instantiate the structural equation g in Figure 1 as
g(x 8) = 9 (X) + aY
(10)
with 91 (X) BT X and ~ NN(0,)) 1). The coefficients 3 are determined by linear regression the original dataset. The hyperparameter a quantifies the performativity strength that we vary in our experiments. The predictions Y are generated from a linear model fe that we modify to illustrate the resulting impact identifiability. We optimize hst in (8) over H being the class of linear functions.

---

104
Finduyed
included as feaore
10a
indudol feature
RASE
RMSE
RASSE
10a
101
10a
10a
3
7 fg
9
11
13
0.0
0.2
0.4
0.6
0.8
1.0
5K
10K
15K
20K 25K
8
number of training dompaiits
(o) Varying functional complexity of fo
(b) Varying degree of noise in fo.
(c) Varying for randomized fo
Figure 3: Ablation study of extrapolation performance. (a) We vary me. Adding Y feature helps as soon as fo is overparameterized with respect 9i (me > 3). (b) We vary the noise the predictions of fo. A small amount of noise is sufficient for identifiability. (c) We vary number of datapoints for training hss.
We start by illustrating a failure mode of supervised learning tion 3). Therefore, we let So be a deterministic linear model fit to the base dataset (X( & B X). This results in MMy not being identifiable from D(fe). In Figure 2(a) we can see that supervised learning indeed struggles to identify a transferable functional relationship from the training data. The meta model returns (X, extrapolation error independent of the strength of performativity. While we only show the error for one fo in Figure 2(a), the error grows with the distance d. (fe. fo). In contrast, when the feature Y not included, the supervised learning strategy returns hsu (X) = (1 + a)3 X. The extrapolation loss of this and is thus strictly smaller than the error the model that predicts from predictions.
non-identifiability setting (Proposi-
(1 + a)YY. instead of identifying g. which leads to a high
petormmaaviit-ggosttte model scales with the strength of performativity (Proposition I)
Next, we move to the regime of our identification results (Proposition 4-6). Therefore, we modify the way the predictions in the training data are generated. In Figure 2(b) use additive Gaussian noise to determine the predictions as Y == fo 1) with v E .N o²). In Figure 2(c) we augment the input to fo with second-degre polynomial features to achieve overparameterization In Figure 2(d) we round the predictions of fa to obtain discrete values. all three cases, including Y as a feature is beneficial and allows the model match in n-distribution accuracy baselines, closing the extrapolation gap that inevitable for performativity-agnne
prediction.
4.2 Strength of incongruence and finite samples
We next conduct ablation study and investigate how the degree of overparameterization and the noise level for randomized fo impacts the extrapolation performance of supervised learning. Therefore, we consider the setup in (10) with a general function gi. We fix the level of performativity at a = 0.5 for this experiment. We optimize hSL in (8) over H. (which we vary).
In Figure 3(a) we investigate the effect of overparameterizationn of fe on the extrapolation eтrоr of hSL- We choose fully connected neural networks with single hidden layer represent the functions gl, fo and hsL. For and H we take a neural network with m = 3 units in the hidden layer. The model I1 is fit to the original dataset. We vary the number of units in the hidden layer of fe, denoted ma- As expected, the extrapolation error decreases with the complexity of fa. As soon as my >> my there is a significant benefit to including predictions as features. In this regime, My becomes identifiable as Proposition 5 suggests. In turn, without access to Y the model suffers an inevitable extrapolation gap due to a concept shift that is independent of the properties of fa. In Figure 2(b) we investigate the effect of the magnitude of additive noise added to the predictions. Here 7. and 91 are linear functions.
We have y = fo(X) (X) + 3n with É (0. 1) and we vary the noise level B. We see that even small sufficient for identification and adding Y as feature to meta-machine lenaring model is effective as soon as the noise in fo is non-zero. In Figure 2(c) we fix the noise level at = 0.5 and vary the number of samples N. We find that only moderate dataset sizes are necessary for predicting from predictions to approximate My our identifiable settings.
amounts of noise
5 Discussion
This paper focused on identifying the causal effect of predictions on outcomes from observational data. We point out several natural situations where this causal question can be answered, but

---

also highlight situations where observational data is not sufficiently informative to reason about performative effects. By establishing a connection between causal identifiability and the feasibility of anticipating performative effects using data-driven techniques, this paper contributes to a better understanding of the suitability of supervised learning techniques for explaining social effects arising from the deployment of predictive models in economically and socially relevant applications.
We hope the positive results in this work serve as a message for data-collection: only if predictions are observed, they can be incorporated to anticipate the performative effects of future model deployments. Thus, access to this information is crucial for an analyst hoping to understand the effects of deployed researcher
predictive models, an engineer hoping to foresee consequences of model updates, studying performative phenomena. To date, such data is scarcely available in benchmark datasets, hindering the progress towards better understanding of performative effects, essential for the reliable deployment of algorithmic systems in the social world.
At the same time we have shown that the deterministic nature of prediction poses unique challenges
for causal identifiability even Y is observed. Thus, the success of observational designs (as shown in our empirical investigations) is closely tied to the corresponding identifiability conditions being satisfied. Our results must not be understood as a green-light to justify the use of supervised learning techniques to address performativity in full generality beyond the scope of our theoretical results.
Limitations and Extensions. The central assumption of our work is the causal model in Figure 1. While carving out a rich and interesting class of performative prediction problems that allows us to articulate the challenges of covariates and predictions being coupled, it can not account for all mechanisms of performativity. This in turn gives rise to interesting questions for follow-up studies.
A first neglected aspect is performativity through social inffuence. Our causal model, relies on the stable unit treatment value assumption (SUTVA) [23]. There no possibility for the prediction of one individual to impact the outcome of his or her peers. Such an individualistic perspective is not unique to our paper but prevalent in existing causal analyses and model-based approaches performative prediction and strategic classification [c.g.. 20, 25, 43, 3. 18, 22]. Spillover effects [cf. 60, 64, 1, 40] yet unexplored in the context of performative prediction. Nevertheless, they have important implications for how causal effects should be estimated and interpreted. In the context of our work they imply that an intervention on f can no longer be explaind solely by changing an individual's prediction. As result, approaches for microfounding performative effect based on models learned from simple, unilateral interventions on an individual's prediction result in different causal estimates than supervised learning based methods for identification studied in this work. A preliminary study included in Appendix C shows that data-driven techniques can pick up on interference patterns in the data and benefit from structural properties such network homophily [19], whereas individualistic modeling misses out on the indirect component arising from neighbors influencing each other.
A second aspect is performativity in non- non-causal prediction. Our model posits that prediction is solely based on features X that causal for the outcome Y. This is desirable situation in many practical applications because causal predictions disincentivize gaming of strategic individuals manipulating their features [43, 3] and offer explanations for the outcome that persist across environments [54, 7]. Nevertheless, non-causal variables are often included as input features in practical machine learning prediction tasks. Establishing a better understanding for the implications of the resulting causal dependencies due to performativity could be an important direction for future work.
Finally, performative effect can also lead to covariate shift and impact the joint distribution P(X,Y) = P(Y X(X) over covariates and labels. We assumed that performative effects only surface in PX)]X). For our theoretical results, this implied that overlap in the X variable across environments is trivially satistied, which enabled us to pinpoint the challenges of learning performative effects due to the coupling between X and Ý. For establishing identification in the presence of a causal arrow Se + X additional steps are required ensure identifiability.
Acknowledgement
The authors would like to thank Moritz Hardt and Lydia Liu for many helpful discussions throughout the development of this project, Tijana Zmic, Krikamol Muandet, Jacob Steinhardt, Meena Jagadeesan and Juan Perdomo for feedback on the manuscript, and Gary Cheng for helpful discussions on differential privacy. We are also grateful for constructive discourse and valuable feedback provided by the reviewers that greatly helped improve the manuseript.
10

---

References
[1] Peter M. Aronow and Cyrus Samii. Estimating average causal effects under general interference. with application to a social network experiment. The Annals of Applied Statistics, 1 11(,), 2017.
[2] Susan Athey. Machine learning and causal inference for policy evaluation. In Proceedings of the 21th International Conference on Knowledge Discovery and Data Mining, KDD '15 page
5-6, 2015.
[3] Yahav Bechavod, Katrina Ligett, Steven Wu, and Juba Ziani. Gaming helps! learning from strategic interactions natural dynamics. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pages 1234-1242 PMLR, 2021.
[4] Joël Berger, Margit Osterloh, and Katja Rost. Focal random selection closes the gender gap in competitiveness. Science Advances, 6(47), 2020.
[5] Joël Berger, Margit Osterioh, Katja Rost, and Thomas Ehrmann. How to prevent leadership hubris? comparing competitive selections, lotteries, and their combination. The Leadership Quarterly, 31(5):101388, 2020.
[6] Xander M. Bezuijen, Peter T. van den Berg, Karen van Dam, and Henk Thierry. Pygmalion and employee learning: The role of leader behaviors. Journal of Management, 3545)1228--126,, 2009.
[7] Peter Bühlmann. Invariance, causality and robustness. Arxiv, abs/1812.002333, 2018.
[8] Michael Brückner, Christian Kanzow, and Tobias Scheffer. Static prediction games for adver-
sarial learning problems. Journal of Machine Learning Research, (3(85):2617-2654 2012.
[9] Flavio Chierichetti, Silvio Lattanzi, and Alessandro Panconesi. Rumor spreading in social networks. In Automata, Languages and Programming, 2009. ISBN 978- 978-3-642-02930-1.
[10] Mina Cikara, Matthew M. Botvinick, and Susan T. Fiske. Us versus them: Social identity shapes neural responses to intergroup competition and harm. Psychological Science, 22(3): 003, 2011.
[11] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021. Code available at: httpp::/ htttpa://githuii tiub comm//yylss/ollkttllll...
[12] Dmitriy Drusvyatskiy and Lin Xiao. Stochastic optimization with decision-dependdnt distribu- tions. Arxiv, 0bs/2011.11172 2020.
[13] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity private data analysis. In Theory of Cryptography, pages 265-284, 2006.
[14] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS 12, page 214-226, 2012.
[15] Dean Eckles, Nikolaos Ignatiadis, Stefan Wager, and Han Wu. Noise-induced randomization in regression discontinuity designs. ArXiv, 0bs:2004409458. 2020.
[16] Dov Eden. Leadership and expectations: Pygmalion effects and other self-fulfilling prophecies in organizations. The Leadership Quarterly, 3(4):271-305, 1992.
[17] João Gama, Indre Zliobaite, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM Comput. Sury,, 46(4), 2014.
[18] Ganesh Ghalme, Vineet Nair, Itay Eilat, Inbal Talgam- and Nir Rosenfeld. Strategic classification in the dark. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 3672-363 3681. PMLR, 2021.
[19] Paul Goldsmith-Pinkee and Guido W. Imbens. Social networks and the identification of peer effects. Journal of Business & Economic Statisties, 21(3):253-264 2013.

---

[20] Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. Strategic classifi- cation. In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science, ITCS 11, pages 111-122, 2016.
[21] Moritz Hardt, Meena Jagadeesan, and Celestine Mendler-Dünner. Performative power. ArXiv, 0eh:2203.17232, 2022.
[22] Keegan Harris, Dung Daniel T Ngo, Logan Stapleton, Hoda Heidari, and Steven Wu. Strategic instrumental variable regression: Recovering causal relationships from strategic responses. In Proceedings of the 39th International Conference on Machine Learning. volume 162, pages 8502 8522. PMLR, 2022.
[23] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[24] Zachary Izzo, Lexing Ying, and James Zou. How to learn when data reacts to your model: Performative gradient descent. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 4641-4650, 2021.
[25] Meena Jagadeesan, Celestine Mendler-Düinner, and Moritz Hardt. Alternative microfoundations for strategic classification. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 4687-4697, 2021.
[26] Meena Jagadeesan, Tijana Zmic, and Celestine Mendler-Dünner. Regret minimization with performative feedback. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 9760-9785, 2022.
[27] Kaggle. Give me some credit, 2011. URL CiveMeSomeCredii Credit/aatrrdiiaaaaaaaaaaaa
com/
[28] Amir-Hossein Karimi, Bernhard Schölkopf, and Isabel Valera. Algorithmic recourse: From counterfactual explanations to interventions. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT 221, page 353-362 2021.
[29] Adam D. Kramer, Jamic Guillory, and Jeffrey Hancock. Experimental evidence of massive- scale emotional contagion through social networks. Proceedings of the National Academy of Sciences of the United States of America, 111 24, 2014.
[30] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in Neural Information Processing Systems, volume 25, 2012.
[31] Bogdan Kulynych. Causal prediction induce performative stability. In ICML. 2022: Work- shop on Spurious Correlations, Invariance and Stability, 2022.
[32] Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin De- tyniecki. Comparison- inverse classification for interpretability in machine learning. In Information Processing and Management of Uncertainty in Knowledge-Bacd -Based Systems. Theory and Foundations, pages 100-111, 2018
[33] David S. Lee and Thomas Lemieux. Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 2010.
[34] Scott Levin, Matthew Toerper, Eric Hamrock, Jeremiah S. Hinson, Sean Barnes, Heather Gardner, Andrea Dugas, Bob Linton, Tom Kirsch, and Gabor Kelen. Machine-leaming- based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index. Annals of Emergency Medicine, 7165):565-574.c. 2018.
[35] Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of bannttuull--andi--bbaeae news article recommendation algorithms. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, WSDM '11, page 297-306,
2011.
12

---

[36] Lihong Li, Shunbao Chen, Jim Kleban, and Ankur Gupta. Counterfactual estimation and optimization of click metries in search engines: A case study. In Proceedings of the 24th International Conference on World Wide Web, WWW '15 Companion, page 929-934, 2015. ISBN 9781450334730.
[37] Yang Liu, Yatong Chen, and Jiaheng Wei. Induced domain adaptation. ArXiv, abs:2107.05911 2021.
[38] Donald MacKenzie. An engine, not a camera: How financial models shape markets. Mit Press, 2008.
[39]
Nikhil Malik. Does machine learning amplify pricing errors in housing marker? economics of ml feedback loops. Information Systems Economics eJournal, 2020.
[40] Charles Manski. Identification of endogenous social effects: The reflection problem. The Review of Economic Studies, 60(3):531-54,, 1993.
[41] Gustavo Manso. Feedback effects of credit ratings. Journal of Financial Economics, 109(2): 535-548, 2013.
[42] Celestine Mendler- Mendler-Dünner, Juan C. Perdomo, Tijana Zrnic, and Moritz Hardt. Stochastic optimization for performative prediction. Proceedings of the 34th International Conference on Neural Information Processing Systems, volume 33, pages 1929-49399 2020.
[43] John Miller, Smitha Milli, and Moritz Hardt. Strategic classification is causal modeling in disguise. Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 6917-6926, 2020.
[44] John P Miller, Juan C Perdomo, and Tijana Zrnic. Outside the echo chamber: Optimizing the performative risk. In Proceedings of the 38th International Conference on Machine Learning. volume 139 of PMLR, pages 77 7710-7720, 2021.
[45] Adhyyan Narang, Evan Faulkner, Dmitriy Drusvyatskiy, Maryam Fazel, and Lillian J. Ratliff. Multiplayer performative prediction: Learning in decision-depender games. ArXiv, abs/2201 1.3398,,, 2022.
[46] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 13:55:3345-13999 2010. doi: 10. 2..109/TKDE.2009.1999
[47] Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4) 1995.
[48] Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, New York, NY, USA, 2nd edition, 2009.
[49] Judea Pearl. Causal inference in statistics: An overview. Statist. Surv., 3:96-146, 2009.
[50] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2832 2011.
[51] Juan Perdomo, Tijana Zmnc, Celestine Mendler-Dünnner, and Moritz Hardt. Performative prediction. In Proceedings of the 37th International Conference on Machine Learning, pages 1199-7609, 2020.
[52] Georgios Piliouras and Fang- Yi Yu. Multi-agent performative prediction: From global stability and optimality to chaos. Arxiv, 0bs/2200111483, 2022.
[53] Aahlad Puli, Adler Perotte, and Rajesh Ranganath. Causal estimation with functional con- founders. Advances in neural information processing systems, 33:5115-5125. 2020.
[54] Mateo Cojas-Carulla. Berhhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. JMLR, 19:1 1110-139922 2018.
[55] Paul R. Rosenbaum and Donald B. Rubin. The central role of the propensity score in observa- tional studies for causal effects. Biometrika, 70(1):41-55, 1983. ISSN 00063444.
13

---

[56] Robert Rosenthal and Lenore Jacobson. Pygmalion in the classroom. The Urban Review, 1968.
[57] Donald B. Rubin. Randomization analysis of experimental data: The fisher randomization test comment. Journal of the American Statistical Association, 75(371):591-595, 1980. ISSN 01621459.
[58] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations treatments: Debiasing learning and evaluation. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, page 1670-1679, 2016.
[59] Yonadav Shavit, Benjamin Edelman, and Brian Axelrod. Causal strategic linear regression. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 8676-8686, 2020.
[60] Michael E Sobel. What do randomized studies of housing mobility demonstrate? causal inference in the face of interference. Journal of the American Statistical Association, 101(47)): 1398-1407, 2006.
[61] Gloria B. Solomon, David A. Striegel, John F. Eliot, Steve N. Heon, Jana L. Maas, and Valerie K. Wayda. The self-fulilling prophecy in college basketball: Implications for effective coaching. Journal of Applied Sport Psychology, 8(1): 8(1):44-55 -9, 1996.
[62] George Soros. The Alchemy of Finance. John Wiley & Sons, 2015.
[63] Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In Proceedings of the 32nd International Conference International Conference on Machine Learning Volume 37, ICML' 15, page 814-823, 2015.
[64] Eric J Tchetgen and Tyler J VanderWecle, On causal inference in the presence interference. Statistical Methods in Medical Research, 21(1):55-75, 2012.
[65] Stratis Tsirtsis and Manuel Gomez Rodriguez. Decisions, counterfactual explanations and strategic behavior. Advances Neural Information Processing Systems, volume 33, pages 16749-16760, 2020.
[66] Stefan Wager, Nick Chamandy, Omkar Muralidharan, and Amir Najmi. Feedback detection for live predictors. In Proceedings the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS' 14, page 3428-3436. MIT Press, 2014.
[67] Yixin Wang and David M. Blei. The blessings of multiple causes. Journal of the American Statistical Association, 1 114(528):5574 1596, 2019.
[68] Killian Wood, Gianluca Bianchin, and Emiliano Dall' Anese. Online projected gradient descent for stochastic optimization with decision -eeeedeenn distributions. IEEE Control Systems Letters, 6: 6:1646-1651, 2022.
14

---

Paper checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see Appendix F.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "iCdhSP9qf9T", "reviewer": "Reviewer_JT8c", "summary": "In the traditional supervised setting, we seek to predict a response $Y$ based on features $X$, say as $\\hat{Y} = f_{\\theta}(X)$. The question this paper asks is the following: What happens when the prediction $\\hat{Y}$ itself influences $Y$? Can we then still rigorously reason about our predictive model? To answer the above question, the paper makes the following contributions:\n\n1) It introduces a causal framework to answer the above question. Instead of the traditional regression estimand $\\mathbb E[Y \\mid X=x]$, the estimand of interest in this setting is the following:\n$$ h(x, \\hat{y}) = \\mathbb E[ Y \\mid X=x, do(\\hat{Y}=\\hat{y})] $$\n\n2) It establishes three practical situations under which the above estimand is identified from observational data and may furthermore be estimated by supervised learning based on weighted empirical risk minimization. These three situations are the following: when noisy predictions are released (e.g., in differential privacy), when the features $X$ are noisy measurements of latent features $U$, and under separability (along with overparameterization or discrete classification).\n\n\n3) A semi-synthetic study is conducted based on census data and Kaggle credit scoring data.\n", "questions": "### Extrapolation error\n\nPlease clearly define what extrapolation error is (I do not think it is ever defined). It may be helpful to state the theoretical result (Proposition 1) with the middle term in the inequality replaced by $R_{f_{\\phi}}(\\psi(f_{\\theta})) - R_{f_{\\phi}}(\\psi(f_{\\phi}))$. Even though the 2nd term is $0$, it may be more instructive to express it as above. In Proposition 1, it seems that smoothness and strong convexity refer to the second argument of the loss function. If so, this should be clarified/explicitly stated.\n\n### Overlap\n\n* Front door adjustment: I did not understand the reference to front door adjustment in both the main text and the proof of Proposition 3. For example, in the proof, it is written that the front door adjustment formula is applied to $\\hat{Y} \\to T_{\\hat{Y}} \\to Y$. But note that here $X$ is actually observed, and is conditioned upon, so front door adjustment does not seem necessary? Instead it seems that the argument boils down to the fact that in this setting:\n$$ \\mathbb E[ Y \\mid X=x, do(\\hat{Y}=\\hat{y})]  = \\mathbb E[ Y \\mid X=x, \\hat{Y}=\\hat{y}],$$\nand the RHS is now estimable/identifiable (while is is not under the conditions of Proposition 2).\n\n\n* Page 6, Line 226: $P(T_{\\hat{Y}} = t \\mid X)>0$: This is not correct, this should be replaced by a statement about the conditional density of $T_{\\hat{Y}}$, or alternatively, stated as in Assumption 1.\n\n### Noisy measurement of covariates\n\n* \"information about the noise distribution of the measurement error $P(U \\mid X=x)$\": Is the measurement error distribution not specified instead by $p(x \\mid U=u)$?\n\n* I do not understand the statement of Proposition 4. What are these weights exactly? They are stated as:\n$w(x, \\hat{y}) = P(U | X=x) / P(U \\mid \\hat{Y} = \\hat{y}),$\nbut this does not make sense to me, since $U$ is a latent random variable (and so the above could not be the density at a fixed $u$). Also why does the beginning of the statement state that knowledge of $P(U \\mid \\hat{Y} = \\hat{y}, Y=y)$ is required even though only $P(U \\mid \\hat{Y} = \\hat{y})$ is included in the weight definition?\n\n* Proposition 4: In the proof of Proposition 4, the equality in line (19) is incorrect. \n\n### Separability:\n\n* Proposition 5: It would be very helpful if the specification of $\\mathcal{H}$ in this statement could be more explicit.\n\n* Example 3.1: \"inferring $\\beta = c_1$ and $\\alpha = c_1 - c_2$\": I think for $\\alpha$ one would infer $\\alpha=c_1$ instead of $\\alpha = c_1 - c_2$.\n\n* Proposition 6: Here the main text claims that results for the regression discontinuity design are used, but then the actual proof cites a result from Wang and Blei (2019) that is unrelated to the regression discontinuity design. Also in the proof, what is $x'$? Why does supervised learning recover the true effect?\n\n### Minor:\n\n* *The supervised learning approach*: This should be in bold instead of italics (to match the bold of other paragraphs in the manuscript).\n\n* In equation (8), why are the weights allowed to depend also on $y$? Would it suffice if they only depend on $(x, \\hat{y})$?\n\n* There is a typo in the statement of Assumption 2.", "limitations": "Limitations of the framework are not discussed. It would be great if a paragraph could be added with possible directions for future work!", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n* Motivation: The setting and problem tackled in this paper are exceptionally well-motivated. It is clear that rigorous reasoning about the causal effects of predictions is of great practical and theoretical importance. \n\n* Clarity: The prose of the paper (but not the formal results, see below) is a joy to read and very clear.\n\n\n# Weaknesses\n\n* Previous work: I believe that the paper misses important previous references. Rigorously thinking about causal effects of predictions and seeking to estimate them is not a new idea, even within the NeurIPS community. For example, the following paper (also see references therein) addresses such a problem, and the underlying method is based on adding noise to the predictors (i.e., very similar to one of the identification strategies in the present paper):\n\n> Wager, S., Chamandy, N., Muralidharan, O., & Najmi, A. (2014). Feedback detection for live predictors. Advances in Neural Information Processing Systems, 27.\n\n* Formal results: Several of the formal results of the paper are not well-stated, have errors (e.g., in the proofs), and make incorrect references to previous work. See points below for a more detailed listing of such issues.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Tl9dRd_sKc-", "reviewer": "Reviewer_e7sJ", "summary": "The setting of the paper is about making predictions from predictions -- given a deployed model and decision subjects may best respond to it, how will their target variable Y change. The goal for the authors is to predict the impact of a new model deployment before actually deploying it. Instead of trying to come up with an optimal solution (e.g. performative optimality), the authors are interested in understanding the underlying causal mechanism of the distribution shifts. ", "questions": "Can the author provide some insights on why overlapping is easier to achieve if $|\\mathcal{T}|$ is small? In general, can we boost the overlapping by including more classifiers? If the underlying causal model is more complicated (e.g. have more variables $X_1, X_2..$ or more complicated causal relationships among each other), what will be a good way to ensure the identifiability of the causal structure?", "limitations": "Please see the strength and weaknesses section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find the problem of identifying $E[Y| do (\\hat{Y} = y), X]$ in the performative prediction setting interesting. Overall the paper makes a novel contribution by providing sufficient conditions to identify the causal effect of predictions under some assumptions. The relation to the literature on spillover effect/social network analysis is also interesting.\n\nTo me, the major weakness to me is that the paper seems to make heavy assumptions in order to have interesting and clean results. For example, the whole paper is built on assuming a particular causal structure (Figure 1),  and the definition of the extrapolation error allows a clean separation between the influence of $X$ and $\\hat{Y}$, and in section 3.3, the author assumes the effect of $X$ on $Y$ and $\\hat{Y}$ is separable as well. Thus it wasn't clear to me how likely this work can be extended to more complicated settings. The authors provide some empirical justification, but I would like to see some theoretical insights on how the results would change/not hold without certain assumptions, and I believe it will greatly strengthen the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QDE3XCY-_A", "reviewer": "Reviewer_Lu7e", "summary": "The paper studies the problem of estimating counterfactual outcomes under a different predictive model, thus drawing a link between causal inference and the recent literature on performative prediction. The paper starts with hardness of performativity-agonostic learning and then describes identification results that enable the supervised learning approach to work. The authors also conduct experiments showing that the identification strategies work well in simulations and mild violations of assumptions are not of concern. ", "questions": "1. In the overparametrization identification result, the author mentions neural network as one of the examples. I am curious about if there is any empirical results illustrating this identification result under this complex model as a lot of the real world models are indeed getting more and more complex.\n2. In proposition 4, is $P(U|Y, \\hat{Y})$ necessary to know or $P(U|\\hat{Y})$ is enough to enable identification?\n3. Could you say more about applicability of knowing $P(U|Y, \\hat{Y})$ in some real world examples, like which examples you mentioned seem more reasonable assuming this?\n\nSome possible typos: line 293 in main text, $f_\\theta$ we will, removing we? line 307, should there be $Y_j$ after the expectation? ", "limitations": "The author is very clear about the limitation of the work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The writing is clear and the problem is well motivated and stated. The identification results are clearly stated with enough explanations and examples. Overall the paper is very clear in exposition. The problem studied is also having potential significance in real world examples as non-stationarity arises, which I particuarly like. The experiements are sound with code available for reproducibility.\n\nWeakness: The scope of the work is somewhat limited since the effect of the predictive model is only assumed to affect $\n\\hat{Y}$ (the authors acknowledge this in section 2.1). When restricting to this particular causal graph, the first two identification results are essentially direct application of existing results, thus limiting the originality of the work. Also, it would be nice to see some real examples in the experiment section. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-FMBHZ9ujnM", "reviewer": "Reviewer_rWDi", "summary": "This work investigates performative prediction through the lens of causal inference, aiming to identify causal structures which allow for counterfactual estimation of performative effects.\n\nInvestigation in this paper assumes a specific causal structure, in which the predicted value $\\hat{Y}$ acts as a mediator between the chosen prediction model $f_\\theta$ and the outcome $Y$, and $f_\\theta$ does not directly affect other variables (Figure 1). For the theoretical analysis, an infinite-sample setting is assumed, where it is assumed that the sample size is infinite. \nThree sets of theoretical results are presented:\n* The first set result assumes a specific performative structure, and quantifies the regret from naively training a model under performative behavior (Proposition 1,2). The result aims to illustrate the negative effects of neglecting a performative causal structure. \n* For the second set of theoretical results, authors identify “overlap/positivity” as the main limiting factor in performative extrapolation. Authors point out three avenues through which this concern can be alleviated - randomization of model outputs (Proposition 3), noisy measurement of covariates (Proposition 4), and incongruence between the effect of $X$ on $Y$ and the effect of $X$ on $\\hat{Y}$ in separable performative structures (“over-parameterization” in Proposition 5, discrete classification in Proposition 6).\n* For the third set of results, authors initiate investigation of spillover effects between users, identifying a spillover structure $G$ through which identifiability is possible.\n\nIn the experimental evaluation, authors present results which validate their claims, showing empirically that results extend beyond the theoretical guarantees.\n", "questions": "* When do we expect the performative causal structure assumption (Figure 1) to hold? When do we expect it not to hold? What are the implications of making a wrong assumption about the causal structure?\n* What is the relation between the different propositions in the theoretical analysis? How do they combine into an understanding of a “bigger picture”? Are they \"complete\" in the sense that they cover all possible influence modes?\n* The discussion claims that “one of the most important lessons from this work is that there is high value to logging the state of the deployed predictor when collecting data for the purpose of supervised learning”. I agree with this claim very much, and am wondering if there is a way to support or quantify it.\n* What is the connection to existing work on learning under distribution shift? Literature that comes to mind is:\n  * “Counterfactual Risk Minimization: Learning from Logged Bandit Feedback” by Swaminathan and Joachims 2015,\n  * “Discriminative Learning Under Covariate Shift”, Bickel et al. 2009\n  * “Actionable Recourse in Linear Classification”, Uston et al. 2018 \nLiterature on online learning and multi-armed bandits (e.g can we think of an “$\\varepsilon$-greedy” strategy as an inducer of identifiability?)\n", "limitations": "* The paper makes a very strong causal assumption, but the current version does not properly contextualize it within the existing literature. I feel that the paper would benefit from discussing its limitations and implications of the causal structure assumption in more depth.\n* Extension of theoretical analysis to the finite-sample case, and the relation to existing machine learning literature on similar topics.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Presentation and mathematical notations are very clear. Causal assumptions are made explicit.\n* Identifying “failure modes” and “success modes” of performative prediction is an approach which can establish strong theoretical foundations for this area of research, and help it further establish applicability in practice.\n* The investigation points out avenues of research which may be interesting for further inquiry, such as the relation between model over-parameterization and performative prediction, or the significance of spillover effects.\n\nWeaknesses\n* The paper claims to address the general problem of \"predicting from predictions\", but in practice assumes a very specific causal structure and does not sufficiently establish its applicability. In particular, the paper assumes a causal model in which $f_\\theta$ does not affect $X$. However, a direct causal path between $f_\\theta$ and $X$ exists in many practical cases, for example in the “Actionable Recourse” setting (Uston et al. 2018). In section 2.1, the authors mention many applications in which a performative structure may be present, but it’s not clear whether they can indeed be approximated using the causal structure presented in Figure 1.\n* The paper presents a collection of interesting insights, but it’s not made clear how they combine into an understanding of the “bigger picture”. Moreover, even though similar problems were investigated in previous literature (e.g “Counterfactual Risk Minimization: Learning from Logged Bandit Feedback” by Swaminathan and Joachims 2015, recent work on over-parameterization and OoD generalization in deep neural nets), these existing results are not discussed in the paper.\n* Theoretical results assume that sample size is infinite, and that predictors are minimizers of the expected risk (i.e $h^*=\\arg\\min_{h\\in\\mathcal{H}} \\mathbb{E}[w(h(x)-y)^2]$) - A quantity which cannot be directly optimized practice. Not clear how the formalism extends to the finite-sample setting. What would change when the dataset size is finite?\n\nA few minor typos: Line 263 (bet -> be), line 307 (inequality expression seems incomplete - missing a random variable next to the expectation operator?), line 374 (lesson -> lessons).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept