Conditional Forecasts and Proper Scoring Rules for Reliable and Accurate Performative Predictions
Authors: Philip Boeken
Onno Zoeter
Joris M. Mooij
Venue: NeurIPS 2025 poster ()
Keywords: performative prediction
conditional forecasts
proper scoring rules
divergence
IPW
causality
Primary area: theory
TL;DR: Conditional forecasts which optimise proper scoring rules guarantee (i) correctness of the forecast (performative optimality), (ii) performative stability under successive retraining and (iii) can be compatible with outcome optimisation.

Abstract: Performative predictions are forecasts which influence the outcomes they aim to predict, undermining the existence of correct forecasts and standard methods of elicitation and estimation. We show that conditioning forecasts on covariates that separate them from the outcome renders the target distribution forecast-invariant, guaranteeing well-posedness of the forecasting problem. However, even under this condition, classical proper scoring rules fail to elicit correct forecasts. We prove a general impossibility result and identify two solutions: (i) in decision-theoretic settings, elicitation of correct and incentive-compatible forecasts is possible if forecasts are separating; (ii) scoring with unbiased estimates of the divergence between the forecast and the induced distribution of the target variable yields correct forecasts. Applying these insights to parameter estimation, conditional forecasts and proper scoring rules enable performatively stable estimation of performatively correct parameters, resolving the issues raised by Perdomo et al. (2020). Our results expose fundamental limits of classical forecast evaluation and offer new tools for reliable and accurate forecasting in performative settings.

Conditional Forecasts and Proper Scoring Rules for Reliable and Accurate Performative Predictions
Philip Boeken University of Amsterdam
Onno Zoeter
Joris M. Mooij University of Amsterdam
Booking.com
Abstract
Performative predictions are forecasts which influence the outcomes they aim to predict, undermining the existence of correct forecasts and standard methods of elicitation and estimation. We show that conditioning forecasts on covariates that separate them from the outcome renders the target distribution porecast-invarian guaranteeing well-posedness of the forecasting problem. However, even under this condition, classical proper scoring rules fail elicit correct forecasts. We prove general impossibility result and identify two solutions: (i) in decision-theorett settings, elicitation of correct and coompiive-oompatble forecasts is possible if forecasts are separating; (ii) scoring with unbiased estimates of the divergence between the forecast and the induced distribution of the target variable yields correct forecasts. Applying these insights parameter estimation, conditional forecasts and proper scoring rules enable performatively stable estimation of performatively correct parameters, resolving the issues raised by Perdomo et al. (2020). Our results expose fundamental limits of classical forecast evaluation and offer new tools for reliable and accurate forecasting in performative settings.
Introduction
In many real-world settings, predictions influence the outcomes they seek to forecast. A forecast of high traffic may cause drivers change routes; grim economic outlook may shift investor behaviour (MacKenzie et al., 2007). These performative forecasts break the classical assumption that the target distribution is fixed instead, the forecasts are part the causal system. This performativity complicates the task of correct prediction, and undermines the standard foundations of scoring rules and statistical estimation. Moreover, correctness of forecast doesn't need align with desirability of the outcome (van Amsterdam et al., 2025).
If forecasts affect the target variable, correct forecasts need not exist: there may be no distributional fixed point where the forecast aligns with the induced outcome, as is the case in self-defeating prophecies. However, show that the forecasting problem becomes well-posee when the forecaster makes conditional predictions- - in particular, predictions given observed covariates that separate the forecast from the target: the target distribution becomes correct forecast is recovered.
and the existence of
This paper formalizes the performative forecasting problem in causal framework and studies con- ditions under which correct forecasting is possible. We prove that even under classical proper scoring rules generally fail to elicit correct forecasts. We provide two concrete solutions: (i) in a deciision-hhoortti setting, proper scoring can be recovered under structural condi- tions on the causal graph, producing incentive compatibility of the forecasters' score and the agents utility; (ii) we introduce proper scoring approach based on unbiased estimates of the divergence between the forecast and the induced distribution of the target variable. We also analyze how correct forecasting parameters can be stably learned from data in performative environments, resolving the problems raised by Perdomo al. (2020).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1.1 Related work and detailed contributions
Our analysis of the existence of correct performative forecasts extends the work of Oesterheld et al. (2023) to conditional forecasts and formalises it in a causal modelling framework (Pearl, 2009) (Section 2). We provide necessary and sufficient conditions for the existence of correct conditional forecasts (Theorem 1). There is vast literature on proper scoring rules for eliciting correct peofrrerfrrmatve)) forecasts, see e.g. Brier (1950); McCarthy (1956); Grünwald and Dawid (2004); Gneiting and Raftery (2007). To our knowledge, the only existing work on scoring rules for performative forecasts in specific decision-theorele
settings (Othman and Sandholm, 2010; Chen et al., 2011, 2014; Oesterheld and Conitzer, 2020; Hudson, 2025). In Section 3 we introduce general theory of proper scoring rules for performative forecasts without restricting how the forecast affects the target variable, contrary to what is done in the aforementioned literature. We then prove an impossibility theorem (2) about the non-existence of proper scoring rules in general performative settings, complementing impossibility result of Othman and Sandholm (2010) that scoring rules are generally not counterfactually strictly proper in decision theoretic settings. In Section 4 we introduce the decision-theoretic setup considered by the earlier mentioned authors. Here, we separately consider the properness of a scoring rule and its incentive compatibility with an agent's utility (which have been merged into single notion in previous works (Chen et al., 201 1; Oesterheld and Conitzer, 2020)), clarifying the different notions, and providing a solution to the issue raised by van Amsterdam al. (2025) that correct forecasts can be harmful. We then provide the additional insight that 'utility scores' (Oesterheld and Conitzer, 2020) can be made strictly proper by adding a suitably scaled strictly proper scoring rule to the utility score; see Theorem 4. In Section 5 we introduce another resolution of finding proper scoring rules in performative settings by scoring based on unbiased estimates of the divergence between the forecast and the observed outcomes; see Theorem 5 and its corollary. We give examples of unbiased estimators of certain strictly proper divergences, for binary and continuous outcome variables. This is compared with an inverse probability weighting method of Chen et al. (2011). Finally, in Section 6 we place this theory in the framework of Perdomo et al. (2020) for parameter estimation performative settings: we show that estimating parameters for conditional forecasts with scoring rule-based estimators resolves the issues raised by Perdomo et al. (2020): the estimated parameters don't change over successive retraining ( ("performative stability') and they induce correct forecast ("performative optimality") (Theorem 6).
2 Existence of correct forecasts through conditioning
Let y' be any measurable sample space of variable and let Py be a set of probability measures on ) We model the effect of forecast F E Py on outcome Y with causal model M (e.g. a causal Bayesian network or a structural causal model (Pearl, 2009)!), inducing a causal mechanism Par(Y do)))).
Let Mp be the set of such causal models M, so whose graph is subgraph of Figure when projected onto F and Y, and which satisfy (PM(Y): M E Mp) S Py.
F
Y
In a non- performaroee setting, if the forecaster believes p* be the true distribution, then forecast F is correct if F === P* This extends to the performative setting as follows.
Figure I
Definition 1. Given causal model M E Mp, we say that the forecast F is correct for M if F = Pas | (F))...
One can interpret correct forecasts as fixed-points of the mapping + PM do(F)), see also Grunberg and Modigliani (1954); Simon (1954) or Oesterheld et al. (2023). A selffullillingg prophecy is a forecast which causes itself to be correct (Merton, 1949). A correct forecast of Y need not exist. For example, for Po. P1 E Py with Po + Pt, let M be such that Po F
(M = P1 otherwise.
then M E MM has no correct forecasts. This phenomenon is well known:
An introduction to causal models is given in Appendix A.
2 For ease of exposition we don't let forecast F' depend on covariates X, nor let it be confounded with Y. In Appendix E, we show that the results from the main paper extend to the setting where F' depends on covariates X, which are allowed to be confounded with Y.
2

---

Example 1. Given a causal model M E MMp a self-defeating prophecy is forecast which causes itself to be incorrect, or more formally, forecast for which F PM do(F)). An example would be a forecast of rising Covid-19 incidences, causing policy makers to implement social distancing or vaccination programs, effectively mitigating many Covid- incidences. When is impossible for the forecaster (the "prophet") to be correct, this problem is sometimes referred to as the prophet's 's dilemma.
Let A be variable with discrete sample space² A and let PA-y be the space of conditional distributions F(Y A) A + Py,a + F(Y A = a). A conditional forecast of Y ||A is set of forecasts
F(Y (F(Y A d) : a PA-Y-
If the conditional forecast F(Y A) E PA-y affects the conditioning variable A and outcome Y then we have a causal mechanism as is graphically depicted in Figure 2a. Let Mpe C MM be the set of such causal models M, so whose graph is subgraph of Figure 2a when projected onto F, A and Y. If A is a set of variables, let Mpe S Mp be the set of causal models whose graph is a subgraph of Figure 2a after merging the variables in A into single multi-dimensional variable (see Appendix for a formal definition) and projecting onto F, and Y.
F
Y
2000 2015
2000
2000
(a) Causal graph of Mpe
(b) Example of conditional forecast
Figure 2
An example of a conditional forecast is the prediction of future average global tempera- tures given various scenarios of air air-pollution, provided by the IPCC (2021) (Figure 2b).
Example 2.
The notion of correctness of conditional forecasts is more nuanced than for marginal forecasts, since we can distinguish between correctness of the forecast for values of A that can be observed, and the 'counterfactual' values of A which cannot be observed.
Definition 2. Given a causal mechanism M E Mpe. the conditional forecast F(Y [A] is: observationally correct if F(Y A a) = (M a, do)) for all E A such that Pa = a a > 0; counterfactually correct F(Y A a) PM (Y = a, do()) for all a e A such that
Pm((= a do(F)) 0;
correct if it is observationally and counterfactually correct.
to Here, we pick the version of the conditional distribution PMr(Y (o,) PM ensure that (Y A a, do(F)) is well-defined for values E A with PAr(A= a do 0. Note that the support of A, i.e. which values of A can be observed after forecasting F, depends on the forecast F itself. The distinction between observational and counterfactual correctness will be useful for defining various types of properness of scoring rules in Section 3, where one may require a forecast only be observationally correct, or also counterfactually correct. In Appendix display similar distinction for non non-performative conditional forecasts.
A way resolve the prophet's dilemma is to make a conditional forecast instead of a marginal forecast. If in the causal graph G of model M have d-separation Y 1 F A, then the target distribution satisfies PM [Y] A, do F)) = PM (Y A), so it does not depend on the forecast itself. This for example holds if A perfectly mediates F and Y and is not confounded with Y
We assume that A is discrete to mitigate measure- theorreic problems.
*This also been noted by Hudson (2025) for the special case where A an action: "One method for avoiding performativity is to elicit predictions conditional on various actions that can be taken in response to the prediction. As this reaction is the causal pathway by which prediction affects outcomes, conditioning on it removes the performative aspect."
3

---

Definition 3. We call the target Y|A forecast-imariiant in M S Mpe pc if the target distribution does not depend on the forecast, so if PM (Y A, do P A) for all M E M'.
Example 3. Consider the prophet's 's dilemma of predicting Covid-19 incidences (Example 1). Denote
with A = 0, 1 whether policy makers implement social distancing, and let Y = 0, 1 respectively denote 'few" and 'many" subsequent Covid- 19 incidences. If for a marginal forecast F E [o.
of the event Y = 1 we have the stylised mechanism A = =1 > 1/2] and Y = 1 - A, then F # P(Y (d(F)) for all F E [0, 1], so the forecasting problem has no solution. However, we consider conditional forecast F(Y A 0), F(Y A 1) with stylised mechanism A arg min, F(Y| A = a) and Y = 1 - A, then the target Y A is forecast-invariant and the conditional forecast F(Y A 0) = 1, F A 1) O is correct, so the forecasting problem is well-posed."
It is immediate that for a given target Y YIA, forecast-invariance implies that Par (Y A) is the unique correct forecast. forecast. One way of motivating forecast-invariane system: the d-separation Y L3 F| | AA implies forecast-invariance as mentioned earlier. If no other assumptions are made, this d-separation even characterises pprecast-invarians and whether the forecasting problem is well well-posed, as shown in the following theorem. All proofs are provided in Appendix B.
-nnerastiiva not necessary for the existence of an (observationally) correct
is by relying on the causal graph of the underlying
Theorem 1. Let M'a S. Mpe be the set of models compatible" with causal graph G, then the following are equivalent:
i) For every M E MMa there exists an observationally correct correct YIA;
it) For every M E MM there exists correct forecast of Y |A;
iii) Y Y|A forecast-iiaaaiinn for all M E &M'gi
iv) Y L3 FI
3 Eliciting correct performative forecasts with proper scoring rules
Without appropriate incentives, forecasters may have motives to report an incorrect forecast. To address this, mechanisms must be designed to elicit correct forecasts: that is, to set the forecaster's incentives so that correct reporting is their optimal strategy. This mechanism is typically modelled as a principal eliciting a probabilistic forecast from a forecaster. They agree beforehand on a scoring rule S Py y - R which determines the payout of the forecaster: after each forecast F and observation y, the forecaster receives S(F, y) from the principal. A scoring rule is called proper if the y) forecaster maximizes the expected score (F,, P) == f ((FF,))Pdd by reporting a correct forecast:
Definition& 4. Scoring rule S is proper relative to Py if for all F, P* E Py we have S(F, S 5(P",P") and strictly proper if the inequality is strict when F is incorrect.
We sometimes refer to such a scoring rule as (strictly) proper in the classical sense. Well known examples of classical scoring rules are the Bregman score S(F,y) y) = (f(())) (x)( R convex and differentiable and where is a density of F. The Bregman score is proper, and strictly proper if & is strictly convex. By picking ((f) = f flg f. the
(f())) for : R
Bregman score reduces to the log-score S(F.y) y) = In(f()). If (0, 1]. by picking p = f2
and setting ʃ == F(Y 1) the Bregman score reduces to the Brier score S(F = - -s y)² (Brier, 1950). If y = Rd, another example is the energy score S(F. y) = Ey'=
yll F Y'Il. where Y, are independent random variables, and the expectations are to be computed analytically of via a sampling scheme (Székely and Rizzo, 2013). The energy score is strictly proper with respect the class of distributions P(Y) with E finite. For other examples we refer to Grünwald and Dawid (2004) and Gneiting and Raftery (2007).
* As an alternative solution to mitigate performativity in Covid-19 forecasting, Bracher et al. (2021) consider marginal, short-term forecasts with "brief time horizons, at which the predicted quantities are expected to be largely unaffected by yet unknown changes in public health interventions." "
"Menning that the Markov property holds: A 1 B|C = A Pa B for sets variables A, B, C. We can have a d-connection Y ( F A but forecast invariance in M due to faithfulness violations after intervention do) (Boeken et al., 2025). However, forecast invariance for all M M's implies Y 1g F A.

---

As straightforward extension of scoring rules to allow for conditional forecasts F E PA-Y,, we refer map S: PA-Y A y R as a conditional scoring rule; after forecasting F and observing and y, the forecaster receives S(F, a. y) from the principal. Any "classical" scoring rule S' induces ==a), a), y); if we consider a classical scoring
conditional scoring rule S via S( F, a, == (F A= rule for conditional forecasts, we always implicitly consider this induced conditional scoring rule. We define propemess of conditional scoring rules for non-performative forecasts Appendix C; this generalised by Definition 5 in Section 3.1 below.
Most existing work on scoring rules assumes that the forecast does not affect the predicted variable, with an explicit mention in McCarthy (1956), assumption (iv): "We assume that neither the forecaster nor the [principal] can influence the predicted event [...]". Our main interest lies in dropping this assumption, as is often required in practice (Perdomo et al., 2020; Bocken et al., 2024).
3.1 Defining properness of scoring rules for performative forecasts
To extend the theory of proper scoring rules to the performative setting we adhere to the same rationale as in the classical setting: if the forecaster knows that M is the true model, then the expected score should be maximised be considered with respect to Pau (Y do (F)). Given a scoring rule S Py x y - R, a marginal forecast F E Py and causal model M Mp the expected score
correct forecast, where now the expected score of forecast F will
(p = / P(F (dy do(F)).
For conditional scoring rule S: PA-YY x A y + R, conditional forecast F(Y A) E PA-Y and model M E Mpe the expected score
(P, = f S(F, a, PM (da, dy do())).
Recall Definitions 1 and 2 of the various types of correctness of performative (conditional) forecasts. Definition 5. For marginal forecasts, a scoring rule S is proper relative to M' C Mp if Sp is maximised at a correct forecast, and strictly proper if all maximisers are correct. We call a conditional scoring rule
observationally (strictly) proper relative to M' S Mpe if Spe is (only) maximised at observationally correct forecasts for all M M';
counterfactually (strictly) proper relative to M' S Mpe if Spe is (only) maximised at counterfactually correct forecasts for all M E M';
(strictly) proper relative to M' C Mpe if it is observationally and counterfactually (strictly) proper, that is, if pe is (only) maximised correct forecasts for all M M''
we consider a class of models M' where F does not affect Y, then Definition 5 equivalent to Definition 4 and its conditional version (Appendix C). For a scoring rule to be (observationally) proper with respect to M', is required that for every M E M' there exists an (observationally) correct forecast P* signifying the subtleties of Section 2: if one only makes assumptions about the causal graph, Theorem implies that the target variable must be forecast-invariant for scoring rule to be proper. Note that if a correct forecast exists, any constant scoring rule is proper; throughout we only consider non-constant scoring rules.
3.2 Impossibility of eliciting correct performative forecasts with classical scoring rules
Few examples of conditional scoring rules exist in the literature; the most straightforward examples are conditional scoring rules S induced by classical scoring rules S' through S a, =F'(F(Y A
"To knowledge, the only related literature on scoring rules for conditional forecasts PA-y Kadane et al. (1980); Schervish et al. (2009) in the non-performative setting, who consider conditional scoring rules of the form ((F(Y) = a), y). and Othman and Sandholm (2010); Chen et al. (2014); Oesterheld and Conitzer (2020) who consider scoring rules for conditional forecasts in the performative setting of the form S(F, y). S(F,, Par (A), a, y) and S(F(Y' A a). y) respectively.
"In decision theooretic context, Othman and Sandholm (2010) and Chen and Kash (201 refer to scoring rules which are both observationally strictly proper and counterfactually proper 'quasi-strictly proper' scoring rules.
5

---

a). v). In this section we investigate whether one can expect these induced conditional scoring rules to be proper in the performative sense, even if they are strictly proper in the classical sense.
As remarked in the previous section, porecast-invarianee is necessary for a scoring rule to be proper; however, it is not sufficient. Typically, the easier the forecasting problem, the higher the expected score of a correct forecast. If traffic jams are hard to predict on a highway called 'Route A', then it can be beneficial for the forecaster incorrectly predict terrible traffic jam on Route A that all cars will take an alternative, easier-to-predie Route B; the forecaster will benefit from a correct forecast on Route B, and will never be punished for the incorrect forecast on Route A. In this mechanism the scoring rule is observationally strictly proper but not counterfactually proper. The following example is adapted from Othman and Sandholm (2010):
Example 4. Let A y {0. 1} and M be such that Y is porecast-invariant with PM 1 A = 0) = 0.5, PM = 1| A = 1) 0.25 and the 'decision rule' A = arg max, F(Y = 1 1 a). If the forecaster reports the correct forecast A is chosen and the expected Brier score -0.25, and if the forecaster reports F(Y = 1| = 0) = 0.2, F(Y= 1|A=1> = 0.25 then A A=1 is chosen the expected score is -0.25- (0.7)
is -5.5 (0.5) -0.5))-0.5-(((( (0.5)²
0.19. The scoring rule is observationally strictly proper since for the observed value A = 1 the score is maximised at a correct forecast, but is not counterfactually proper since it pays off to misreport for the value A 0, which is not observed.
Inspired by the self- self-defeating prophecy, we can construct even more pathological examples where the scoring rule is neither observationally nor counterfactually proper. Consider mechanism [A] | do))) which can pick between two values of A: one for which Y is easier to predict, giving higher expected score, and one for which Y is harder to predict, giving a lower expected score. If for any (close to) correct forecast the mechanism (M do(F)) picks the barder-to-predicc action, it is beneficial for the forecaster to misreport since this induces an casier-to- torieritt--reiictt action, yielding a higher expected score. Hence, in this mechanism the scoring rule is observationally and counterfactually improper. Notably, the mechanism PM (o)F)) can be, but need not be, deterministic. In Appendix F.1 we provide examples and 3D-plots of this phenomenon.
The following theorem shows formally that in general performative settings, non-constant proper scoring rules don't 't exist. The proof relies on the phenomenon of the self-defeating prophecy, as just explained.
Theorem 2. Let M' C MMp be the set of models such that Y|A is forecast-invariant where (A | do(F)) has full support for all F E Py and all M E M', or where PM (A| do(F)) is deterministic for all M E M'. If S is scoring rule with respect to Py such that there are Po, P1 E Py and forecast F Py with F + P1 and ((o,, Po) < S(F P1) S(P,, then S is not observationally proper for predicting Y|A in M'.
Notably, this scoring rule is also not observationally proper for any larger class of models, for example when one does not require Y|A to be forecast-invariant or if no assumptions are made on the mechanism PM (A | (F). We will cover two solutions which go beyond classical scoring rules: in Section 4 we consider a secissonn-heorret class of models with particular deterministic mechanism (A (F)) (the same for all M) for which proper conditional scoring rules exist, and in Section 5 we relax the setting by allowing the principal to score forecasts using estimated properties of the distribution (A, Y (F)).. Another notable solution is given by Hudson (2025), who relaxes the setting by considering multiple forecasters who compete in zero-sum game.
4 Properness and incentive compatibility in a decision-theoretie setting
Often, the forecast provided to an agent who acts based on the forecasted information." 10 The quality and reliability of the forecast thus have a direct impact on the agent's behaviour and, consequently, on outcomes which may be of interest to the principal. More specifically, given conditional forecast F, let the agent take an action which maximizes the expectation of a utility U(a, v), so we have aF := arg max,e A U(a, y) F(dy a). " This requires the forecast F(Y (A) to be
10 Note that this agent might be the principal herself. "Thee action of is often referred to as the Bayes act, see also Dawid (2007); Brehmer and Gneiting (2020); Dawid (2021).

---

forecast of the causal effect PY do(A))) which holds in our setting since we assume that there is no confounding between A and Y: let Mat S Mpc be the set of causal models M with PMr | do(F)) = Sap.
Definition 6. Given a utility U and a set of causal models M' S Mpps we call a conditional scoring rule S incentive compatible" with U if for all M E M' we have arg max, pF = arg m J' U (aa,y))))))))))))) Pas (da, dy do(F)).
The goal of the agent is to maximise its expected utility, and the goal of the forecaster is to maximise its expected score. As also noted by Othman and Sandholm (2010) and by van Amsterdam et al. (2025) for marginal forecasts, these two incentives don't 't necessarily align:
Example 5. Consider Example 4, interpreted as forecast being given to an agent with utility U(a, y) = y: then indeed A = ar = arg max, F(Y = 1 = a). The optimal action for the agent is a" 0, but to maximise the score the forecaster reports F(Y = 1 A 0) = 0.2 and F(Y A = 1) = 0.25, forcing the agent to take the suboptimal action ap = 1. Despite the forecast being observationally correct, it is not incentive-compatible with U.
It is not hard to see that every strictly proper conditional scoring rule incentive compatible with any utility. Generalising the result of Othman and Sandholm (2010) (Theorem 4) to our setting, we moreover have that Mat proper conditional scoring rules must have the form S(F,a, a, y) = S(F(Y A = ap), 4r, y), so they may only depend on forecasts for unobserved actions through ar- and they can never be counterfactually strictly proper.
Nevertheless, proper, incentive compatible conditional scoring rules exist; we will later show that they can even be made observationally strictly proper and incentive compatible. In the non-performative setting where marginal forecasts F(Y) affects A but where A does not affect Y. it is well known that the utility score S(F, y) == U (ap, y) is proper (Dawid, 2007: Brehmer and Gneiting, 2020). In the performative setting with conditional forecast, we can straightforwardly consider the conditional scoring rule S(F,, a. y) = U U'(aF, y): since the expected score then equals the expected utility, is automatically incentive-compatibe
- not only in Mas. but even for all M E .Mpe- so for any (possibly stochastic) policy Par (A do))).. As shown by Oesterheld and Conitzer (2020), this utility score is proper in MMi:
Theorem The utility score incentive- comppail- with U, and proper if IA is forecast-invarian in M' S Ma for given causal graph G one considers the class of models M' S Maa which are consistent with G, then forecast invariance of Y|A necessary for the utility score to be proper.
A main restriction of the utility score is that it is not observationally strictly proper: there can be multiple observationally incorrect forecasts which maximise the score. However, in the following we show that there does exist a conditional scoring rule which is incentive-compatible and observationally strictly proper. If the principal knows that the expected utility between any two actions differs with at least A > 0, then the principal can elicit incentive-compeee compatible and observationally correct forecasts by first scoring using the utility score to ensure that the optimal action a* is taken, and subsequently add scaled strictly proper scoring rule S' for F(Y A = a²) to ensure that the maximum is uniquely attained at the observationally correct forecast (Y A = a"), where the magnitude of S' is bounded by A to ensure that it does not incentivise a forecast which induces an action different than a". This utility gap A can for example be assumed if A is binary indicator of 'high-risk, high-reward' action, like risky operation or bold marketing campaign.
Theorem 4. Let YIA be forecast-invariant in M' S Ma let U E 10, let S' E [0, 1] be a proper scoring rule in the classical sense and let there be A > 0 such that min {) Y) l) (u Y) [a') & d'} > for all M E M', then S(F, a, y) := U(aF v) + A ('(( A (F), 3) is proper and incentive-comee cocentive-compatbbl with U. If S' is strictly
proper in the classical sense, then S is observationally strictly proper.
Eliciting correct forecasts with estimates of the performative divergence
Given a scoring rule S: Py x y - R, forecast F E Py and distribution P E Py, the generalised entropy is defined as HP)) == - P) and the generalised divergence as D(F, P) == -H()
We use the term 'incentive compatibility" different than Chen et al. (2011): they let it refer to strict properness of conditional scoring rule, which implies our notion of incentive compatibility.
7

---

S(F,P) - ( S(F, P(dy), following Grünwald and Dawid (2004). If S is strictly proper, the entropy is the maximal attainable expected score - see also Example 4, where the forecaster manipulates towards easier problems, i.e. problems with lower entropy. Recall the examples of classical scoring rules from the beginning of Section 3: the Bregman score induces the Bregman divergence, the log-score induces the Shannon entropy and Kullback-Leibler divergence, and the (negative) expected Brier score equals the mean square error -S(F, P) = Ep - y) (p f)a+ + p(1 - p), which indeed decomposes into the variance H(P) = p p) and the bias term P) = (p- f)?. The entropy of the energy score is given by (P) = 3Ey,y'=Pll
Y and - =F
the divergence by D(F, P) = EY=FYY p -YIl -
- || - -Y'll --'1
Y'Il
Definitions of generalised entropy and generalised divergence for non non-performative conditional forecasts are given in Appendix C.
In the classical setting, maximizing proper score is equivalent to minimising the divergence: the correct forecast attains zero divergence (Grünwald and Dawid, 2004; Gneiting and Raftery, 2007). Inspired by this, we aim at proper scoring of forecasts with the performative divergence. To this end, we show that the performative divergence is positive and optimised at observationally correct forecasts (Theorem 5). We subsequently suggest how the principal can calculate the divergence (Corollary and Lemma 1), which is non-trivial problem since the causal mechanism M is unknown to her.
Indeed, for marginal performative forecasts, we see that the deviation of the forecast F from the induced distribution Par (Y do(P)), given by the performative divergence Dp M) := D(F, Pas (Y ddo())) is also positive. We have DD(F,M) holds with equivalence if the divergence is induced by a strictly proper scoring rule. We suitably extend this conditional performative forecasts as follows:
if F is correct for M, and this
Definition 7. Given a scoring rule S, conditional forecast F(Y |4) E PA-Y and causal model M E Mpe. the performative divergence is defined
Dpc (F, = / (S(PM a, d, - (((()) A = a), Pas (da, dy do(()).
f With performative entropy Hpe(M cdo(F)) = H(M |A=
= a, do)) do)))), the performative divergence can be written as (F, M) = - (M [do(F)) - ppc A), M).
Although the divergence cannot be written as the expected value of a scoring rule, we use the same nomenclature of Definition 5 and refer to the divergence as inneevationallyycounteffaacc proper, keeping in mind that the divergence is minimised at correct forecasts, instead of maximised.
(strictly)
Theorem 5. Let M' C Mpc be such that for every M E .Me there exists an observationally correct forecast for Y|A. If is proper in the classical sense, then the divergence Dpe is positive and proper. If S is strictly proper in the classical sense, then Dpc is observationally strictly proper.
Notably, Theorem 5 does not require the target to be forecast-invariant. Also note that full support of PM (A do(F)) implies strict propernsss of the divergence.
When scoring the forecast directly with conditional scoring rule, this is straightforward for the principal to carry out: upon receiving a conditional forecast F(Y A) and observing a and y. the resulting score is S(F,a, v). The forecaster picks the forecast which optimises the expected value 3pc (M). When scoring with the performative divergence however, upon receiving the forecast F(Y A) and observing a and y. she should score with S(PMs (Y A = a, do(F)) y) ((F(Y A = =a)) This either requires access the distribution PM | A a, do ()) (in which case forecasting is redundant), or more practically, it requires the principal to estimate the entropy, i.e. the expectation of the first term. More generally, the principal can estimate the divergence. If the employed estimator Dpc is unbiased, the forecaster optimizes EM [Dp do(F)) = Dpc M)) which renders the scoring method observationally strictly proper by Theorem 5.
Corollary 1. Let M' C Mpc be such that for every M E MM there exists an observationally correct forecast for YIA, let S be proper in the classical sense, and let (A₁, Yi).,..... (An, Yn) ~ (A, do(F)). Any unbiased estimator Dpo(Ai,..,,,, Yn) of the performative divergence (Pc(F, M) is proper. for any sample size E NN such that Dpc is well-defined If S is strictly proper in the classical sense, then Dpe is observationally strictly proper.
Notably, the variance of the estimator and the sample size are irrelevant for the properness of the method. To demonstrate the applicability of this result, we show that for finite A and binary or
8

---

continuous Y, there are strictly proper classical scoring rules with corresponding unbiased estimator for the performative divergence.
Lemma 1. Consider the setting of Corollary 1. For binary Y and S the Brier score, the estimator
11
pa, (1 Pa:))
DBriee pe
Σ n2
(Pa: - -Sa.)²
Ra. -1
is unbiased, where we write na == E'=1 aa; = a}. Po ==
1 Ei=
a} and fa ==
ne
F(Y = 1 A u). For Y E Rn with integrable norm and S the energy score, the estimator
Σ
jzi 1 =
U5ll
1 BY.
Denergy
Ey Fe, I|Y -B-ll
will
|| Il
2
Rai
2
is unbiased.
Note that both estimators require n, 2 2 for all A. The Brier score is strictly proper, and the energy score is strictly proper with respect to the class of distributions with finite first moment (Gneiting and Raftery, 2007). The estimator for the energy score requires an evaluation of the expectations, either analytically or via an (unbiased) sampling scheme.
Instead of making scoring rule proper by subtracting the estimated entropy, one can multiply a classical scoring rule with estimates of the inverse probability weights (IPWs) 1/PM(A = S(F(Y) A a), PM (A
(do(F)) and properly score with SIPw(F, a. proposed by Chen et al. (2011). In Appendix D we prove for completeness that if Pas (A ()(F)) has full support and S is (strictly) proper scoring rule in the classical sense, then IPW score SiPw is (strictly) proper in the performative sense. For properness of the IPW score the principal either requires knowledge of PM (Al do(F)), must be able to estimate it with an unbiased estimator. is unclear whether such an unbiased estimators exist. In Appendix G we provide an empirical analysis of unbiased estimates of the divergence on synthetic data, and we compare with biased plugin estimators of the divergence and IPW score.
(d)FF)) as
Parameter estimation based on scoring rules
In the previous sections we have taken the viewpoint of the principal by addressing how to elicit correct forecasts via proper scoring rules, assuming that the forecaster picks a forecast which maximises the expected score. In this section switch to the viewpoint of the forecaster who aims at estimating the parameter of the forecasting model from data. We show that Y ||A is forecast-invarian and the parameter of the conditional forecast is estimated using the divergence related to strictly proper scoring rule, then this yields correct forecast which is immediately performatively stable, performatively optimal, and subsequently maximises the expected score when the forecaster is scored by a principal using any proper scoring method.
Formally, consider parametrised forecasts (Fa 0 }} for some parameter space O C Rd. If Fo does not affect Y and has data (y1,-.., Va) with empirical distribution P., one employ scoring rules for parameter estimation by minimising the divergence â = arg min, D(Fe, P). which is equivalent to minimising the empirical negative score = arg ming Σ. (Fe. y1) which satisfies the estimating equation Σnm VoS y:) = 0, and therefore defines an M-sstmmator (Dawid and Musio, 2014), which (under regularity conditions) is consistent estimator for a correct parameter 0*, i.e. a parameter such that the forecast Pg- is observationally correct. For the log- score this amounts to maximum likelihood estimation. In machine learning, a popular scheme is score matching with the Hyvärinen score, amounting to the minimisation of the related divergence M)) Jyy vvp(v) V fe Pdy (Hyvärinen, 2005). We investigate whether estimation based on scoring rules can also be employed when the forecast is performative, by formulating this question in the framework of Perdomo et al. (2020).
Definition 8. Given a loss function ((O, a, y). the corresponding performative risk is defined as R0) = J E(A a, y) Pad (da, dy (do))))), and minimiser 9ro of the performative risk is re- ferred to as performatively optimal. The decoupled performative risk is defined as Rd (0 0,) = I [(00+1, a, y) PM (da, dy do( )o. and parameter Ops is performatively stable if Ops arg ming Rd (0, 0 Pp)).

---

The preceding sections concern the elicitation of correct forecasts through scoring rules, calculated as the score of forecast with respect to the distribution that forecast induces. These metrics (expected negative score R25(0) := S pe (L's, M), divergence R (0) == pe (Fe, M),, ), and analogue definitions for utility score and IPW score) can be interpreted as a generalised type of performative risk, and if these methods are observationally strictly proper, then observational correctness of the forecast is equivalent to performative optimality of the parameter.
For parameter estimation the decoupled performative risk is used, which does not correspond to the 'performative' metrics as just mentioned. Instead, they correspond to more classical metrics, with an additional dependency on the previous parameter:
Rig (0.11, B1) / S(Fo.+ (Y (YIA= o), PM A a, do (h (da (o.)))
BC (0c+1,,, = / (F (Y A= =0) PM ( = a, do( Fe. ()) Pas (da (F.)).))
If S and D are strictly proper in the classical sense then they are observationally strictly proper for conditional forecasts (see Appendix C) so upon setting Bz+1 = arg ming R$. (0, 8,) or equivalently 8t+1 = arg ming RD (0. ds) we obtain the recurrence relation Fe.,;
for which we indeed have that (0, 0) = 12 (0) and -0,0)
PM (Y) A, (o(Fe,)) (which holds (M do Fo.))- almost everywhere), assuming there is no model misspecification. Now, if 0r is observationally correct then we obtain Fo. Pas A, do Fo, )) and hence Fo.., = Fo.. i.e. performative stability. If Y| A is forecast-invariant then we obtain Fo aforementioned recurrence relation, so if the supports of PM (A (F.)) and Pas do Fo. )) coincide we see that Fe...1 is observationally correct. This parameter is in turn performatively optimal for any performative risk for which performative optimality corresponds to correctness of the parameter. We summarise these insights in the following theorem:
PM A) (PMr (A do Fo, )-almost everywhere) in the
Theorem 6. Suppose that for every model M in the class M' C Mppe and every forecast F E PA-Y the distribution PM (A F)) has full support and that Y|A is forecast-invarians and let D be the divergence induced by a strictly proper scoring rule. For any parameter 02 for conditional forecast Fe, (Y A).. we have that oz+1 == arg ming R3 (0, 0t) yields correct forecast which is performatively stable and performatively optimal with respect to (D(
Assume that the forecaster will be scored by the principal using a performatively proper scoring method, e.g. with the utility score from Theorem 4, or with an unbiased estimate of the performative divergence as in Corollary 1. If the forecaster has merely estimated her parameter using the procedure Theorem 6 and does not have an estimate of Par (A do)F)), she is unable to evaluate (and hence optimise) her expected score directly. However, since the expected score is optimised by the correct forecast, the procedure from Theorem 6 optimises her expected score.
Corollary 2. Consider the setting of Theorem 6. If the forecaster is scored with a strictly proper scoring rule, then the parameter 0141 := arg ming Rej (0, 0c) is an optimiser of the expected score.
7 Discussion
We develop theory of performative forecasting that describes when and how making predictions can influence the very outcomes being predicted, showing that classical scoring rules need not be proper nor compatible with outcome-optimisation under performativity, and we propose new ubie and divergence-based scoring methods that incentivise correct forecasting. We further prove that these methods yield stable and optimal parameter estimates under repeated retraining. We hope that our impossibility result and our prominent emphasis on incentive compatibility instill caution when scoring rules are applied in practice, and that the potential solutions inspire further research on reliable and accurate performative predictions.
The provided proper scoring methods depend on couple of restrictive assumptions which should be scrutinised in practise, and which open up new paths for further research. First, the assumption of inrecast-ivvaringg may be hard to motivate in practise. We also assume a discrete sample space A; extension to the continuous case would be of interest. We conjecture that our theory for probabilistic forecasts can be extended to point forecasts (or actually, various functionals of PYr(Y A, do FF)) through consistent scoring functions (Lambert et al., 2008; Gneiting, 201 1; Holzmann and Fulert, 2014).
10

---

Acknowledgments and Disclosure of Funding
This work is supported by Booking.com We thank Jason Hartline for helpful discussions, and Tilmann Gneiting, Sourbh Bhadane and anonymous reviewers for helpful remarks.
References
Boeken, P.. Forré, P., and Mooij, J. M. (2025). Are Bayesian networks typically faithful? arXiv:2410.16000 16004.
Boeken. P., Zoeter, O., and Mooij. J. (2024). Evaluating and Correcting Performative Effects of Decision Support Systems via Causal Domain Shift. In Proceedings of the Third Conference on Causal Learning and Reasoning, pages 551-569 PMLR.
Bongers, S., Forré, P., Peters, J., and Mooij, J. M. (2021). Foundations of structural causal models with cycles and latent variables. The Annals of Statistics, 49(5))2885-29)) 2915.
Bracher, J., Wolffram, D., Deuschel, J., Görgen, K., Ketterer, J. L.,, Ullrich, A., Abbott, S., Barbarossa,
M. V., Bertsimas, D., Bhatia, S., Bodych, M., Bosse, N. I., Burgard, J. P., Castro, L.., Fairchild, G., Fuhrmann, J., Funk, S., Gogolewski, K., Gu, Q., Heyder, S., Hotz, T., Kheifetz, Y., Kirsten, H., Krueger, T., Krymova, E., Li, M. L.,, Meinke, J. H., Michaud, J., Niedzielewski, K., Ožański, T., Rakowski, F., Scholz, M., Soni, S.. Srivastava, A., Zieliński, J., Zou, D., Gneiting, T.,, and Schienle, M. (2021). A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave. Nature Communications, 12(1):5173.
Brehmer, J. and Gneiting, T. (2020). Properization: Constructing Proper Scoring Rules via Bayes Acts. Annals of the Institute of Statistical Mathematics, 72(3):659-60 673.
Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability, Monthly Weather Review, 78(1):1-3.
Chen, Y., Kash, I., Ruberry. M., and Shnayder, V. (2011). Decision Markets with Good Incentives. In Chen, N., Elkind, E., and Koutsoupias, E., editors, Internet and Network Economics, pages 72-83, Berlin, Heidelberg. Springer.
Chen, Y. and Kash, I. A. (2011). Information elicitation for decision making, In The 10th International Conference on Autonomous Agents and Multiagent Systems Volume 1, AAMAS '11, pages 175- 182, Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems.
Chen, Y., Kash, I. A., Ruberry, M., and Shnayder, V. (2014). Eliciting Predictions and Recommenda- tions for Decision Making. ACM Transactions on Economics and Computation, 2(2).6:16::77
Cover, M. and Thomas, A. (2006). Elements of Information Theory. Wiley-Interscience, Hoboken, N.J, 2nd edition.
Dawid, A. P. (2007). The geometry of proper scoring rules. Annals of the Institute of Statistical Mathematies, 59(1):77-93.
Dawid, P. and Musio, M. (2014). Theory and applications of proper scoring rules. METRON, 1292):169-183
Dawid, P. (2021). Decision-theoretic foundations for statistical causality. Journal of Causal Inference, 9(1):39-77.
Forré, P. and Mooij, J. M. (2025). A Mathematical Introduction to Causality. Lecture notes.
Gneiting, T. (2011). Making and Evaluating Point Forecasts. Journal of the American Statistical Association, 106(494):46--62.
Gneiting, T. and Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477):35-337.
Grunberg, F. and Modigliani, (1954). The Predictability of Social Events. Journal of Political Economy, 62(6):465-473
11

---

Grünwald, P. D. and Dawid, A. P. (2004). Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory. The Annals of Statistics, 13744:333.
Holzmann, H. and Eulert, M. (2014). The role the information set for forecasting- applica- tions to risk management. The Annals of Applied Statistics, 8(1):595-621 621.
Hudson, R. (2025). Joint Scoring Rules: Competition Between Agents Avoids Performative Predic- tion. Proceedings of the AAAI Conference on Artificial Intelligence, 39(26))7733-9211 2309222466...
Hyvärinen, A. (2005). Estimation of Non-Normalized Statistical Models by Score Matching. Journal of Machine Learning Research, 6024):695-709.
IPCC (2021). Summary for Policymakers. Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA.
Kadane, J. B., Dickey, J. M., Winkler, R. L., Smith, W. S., and Peters, S. C. (1980). Interactive Elicitation of Opinion for Normal Linear Model. Journal of the American Statistical Association, 75(372):845-854,
Lambert, N. S., Pennock, D. M., and Shoham, Y. (2008). Eliciting properties of probability dis- tributions. In Proceedings of the 9th ACM Conference on Electronic Commerce, EC '08, pages 129-138, New York, NY, USA. Association for Computing Machinery.
MacKenzie, D. A., Muniesa, F., Siu, L., al. (2007). Do economists make markets?: on the performativity of economics. Princeton University Press.
McCarthy, J. (1956). Measures of the value of information. Proceedings of the National Academy of Sciences, 4259):654-655
Merton, R. K. (1949). Social theory and social structure. New York: Free Press.
Oesterheld, C. and Conitzer, V. (2020). Decision Scoring Rules. In Web and Internet Economics: 16th International Conference, WINE 2020.
Oesterheld, C., Treutlein, J., Cooper, E., and Hudson, R. (2023). Incentivizing honest performative predictions with proper scoring rules. In Proceedings the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, pages 564--1774 PMLR.
Othman, A. and Sandholm, T. (2010). Decision Rules and Decision Markets. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems.
Pearl, J. (2009). Causality. Cambridge university press.
Perdomo, J., Zmic, T., Mendler-Dünner. C., and Hardt, M. (2020). Performative Prediction. In Proceedings of the 37th International Conference on Machine Learning, pages 7599-7609. PMLR.
Schervish, M. J., Seidenfeld, T., and Kadane, J. B. (2009), Proper Scoring Rules, Dominated Forecasts, and Coherence. Decision Analysis, 6(4):202-
Simon, H. A. (1954). Bandwagon and Underdog Effects and the Possibility of Election Predictions. The Public Opinion Quarterly, 18(3):245-22 253..
Székely, G. J. and Rizzo, M. L. (2013). Energy statistics: A class of statistics based on distances. Journal of Statistical Planning and Inference, 149-))222
van Amsterdam, W. A. C., van Geloven, N., Krijthe, J. H., Ranganath, R., and Cinà, G. (2025). When accurate prediction models yield harmful self fulfillinn prophecies. Patterns (New York, N.Y.), 6(4):101229.
Verma, T. (1993). Graphical aspects of causal models. UCLA Cognitive Systems Laboratory, Technical Report (R-191).
12

---

A Introduction to causal models
The causal models that we consider can be interpreted as causal Bayesian networks or structural causal models. For completeness, briefly present both modelling frameworks. This material can largely be found in Pearl (2009); Bongers et al. (2021); Forré and Mooij (2025); Boeken et al. (2025).
A.1 Graphs
A directed acyclic graph (DAG) is tuple G - (V, E) with V finite set of vertices and E C V V set of directed edges such that there are no directed cycles. An acyelic directed mixed graph (ADMG) is a tuple G = (V. E, I.) with vertices V, directed edges F, and bidirected edges I. C V x V such that there are no directed cycles.
Given DAG G with path π *** * b, a collider is vertex with - U in π. For sets of vertices A, B3 C. V we say that A and B are d-separated given C, written 1 B C, if for every path т = +- *+ b between every E A and b E B, there is a collider on т that is not an ancestor of C, or if there a non-collider on C. The sets A and they are d-separated, written A Lg B|C.
d-connected given C'
Given a DAG G over V U W, the latent projection G onto is the ADMG Gy with vertices V, directed edges
b if there is a path a Wh - ... Whp - in G with wi W for all
i = I,..., n (if any), and bi-directed edges a <+ b there is a bifurcation a t- W1 < f wk 7
W. b in G with w; E W for all i 1, b.... (Verma, 1993).
A2
A2
A1
Y
F
A
Y
A3
L
As
(a) Graph G
(b) Latent projection onto F, A1, A2, As,
Figure Example of DAG G and latent projection.
For an ADMG G over V, a set of variables A An S V can be merged into single variable A & V by removing from G all vertices A,, adding vertex A, and for every E V {Ai,..., An add the edge v + A/ 4 V + v there is an {I, n. such that + A/ A, A; in G.
F
Y
Figure Merging of variables A1, A2, As of the graph in Figure 3b to single vertex A.
In Figure 3a, we see that F FL Y A2, A2,AA. As. so defining A = A2, we indeed have forecast invariance of Y|A (see Theorem 1). Note that testing for forecast-invariance the original graph (where the set of variables A is not merged) is stronger than testing it after the merging of A: we have F $1 y A1, A2, Ag in Figure 3a and equivalently in Figure 3b, but we have F y y A in Figure
A.2 Bayesian network
Given a DAG G, writing pa(v) for the set of parents of v in G. a Bayesian network over G is tuple of Markov kernels (P(X. Xpa(n))eev. We refer to the joint distribution P(Xv) 8 vev P(X, [Xpa()) as the observational distribution.
13

---

Given a Bayesian network and subset of variables T S V and values 2T E Xr, the intervened Bayesian network given by the set of Markov kernels
82z (Xx) P(X, paa(t) = P((x) Xpa(t)
i u
This gives rise the interventional distribution P(Xv do = and the graph of the intervened Bayesian network is the graph G' without any edges into any of the variables E T. In the main paper we slightly deviate from this notation by writing e.g. PY F)) to mean Par do(XF IF)), so only for F we make the exception to always indicate a value y instead the random variable Xp.
BEEv P(X,) X paa(v))
A.3 Structural Causal Models
Alternatively, the causal models can be interpreted as structural causal model (SCM), which formally consist tuple (V, W, f, P) where
V the index set of endogenous variables
. W is the index set of exogenous variables
= II Ne x IIwew Nw is a product of Polish spaces (c.g. {0,1),
Jv : Xv x Xw Xy set of structural equations
P(Xw)= = X NEW P(Xw) is a product of Borel probability measures.
We call j E EVUW W parent of i E V if the structural equation fi truly depends Ty, i.e. if there is measurable map Ja X x Xwyes -> Xr such that for P(Xw))almost all sw and all zy we have
x; (xv
= x. [looao)
The augmented graph of an SCM M is a directed graph Go with vertices V W and for all j E V U W,ie V an edge > i if is a parent of i. The graph G(M) of an SCM M is the latent projection of GM(M)) onto V. We call an SCM M acyclic if Go (M) (or equivalently, G(M)) is acyclic.
Given an SCM M (V, W. P) and subset of variables T S V and value 2r E Xr, the intervened SCM M = (V, W,K,J, has for every v E the structural equation
f. xr
e if T.
Given subset T C V and writing O = V T, solution function with respect O is measurable map go Xr x Xw + Xo such that )-XX)-almost all E Aww and for all øy E X7 we have (0 zw) fo (IT, go (xT, ). Given random variable Xw with distribution P(Xw). the random variable Xy == gv (XXw) defines the observational distribution PM (xy)), and the random variable Xo == go (æT, Xw) defines the interventional distribution PM (Xo do(Xy- = ar) )). If M is acyelic, go can always be obtained by recursive substitution of the structural equations of Mao(Xr=er):
A.4 Markov property and faithfulness
For any Bayesian network or acyclic SCM with DAG G and observational distribution the global Markov property holds:
A A B C = X Xc
for A,B A Bayesian network is called faithful if for all A, C V we have
d
AfBC = XA Xn I |c.
also Boeken et al. (2025).
We refer to distribution P(Xy) as compatible with ADMG with vertices if the Markoy property holds.
14

---

B Proofs
B.1 Section 2
Theorem 1. Let Ma S Mpe be the set of models compatible" with causal graph G. then the following are equivalent:
For every M MMa there exists an observationally correct forecast of [A:
ii) For every M E Maa there exists a correct forecast of Y IA;
ii)) |A is forecast-invarionn for all M MM'g
iv) Y 16 A
Proof. iv)
ii:: If F Y A then for every M MM have PY A, do())
PM A) by the third rule of the do-calculus (Pearl, 2009).
iii) ii:: If Par(Y A, do( (Y A) for every M M'g, then Par(Y [A) is correct
forecast.
ii)
i): This follows immediately from Definition 2.
iv): We prove the contrapositive. If F LG Y I then there must be a path F V1 +
vn + Y in G with V, & A for all i 1,..., Tb. Let Po, Pi E Py with Po o Pt. Set
60
if F(Y | - a) P1 for some E A otherwise
PM (V()
8,
80
if V; =0 otherwise
PM (V441 (do(:))
=
8,
for all i = 1,,.,
60 if V =0
PY V.))
66
otherwise,
and pick for every other variable Z (including the variables in A) some independent distribution (2(() Then M Markov with respect to so M Ma. and PM ( do())) P1 iff Pi & F(Y A), so there is no observationally correct forecast for M.
B.2
Section 3
Theorem 2. Let M' Mpp be the set of models such that YIA is forecast-invarianl, where Pas dF(F)) has full support for all F E Py and all M E M', or where PM (A] do(F)) is deterministic for all M E M'. If S is scoring rule with respect Py such that there Po, P1 E Py and forecast F EET Py with f + P1 and S(Po, Po) < S(F, P1) S(P1, Pt), then S is not observationally proper for predicting Y|A in M'.
Proof. Let Po, Py with Po # PI, and let M be such that PM do 1)) P1 and PM (Y do(A # 1)) = Po. For the case where (A do(F)) is deterministic, let
{1 if A=
(A (o(F))
(80 otherwise,
then Y|A forecast- forecast-invariant invariant in M with (Y A =) = Pi and F (Y #1) = Po the correct
forecast, and the incorrect forecast F(Y A = 1) = 5 and F(Y + 1) = Po has
S(F*, M) S(P,, Po) < S(F, Pı) S(F,
so S is not proper with respect to M E M..
For the case where Par(A do has full support, 1et ((A) be a probability distribution on A with full support and such that Q(A 1) < 3(P..P))-((P
-(F.P.)-S Po.) note that this right-hand side is strictly
for sets variables A, B, C. "We can have a d-connection Y & F A but forecast invariance in M due to faithfulness violations after intervention do(F)) (Boeken et al., 2025). However, forecast invariance for all M M's implies Y 1g A.
B3 MManing that the Markov property holds: A 1 BIC = AL
15

---

positive by assumption. Then the mechanism
2- F otherwise
PM((Ado(F))
QQ222
has full support for all forecasts F and A| is forecast-invarion in M, and we have correct forecast F (Y A = 1) P1 and (Y A + 1) Po and observationally incorrect forecast F(Y A 1) = and F(Y A + 1) Po. and we have expected scores
F MM = (Q(A + 1) + 1) + Q(A= 1)
< 1))((o, (Q(4 (00)) Pı) = 2),
so S is not observationally proper.
B.3
Section 4
Theorem The utility score is nucentive-compatibee with U, and proper if [A is forecast-invariant in M' S Mar- for a given causal graph G one considers the class of models M' Maa which are consistent with G, then forecast invariance of Y|A is necessary for the utility score to be proper.
Proof. Since ((F [ (U(aF.y)PPs(C (d(()) f [ PMr dy do(F)) we auto- matically have
arg max (F,, M) = arg max / U(a Prr dy do(()),
so S is incentive compatible with U. If Y forecast-invariant, for every M E M' the forecast F(Y AA) = Par (Y A) is correct forecast. By definition of the Bayes act ap we have (Y = arg max,, f [a, y) (dy A a) so for F E PA-YY we have
S(F(Y M) = / U U(r u) PMr (dy A = aF)
< / Ulpss v) Pardyl A S(PM)) M).
Since the expected score is maximised at correct forecast we have that S' is proper. Y|A is not forecast-invariant in M'.. then by Theorem there does not exist an observationally correct forecast, so the utility score is not proper. (Note that in the proof of Theorem 1, the counterexample for which no observationally correct forecast exists can easily be altered to let A = OF be the Bayes act, so that the theorem holds in the class M' S Mar-)
Theorem 4. Let YIA be forecust-invariant in M' S. Mas let U E [0, let S' [0, be a proper scoring rule in the classical sense and let there be A min (U Y) [a) - EM (U Y) v))aa)):a a d'} > A for all M e M', then S(F, a,y) == U(F A S (FY A oF). 3) is proper and incentive-compatiae proper in the classical sense, then S is observationally strictly proper.
>> 0 such that
with U. If S' is strictly
Proof. We have
S(F.M) - f U(a, y) dy do(F)) + AF
so if F* = arg mag Ula PM (da, dy do(F)), then for any other F we have
S(F*,M) S(F,M) / U(a, v) Par dy (do(F")) - U(a,v) dy dyloFF)) (d)(F)
+ A (F* M) 3(FF
A + A (F' (F*, M) - (F, 20
16

---

since S" E [0, 11, hence
arg max ((F = arg max / |U(a Par dy do(F)),
F
F
so S'is incentive compatible with U. If A forecast-invariant then for every M M' the forecast
F' (Y A) ( is correct forecast. Note that any optimiser F of S(F, M) must induce the action OF = a" == aF.. If F(Y A a²) + (Y A a²), then 3 (F, M) < 5 (F*) M) so
also (F,, M) < ( M). hence is observationally strictly proper.
B.4 Section 5
Theorem 5. Let M' C. Mae be such that for every M Mpe there exists an observationally correct forecast for Y|A. I is proper in the classical sense, then the divergence D is positive and proper. If S is strictly proper in the classical sense, then Dpc is observationally strictly proper.
Proof. If S is proper in the classical sense, we have for every F(Y E PA-tY and every a EA that
S(F(Y AA = A a,, S SY A = a, (F),, PY = d.d)()))
=: H( A a, do(())))
so the divergence is positive:
(po 1A= 5(F())((FF(())))) |A = a). Par dy do(F))
= / (M(PM(()1A= 2 O.
a), PY A a, do(())))) PM (da do(F))
(I)
If F" (Y [A) is correct for M, then F* FY AA a)
A a, for all a A such
that PM (A = a do(F")) > 0, so for every such a we have
S(F (Y A= a= Pas (Y] A do(F = (Y A= a, do(F))) PM PMMMYY A= a, do()))))
= HPMM (Y a, do()))
and by plugging this into (1) we get (pe M)) = 0, s0 -Dpe is indeed proper. If S is strictly
proper, then for any F(Y A a) + (Y A a,do(F)) for some such that Pas (A a do (F)) > 0 then we have
H(Pas (Y A a, (F)))) ((F(( A= a), PY A A= a, doF)(F))) 0 and hence Dpo (F(Y (PY
A), M) > so - -D is indeed strictly observationally proper.
Corollary 1. Let M' C Mpc be such that for every M E MPS there exists an observationally correct forecast for YIA, let S be proper in the classical sense, and let (A1, Yı)..., . (An, Yn) (A, Y do(F)). Any unbiased estimator Dpe(Ar,..., Y,) of the performative divergence (F,
(Fc M) is proper, for any sample size E N such that Dpc is well-defined. If S is strictly proper in the classical sense, then Dpe is observationally strictly proper.
Proof. By unbiasedness of the estimator we have I Dpc Dpo (F, M). the result immediately follows from Theorem 5.
(A,))
Lemma 1. Consider the setting of Corollary For binary Y and S the Brier score, the estimator
1 (Pas Saa)² Pa. (1 Pa.)
Brier
n
na. 1
17

---

is unbiased, where we write na := Ei= 1a: = a}, Pa ==
Ei a} and fa ==
no
F(Y = 1= a). For Y E R with integrable norm and S the energy score, the estimator
1 Σ n
Σ
iti =u:
yill
Benergy pe
N For IIY - will
By. (|),00111 || Y'
2
na:
2
1=1
is unbiased.
Proof. Note that we have Derier expectation over the data, we have
(e, (F,
EocA(Da (aa) a (o](o(F)). By taking the
pe
Po, (1 - Pa;) (Pa. -Fa))
D Brier pc
E
E
no.
aae
Writing Pa : P(Y = A, (ddo(()))))))) for given a E A in the summation and by conditioning
we have (f, Ja) na) = (Pa -Saaaaaa 1 Pn (1((1 Pa)) and EE - P.)) ma)] = _
Pa - -Pa)
and hence E1 (pa, )a Pa. (1-Pa)) [na] nu (Pa fa)². Integrating out the na have
no;
El na PM (A ()) and hence E Brierr] EPEd fa) P(A do(()) DBrier (F, M). which is the desired result.
For the Energy score, the divergence D DEnergy is equal to
De
Σ E EY-F, II|Y Y'll By.yr - - -Y'll - -By,y 11|l -XaI- PMr aldo)F)).
1
Y'mP.
a
Taking the expectation of the first term in the estimator we obtain E [ -ー= By Fo; || vill]
ExEA By P. [|Y Y'Il second term that
Par a do()). Denoting Ta = {i: A, a). for the
Σ 4, a
juti
11ys
yyll
1
Σ infy
||Y; - Yyll
na n
72
na:
nh aanna - 1)
aEAA
where the part between brackets a standard U-statistic for the norm (also known as Gini's mean difference) of independent random variables Y. ~ PM (Y) A = do()),, so the expectation of
this term equals EnEA 2 EE,,'' I||Y - P'I[] MM (A a (F)) (Székely and Rizzo, 2013). The
expected value of the last term clearly equals Σ a 2 By,r [IY Y'Il] PM (A = a (do(F))) so
upon combining these results that E
Energy
D Energy
pc
pc
B.5 Section 6
Theorem Suppose that for every model M the class M' C Mpe and every forecast P A-- the distribution (A do( F)) has full support and that Y|A is forecast-invariin the divergence induced by strictly proper scoring rule, For any parameter 0, for conditional forecast Fg, | A), we have that 0z+1 := arg ming Ret (0, 0c) yields correct forecast which is performatively stable and performatively optimal with respect to RD(
and let D be
Proof. Since (A do(Fe,)) has full support, have by Theorem 8 that the decoupled perfor- mative risk Rd. is strictly proper, hence we have that Fer., is correct. Similarly we have that 0.+2 == arg ming (0, 1++1) is correct as well, hence 0c+1 = Oc+2, so 0.+1 is performatively sta- ble. By Theorem we have that the performative risk Rp is strictly proper as well, so RD (6++1) = hence 9,-1 is performatively optimal.
C Proper scoring rules for non-performative conditional forecasts
As a special case of Definition 2, given a Markov kernel ((Y|A) PA-yy and distribution P(A) E PAs the conditional forecast F(Y [A] E PA-y is:
18

---

observationally correct if F(Y A a) = P(Y A= a) for all E A such that P(A
a) >
counterfactually correct if F(Y A a) P(Y a) for all A such that
aa =0;
correct if it observationally and counterfactually correct.
Given conditional scoring rule S PA-y - + R. conditional forecast F(Y (A) PA-Y and model (P(Y |A, P(A)) E PA-y x PA. the expected score is
( A), P(,,)))) =
where P(A,Y) == P(Y A) & P(A). Definition 9. We call a conditional scoring rule
observationally (strictly) proper relative to PA- x PA if S. is (only) maximised at observationally correct forecasts for all M E .M;; counterfactually (strictly) proper relative to PA-y PA 3g is (only) maximised counterfactually correct forecasts for all M E .M;; (strictly) proper relative to PA-Y PA if is observationally and counterfactually (strictly) proper, that is, if S. (only) maximised at correct forecasts for all P(A)) E PA- x P.A
Recall that any scoring rule Py x induces a conditional scoring rule through S'(F, v) == S(F(( A a),
Theorem I is proper scoring rule for marginal forecasts with respect to Py. then the induced conditional scoring rule is proper with respect to PA-Y PA. If S is strictly proper. then the induced conditional scoring rule observationally strictly proper.
Proof. Given (P(Y A), P(A)) E PA-y x PA we have for every a E A with P(A - a) > 0 that F(Y A a). P(Y A= E Py, and since S proper in the classical sense we have
3(F(Y A = a), P(Y | = a)) S 3(P(Y A a), P(Y A A=
(2)
Integrating with respect to P(A) gives
/ S(F(Y A- a), P(Y A = P(da)
Sa(P(Y A), (AP(YYA), P(A,Y)
$ 5(P(Y a), P(Y = P(da) (c((
and since P(Y | A) is correct, S is proper. If S is strictly proper in the classical sense, then for F(Y A) * P(Y [A) there is an a with P(A = a) > such that F(Y AA = a) / P(Y | A = a), in which case have strict inequality in (2), giving strict inequality in (3), S is observationally
strictly proper. Note that any other Q(Y A) E PA- such that Q(Y A = a) = P(Y = a) for
all A with P(A a) obtains the same expected score but need not be counterfactually correct, so S is not counterfactually strictly proper.
C.1 Divergence for non-performative conditional forecasts
For conditional forecast F(Y A) E PA--Y and model (P(Y A), P(A)) E PA- X PA. we define the generalised conditional entropy and generalised conditional divergence as
H)( = H(P A P(da)
D.(P(I1A), = H.(P) (F)
= S(P(Y = a). S(F(F) a)) P(da, dy).
For the log-score, the entropy H. is also known the conditional entropy (Cover and Thomas, 2006). For the divergence, we obtain theorem similar Theorem 7, that the generalised conditional divergence is observationally (strictly) proper if it is induced by a (strictly) proper scoring rule.
19

---

Theorem 8. If S is proper scoring rule, then De is proper for conditional forecatss If S is strictly proper, then D. observationally strictly proper.
Proof. Given (P(Y|A) P(A)) E PA-Y x PA and proper scoring rule S we have
De(F(Y A), P(A,Y) D(F(Y A P
2
(4)
and (2(F(Y A), P(),Y)) 0, D. is proper. If strictly proper then for every A with
P(A = a) > O we have ( a), P(Y A = a)) = 0 if and only if F(Y A = a)
P(Y A a), hence (2 P) = 0 only if F observationally correct.
D Proper scoring with inverse probability weights
By scoring with divergence, the utilised scoring rule made performatively proper by subtracting the (estimated) entropy. As alternative approach, one multiply the score by (estimates of) the inverse probability weights (IPWs) (A do(F)), and consider the expected IPW score:
/ S(F(Y A = a).9) P(da, dy do(F))
SIPW ())
PM (A a do(F))
Σ S(F(Y A |AFY|A=10 a))) P(dyla)
aEA
Chen et al. (2011) introduced the IPW score in settings where the principal knows Pas (A) (which does not depend on F), who can then after every observation (a, y) properly score with SIPw(F, a, := S(F(Y) A = a), PM = a). the distribution Pas (A do(F)) has full support, then this procedure is strictly proper:
Theorem 9. Suppose that for every model M in the class M' C Mpe and every forecast E E PA-Y the distribution Par (A do(F)) has full support, and that Y|A is forecast-invarian in M E M'. S is (strictly) proper in the classical sense, then Siew is (strictly) proper in the performative sense.
Proof. By forecast-invariance of YIA we have
3IPw (W, = S(F(Y A PM. A a)da
= S(F(Y IA PM, PM (Y = a))
so for every E the optimum of the integrand lies at the correct forecast F A a)
(V = a), and hence PM (Y A) maximises Sirw, so Sirw proper. If S is strictly proper, then for every F with F(Y A = a) * (Y A = a) for some E A we have S(F(Y a). (Y A a)) < S(F* (Y A = a), Pa (Y A a)), so (F, < SIPW(F) (F", M).
hence SIPw is strictly proper.
In Section F.3 we provide an example where SiPw not counterfactually proper when PMr do())) does not have full support.
similar principle as for the divergence applies: the principal does not have access to (M do(F)). unbiased estimate of the mean IPW score suffices for the method to be (ob- servationally strictly) proper. It is unclear whether unbiased estimators for the mean IPW score exist. In Section G provide brief empirical investigation into the consistency of the plugin estimator for the mean IPW score.
E Forecasts depending on covariates
E.1 Causal model including covariates
We extend the results from the main paper to the setting where we let the forecast depend explicitly covariates X. In the main paper, we use the notation Par(Y do(F)) such that for
20

---

every forecast F Py we can meaningfully talk about the distribution of Y, given the forecast F. If we let F depend on covariates X. then intervening on would make the forecast invariant of X, which is not what we intend to do. Therefore, we consider a parameter 0 E θ C TR for some d which specifies a forecasting model Fo X) E Px-y. y. In this setting, we will instead of PM (do(F)), for each observed E X' consider the causal effect of specific parameter 0 and its induced forecast FY |X = x) on Y. which we express via PM (Y do(())) which is equal to PM do(0, Fx PM (dz) by consistency.
Formally, for marginal forecasts we a parameter induce the forecasting model (Y |X) Px-y', where for every observed value and parameter O the reported marginal forecast is Fo X 2) E Py. For this setting let Mp+ be the set of causal models whose projection onto X,8,F,Y is subgraph of Figure 5a.
For conditional forecasts, let induce the forecasting model Fo A, X) E PAxx-yy. where for every observed value x and parameter the reported conditional forecast is Fo X x) E PA-y:: let Mpe+ be the set of models whose projection onto X, 0, F, A, is subgraph of Figure 5b.
X
F
(a) Ma+: Marginal forecasts
(b) Mpc+: Conditional forecasts.
Figure 5: Causal graph of performative forecast depending on covariates X.
E.2 Section 2: Correctness of forecasts
In the following sections, in each definition or theorem, the corresponding definition/theoremm number from the main paper is added in parentheses.
Definition 10 (Definitions 1, 2). For marginal forecast FY X x), the parameter is correct for M E Mpe+ if FY X = z) = (Y | = x, do(0)) holds for (M )-)-mooosttt all e X. For a conditional forecast FY A, X z), the parameter 0 is
observationally correct if for PM(X ) >-almost all E X, for all a E A such that P(A a X x, do(()) > 0 have FY |A = a, x) P(Y A a, X x, do(O)): counterfactually correct if for PAr -almost all X, for all A such that P(A a X = x, do ()) = O we have Fo A = a, = x) P(Y A = a, = x. do(O)); correct if is observationally and counterfactually correct.
Definition 11 (Definition 3). We call the target Y A, X forecast-invariant in M' C M if the target distribution of Y Y|A, X does depend on the forecast, if PM A, do())))))))))))) PM (Y A, X) for all M E M'.
Theorem 10 (Theorem 1). Let M' S Mpc+ be the set of models compatible with causal graph G, then the following are equivalent:
For every M E EMa there exists an observationally correct parameter for Y|A,X;
it) For every M MMg there exists correct parameter for Y Y|A,X;
ii)) Y Y|A, X is fisorrcaattnnaarrann for all M E MMa:
iv) 0 011 1 A,X.
Proof. The implications iv) = iii) = ii) = i) apply without any change the proof; for iv) one has to also specify an independent distribution PMr (X) to finish the proof.
21

---

E.3 Section 3:
Scoring rules
Consider a scoring rule S Py x x x + R, where given a marginal forecast FY x 2) and an observed outcome the forecaster receives the score S(Fo(Y X = 2), v)). Considering M E Mpe+. the expected score is
(Fo M) / S(Fo(Y|X = 2)) dy do())).
For conditional forecast FY A, X = x), conditional scoring rule S PA-Y AX x + R. observed action a and outcome y, the forecaster receives the conditional score S(F(Y A, x)) a, x. y), and given a model M EMpo the expected score is
3 pec (Fo, M) MF) = / S(Fo X x)), a, 0, PM (da, dr, dy do(O)).
Definition 12 (Definition 5). For marginal forecasts, a scoring rule S is proper relative to M' C Mp+ if Sp+ is maximised at correct forecast, and strictly proper if all maximisers are correct. We call a conditional scoring rule
observationally (strictly) proper relative to MM S -Mpp+ if Spc+ is (only) maximised at observationally correct forecasts for all M E MM;;
counterfactually (strictly) proper relative to MM' C MMo+ if Spo+ is (only) maximised at counterfactually correct forecasts for all M MM;
(strictly) proper relative to M' S Mpp++ is observationally and counterfactually (strictly) proper, that is, if 5 pc+ is (only) maximised correct forecasts for all M .MM.
Theorem 11 (Theorem 2). Let M' C. Mpc+ be the set of models such that Y|A, X is forecast- 2, do (f)) has full support for all 0 E θ,
invariant, where (A X and all M E M', or where PM (A] X = x, r,,o(()) is deterministic for all 0 & 9, Pas Px)(X)-almost all E X and all M E M'. S is scoring rule with respect Py such that there are Po; Pi E Py and forecast F E Py with F + P1 and S(Pu, Po) < ((F, S(P1, Pt), then is not observationally proper for predicting Y|A, X M'.
all X
Proof. The proof proceeds in the original proof of Theorem with adjustment: the constructed counterexample specifies a mechanism for F' + A + Y; specifying any independent distribution Pas (X) completes the proof.
E.4 Section 4: Decision theory
Given conditional forecast Fe, let the agent take an action which maximizes the expectation of a utility U(a, x. y). so we have A = fA(Fa x) = arg max Ú(, x,, Fo x, a,,). Let Mat+ S Mpet be the class of models with PMr(A X = x, do(O)) Sas_.
Definition 13 (Definition 6). Given utility U a set of causal models M S M Mpe+. we call conditional scoring rule incentive compatible with U if for all M E M' we have arg maxg Spc+ (Fo; M) arg maxg J. U(a, x. Pns (da, da, dy do)))).
The utility score is given by S(Fo. a, a. = (Ff x), x. v)
Theorem 12 (Theorem 3). The utility score is incentive cooniive-commatbblee with U, and proper Y X is forecast-invarion in M' C Maa++ If for a given causal graph G one considers the class of models M'a S Ma++ which are consistent with G, then forecast invariance of Y|A, utility score to be proper.
A, X is necessary for the
Proof. The proof proceeds analogous to the proof of Theorem
Theorem 13 (Theorem 4). If YIA is Dorecat-iivariian in M' S Matt
[0,1] is a proper scoring rule in the classical sense and min ()) - x,,d,,,Y))))) xY [ a (a'a) a'} A > 0 for (x) all x E X and all M E M', then S(F, a, = UU(f((Fe
U. E 10, 1]. S'
- -A (FY
fA 2)), is proper and incentive-come compatible with U. If S' is strictly proper in the classical sense, then observationally strictly proper.
Proof. The proof proceeds analogous to the proof of Theorem 4.
22

---

E,5 Section 5: Divergence
Definition 14 (Definition 7). Given a scoring rule S Py x y R. conditional forecast FY AA = x) E PA+y and causal model M E Mpe+. the performative divergence is de- fined as
Ppc () = /
(S (S(PM(Y a, a, x, S(Fo( a))) Par (da, dz, dy |do()))
With performative entropy Hpa (M do(o))
[PM a. 2, do()))) dr do(0)). the performative divergence can be written as Dpc (P,M)
- -pp (M do(0))
Spc+ (F&(Y) M))
Theorem 14 (Theorem 5). If every M E M' S Mpe- there exists an observationally correct
forecast for YIA, X and if S is proper the classical sense, then the divergence Dpc+ is positive and proper. If S is strictly proper in the classical sense, then Dpct is observationally strictly proper.
Proof. The proof proceeds analogous to the proof of Theorem 5.
Corollary Let for every M E M' C. MMo+ there be an observationally correct forecust for YIA, X, let S be proper in the classical sense, and let (A1, X,,Y)), Yi), (An, Xns Yn) of the performative divergence Doc+ (Fo, M) is proper, for any sample size n E N such that Dpct well-defined. If S is strictly proper in the classical sense, then Dpct is observationally strictly proper.
Pas (A, Y do((F))) Any unbiased estimator
E.6 Section 6: Parameter estimation
Definition 15 (Definition 8). Given a loss function E(O, a, x. y); the corresponding performative risk is defined as R(0)= == f {(O, a, x. y) PMr dr, dy do(O)), and a minimiser 0ro of the per- formative risk is referred to as performatively optimal. The decoupled performative risk is defined as Rd (f 02) := J [( a, x. y) PM (da, dr, dy do(),)), and parameter 0ps is performatively stable if 0 == arg ming (0, 0ps).
We define the performative divergence and decoupled performative divergence as:
RD+ (0)= D Ppc+ (Fo, M)
RD RD (0++1, = /
D D(Fo (Y I,,x), Pas (Y a, a, do(O:))) PM (da, dx do((.)).
Theorem 15 (Theorem 6). Suppose that for every model M in the class M' C. MMot, every parameter O E ④ and PMr(X) -Xalmmot all x E X the distribution Pas[A] X I. do(O)) has full support, and that Y|A, X is forecast-invarian in M E M', and let D be the divergence induced by a strictly proper scoring rule. For any parameter for conditional forecast Fs, (Y A, = x), we have that 0.41 = arg ming Rd) (0,0,) is performatively stable and performatively optimal with respect to RD+ (f).
Proof. The proof proceeds analogous to the proof of Theorem
F
Examples
Python code to generate the plots of the examples in this section is provided in the supplements. The code generates interactive 3D plots, which can be helpful for visual aid.
For interpreting these plots (see e.g. Figure 6). the reading guide of these plots is as follows. Global optima are indicated with blue line or dot, the correct forecast is indicated with green dot. (Strict) Properness means that (only) the correct forecast has the same expected score as the global optima (z-axis). The grey, projections of the true and optimal forecasts can be used to read the score of these forecasts (on the z-axis). Observational/counerffiii optima are (only) at the correct values of F(Y = 0) and F(Y A = 1). The bottom projections can be used to read off whether the optimal forecasts coincide with the true forecast. This can
(strict) properness means that the global
23

---

separately be checked for A = 0 and A = of A (that is, the values A such that PM a do(F)) 0) can be read off by checking the direction in which the expected score at F is not piecewise constant.
1. In these examples, at any F, the 'counterfactual value
F.1 Classical scoring rules which not performatively proper
The following example is provided in the main paper (Example 4):
with PM = 1 1|A=00 0.5, Pas = 1 1) 0.25 and the 'decision rule' A arg max, F(Y =1 a). the forecaster reports the correct forecast the expected Brier score is - 0.. (..5 0..5 -02 and if the forecaster reports F(Y
Example 6. Let A y {0. } and M be such that Y [A is forecast-invario
1] = 0.25 then the expected score is -0.25 (0.75)² 0.75 (0.25)² = 0.19. See also Figure 6: the scoring rule is observationally strictly proper since for the observed value A = 1 the score is maximised a correct forecast, but it is not counterfactually proper since it pays off to misreport for the value A = 0 which not observed. This can also observed in the figure using the bottom projection the optimal forecasts and the unique, correct forecast.
= 0) 0.2, F(Y =
The scoring rule in this example is not (counterfactually) proper. but it is observationally strictly proper. As an extension of the self-defeating prophecy, if we pick the harder predict value of A when the forecast for the easier to predict value of A gets close to being correct, this provides an example where the score is not observationally proper.
Example Consider Example 6, with the decision rule
A = [1 if F(Y 1 A 0) - 0.4 and F(Y - A= 1) 2 0.4
otherwise.
Since PY = 1 = 1) = 0.25 is much easier to predict than Pas = 1 = 0.5. even an incorrect prediction F(Y = A 0) F(Y
= 1) 0.4 made such that we observe A = the expected score -0.25 (0.6) - 0.75 (0.4) -0.21 is higher than for correctly predicting and observing Pv 1
0) 0.5: if the forecaster reports her true belief the expected score is -05 (0.5)² - 0.5 (0.5)² = -0.25. Hence, the scoring rule is neither observationally nor counterfactually proper. See the bottom projection in Figure 7a: the optimal forecasts have neither F(Y A nor F(Y A = 1) correct.
One could expect this problem to be resolved if one considers a stochastic policy for deciding A given F. However, the following example shows that full support of PMr (A) do(F)) does resolve this issue:
Example 8. Consider the decision rule which is a mixture of the mechanism of Example 7 and uniform distribution over A 0, 1: with probability 1/3 there uniform 'exploration', and with
0.2
-.3
0.4
0.5
0.6
007
0
1.0
1.0
0.8
0.6
0.4
0.6
0.4
0.2
F(Y=1|A=1)
F(Y=1/A=0)
0.0
Figure 6: Expected Brier scores Spc for the mechanisms PM as defined in Example 6. is observationally strictly proper, counterfactually improper.
24

---

0.2
0.3
0.3
0.4
0.6
0.7
0.8
0.8
0.60
0.8
1.0
1[Y=1)A=0)
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.000
F(Y=1[A=1)
0.6
1.0
F[Y=1|A=0)
0.8
0.6
0.2
0.0
0.0
0.2
0.4
F[Y=1]A=1]
0.6
(a) Example observationally improper, counterfactually improper.
(b) Example 8:
observationally improper,
counterfactually improper.
Figure 7: Expected Brier score for the self-defeating prophecy, without positivity and with positivity. In both mechanisms expected score is neither observationally proper nor counterfactually proper
probability 2/3 the decision rule
A = {
if F(Y P
0.4 and F(Y 1 = 1) 2 0.4
otherwise
used. For this mechanism the correct forecast obtains the expected score of ( 1 + (-0.25)
(-0.25)))
2
1 (-0.19) x -0. -0.24, and if the forecaster reports F(Y = 1 A = 0) = F(Y = = = = 0.4
then the expected score is () - (-0.25) + ( + ()) (-0.21) = -0.22, so the scoring rule is not observationally proper. See Figure 7b.
F22 Incentive-compatible and proper utility scores
Example 9. Consider Example 6, interpreted as forecast being given to an agent with utility U(a, y) y, then indeed A = op arg mmx, F(Y = 1 A a). The optimal action for the agent a* = but to maximise her score (with respect to the Brier score, see Figure 6) the forecaster reports F(Y A 0) 0.2 and F(Y A 1) 0.25 inducing the agent to take the suboptimal action af 1. Despite the forecast being observationally correct, it is not incentive- compatible with U.
However, if we use the utility score then every forecast with F(Y = induces the optimal action A a so it obtains the maximum expected score: Figure 8a. (To improve visibility we have not depicted the global optima.) Since all forecasts that induce the action A = obtain the same score, it is clear that the utility score is proper but not strietly proper. If we add any < 0.25 times the Brier score to the utility score then the resulting score becomes observationally strictly proper: the score is uniquely maximised for the correct value of F(Y 0); see Figure 8b.
1 11 > F(Y = A=
F33 Scoring with divergence and IPW score
Example 10. Consider the setting of Example 7 where there is no positivity of PM (A do do(F)). Recall that the Brier score is neither observationally proper nor counterfactually proper. However, we consider the performative divergence related to the Brier score, as depicted in Figure 9a, we see that the correct forecast an optimum, optimum we see that the action A = 0 is chosen (since the score does not depend on forecasts for A = and every optimal forecast correct for A In Figure 9b we see that the IPW score is neither observationally nor counterfactually proper, similar as in Figure 7a.
is proper. Moreover, for any forecast that attains the
When adding uniform exploration with probability 1/3 we have positivity as in Example 8, see that the the divergence and IPW score as forecast is the unique optimiser.
depicted in Figure and 9d are strictly proper: the correct
25

---

0.4
0.50
0.45
0.2
0.40
0.1
0.35
0.0
0.30
0.25
0
0.0
0.0
1.0
0.8
0.6
0.
0.6
1.0
0.8
0.4
F[Y=1
0.4
0.0
0.0
0
F(f=1|A=1)
F[Y=1/A=0)
0.4
0.2
0.0
0.0
0.2
F(Y=1/A=1)
(a) Utiliyy score:
(b) (Utility score) + (Brier score): observationally strictly proper.
observationally proper.
counterfactually proper.
counterfactually proper.
Figure 8: Expected scores of Example 9. Both decision scoring rules are incentive compatible with
utility U, since every forecast which attains maximum score induces the optimal action A a 0.
0.0
0.2
0.1
0.3
-02
-0.4
-0.4
0.5
0.7
0.8
0.
1.0
F[Y=1(A=0)
0.8
0.6
0.4
0.2
0.0
0.0
02
0.4
F(Y=1|A=1)
0,
1.0
R(==1(A=0)
0.8
0.6
0.2
0.0
0.0
0.2
0.4
F[Y=1AA=1)
0.6
(a) Divergence: observationally strictly proper,
(b) IPW ohservationally improper,
counterfactually proper
counterfactually improper
00.4
-0.
-0.6
0.2
-0.8
0.3
-1.0
0.4
-0.5
-1.2
1.0
0.8
0 0.
0.
0.000
1.0
0.6
04 0.
0
0.4
0.2
0.0
F(*=1AA=1)
0.4
0.2
0.0
0.2
F(Y=1/A=1)
F(Y=1(Ac0)
F(f-1AA00)
(c) Divergence: strictly proper
(d) IPW score: strictly proper.
Figure 9: Plots related to Example 10. If there is positivity (9a and 9b) then the divergence is observationally strictly proper and the IPW score is not observationally proper. there positivity (9c and 9d) then the divergence and IPW score are strictly proper.
26

---

G Bias and consistency of estimates of divergence and IPW
For general scoring rules S. plugin estimators for the divergence and IPW score are given by
Dpc(F.M) M) Σ
s (P(Y A = az, do(F)) ws) S (F(Y A= A=a;), v:)
2=1
S(F(Y A = a.;;)
(F) =
P(A 0; do ())
using estimates of P(Y | A= a, and P(A = aj do())). To investigate the variance
and the reliability of these scoring methods at various sample sizes, we consider the mechanism of Example so where Pw (A P)) has full support for every E Py. In this setting, the unbiased estimator for the divergence is given by
1 n
Pa. (1 Pa.)
D Brier pc
- (pa, - s2.
(5)
T2 1=1
ne. - -1
where we write nhe a).
Ei= 1(u
1 a}, po na Ei yil a) and Sa = F(Y = A
We consider two forecasts: the correct forecast F*(Y | A 0) 0.5, FY
0.25.
and an incorrect forecast FY A = 0) = 0.7, F(Y A = 1) = 0.45. For both methods, the correct forecast should obtain lower score than the incorrect forecast. For the sample sizes n E {2, 3, 4, 5, 6, 7, 8, 10, 21, 46} we simulate for both the correct and the incorrect forecast, 10.000 datasets of (A1, Y))..., (An, Y..) ~ (M Y do(F)). for F = F and F = F. We every dataset the divergence and IPW score, and from those 400 repetitions we take the median and 5 and 95-percentiles. The results are depicted in Figure 10. This verifies that the plugin estimators for thr Brier divergence and IPW Brier score are biased but asymptotically unbiased, and that the estimator (5) for the Brier score is indeed unbiased.
subsequently estimate
These experiments can be run within 15 seconds Python code to run these experiments is provided in supplementary files.
Apple M2 Pro processor with 16GB RAM.
Eximmted
Extimated
0.6
Thue score
0.6-
Thue scone
0.6
Diveigence
Pavergence
0.4
0.4
0.4
IPW
Estimated
6.0
0.0
0.0
Extimated score
The scone
-0.2
-02
-02
10a Sample size In)
10a Sample size (n)
10a
Sample size
(a) Divergence (plugin)
(b) Divergence (unbiased)
(c) Mean IPW score (plugin)
Figure 10: Estimated divergence/IPW score (solid line) and true divergence/IPW score (dashed line), for a correct forecast (green) and an incorrect forecast (red), with 90% confidence intervals.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the abstract are further specified in Section 1.1, where each claim refers to specific theorem. Each theorem is subsequently proven in the appendix, Section B.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
28

---

Answer: [Yes]
Justification: All formal proofs are deferred to the appendix (Section B), but the intuition behind the proofs is provided in the main paper, either as example, or in the text surrounding the theorem.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The empirical aspects consist of quantitative examples, where analytically calculated scores are plotted, and an experiment where samples are generated and scores are estimated. These examples and experiment are accurately described in the paper and appendix. Additionally, Python code is provided the supplements re-create the plots.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
29

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Two Python scripts are included as supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Suuidee/CodeSumiise
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiments show estimates of empirical divergence and IPW score in a setting with binary variables. This setting involves no model training, data splitting or (hyper)parameter optimisation.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The experiments are repeated 400 times, and the median is plotted with 5-95 percentile confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: No heavy computing resources are necessary: it is mentioned in the supple- ments that the experiment can be run in minutes on an MacBook M2 Pro processor with 16GB RAM.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'ttnt make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https http://wwwp....popoa//cccccccaaa
Answer: [Yes]
Justification: Our replies to questions 1-8 and 10-15 of this checklist reflect that our paper complies with all topies mentioned in the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
31

---

Justification: In Section 7 we discuss the potential impact of the provided theoretical results on practical applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not introduce data or models that have high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
32

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
33

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: /tt////eurippppepe http://rppp..ccomeeeeeree/////// wctcConfeeeencc///25/////////// for what should or should not be described.
34


Reviews:
[{"review_id": "QsNIL7NVJB", "reviewer": "Reviewer_z9DJ", "summary": "this paper studies how to design scoring rules to elicit correct forecasts in the performative prediction setting. To overcome the impossibility described in self-defeating prophecies, the authors considers a causal framework where a conditional forecast F(Y|A) affects the conditioning variable A and the target variable Y, with a focus on the case where the conditional distribution Y|A is forecast-invariant --- which they show in Theorem 1 to be an equivalent condition for causal models to possess correct forecasts. The paper shows that even when Y|A is forecast invariant, classical proper scoring rules might no longer be proper in the performative setting. The authors then consider two approaches to construct proper/\"incentive compatible\" scoring rules, one is for decision-theoretic class of causal models, and is based on the combination of utility score and strictly proper scoring rule, under the assumption of a nonzero gap between optimal and suboptimal actions. Another is based on divergence (difference between entropy S(P,P) and score S(P,Y)), which avoids penalizing predictors that correctly predicts a high-entropy target, but only works in the batch setting where the evaluator has access to a large sample of events (A,Y) to estimate the entropy. Finally, the authors study parameter estimation, and showed that using divergence as the loss function results in performatively stable and optimal solutions when A|F has full support.", "questions": "1. Could you comment on the finite-sample guarantees of the divergence-based approach (Algorithm 1)? Please see weakness 3 -- did I miss anything?\n2. Could you motivate the utility gap assumption in theorem 4 and the full support assumption in Theorem 6?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper studies the important question of how to evaluate and incentivize forecasts in a causal system where the target distribution is not fixed but is a response to the forecast itself. The authors identify conditional forecast as the correct notion that bypasses impossibility results, and developed some theory towards restoring proper scoring rules in this setting. The problem is theoretically interesting and practically relevant.\n\nWeaknesses:\n1. At a high level, while the paper covers multiple aspects of performative prediction (decision-theoretic, batch setting, parameter estimation), the theoretical depth for each aspect is limited. Most results are relatively straightforward extensions of classical scoring rule theory. The contributions feel more like a collection of observations rather than deep theoretical insights.\n2. regarding the decision-theoretic approach: The existence of a utility gap might require some motivation.\n3. regarding the divergence-based approach: The divergence used in Def 7 is guaranteed to be proper, but requires knowledge of the conditional distribution to compute the divergence, which might be impractical (especially when P(A|F) has full support, which implies that the principal already knows all the conditional distributions and would not need an external forecaster). While the finite-sample version in Algorithm 1 attempts to address this, the estimation of entropy introduces bias in the finite-sample regime, which might change the forecaster's incentives. For example, consider the Brier score and Y|A,do(F) ~ Bernoulli(p), the expected estimated entropy is (n-1)/n times the true entropy. This means that even though the performative divergence is proper in theory (Definition 7), the finite-sample approximation used in practice may not preserve properness.\n4. regarding the parameter estimation results: Thm 6 requires A|Y to have full support, which may be unrealistic. This assumption (together with the fact that using divergence requires access to true distribution) also mildly trivialize the problem and makes it such that a single step of updating $\\theta_t$ already yields the optimal solution, which seems far from practice.", "was_revised": true, "final_justification": "The authors' rebuttal partially addressed my concerns regarding the finite-sample properness of Thm 5 by proposing a sufficient condition --- the existence of an unbiased estimator for the entropy term. However, such unbiased estimators are not guaranteed to exist in general, and the authors only provided a few examples without a more general characterization. My concerns regarding the full support assumption also remains --- although being sharp, it is a quite strong assumption that leads to unnatural learning results. Although I appreciate the contributions of this paper, I think these two issues limit the applicability of the results in the batch setting and the parameter estimation setting.", "rebuttal": "Dear reviewer, thank you for your helpful remarks and questions.\n\n**Q1:**\n\nThank you for this interesting question. Indeed, if one utilises a biased estimator for the divergence, then the method need not be proper anymore. We will make a note of this in the main paper and provide examples in the supplements, similar to Figure 6 but then with biased estimates of the divergence and IPW score.\nHowever, due to your question we got the important insight that in some cases **this problem can be fully resolved: the estimated divergence is proper if one uses an unbiased estimator for it.** The forecaster optimizes the expected estimated divergence, which is equal to the true divergence in case the estimator is unbiased. This does not even require a large sample: any sample size for which the unbiased estimator is well-defined ($n>=2$ in your Bernoulli-Brier example) will do. In particular, the principal does not have to *know* the distribution $P(Y|A, do(F))$ – an unbiased estimate of the divergence is sufficient. We will:\n- remark that the method is not necessarily proper if one uses a biased estimator for the divergence;\n- add a corollary of Theorem 5 that the (unbiased) estimated divergence is proper for any sample size for which the estimator is well-defined;\n- make appropriate changes to Algorithm 1 to use an estimator of the divergence (and not the plugin estimator);\n- give an example of the unbiased estimator for the divergence based on the brier score;\n- note that the existence of an unbiased estimator is not guaranteed for certain choices of scoring rules (e.g. KL divergence)\n- add a corollary to Theorem 9 in Appendix B that the (unbiased) estimated IPW score is proper for any sample size (improving the applicability of the results of [1]);\n- change Figure 7 in the supplements to use an unbiased estimator for the divergence.\n\n**Q2:** \n- In Theorem 4, the principal may for example assume a utility gap if $A$ is binary and denotes taking an action which definitely has some effect on the utility, but it’s unclear whether it will be good or bad (like rolling out a bold marketing campaign, performing a risky operation, etc). The principal might assume that the expected utility under that action will surely differ from expected utility when not taking that action, it’s just unclear whether the utility would improve or not. In such a setting, the principal might want to elicit a forecast of what the effect of the action will be, in which case Theorem 4 gives a scoring rule to make that forecast truthful. We will add this motivation of the utility gap to the paper.\n- In Theorem 6, the assumption of having full support can be for example satisfied if actions $A$ are taken based on more information than the forecast only (or the features that the forecast take into account, as considered in the Supplements, section 5). For example, a doctor can decide to operate based on conversations with the patient -- information which is not available to the forecaster.\\\nWe note that the assumption of having full support is even *necessary* for having performative stability: if $\\mathcal{A} = \\mathbb{N}$, one could otherwise construct a model $M$ such that for every $\\theta_t$ obtained trough retraining we have $\\mathrm{supp}[P_M(A | do(F_{\\theta_t}))] = ${$1, …, t+1$}  (i.e. the support of A increases for every newly obtained parameter) and $\\theta_{t+1} \\neq \\theta_t$, so we don’t have performative stability. We will prove the necessity of the assumption of full support in the appendix.\\\nAlthough unapplicable in some scenario’s, we regard the setting of this theorem as a nontrivial but fairly general model class in which performativity can be completely mitigated. There might be other regularity assumptions on the model class (e.g. of the map $F\\mapsto P(A, Y | do(F))$, as considered by [2]) which do not require the assumptions of full support or forecast-invariance to obtain (approximate) performative optimality or stability. We leave this to future work, and will make a note of this in the discussion section.\n\n\nWe hope that these further explanations and insights resolve some of your issues with the paper. If any aspects remain unclear, we’d be happy to further elaborate during the discussion phase.\n\n[1] Chen, Kash, Ruberry, and Shnayder. (2011). Decision Markets with Good Incentives.\n\n[2] Perdomo, Zrnic, Mendler-Dünner, and Hardt. (2020). Performative Prediction."}, {"review_id": "fKsyItlqdc", "reviewer": "Reviewer_DFQj", "summary": "This paper addresses the problem of performative prediction, where forecasts influence the outcomes they aim to predict, undermining classical notions of correctness and proper scoring. The authors show that conditioning forecasts on suitable covariates restores well-posedness but still renders classical scoring rules inadequate. They prove an impossibility result for standard scoring and propose alternative approaches—decision-theoretic and batch-scoring methods—that enable truthful and stable forecasting in these settings. The framework also resolves instability in parameter estimation identified in prior work.", "questions": "1. On page 8, paragraph below Algorithm 1. The authors mention that Chen and Kash assume that P_M(A) does not depend on the forecast. The distribution on the actions in Chen and Kash indeed depends on the forecast, according to my understanding. Their notation uses decision rules that take a forecast and output a distribution of actions. Hence, it is unclear how Theorem 5 generalises results in Chen and Kash. Perhaps the contribution of Theorem 5 needs to be clarified in the context of prior work.\n\n2. Authors mention that to elicit correct forecasts, P(A|do(F)) needs to have full support, similar to previous results in Chen and Kash et al, which assumes that the random decision rule also has full support. How related is this assumption to randomised controlled trials?", "limitations": "As mentioned above, limited empirical evaluation of the theoretical results proposed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths And Weaknesses:\n- The authors propose an elegant formalisation of a hard problem, combining ideas from several fields of causality, scoring rules, and performitivity in a novel way.\n- The main strength of the paper is its theoretical contributions on the characterisation of structure in which perfomitivity influences the real outcome and hence truthful scoring of the forecast.\n- What is the connection between the formalisation proposed by the authors and the one proposed by Chen and Kash et al [1]? The way I see it, their notions of observationally strictly proper and counterfactually strictly proper and correct are related to the notion of quasi-strict properness and strict properness of forecasts in Chen and Kash et al. Placing the current formalisation in the context of prior work is important.\n- Minimal evaluation of the theoretical results proposed by the authors. While the theoretical results are significant, it is still unclear how the proposed characterization of proper scoring rules is implemented in practice with respect to repeated risk minimization. Evaluation on the credit scoring dataset [3] similar to Perdomo et al [3] would strengthen the contributions of the paper even further.\n\n1. Chen and Kash et al. Information Elicitation for Decision Making\n2. Give me Some Credit. https://www.kaggle.com/c/GiveMeSomeCredit/data, 2012.\n3. Perdomo et al Performative Prediction, ICML 2020", "was_revised": true, "final_justification": "I remain very positive about this paper. The rebuttal addressed my concerns. Thank you.", "rebuttal": "Dear reviewer, thank you for your helpful remarks and questions.\n\n**S&W, point 3:**\n\nWe agree that placing our formalisation in the context of prior work is important. To this end, we already make this comparison in footnote 9: “In a decision-theoretic context, [3] refer to observationally strictly proper and counterfactually proper scoring rules as `quasi-strictly proper' scoring rules.” (We think that separately considering observational and counterfactual properness provides a clearer language.) We will add a reference to [1] in the footnote. \n\n**Q1:**\n\nWe don’t claim that Theorem 5 generalises [2]: it proves strict properness of the performative divergence, which is a different notion than the IPW score as considered by [2]. As far as we know, we are the first to consider the performative divergence, so Theorem 5 is entirely novel.\\\nFor completeness, we have proven in Theorem 9 in Appendix B that the IPW score (as considered by [2]) is strictly proper, and in the paper we claim that Theorem 9 generalises [2]. We realise that we have initially proven the same result as [2], albeit in a more explicit formalism.\nHowever, inspired by Q1 of Reviewer z9DJ we will add a corollary to Theorem 9 that unbiased estimates of the IPW score render it proper (see our rebuttal to Reviewer z9DJ), so this shows wider applicability of [2] in settings where $P(A|do(F))$ is not *known* to the principal but instead where the IPW score can be unbiasedly *estimated*, ultimately justifying the claim that we generalise [2].\\\nWe will better emphasize which results are novel and which results already exist, and we will adjust our claim of generalising [2] appropriately.\n\n\n**Q2:**\n\nIf A is determined in an RCT then $F$ typically has no effect on $A$, so $P_M(A | do(F)) = P_M(A)$. In this case the forecast is not performative, and a classical strictly proper (marginal) scoring rule is observationally strictly proper for conditional forecasts (Appendix A, Theorem 7).\\\n(If the reviewer is interested in a motivation of the assumption that $P_M(A | do(F))$ has full support, see also our rebuttal to Reviewer z9DJ, Q2.)\n\nIf any aspects remain unclear, we’d be happy to further elaborate during the discussion phase.\n\n\n[1] Chen and Kash (2011). Information Elicitation for Decision Making.\n\n[2] Chen, Kash, Ruberry and Shnayder (2011). Decision Markets with Good Incentives.\n\n[3] Othman and Sandholm (2010). Decision Rules and Decision Markets."}, {"review_id": "TGbjynKjDg", "reviewer": "Reviewer_soxu", "summary": "The paper studies forecasting through the lens of performativity: it identifies limits of classical forecasting under performative feedback and proposes new tools for more reliable forecasting under performativity. It shows that classical proper scoring rules fail to be correct under performativity. Under suitable conditions, it also shows the existence of proper incentive-compatible scoring rules. Finally, the paper shows how correct forecasts can achieve performative stability and performative optimality simultaneously.", "questions": "1. Can you explain counterfactual correctness? I don't understand how we are able to talk about the conditional distribution Y|A=a, do(F) when P(A=a|do(F)) = 0. I don't understand this criterion conceptually or mathematically.\n2. Is Theorem 1 a classical result?\n3. In Theorem 6, \\theta_{t+1} is performatively stable and optimal for *all* t, correct?", "limitations": "Yes, these are explicitly discussed in Section 7.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This work is very well-motivated. In a lot of problems motivating performative prediction the goal is to make accurate forecasts; for example, in policy problems, such as when we are predicting a distribution of future covid cases. The paper establishes a clear foundation for studying forecasting under performativity, and it establishes basic first results contrasting classical forecasting with forecasting under performative feedback.\n\nThe current results are quite abstract and not very practical and prescriptive, but I think they are a good starting point. The examples in the paper were very helpful to me for grounding the discussion, and I would appreciate it if there were even more such concrete examples where all variables and desirable criteria would be fully explained and instantiated.\n\nIt's worth mentioning some classical results on the possibility of accurate forecasts under performativity; for example, \"Bandwagon and Underdog Effects and the Possibility of Election Predictions\" by Herbert Simon (1965).", "was_revised": true, "final_justification": "I was positive about this paper from the beginning. I'm arguably even more positive now after the discussion phase, knowing that Theorem 1 is novel. The paper is original and well-motivated.", "rebuttal": "Dear reviewer, thank you for your helpful remarks and questions.\n\nReviewer Tz7S made a similar comment, that the paper is a bit dense. Due to space constraints we had to put visual examples in the supplements. We would like to spend part of the additional page to add more elaborate motivations, explanations and examples to the paper.\n\nThank you for pointing us to the paper by Herbert Simon. We will add a reference (also to Grunberg, E. and Modigliani, F. (1954). The Predictability of Social Events.) to the paper.\n\n**Q1:**\n\nConceptually, counterfactual correctness means that the conditional forecast is correct for all actions $a\\in \\mathcal{A}$ which *cannot* be observed. This should be contrasted with observational correctness: that the forecast is correct for all actions which *can* be observed. Mathematically, the causal mechanism prescribes a distribution $P_M(Y|do(A=a, F))$ for all possible values of $a \\in \\mathcal{A}$, so also if $P_M(A=a | do(F))=0$. In the setting that we consider there is no confounding between $A$ and $Y$, so we can pick a version of the conditional distribution which satisfies $P_M(Y | A=a, do(F)) = P_M(Y|do(A=a, F))$ for all $a\\in\\mathcal{A}$. \nWe will elaborate on this nuance in the paper.\n\n**Q2:**\n\nTo our knowledge, Theorem 1 is entirely novel.\n\n**Q3:**\n\nWell, it might be the case that for $t=1$ the parameter is incorrect and not immediately performatively stable, but if the next parameters $t>=2$ are obtained by repeated minimization of the divergence, then $\\theta_t$ is stable and optimal for all $t>=2$. \n\nIf any aspects remain unclear, we’d be happy to further elaborate during the discussion phase."}, {"review_id": "S0fpBxn5Qt", "reviewer": "Reviewer_Tz7S", "summary": "This paper studies the problem of how to formalize performative prediction and various issues and solutions that are particular to it (as opposed to other forms of forecasting). This problem is essentially that when a forecaster makes a forecast $F$, it alters the distribution of the outcomes. This is a problem because in such cases a forecast might never be correct in the sense that once it is predicted, it doesn't really match the \"real-world\" distribution of the event.\n\nTo avoid this issue, the authors formalize this problem via a causal mechanism using conditional forecasts and causal inference. Concretely, they focus on forecasts which are conditional on given actions and pose a causal graph on the problem where there is a path from $F$ to actions $A$ and then $Y$, and optionally one path from $F$ to $A$ directly.\n\nOnce this causal formalization has been introduced, the authors proceed to formalize various notions that have to do with these types of forecasts, such as three types of correctness (observational correctness, counterfactual correctness, and the combination of these two) and the notion of forecast invariance, which means that in the causal model $P(Y \\mid A, \\text{do}(F)) = P^M(Y\\mid A)$. Forecast invariance is important as it solves to some degree the issue, which is a priori intuitive but also given the examples the authors provide. The authors also show that this notion is equivalent to d-separation of $Y$ and $F$ by $A$ in Theorem 1.\n\nAfter completing the general setup, the authors proceed to discuss various problems in this setting.\n\nThe first one is the problem of proper scoring rules. The authors first straightforwardly extend the definitions for a reasonable proper scoring rule to the conditional setting and then show in Theorem 2 that all non-trivial proper scoring rules designed for the non-conditional setting fail for performative prediction. The intuition is that the forecaster may produce conditional forecasts that bias the actions to a world where it can more easily predict $Y$. Their first proposed solution is to align decisions by using utilities to create proper scoring rules. In Theorem 3, they show this score aligns with the goal of maximizing utility and is observationally proper in the $Y \\mid A$ invariant setting. Theorem 4 then deals with some technicalities to make it strictly proper.\n\nThe second solution that they propose is batch scoring, which is essentially a divergence that measures the gap between the proper score of $P(Y \\mid A=a, \\text{do}(F))$ and the proper score of $F(Y\\mid A= a)$, assuming the integral is with respect to $P(Y, A \\mid \\text{do}(F))$. They show that this scoring rule is proper, although it has the disadvantage of requiring access to $P(Y \\mid A=a, \\text{do}(F))$, which makes it impractical for many scenarios.\n\nFinally, in Section 6, they propose an iterative procedure that extends these ideas to parameter estimation. In Theorem 6, they guarantee that this procedure is stable (so that it attains a fixed point) and is optimal in the sense of minimizing risk.", "questions": "I have no major questions beyond those stated in the weakness section.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**: I thought the paper was very clear and well-written. I really enjoyed reading it.\nI think the contributions that the paper makes are interesting. To state some:\n- The rigorous causal framing to ground the framework is interesting and a good contribution as it clarifies the problem. While intuitive, formalizing is important.\n- The impossibility result for classical scoring clearly articulates what might go wrong, and the authors do a good job explaining it to the reader both with examples and formally.\n- The authors do a good job of both formalizing the criteria for proper scoring rules to work in this setting and also propose two ways of constructing proper scoring rules.\n- The authors propose a method for parameter estimation in this setting and show that it reaches optimality and is well-behaved.\n\nWeaknesses\nIn my opinion, there are no major weaknesses with the paper; however, some small ones are the following:\n\n- Although the paper is well-written, I think there are some things that could be improved notationally. For example, the paper tends to use integrals all the time, which could be more cleanly expressed as expectations. This is especially the case in Section 6, where parsing the integral is quite dense and makes it hard to see what's happening at a glance. In this section as well, I felt that an algorithm box could have clarified things, although I understand the authors might have chosen not to do that due to space constraints.\n- In Section 6, I didn't really understand the connection to score matching, and I think this could be spelled out more.\n- One limitation of the batch scoring metric they propose is that it is hard to estimate from data in practical settings. I think it could be good if the authors devoted some more time to explaining this.\n- I think one possible weakness of the paper is that a lot of the framework is fairly intuitive, and some of the contributions are mostly about formalizing things: coming up with the right definition and such. However, I think this work always needs to be done, so I wouldn't hold it strongly against the paper even if aspects of it are somewhat intuitive.\n- A minor weakness is that this paper focuses on a very specific setting, so it is unclear to me how high the impact would be.\n\n**Overall**\nAs I said, I really enjoyed reading the paper. While I think that due to being a bit dense, there could be more effort put into clarifying some notation and concepts, especially at the end, I think this is an interesting contribution.", "was_revised": true, "final_justification": "The rebuttal did not change my view on the paper and all of my previous comments and positive opinion  of the paper stands so I retain my acceptance score of 5.", "rebuttal": "Dear reviewer, thank you for your helpful remarks and questions.\n\n**W1:**\n\nFor example, before Definition 5, in the integral $\\int S(F, y)P_M(d y | do(F))$ we want to explicitly denote the model $M$, the dependence $do(F)$, and the random variable $Y$ that we integrate over. Alternative notation would be $E_{P_M(Y | do(F))}\\left[S(F, Y)\\right]$, but since this does not seem much more intelligible we chose for the integral notation.\nWe would like to spend part of the additional page to add an Algorithm box to Section 6.\n\n**W2:**\n\nHere, score matching is a name of the parameter estimation method using the Hyvärinen score $S(F_\\theta, y) = \\frac{\\Delta \\sqrt{f_\\theta(y)}}{\\sqrt{f_\\theta(y)}}$ (where $f_\\theta(y)$ denotes the density of the distribution $F_\\theta(Y)$ and $\\Delta$ denotes the Laplacian), as introduced by [1] (see also [2]). Perhaps the confusion stems from ‘propensity score matching’ in causal inference? This is an entirely different method. If deemed necessary, we can add a footnote to prevent confusion.\n\n\n**W3:**\n\nAs also remarked in our rebuttal to Q1 of Reviewer z9DJ, we will add the important result that for the estimated divergence to be proper, one does not have to estimate it accurately; the only requirement is that an *unbiased* estimator is utilised. In that case, for any sample size that will be used to estimate the divergence, the forecaster's expected estimated score (divergence) equals the true expected score (divergence), and hence the method is proper.\n\n**W5:**\n\nNote that a setting where the forecast depends on covariates is analysed in the supplements, which greatly improves the applicability. We will move this section from the supplements to the appendix for greater visibility.\n\nYour final comment is similar to ‘weakness 1’ of Reviewer soxu. We would like to spend part of the additional page to add more elaborate motivations, explanations and examples to the paper.\n\nIf any aspects remain unclear, we’d be happy to further elaborate during the discussion phase.\n\n[1] Hyvärinen, A. (2005). Estimation of Non-Normalized Statistical Models by Score Matching.\n\n[2] Dawid, A. P. and Musio, M. (2014). Theory and applications of proper scoring rules."}]

Author rebuttal:

Meta-review: This paper explores truthful elicitation and proper scoring rules in the context of performativity, where a forecast can influence the outcome it predicts. It demonstrates that traditional scoring rules are ineffective in this setting. The paper presents a general impossibility result and proposes two solutions: (1) In decision-theoretic settings, truthful and optimal scoring is achievable if forecasts are separating. (2) Batch-scoring methods, which use empirical divergence or inverse probability weighting, can produce accurate forecasts. These findings are then applied to parameter estimation and conditional forecasts, showing that proper scoring rules can lead to performatively stable estimation of correct parameters.

The reviews are overall positive. Nevertheless, Reviewer `z9DJ` remains skeptical about the finite-sample properness of Theorem 5 and the full support assumption. They cited that "*these two issues limit the applicability of the results in the batch setting and the parameter estimation setting.*" I find the concerns raised by Reviewer z9DJ reasonable. On the other hand, Reviewers `DFQj`, `soxu`, and `Tz7S` are in favour of accepting this paper, praising its originality and novelty. However, Reviewer `Tz7S` also pointed out that a lot of effort went into 'formalising things' rather than addressing non-trivial challenges of the problem. Regardless, Reviewer `soxu` said, "*they are a good starting point*" and appreciated the novelty of Theorem 1, which is corroborated by Reviewer `Tz7S` who stated, "*this work [formalising things] always needs to be done, so I wouldn't hold it strongly against the paper even if aspects of it are somewhat intuitive.*" This last comment seems to align with the concern raised by Reviewer `z9DJ`.

After reading the paper, it's clear that, despite some limitations, the paper has made a significant contribution in truthful elicitation of performative forecasts, elucidating the important challenges from the novel causal perspective and providing preliminary solutions to mitigate some of these challenges. This paper is arguably one of the few papers to thoroughly study the relationship between probabilistic forecasts, proper scoring rules, and performativity from the causal perspective. The main contribution is the new theoretical framework, while other areas, like the applicability of the results to batch processing and parameter estimation, remain less developed. As a result, it deserves a presentation at the conference.
Decision: Accept (poster)