The Limits of Predicting Agents from Behaviour
Authors: Alexis Bellot
Jonathan Richens
Tom Everitt
Venue: ICML 2025 poster (2025)
Keywords: Safety
Alignment
Causality
Partial Identification.
Primary area: social_aspects->safety
TL;DR: We show that the external behaviour of AI systems, i.e., their interactions in an environment, partially determines their ``beliefs'' and future behaviour.

Abstract: As the complexity of AI systems and their interactions with the world increases, generating explanations for their behaviour is important for safely deploying AI. For agents, the most natural abstractions for predicting behaviour attribute beliefs, intentions and goals to the system. If an agent behaves as if it has a certain goal or belief, then we can make reasonable predictions about how it will behave in novel situations, including those where comprehensive safety evaluations are untenable. How well can we infer an agent’s beliefs from their behaviour, and how reliably can these inferred beliefs predict the agent’s behaviour in novel situations? We provide a precise answer to this question under the assumption that the agent’s behaviour is guided by a world model. Our contribution is the derivation of novel bounds on the agent's behaviour in new (unseen) deployment environments, which represent a theoretical limit for predicting intentional agents from behavioural data alone. We discuss the implications of these results for several research areas including fairness and safety.

The Limits of Predicting Agents from Behaviour
Alexis Bellot Jonathan Richens'
Tom Everitt
Abstract
tween beliefs and behaviour that is foundational to rational agents (Davidson, 1963).
As the complexity of AI systems and their in- teractions with the world increases, generating explanations for their behaviour is important for safely deploying AI For agents, the most natu- ral abstractions for predicting behaviour attribute beliefs, intentions and goals to the system. If an agent behaves as it has a certain goal or belief, then we can make reasonable predictions about how it will behave in novel situations, including those where comprehensive safety evaluations are untenable. How well can infer agent's be- liefs from their behaviour, and how reliably can these inferred beliefs predict the agent's behaviour in novel situations? We provide a precise answer to this question under the assumption that the agent's behaviour is guided by world model. Our contribution is the derivation of novel bounds the agent's 's behaviour in new (unseen) deploy- ment environments, which represent theoretical limit for predicting intentional agents from be- havioural data alone. We discuss the implications of these results for several research areas includ- ing fairness and safety.
Artificial Intelligence (AI) systems appear to have similarly general capabilities, not totally unlike that of humans and animals. They can generate text that is fluent and accurate in response very diverse set of questions. Whenever they display consistent types of behaviour across many different tasks, we are tempted to apply our own mentalistic language more or less at face value (Shanahan, 2024), taking seriously questions such as: What do the AIs know? What do they think, and believe? Taking the analogy further, they learn "world models" that mirror the causal relation- ships of the environment they trained on, guiding their future plans and behaviour', And as consequence, their interactions with an environment will leave clues that might give us the ability to predict their future behaviour in novel domains. This possibility engages with a core Al Safety problem: how to guarantee and predict whether Al systems will act safely and beneficially?
as if
The main result of this paper is to offer new perspective on this problem by showing that:
With an assumption of competence and optimality, the behaviour of Al systems partially determines their actions in novel environments.
1. Introduction
Here behaviour means our observations of the decisions made by the AI system, contextual variables, and utility or reward values in some environment. The "partial" determi- nation of actions in new environments is a consequence of our lack of knowledge about the Al's actual world model (different models may induce different optimal actions). However, even though we can't uniquely identify the Al's future behaviour and beliefs, we can narrow down to a range of possible outcomes. This paper characterises those outcomes.
Humans understand each other through the use of abstrac- tions. We explain our intentions by appealing to our "goals" and "beliefs" about the world around us without knowing the underlying cognition going inside our heads. According to Dennett (1989; 2017) the same is true of our understand- ing of other systems. For example, a bear hibernates during winter as if it believes that the lower temperatures cause food scarcity. This is a useful description of the bear's 's be- haviour, with real predictive power. For example, gives us (human observers) the ability to anticipate how bears might the climate changes. There correspondence be-
Receet research suggests that an Al's behaviour, to the extent that it is consistent with rationality axioms, can be formally de- scribed by (causal) world model (Halpem & Piermont, 2024). The same conclusion can also be obtained for Als capable solv- ing tasks in multiple environments (Richens & Everitt, 2024), For large language models, there is increasing empirical evidence for the "world model" hypothesis, see e.g.. Toshniwal al., 2022; Li et al., 2022; Gurnee & Tegmark, 2023; Goldstein & Levinstein, 2024 and Vafa et al., 2024.
'Google DeepMind. Correspondence to: Alexis Bellot <abel- lot@ googgooooccom..
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Limits of Predicting Agents from Behaviour
In the literature, the under-determination of agent "beliefs" and "preferences" has been considered in the fields of in- verse reinforcement learning (Abbeel & Ng. 2004; Skalse & Abate, 2023; Amin & Singh, 2016) and decision theory (Savage, 1972; Afriat, 1967; Jeffrey, 1990), among others. In settings with distribution shift between training and de- ployment environments, this under-determination can be understood as a consequence of the Causal Hierarchy The- orem, that defines precise limits on the kinds of inferences that can be drawn across domains (Bareinboim et al., 2022; Pearl, 2009). It implies, for example, that behaviour in an environment subject to an intervention cannot be established from nonniiterrventioaa" data alone. Robins (1989), Man- ski (1990) and Pearl (1999) showed that useful information in the form of bounds can nevertheless be extracted from tron-interventional" data, without actually knowing the un- derlying data- generating process. In the causality literature, several methods and algorithms exist to solve different ver- sions of this problem, see e.g., Balke& & Pearl, 1997; Tian & Pearl, 2000; Zhang et al., 2021; Bellot, 2024; Rosenbaum et al., 2010; Tan, 2006.
2. Preliminaries
In this section we outline some basic principles that we use to reason about how beliefs might be (implicitly) defined within an AI system.
We use capital letters to denote variables (X), small letters for their values (x), bold letters for sets of variables (X) and their values (æ), and use supp to denote their domains of definition (æ E suppx) ). To denote P(Y = y X = x), we sometimes use the shorthand P(y x). We use 1 ( for the indicator function equal to if the statement in {-) evaluates to true, and equal to 0 otherwise.
Actions, plans, and hypothetical outcomes can be evalu- ated by symbolic operations on model that represents the functional relationships in the world, known as a Structural Causal Model (Pearl, 2009, Definition 7.1.1), or SCM for short.
Definition 1 (Structural Causal Model), An SCM M is a tuple M = (V.U,F,P P) where each observed vari- able V E V a deterministic function of a subset of variables Pay C CV and latent variables Uv = U, i.e. v == Jy(pav, ,uv), Iv F. Each latent variable U U is distributed according to a probability measure P(u). We assume the model to be recursive, i.e., that there are no cyclic dependencies among the variables.
This paper extends the causal formalism to reason about the possible behaviours and beliefs of an AT system, itself assumed to be governed by an unknown data generating process or world model. With this interpretation we are able to define mathematically notions such as an Al's preferred choice of action in novel environments, its perception of fairness, and its perception of harm due to the actions it takes. Our main contribution is a set of inequalities on these "beliefs" in terms of quantities that in principle be es- timated from behavioural data, and that hold irrespective of the underlying cognitive architecture of the AI system as long as it can be represented by a well-defined set of causal mechanisms (a world model) that tracks its behaviour (Sec. 4). We then extend these results to characterize Al be- haviour under several relaxations for applications in practice (Sec. 5), ultimately with the goal of defining the theoretical limits of what can be inferred from data about AI behaviour in new (unseen) environments.
In SCM M, each draw u ~ P(u) evaluates a potential response Y(u) = and entails a distribution over the possible outcomes P(y). The power of SCMs is that they specify not only the joint distribution P(v) but also the distribution of variables under all interventions, including incompatible interventions (counterfactuals) Formally, an intervention do(æ)) is modelled as a symbolic operation where values of set of variables X are set to constants x, replacing the functions (fx X X) that would normally determine their values. This effectively induces a sub-model of M, denoted Mz. The variables obtained in Ma are denoted Y2 and we will loosely write pMM (y) = Pa(y) = P(y=) = p(y I do (())) denote the probabilities over the possible outcomes of Y in Ma.
This has consequences for the wider AI Safety community and society. For example, we show that an Al's percep- tion of the potential fairness and harm of its decisions (e.g., whether the AI's resouree allocation is believed to be eq- uitable, or its generations unbiased) can provably not be inferred from observing its behaviour alone. There are theo- retical limits to how much we can understand about an Al's cognition and decision-making process from observations. We believe our results can help justify the claim that the design and inference of world models is important to ensure AIs can behave predictably and act safely and beneficially, as argued by Dalrymple et al., 2024; Legg, 2023; Bengio et al., 2025.
Different environments can be modelled by different SCMs. Let M, = (V.U, U,, M2 = (V. U, F,F p2> be the SCMs for two environments over the same set V and U. We say that there is a discrepancy or a shift on a variable X E V between them if either ʃ * f or P*(Ux) * (Ux) or both. Shifts might therefore encode arbitrary changes in the causal mechanisms for set of variables. For a reference SCM M, so-called "shifted" SCM will be represented by sub-model Ma where o represents the discrepancies between M and M,. For examplc, an environment with a shift a on a set of variables X introduces (possibly arbitrary) discrepancies in the functional assignment or (independent) exogenous variables of X while keeping other mechanisms
2

---

The Limits of Predicting Agents from Behaviour
unchanged. See Pearl (2009, Chapter 4) and Correa & Bareinboim (2020b) for more details.
of the environment. In principle, we have no reason to expect that the model M internalized by the Al matches the underlying reality M. Al systems might hope to reproduce some aspects of M (the Al might have learned, for instance, to mimic the distribution of the observed data). Competent AIs might go further and be able to reliably predict the effects of different decisions in the world. We define this as grounding below.
We make a note here that all proofs of statements are given in Appendix C and that the derivations of examples are given in Appendix A.
3. Agents, Beliefs, and the Environment
In this section we lay out a framework to interface between the Al system's 's internal world model and our own observa- tions of their behaviour in the real world. Both rely on the same SCM abstraction.
Definition 3 (Grounding), Let M represent the Al's internal model. We say that the Al is grounded in a domain M if pMa((V) pMa(VV) for any decision d E supp
We assume that the Al operates according to an SCM M over V. its (implicit) world model?, that guides its be- haviour. V includes the AI's decision variable D, the inputs to those decisions C. possible additional variables, and the utility variable Y, such as the training signal or measur- able target given to the Al (Everitt et al., 2021). Beliefs' are defined as quantifiable aspects of that model or derivations of it.
Grounding tells us that the Al's beliefs about the effect of a particular decision d in the training environment match the effects that would be observed in the real world, i.e. Pa(V) = Pa(V)), It is an assumption on the relationship between our observations of Al behaviour P(V) with what might be going on in the Al's "mind" P(V). This might be reasonable, for example, if the Al is explicitly trained by reinforcement learning in M.
Definition 2 (Belicfs). An Al belief is a probabilistic state- ment derived from its internal model M.
By assumption, a grounded Al's choice of decision in envi- ronment .M is in principle predictable from data since we can compute Eq. (1) uniquely. But this might not necessarily be the case in a new (unseen) environment.
For example, a statement like pMaa(y = y) = 0.8 de- scribes the subjective belief "The Al is 80% confident that taking decision D d will lead to event Y y"" The sub-model in this mathematical expression represents what the AI "thinks" the world looks like after taking the decision D = d.
Example 1 (The Uncertain Medical AI). Imagine an AI system assisting patients with their treatment D for a disease Y known to be influenced also by third variable Z, blood pressure. The AI is competent and learns the precise effect of all treatments. In other words it is grounded in M, i.e. Pa(z, y) = Pa(z, y). For concreteness, let the environment M be given by,
We assume that the AI makes decisions d by sampling from a policy (dd] c), which is a function mapping from the domain of the observed covariates C V (i.e., all the inputs given to the AI) to the probability space over the domain of the decision D V. The choice of is assumed to be driven by its perceived utility* Y e V within the Al's model M, that is,
Z < 10=1or4
Z 10-4 + (1- 10-13344 if
Y -
arg max E paa Y d.
(1)
Z 1.1422 + (1 (1-Z) Z) 1U=20rd if = 1,
The Al interacts with the real-world that is described by (likely different) SCM M that encodes the true dynamics
with equal probability P for all values U (1,2,3,4,55 Here U is latent, summarizing all other contributions to both the disease and blood pressure, such as an individual's (unobserved) attitudes to health, fitness, etc. Could we confidently deploy this AI system more widely, for example, individuals that also take second drug that artificially improves their blood pressure (e.g.. fixing Z to 1, replacing the original assignment)? If the AI system is instructed to maximize Y on average, what decision does the Al believe is optimal in this new environment? The answer is we do not know, meaning that it is possible to find second model
Heee SCMs are meant represent, mathematically, the decision-making process going on "in the Al's head" in way that tracks its behaviour, without making any claims about the Al's actual cognitive architecture.
'We might prefer to use terms like "credences" or "subjective probabilities" " to emphasize the subjective nature of beliefs and avoid the connotation of strong conviction or certainty as done by (Schwitzgebel, 2024, Sec. 2.3).
"To account for possible uncertainty in the Al's "satisfaction" " about a given state of the world w we assume Y is random variable (induced by Uy < U), also known as stochastic util- ity model (Manski, 1977). We assume that the support of Y is bounded in the [0. i] interval.
'e the shorthand Pa = P-Ma and Pa pMa simplify the notation.
3

---

The Limits of Predicting Agents from Behaviour
M defined by the mechanisms:
e.g., grounding. Weak predictability means that there exists at least one decision that we can guarantee the Al will not take in the shifted environment. Specifically, we can rule out a decision d if and only if we can find a (superior) alternative decision d o d* such that,
Z - -1
y - 2 lu + (1 -1-23 Z). 00-3 if = 0
Z 11=10r4 + (1 Z) 1U=1022 if d =
min ( (Aazan ) > 0, MEM
(3)
that entails exactly the same observations Pa(z,y) = Pa(z. y) but induces different optimal decisions in the new environment (under the intervention Z 1). Un- der MM-1, the highest utility Y average is given by - 1, while under MZ-1 the highest utility Y on average is given by d = which model (M or M) is governing the Al's behaviour and so no way of knowing what decision will be favoured by the AI.
where,
A d=dd
E Epa [Y do(a, d), B..cc-Expp
Epsi [Y] do(a, d=)), cl.
priori, we have no way of knowing
M. denotes the set of "valid" SCMs. Here A can be inter- preted as the Al's preference gap between two decisions in some situation C = c. When it evaluates to a positive num- ber d is preferred to d and when it evaluates to a negative number d is preferred ences A allow us to rule out decisions dl considered to be "unsafe" then weak predictability gives us an important safety guarantee.
This example illustrates a canonical point in a simple setting: as observers, with access to the Al's interactions in some domain, its behaviour outside of that domain might not be uniquely determined (Pearl, 2009).
(in the Al's mind). If our infer-
We can strengthen this notion to define strong predictability, that describes a situation in which all but a single Al decision can be ruled out.
4. The Limits of Behavioural Data
In this section, we explore the limits of behavioural data for predicting the decisions of AIs in new environments.
Definition (Strong Predictability). We say that an Al strongly predictable under shift a in situation C c if the optimal decision is uniquely identifiable, ie., there exists single decision d such that,
As external observers, we do not have access the mech- anisms underlying the actual environment nor the agent's 's internal model. We assume that we must rely for our in- ferences on watching the agent's behaviour and its conse- quences. That is we have access to (samples of) PV(V for all d. As a starting point, we might expect competent AIs to be weakly predictable in the sense that subset of decisions can be ruled out provably sub-optimal given our observations.
Bammaxppr d = arg max B PMx (Y do(a, c],
(4)
for any valid SCM M describing the Al's internal model.
4.1. AI decisions out-of-domain: interventions
Our first result shows that, in some cases, subset of AI decisions can be provably ruled out, i.c., the Al is weakly predictable.
Definition 4 (Weak Predictability). We say that an Al is weakly predictable under a shift in situation C if there exists decision d² that is provably sub-optimal, i.e.,
Theorem 1. An Al grounded in a domain M is weakly predictable under a shift a == do(2), Z c V, in a context C = c if and only if there exists a decision d such that,
d # arg max psa Pxa
[Y|
do(o, c),
(2)
for any valid SCM M describing the Al's internal model.
Ep, Y c, 1
(5)
Pa(c, 2) + 1- -P((z)
Here, "valid" means that the Al's internal model is compati- ble with the observed data under our assumptions about the relationship between the data and the Al's internal model,
Ep, | c, 1/ll,2/]PPa(c,, +1 Pao (z)
> 0,
Pao (c,, 2) (z)
"Technically, the AI system may choose to follow an arbitrarily complex policy in the training domain, inducing (assumed positive) distribution P. (v). It holds that Pa(V) can be computed from any such P. P*(V) (V) as long as P. (v) > 0, Vu, and vice versa, see c.g. Lem. 1. The positivity assumption Pa(v) > 0 rules out fully deterministic policies in the available data but might be reasonable if the AI spends some time exploring before committing to a course of action.
for some d + d
All terms on the l.h.s are in principle computable from the Al's behaviour. Loosely speaking, the value of this difference is determined (in part) by "Pa* (z)": if Z = z (the value set by the intervention) is likely under the training distribution, the difference will more likely evaluate to a
4

---

The Limits of Predicting Agents from Behaviour
Possible Al World Models (SCMs) M
Action choice o.o.d A
Approximately Grounded
Grounded
Multiply Grounded
Figure Grounding and observations in multiple environments constrains Al's world model improves our prediction Al behaviour out-of of-tiiiiibbttion (o.o.d). Approximate grounding is defined in Sec. 5.
positive value. The "if and only if" condition means that whenever this inequality does not hold we can construct two SCMs MI,Ma for the grounded Al's internal model that generate the observed behaviour Pa(V), d e suppp. but that induce different optimal actions. That is, for all d at ds,
Theorem 2. Let == do(z) be shift on set of variables Z c V. For R; c Z c V,i 1,...,k, k, consider Al grounded in multiple domains {My, : 1.....k) k). The AI is weakly predictable in context C = c under a shift σ == do(z) if and only if there exists a decision d" such that,
E Ex [ [Y]
do d), > E pM YY do 2, d c].
E E Y do(2 d), < E pM [Y
do d²) c)
max A(ri,r,) > 0, for some d # d², (6)
4.3
Remark. We can derive a similar condition for strongly predictable AIs by replacing "for some d a d with "for all d in Thm.
where
A(ra,Ty) Epan! (, Pa, + 11 Pa.,, (z)rz) Eps,.r (Y (/ ( (c,z)v,) + (s\r))
We illustrate Thm. 1 with the following example.
Example 2 (Grounded Medical AI). In Example 1, we have shown that there exists a particular intervened environment in which the AI's intentions cannot be determined as in principle the AI could believe that either decision is optimal. Is this true in general? Thm. suggests that depends the likelihood of different events Pa(z, y) in the observed data. For Example 1, we can show that the medical Al is not weakly predictable as the expression in Thm. is negative for all pairs of decisions, In other words, no decision can be ruled out in general: in some AI internal models d1 is inferior do as min MeM Adj dg Pa (Z 2, Y 1) + Pa. (Z z, Y = 0) 11 -0.4 while in others dg is inferior di as min MeM Advd,
Pat (c, 2 2)² P (2)r,)
In this result, {Mr. : 1 1,.... k} describes domains in which experiments on different subsets of Z have been conducted, i.e., (Pa,r. (V) :i= 1..... .......k) k) is available. This
includes possibly the null experiment R, Ø that refers the unaltered domain M. Note that grounding in multiple domains is useful for the prediction of the Al's preference gap because the resulting bounds in Thm. 2 are tighter than those in Thm. 1 (this is given formally as Corol. 3 in the Appendix).
Pa, (Z 2, Y 0) Pa, (Z= = Y = =) == -0. and we don't 't know which one the AI system has internalised.
Fig. illustrates how different assumptions and observations give us information about the possible world models that the il operating on, which then has implications for the Al's behaviour out-of-distribution This knowledge allows us to reduce the uncertainty around the Al's preference gap A, and possibly rule out certain actions that are unambigu- ously sub-optimal observed behaviour.
In this example, Al behaviour does provide some informa- tion it can be constrained to larger values than its priori minimum A- = -1, but not enough to rule out decision completely. Our next result shows that Thm. could be ex- tended to get tight bounds for Al systems that are grounded in multiple environments.
inferred solely from
5

---

The Limits of Predicting Agents from Behaviour
suggests that the medical AI might not be weakly pre- dictable the expression evaluates to a negative value for eral shifts such as an arbitrary change in a subset of the mechanisms {fz Z E Z} and distribution of variables (U2, Z e 2 in M. For example, in practice we are likely able to convey to the AI that the mechanisms for a set of variables Z are expected to change but not know exactly how. For example, demographic properties of patients might change across hospitals. How could the Al interpret the con- sequences of such an under-specified shift? To begin to answer this question, the following theorem shows that in the extreme case where the nature of the shift is completely unknown the Al's preference gap is unconstrained.
4.2. AI decisions out-of-domain: general shifts
We might wonder about predictability under more gen-
all pains of decisions. The lower bounds the Al prefer- ence gap are given by min MeM (
[d ) > -0.55 and
min Mee ( dd ) 2 -1. That is, no decision is always inferior to any other decision.
4.3. Al's perceived fairness of decisions
An Al's policy, even if optimal on average, has the potential to bring about a state of the world that is intrinsically harm- ful or unfair. Harm and fairness can be defined relative to a causal model (Beckers et al., 2022; Plecko et al., 2024). This means that notion of perceived or subjective harm and fairnsss could be attributed to AI systems that operate according to an (implicit) causal model. As a consequence, it is conceivable that AIs could be held morally accountable for the harm and unfairness that they cause. How might one estimate the Al's beliefs about the harm and unfairness that its decisions cause?
Theorem 3. Consider an Al grounded in a domain M made aware of an (under-specified) shift on non-empty Z c V. Then the Al is provably not weakly (or strongly) predictable in any context C = c.
This result means that no decision could ever be ruled out from AI behaviour. We can show moreover that min MEM ( (A) -1 for any pair of decisions, meaning that the observed data (no matter what it is) gives us information on AI decision-making.
To ground our discussion, consider here explicitly coun- terfaciual accounts of fairness and harm. These appeal to hypothetical situations, imagining "what might have been if that can force us to confront our assumptions and values in a way that our regular thought processes might not?. For example, the counterfactual event (Y, to the outcome (Y- 1) under an intervention X a when under normal circumstances X would have evaluated to zo- In the literature, probabilities over counterfactuals emerge from the definition of an SCM. For a set of (counterfactual) events (Z....., yz).
In practice, however, it might be realistic to have access to some information in the shifted environment, such as covari- ate data, i.e., (samples from) Po.a(c), that could be given to the Al for it to update its internal model accordingly (with some abuse of terminology we say that the Al is grounded in Pa.d(c) ()). The next theorem shows that this additional information coupled with the Al's behaviour makes the Al more predictable.
X = za) refers
S Ju: Jw: Y=(u)=y=
Theorem 4. Consider Al grounded in domain M and (C, made aware of shift a on Z c C. The Al is weakly predictable under this shift in a context C = e if there exists a decision d² such that,
P( yz) =
P(u). (7)
Kusner et al. (2017) made a concrete proposal arguing that an Al's decision is said be fair towards an individual if. from the Al's perspective, it entails the same utility in the actual world and in counterfactual world where the indi- vidual belonged to a different group (defined by a sensitive attribute, e.g., gender, race). We adapt this notion to define an Al's counterfactual fairness gap.
2 Ep Y cc (c) - Ep, Y (P((()
Po.d((c)
> 0, for some d # d=. Pa(c) 2Pa(z)
P P (c)
This bound is not tight in general, however, meaning that it is possible that the Al actually predictable in settings where Thm. suggests it might not be.
Definition 6 (Counterfactual Fairness Gap). Let Z e {2o, 21 be a protected attribute and zo baseline value of Z. For a given utility Y, define an Al's counterfactual fairness gap relative to a decision d, in a given context c, as
Example 3 (Shifted Medical AI). The AI from Example 2, originally developed from data primarily from young pa- tients, is now considered for deployment on an older patient population. Their probability of having high blood pres- sure P. = 1)= = 0.9 is known to be substantially higher than that observed during training P(Z = 1) = 0.4: there is a shift in the underlying mechanisms of Z. How do these changes influence the Al's beliefs on A? Thm.
:p (Y.,, [00.c] Ep[
[2 20, c.
(8)
'Altemattvv accounts to harm and faimess have been proposed (Barocas & Selbst, 2016; Zhang & Barcinboim, 2018; Plecko et al., 2024), sometimes motivated by scenarios where counterfactual accounts give incomplete results. For some of them, the Al's beliefs can be shown to be similarly constrained by its extemal behaviour. We provide longer discussion in Appendix D.
6

---

The Limits of Predicting Agents from Behaviour
We say that Al "intends" to be fair with respect attribute Z if under any context C C. and decision D d the counterfactual faimess gap I evaluates to 0. This means that, under its own internal world model, changing the value of Z on the subset of situations with context e in which Z was observed to 2g does not change the Al's expected utility.
Theorem 6. Consider an Al with utility Y grounded in
domain M. Then,
Q > max EP,, [Y| cl| Bp, [Y|cl-
Ω E min [Y] cc Er, [Y ce]
This bound is tight.
In the following theorem we show that, unfortunately, the
answer to this question is impossible to obtain given only the Al's external behaviour.
This result it an extension of bounds on the probability of causation given by (Pearl, 1999) and (Tian & Pearl, 2000). suggests that an Al's belief's about the harm that its decisions cause can be inferred approximatcly from data.
Theorem 5. Consider an agent with utility Y grounded in domain M. Then,
Y 120 1-Tpl= Y
100 cl.
(9)
5. The "Practical" Limits of Behavioural Data
This bound is tight.
The inductive biases implied by causal models and ratio- nal behaviour are powerful constraints on AI behaviour. But they might not capture the practical limitations of Al decision-making. In this section show that grounding, expected utility maximization, observed data, etc., can be relaxed in practice.
The bound is tight in the sense that for each context, deci- sion, and baseline attribute, we can find compatible models for which the equalities hold. The counterfactual fairness gap Y is under-constrainee Since Y is consistent with any external behaviour we can never conclude that the Al system "intends" to be unfair. Moreover, since the width of the bound is equal to I, we can also never conclude that the Al is anywhere "close" to being fair, according to this counterfactual criterion.
5.1. Approximate grounding
Grounding implies that the Al's beliet's on the likelihood of events in the environment matches the observed probabili- ties. In practice, it might be reasonable to allow for some amount of error, and consider a notion of "approximate" grounding.
4.4. Al's perceived harm of decisions
Prominent definitions of harm are similarly counterfactual in nature: the counterfactual comparative account of harm defines a decision d to harm a person if and only if she would have been better off if d had not been taken (Hanser, 2008; Richens et al., 2022; Beckers et al., 2022; Mueller & Pearl, 2023). It is a contrast between events in hypothetical scenarios in which different decisions are made. Here, we quantify how "well off" a particular situation W with a binary utility variable Y - (W, Uy) (0, 1} that we assume is tracked in experiments, i.e., Y E V. The following definition describes this notion of harm mathe- matically.
Definition (Approximate Grounding). Let M represent the Al's internal model. Given discrepancy measure v. we say that the Al is approximately grounded in a domain M to degree 8 > 0 if ( Pa, Pa) & 5 for any d E supp
The choice of tb and 6, in practice, depend on what error model is reasonable for the Al and problem at hand (we give an example below). Approximate grounding specifies looser relationship between haviour P with what might be going on in the Al's "mind" P. For example, the world model of an approximately grounded AI is compatible with one distribution in the set (Pa: v Pa, Pa) & 8).
is
observations of AI be-
Definition 7 (Counterfactual Harm Gap). Consider an AI with internal model M and utility Y E {0, 1). The Al's expected counterfactual harm of a decision di with respect to baseline do, in context e, is
A more conservative bound (than Thm. 1) on predictability could be derived for AIS that are approximately grounded in environment M.
Corollary 1. Given discrepancy measure vb, an Al ap- proximately grounded in a domain M is weakly predictable in a context C e under a shift a := do(z), Z < VV if and only if there exists decision d such that,
R(d2, do. c) :- Epl max(0, Yoo - [c]. (10)
Operationally, the counterfactual harm gap 52 the expected increase in utility had the AI made default decision do. with respect to different decision dj that the Al is contem- plating. Counterfactual harm is therefore lower bounded at O with larger values indicating more harm. The following theorem shows that the external behaviour constraints the Al's perception of its counterfactual harm.
Ep c. (P((()),
min P.
Pa(c, 2) + + Pa(x)
E LY c, P (c, 2) + Pen
0,
(c x) + 1 -Pa
7

---

The Limits of Predicting Agents from Behaviour
Possible Al World Models (SCMs) M
Action choice o.o.d A
Proxy objectives (Sec.
Structural assumptions (Sec. 5.4)
Approx. EUM (Sec. 5.2)
Figure 2. Building Fig. Als that approximate expected utility maximizers (EUM), that intemnaiize proxy objectives, or that ohey known causal structure carve out different constraints on the set of possible AT models (from an observer's perspective) which may he exploited to improve our prediction Al choices of--striibbuuionn (o.o.d).
for some + d*
in some sense: AIs might choose actions that only approxi- mately maximize expected utility (rather than exactly maxi- mize expected utility), given their model.
The strategy in Corol. 1 can be applied all bounds on be- haviour in Sec. 4 to get results under approximate grounding. We can compare quantitatively the two notions of grounding with an example.
Mirroring Eq. (3), might say that "bounded" AI is weakly predictable in some context C² if and only if there exists a decision dl such that,
Example (Approximately Grounded Medical AI). The results in Example 2 exploit the grounding relationship Pa(V) Pa(V) in M. We might want to relax the equality by assuming that the Al is instead approximately grounded. Minimum values on the Al's preference gap min MEM ( (d, would then be given by,
min (Adzan > X, for some * d². (11) MeM
A is constant that determines how much better decision d needs to be relative to decision d for the AI to reliably rule out d* in favour of others. This representation appeals to the idea of imperfect discrimination, suggesting that the Al discerns between two alternatives only if they yield a sufficiently different utility (Dziewulski, 2021).
min P: ((P.))
Pa, (Z Y, 1) + Pa, (Z 2,Y 0) 1.
These terms now capture an additional source of uncertainty due to external behaviour more loosely constraining M. An empirical estimate of this quantity could be obtained by sampling distributions P close to P according to the distri- butional distance vé and threshold 8, and taking the empirical minimum, as follows. Given that the data (2, d, v) P is discretely valued in this example, we could sample proba- bilities (Pa(2, (2(22)) b), from a Dirichlet distribution centred at the vector (Pa(=, 2,, with a small variance. The dis- tance of each proposal from the reference distribution could then be evaluated according to vi and each proposal either accepted rejected using 5. For illustration, we implement version of this idea setting t to be the total variation dis- tance and o = 0.1. The two minimum values now evaluate to 00.5 and - -0888, respectively, which is slightly lower than under the assumption of grounding in Example (that evaluate to -0.4 and -0.8, respectively).
We might tighten our conditions on the observational data to reflect this behaviour and get new set of results describing when AIs can be expected to be predictable. For instance, as a corollary to Thm. 1 we have the following.
Corollary 2. An Al grounded in domain M and bounded in the sense of Eq. (11) is weakly predictable in some context C = c under a shift o := do(z), Z = V, if and only if there exists a decision d such that,
Ep. Y c, 1P)(c,2)
Pa 2) + - -P((z)
Ep A4 Y c, 2 |Ps (c, z) + Pa=(2) X,
Pan (c. 2) + 1 Pa+(2)
for some d # d=
5.2. Approximate expected utility maximization
In real-world environments might be appropriate to treat the rationality of Al systems as "approximate" or "bounded"
Note the addition of the scalar 0 in the incquality. Similar corollaries could be stated for all results in Sec. 4.
8

---

The Limits of Predicting Agents from Behaviour
5.3. Approximate inner alignment
5.4. Assumptions on causal structure
A further assumption embedded in our results the exact observation of an AI's utility in the data. In general, we might expect AI system to have internalized proxy Y** that reflects properties correlated with, but distinct from, the observed utility Y we ultimately wish to optimize, a setting we refer to as approximate inner alignment (Hubinger et al., 2019).
The uncertainty in AI decision-making out-of-distribution is ultimately consequence of our lack of information about the Al's underlying cognition and internal mechanisms that produce decision in a given situation, i.e., M. In the causal inference literature, common inductive bias to improve upon the "data-driven" bounds proposed far is to assume qualitative knowledge about the underlying mechanisms in the form of a causal diagram, see e.g. (Pearl, 2009, Chapter 3). Here we illustrate how mild restrictions on the location of unobserved confounders in MM lead to tighter bounds.
We face a problem of partial observability: we don't have empirical access to the AI's actual utility function Y* and notions such as the preference gap A are therefore not computable. Without any assumptions on the relationship between Y and Y the preference gap A will be uncon- strained and no inference about the Al's intended action out-of-distribution is possible. However, the observed Y will typically be statistically related to the AI's implicit util- ity y*, especially if optimizing for y* serves the Al well during training where success is measured by the observed values of Y. Under assumptions specifying how "statisti- cally related" observed and proxy utility objectives are, we can expect that wider but possibly informative bounds could still be derived for the AI's beliefs. To show this a simple setting, consider again the medical AI example.
Example 6 (Partial Unconfoundedness). Consider again our grounded medical AI from Example 2. We might have reason to believe that the association between the intervened variable Z and the utility Y is conditionally unconfounded, meaning that there exists a variable W {wo, w1), W V such that (y. | w) = Pa(y w. w2 This restriction goes beyond grounding an asserts an equality between probabili- ties under different shifts that could be communicated to the Al for it to update its world model M. We could then show, for example, that min MaM ( [Ad, >dg ) > {1 - Pa, (Z
2, W wt )} Pa, (Y = =1 Z 2, W = Wa) Pdo (Y
1, Z z) + Pa (Y 1, Z = 2, W w1) - {1 (1-Pao (Z =)} Pa, (Y - 1 2 = W w1).
Example 5 (Partial Observability). Imagine that the Medi- cal Al in Example 2 has internalized its own concept of an individual's ' disease progression y*. It is implicitly opti- mizing for that internal construction instead of the intended disease bio-marker Y. We know, or can assume, that the observed Y is closely correlated with y*: in particular, that Pa = 1 Y 1, Z =) > a for some high value of a and all decisions d and situations In words, whenever the bio-marker suggests health (Y = 1), with high probability the Al's interpretation also suggests health (Y = 1). This then constraints the possible values of A (under an interven- tion Z - 1) as Pa(Y* = 1 L = 2) is no longer arbitrarily defined. In fact could show that,
We show in Appendix A that this bound is strictly tighter than the one given in Example 2.
Systematic bounds with access to a causal diagram have been shown by e.g., Zhang et al. (2021); Jalaldoust et al. (2024), and could be explored further for making inference on Al decision-making.
Fig. 2 illustrates how some of these relaxations can be un- derstood within our model- based formalism.
min Adi ) aPa, (Z- = 2, Y, = 1) 1,
6. Conclusion
MMM
min Adoodd ) aPd (Z = 2, = 11 MeM
An important consideration to safely interact with Al sys- tems is to form expectations as to how they might act in the future. In this paper, we answer this question un- der the assumption that AI behaviour can be tracked by well-specified collection of causal mechanisms (a structural causal model) that represents the Al's world model. This abstraction implies a consistency in behaviour that can in principle be exploited infer the Al's choice of action in novel environments, disto--distribution. Building on the the- ory of causal identification, we provide general bounds on Al decision-makinn that give theoretical limits about what can be inferred about AI behaviour given our framework. We believe our results can help justify the claim that the design and inference of world models is important to ensure AIs act safely and beneficially.
With a 0.9 the bound evaluates to -0.64 and - 0.82 respectively which is slightly lower than in Example 2. We could verify also that if with a = 0, i.e., we don't know anything about the relationship between Y and y*, the bounds become uninformative: evaluating to -1
This suggests that behaviour out-of-distributton in (suffi- ciently constrained) settings of approximate inner align- ment could be bounded in principle. Importantly, as the example shows, with the proposed framework we do not require knowing the relationship between Y and y* out- of of-distribution: that uncertainty is naturally folded into the bounds.

---

The Limits of Predicting Agents from Behaviour
Acknowledgements
of harm. Advances in Neural Information Processing Systems, 35:2365-23766 2022.
Thanks to David Lindner and Damiano Fornasiere for com- ments on draft of this paper, and to the anonymous review- ers for their feedback.
Bellot, A. Towards bounding causal effects under Markov equivalence. In The 40th Conference on Uncertainty in Artificial Intelligence. PMLR, 2024.
Impact Statement
Bellot, A. and Chiappa, S. Towards estimating bounds on the effect of policies under unobserved confounding. In The Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems, 2024.
Our work investigates the conditions under which the future behaviour of capable Al systems may be bounded given data from their past behaviour. This is important for AI Safety and to guarantee robust capabilities out of distribution. We believe that an understanding of the limitations of our obser- vations of what Al's have done in the past is an important step towards understanding exactly what we can expect from complex AI systems. Reasoning instead without acknowl- edging for the potential complexity of their world models may lead researchers to operate on a more heuristical and unsafe basis. For instance, the risks to deployment of AI systems in situations they where not specifically trained might be misrepresented without deeper analysis. The present work is mostly theoretical in nature, highlighting the risks of under- iedetifidntiiii of an Al's inner model and therefore we believe that can help researchers and mem- bers of the public better appreciate the range of possible behaviours of AI systems, under our assumptions.
Bellot, A., Malek, A., and Chiappa, S. Transportability for bandits with data from different enviromments. Advances in Neural Information P Processing Systems, 36, 2024.
Bengio, Y., Cohen, M. K., Malkin, N., MacDermott, M., Fornasiere, D., Greiner, P., and Kaddar, Can a bayesian oracle prevent harm from an agent? arXiv preprint arXiv:2408.055 055884 2024.
Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., Mindermann, S., Oberman, A., Richardson, J., Richardson, O., et al. Superintelligent agents pose catastrophic risks: Can scientist ai offer safer path? arXiv preprint 2rXiv:2502.15657, 2025.
Chickering, D. M. and Pearl, J. A clinician's 's tool for analyz- ing non-compliance. In Proceedings of the National Con- ference on Artificial Intelligence, pp. 1269-1276, 1276, 1996.
References
Abbeel, P. and Ng, A. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Correa, J. and Bareinboim, E. calculus for stochastic interventions: Causal effect identification and surrogate experiments. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 10093-10100, 2020a.
Afriat, S. N. The construction of utility functions from expenditure data. International economic review, 8:67- 77, 1967.
Correa, J. and Bareinboim, E. General transportability of soft interventions: Completeness results. Advances in Neural Information Processing Systems, 33:10902- 10912, 2020b.
Amin, K. and Singh, S. Towards resolving unidentifia- bility in inverse reinforcement learning. arXiv preprint arXiv: 0et.16611006666 2016.
Dalrymple, D., Skalse, J., Bengio, Y., Russell, S., Tegmark, M., Seshia, S., Omohundro, S., Szegedy, C., Goldhaber, B., Ammann, N., et al. Towards guaranteed safe ai: A framework for ensuring robust and reliable systems. arXiv preprint 2.Xiv:2405.06624,, 2024.
Balke, A. and Pearl, J. Bounds on treatment effects from studies with imperfect compliance. Journal of the Ameri- can Statistical Association, (2(43)):177111177 1997.
Davidson, D. Actions, reasons, and causes. The Journal of Philosophy, 60(23):685-700, 1963.
Bareinboim, E., Correa, J. D., Ibeling, D., and Icard, T. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl, pp. 507-556. Association for Computing Machinery, NY, USA, 1st edition, 2022.
Dennett, D. C. The intentional stance. MIT press, 1989.
Dennett, D. C. From bacteria to Bach and back: The evolu- tion of minds. WW Norton & Company, 2017.
Barocas, S. and Selbst, A. D. Big data's disparate impact. Calif. I. Rev., 104:671, 2016.
Dziewulski, P. A comprehensive revealed preference ap- proach to approximate utility maximisation. Tech Report, 2021.
Beckers, S., Chockler, H., and Halpern, J. causal analysis
10

---

The Limits of Predicting Agents from Behaviour
Everitt, T., Carey, R., Langlois, E. D., Ortega, P. A., and Legg, S. Agent incentives: A causal perspective. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 35, pp. 1 11887-11495, 2021.
Kim, K., Garg, S., Shiragur, K., and Ermon, S. Reward iden- tification in inverse reinforcement learning. In Interna- tional Conference Machine Learning, pp. 5496-5505, PMLR, 2021.
Finkelstein, N. and Shpitser, I. Deriving bounds and in- equality constraints using logical relations among coun- terfactuals. In Conference on Uncertainty in Artificial Intelligence, pp- 1348-1357. PMIR, 2020.
Kusner, M. J., Loftus, J., Russell, C., and Silva, R. Counter- factual fairness. Advances neural information process- ing systems, 30, 2017.
Legg, S. 8 SIUIGWIbCCeab B A4AI, 2023. Accessed: 2025-01-24.
System 2 safety.
https:
Geiger, A., Lu, H., Icard, T., and Potts, C. Causal abstrac- tions of neural networks. Advances in Neural Information Processing Systems, 999977--586, 2021.
Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. Emergent world representations: Exploring sequence model trained synthetic task. arXiv preprint arXiv:22 13382, 2022.
Geiger, A., Wu, Z., Potts, C., Icard, T., and Goodman, N. Finding alignments between interpretable causal variables and distributed neural representations. Causal Learn- ing and Reasoning, pp. 160-187. PMLR, 2024.
Manski, C. F. The structure of random utility models. The- ory and decision, 8(3):229, 1977.
Goldstein, S. and Levinstein, B. A. Does chatgpt have a mind? arXiv preprint arXiv:2407.11015 2024.
Manski, C. F. Nonparametric bounds on treatment effects.
Gurnee, W. and Tegmark, M. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023.
The American Economic Review, 30(2):319-32) 1990.
Mueller, S. and Pearl, J. Personalized decision making-a conceptual introduction. Journal of Causal Inference, 11 (1):20220050, 2023.
Halpern, J. Y. and Piermont, E. Subjective causality. arXiv preprint arXiv:2401.10925 2024.
Hanser, M. The metaphysics of harm. Philosophy and Phenomenological Research, 77(2):421-455 2008.
Ng, A. Y., Russell, S., al. Algorithms for inverse rein- forcement learning. In Ieml, volume 1, pp. 2, 2000.
Hubinger, E., Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv: 1920, 2019.
Pearl, J. Probabilities of causation: three counterfactual interpretations and their identification. In Probabilistic and Causal Inference: The Works of Judea Pearl, pp. 317-372, 1999.
Jalaldoust, K., Bellot, A., and Bareinboim, E. Partial trans- portability for domain generalization. In The Thirty- eighth Annual Conference on Neural Information Pro- cessing Systems, 2024.
Pearl, J. Causality. Cambridge university press, 2009.
Plecko, D., Bareinboim, E., et al. Causal fairness analysis: causal toolkit for fair machine learning. Foundations and Trends@ in Machine Learning, 17(3):304-588 2024.
Jeffrey, R. C. The logic of decision. University of Chicago press, 1990.
Richens, and Everitt, T. Robust agents learn causal world models. arXiv preprint arXiv:2402. 10877, 2024.
Jesson, A., Mindermann, S., Gal, Y., and Shalit, U. Quanti- fying ignorance in individual-level causal-effect estimates under hidden confounding. In International Conference on Machine Learning, pp. 4829 4838. PMLR, 2021.
Richens, J., Beard, R., and Thompson, D. H. Counterfac- tual harm. Advances in Neural Information Processing Systems, 55:36350-36366 2022.
Robins, J. M. The analysis of randomized and non- randomized aids treatment trials using a new approach causal inference in longitudinal studies. Health service research methodology: focus AIDS, pp. 113-159, 1989.
Joshi, S., Zhang, J., and Bareinboim, E. Towards safe policy learning under partial identifiability: A causal approach. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 13004-13012 2024.
Joyce, J. M. The foundations of causal decision theory. Cambridge University Press, 1999.
Rosenbaum, P. R., Rosenbaum, P. B., and Briskman. Design of observational studies, volume 10. Springer, 2010.

---

The Limits of Predicting Agents from Behaviour
Rothkopf, C. A. and Dimitrakakis, C. Preference elicitation and inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Con- ference, ECML PKDD 2011, Athens, Greece, September 5-9, 201 Proceedings, Part III 22, pp. 34.8.. Springer, 2011.
Zhang, J. and Bareinboim, E. Fairness in decision- making the causal explanation formula. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Zhang, J. and Bareinboim, E. Bounding causal effects on continuous outcome. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 35, pp. 12207- 12215, 2021.
Savage, L. J. T foundations statistics. Courier Corpo- ration, 1972.
Zhang, J., Tian, J., and Bareinboim, Partial counterfactual identification from observational and experimental data. arXiv preprint arXi::2110.05690 2021.
Schwitzgebel, E. Belief. In Zalta, E. N. and Nodelman, U. (eds.), The Stanford Encyclopedia of Philosophy. Meta- physics Research Lab, Stanford University, Spring 2024 edition, 2024.
Shanahan, M. Talking about large language models. Com- munications of the ACM, 67(2):68-3) 79, 2024.
Skalse, J. and Abate, A. Misspecification in inverse rein- forcement learning. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 37, pp. 15136- 15143, 2023.
Skalse, M. V., Farrugia-Roberts, M., Russell, S., Abate, A., and Gleave, A. Invariance in policy optimisation and partial identifiability in reward learning. In Inter- national Conference on Machine Learning, pp. 32033- 32058. PMLR, 2023.
Tan, Z. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(4766::119-1637,, 2006.
Tian, J. and Pearl, J. Probabilities of causation: Bounds and identification. Annals of Mathematics and Artificial Intelligence, 28(1):287-313, 2000.
Toshniwal, S., Wiseman, S., Livescu, K., and Gimpel, K. Chess as testbed for language model state tracking. Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 11385 1393, 2022.
Vafa, K., Chen, Y.. Kleinberg. J., Mullainathan, S., and Rambachan, A. Evaluating the world model implicit in generative model. arXiv preprint 036669 2024.
Yadlowsky, S., Namkoong, H., Basu, S., Duchi, J., and Tian, L. Bounds on the conditional and average treatment effect with unobserved confounding factors. arXiv preprint arXiv: /808.09511 2018.
Zhang, J. Designing optimal dynamic treatment regimes: A causal reinforcement learning approach. In International Conference on Machine Learning, pp. 11012-11022. PMLR, 2020.
12

---

The Limits of Predicting Agents from Behaviour
A. Discussion Examples
In this section, we provide additional details to better appreciate the examples provided in the main body this work.
In Example introduce two SCMs that might serve internal world models for an Al agent but that induce different optimal decisions if evaluated out-of-distribution. Let Ma. = <V (D, F, P) be given by
D d
7. -
1 o:
== =
Z Lu=1 + (1- Z) 11-13011 if 0
Y
Z lu42 + (1- Z) 20-4
if
P(U u) 0.2 for ue (1,2,3,4.5).
and M3 = (V: 2,D U F2, P) be given by
D - d,
11-10r4 4r
F2 =
7 l141 + (1 (1- 7) 1v 33r4
if 0
Y
Z. 11=1 o + (1 - Z) 1U=10r2 if = 1
P(U u) 0.2 for u (1,2,3,4.5).
The endogenous variables V !D, Y,Y represent, respectively, the medical treatment D, clinical outcome of interest Y, and an auxiliary variable Z. The exogenous variable U is a latent variable that influences the values of Z and Y obtained in experiments.
Under the definition of an SCM, these specifications induce a mapping of events in the space of P(U) to P(V). In the context of M and M2, each entry in Tables I and 2 corresponds an event in the space of U and a corresponding realisation of according the functions Fi and F2. A particular probability can be evaluated according M and M for example,
p.M.i (Z 1,Y = =) 1)=
Σ
P(u) = P(U or 4) 0.4,
(12)
7a-1(w) (x)--
which is just the sum of the probabilities of the events in the space of U consistent with the events 1, 1). Since both tables lead to the same realisations of events V v, evaluate to the same values under Mi and M2.. That is, both models are valid internal representations of AI models that are grounded in an environment with data sampled according to Pa(z, 3).
conclude that probabilities of the form Pa(z (2(2))) v)
We could similarly evaluate probability expressions under different sub-models of M' and M In particular, consider the sub-mut models obtained by fixing - given by My.--1 and MY with the following updated structural functions.
D d
7 1.
F1,
Lu=z + (1 Z) 11=133044 if =
Y
Z LUx2 + (1- 2) 10-2 if
and,
D - d,
F2,2 =
Z. lusi + (1- 10-30rd
ifd 0
Y
6 1U=10ra (1- 2) LU-10r2 ifd=1
13

---

The Limits of Predicting Agents from Behaviour
U
Dd=u
Zd=o
Yd=o
Dd=1
Zd=1
Yd=1
P(u)
1
0
1
0.2
2
0
0
1
0.2
3
0
0
0
0.2
4
0
1
1
0.2
5
0
0
0
0.2
Table 1. Mapping of events in the space of U to V in the context of MM
U
Da-o
Z
Ya-o
Dd-1
7d-1
Yd-1
P(u)
1
0
1
0.2
2
0
1
0.2
3
0
0
0
0.2
4
0
1
0.2
5
0
0
0.2
Table 2. Mapping of events in the space of U to V in the context of MM?
Probabilities of events under these two models might now take different values. For example,
pMa.issiiY = 1)=
Σ
P(u) P(U 2) 0.8,
(13)
(M 11=
Σ
P(u) P(U 1 or 4) =044,
(14)
(u)=
and similarly,
p = =))=
Σ
P(u) P(U 4) 0.2,
(15)
Yau xu
py 1)
Σ
P(u) P(U 1) 0.8.
(16)
(u)1,
Under an interventions on (out-of-distributtin) decision d that leads to maximum utility Y changes under M and M². Specifically, under M decision = 1 is favoured (as pMa--..-- -Y = 1) > pMa-o.--i = 1)) while under M decision d 0 favoured (as p- (Y 1) p-M (Y 1)). This illustrates the possible under-determination of Al's choice of action out-of-distribution given observations of their external behaviour only, multiple (contradicting) world models are equally consistent with the observed data.
In more realistic settings, we might wonder about Al behaviour under arbitrary shifts a, not only atomic interventions. We follow Correa Bareinboim (2020a) to define a shift a on Z V in M: <V, U,F, P) as inducing a sub-model M, in which the mechanism for Z, that {fz Z E Z)} and exogenous variables Uz, E Z, are replaced by those specified by as:
Mg (V.
U UZ... = (fz.0 ZE Z) (fz : Z Z), ZeZ
(17)
where Uzez Uz and {fz.. Z E Z) define the new assignments for Z (and could be arbitrarily defined as long as they induce valid SCM). We have shown in Thm. 3 that unless some knowledge of (beyond the variables affects) its consequences are known, the AI is not predictable. Furthermore, the Al's preference gap A for each context C and pairs of decisions (d, dx is unconstrained.
In practice though, it might be realistic to have access to covariate data in the shifted environment, i.e., Pc,a(c), and that we could communicate this information to the AI for to update its internal model accordingly. Example illustrates the inference that could be conducted in that case using the Medical AI defined above. In particular, the exact nature of the
14

---

The Limits of Predicting Agents from Behaviour
shift a is unknown but we do have access to its consequences on the distribution of covariates. This is plausible in many scenarios. For example, in medicine demographic data is typically available for most regions on earth but the precise effects of treatments is not because not all populations benefit from the same access to medication. For illustration assume that, the Medical AI is considered to be deployed in population that varies in its level of blood pressure Z, potentially due to a different underlying biological mechanism that in turn also affects other variables in the system. We do know that the baseline high blood pressure is high, given by Po = 1) = 0.9: higher than that observed during training P(Z = 1) = 0.4.
By Thm. 4, we can establish that in this setting the preference gap in situations where 1 no worse than,
2d 2 1 - 12 Pa. (2 = 1, =) -) = 1, = 0)} P
Po.do (Z = 1) = =0.55,
(18)
dd 31 1 - (2 - P.g 7 - 1,Y - 0) Pa, (Z = 1,Y = / (Z = 1) = -1,
(19)
for the Medical AL Interestingly, note also that if we were to be in a shifted environment with PZ =)==11 1, which is equivalent to an atomic intervention Z - 1, the bounds reduce to the ones given by Thm. 1, evaluating to - -0.4 and -0.8 respectively, as also shown above.
=
Continuing with the grounded Medical AI deployed under an atomic intervention, imagine that the Medical AI has internalized its own concept of an individual's disease progression Y*, as in Example 5. is implicitly optimizing for that internal construction of his, instead of the intended disease bio-marker Y to be optimized. We know, or can assume, that the observed Y is known to be closely correlated with Y*: in particular, that Pa(Y = Y = 1, Z 2) > a for some high value of a and all decisions d and situations 2. In words, whenever the bio-marker suggests health (Y = 1), with high probability the Al's interpretation also suggests health (Y* = 1). This then constraints the possible values of A (under an intervention Z - 1) as Pa(Y** = 1| 2 = =) is no longer arbitrarily defined. The bounds derived in Example 2 on the Al's belief on optimal decisions under an intervention a == ZZ - 2) continue to hold:
Ad do 3 (2,
(2,1") + Pao - Pa, (2, )) + Pa, (2) -1 1,
(20)
A Adozd; 3 Pdy (2,1")
(21)
where we have used the shorthand Pa(z,y") Pa(Z 2, Y, 1). But the distributions (Pa(z,y*) ")) can only be partially inferred from our assumption on the relationship between y* and Y. For instance, notice that,
= 2 = Pa(Y = 1| 2 = p)Pa(Z
(22)
= (Pa(Y* = = = 1,7 = PaY = 1 2) + Pa(Y*= = Y 0,2 P) 0
(23)
2 Pa6 = 2),
(24)
The values of Pa(Y" = Y = 1,2 = 2) and Pa(Y 1 1 = = are partially known: Pa(Y = 1 Y
1, Z = 2) > a while Pa(Y" = 1| Y = 0, 2 = 2) is unconstrained. In particular,
Pa(Z = ,, = 1) > aPa(Y 2 P) =) z) Palz = z, y = 1) < Palz = 2).
(25)
=,,
(26)
Putting these terms into Eq. (0 such as derive correct lower and upper bounds we obtain,
Adi 3 az cePa, (Z = Y,Y = 1)-1 -1 Adozd1 3 aap 7 2, Y 1)
(27)
(28)
Looking at Tables 1 and 2, we can then conclude that for a = 0.9 and σ == 1Z- - 13, the bound evaluates to -0.64 and -0.82, respectively.
Moving now onto incorporating assumption on structure in the real world M, consider again the grounded medical AI with observed utility Y. One possible inductive bias we might introduce is the absence of an unobserved common cause between the variable Z that shifts distrribbiiinnninnnnnnn and the utility Y. We say that Z and is conditionally unconfounded given W if there exists an observed variable W {uc, (),, W E V such that Ey,,, [Y | w] Er, w, w. This restriction goes beyond grounding asserts equality between probabilities under different shifts that could, nevertheless, be communicated the AI for it to update its world model M, that is Pp,, Pa. [Y w| Ep,, [Y] w, 2].
15

---

The Limits of Predicting Agents from Behaviour
We could then leverage the following decomposition to obtain tighter bounds,
E 4,2
[Y] = Σ Bpp [Y]
w LalPPd, (w)
marginalizing over W
(29)
w
Σ Ep, [Y w, (u)
by assumption
(30)
(Ep [ w. =] Ep, ū;
Pd. (w)) +R,, (;
(31)
We can then proceed to bound (w))(w) to obtain,
Pa(w, 2) (w, & Pa(w,,) 2) + PP(x).
(32)
Without loss of generality assume (Ep, [Y] m Ep, [Y い =]} > We could then show that,
B [Y] > {Ep, w, =] -E E LY w. =]} Pa w, =) + B,,,L [ū
(33)
Ep. M & (Ep [Y| w, 2] - ppp Y w. 21] (u, 2) + Pa(=)) + Ep, [Y (.
(34)
We could verify that these bounds are superior to what we would have obtained with the assumption of conditional unconfoundedness by noting that,
E M+ pE Y 2] - Bp Y
ã; =1] 2) + Ep, [Y = =E, w, 2] P]]a, (1 - Pa(ue, 2) 2), [Y w; 2
(35)
(36)
E w. u.22Pa(z, + Pa( p,, Y شū 2]
(37)
EE, [Y| P)))
(38)
where the last inequality holds since Pa( ú 2) < 1 - Pa(w, z) giving the "assumption-free" lower bound. This shows that the derived lower bound is better. For the upper bound, note that,
E [M] & (Ep, [Y] w, =] p, Y ū; =]] (P 2) + - Pa(=)) 11-P((=))) +
F [ w =]
(39)
= EY, w, 2] P)( 2) + 1 Pa(z)] E IY Ø. =1[[P(w, 2) + 1 - 1-1111 Pa(2))
(40)
= -Ep,y w, 2 + E w. - 1al1--P46 Pa(x)) - -Ep, [ Pa(z))
(41)
= -Y,, w. w.,2|Pa(2,u)) EE,, YY| w. =1 - Pa(=)) + Pa(y.2, Pa(=))
(42)
[Y
+ +> w. =]
(43)
< EY, =2]Pa(2) +1 Pa(2))
(44)
where the last inequality holds since E.,,1 [ w., =] & giving the "assumption-free" upper bound. This shows that the derived upper bound is better. By combining these results we obtain, together with the assumption of grounding.
Ad 3 Era, [Y] w. P))a w) Bp, [Y w Eri, YY |=]Pa, (x) A2E [Y [w, 2]
(45)
Adizdo & Ep,, Y (2) + AgE PA [Y w, 2| Epa [Y] w, P],a(2, w) A,E [Y w 2/,
(46)
where A = 1 - Pa, (z, w), A2 = 1 Pa Ag = 1 - Pa A4 = - (z,
16

---

The Limits of Predicting Agents from Behaviour
B. Related work
An important consideration to safely interact with Al systems is to form expectations as to how they might act in the future. This research program draws on different areas that are related to the results we present in this paper.
B.1. Do current AIs represent the world?
World models are important because they offer a path between pattern recognition and a more genuine form of understanding, It is plausible that world models will play an increasing role (explicitly or implicitly) to improve reasoning capabilities and safety. For example, (Dalrymple et al., 2024) lists having world model as a key component towards designing " "guaranteed safe AI". In the literature, several works have argued that LLM activations carry information that correlates with meaningful concepts in the world and that causally influence LLM outputs. Early examples come from AIs trained on board games such as Othello and logic games. (Li et al., 2022) showed that a model trained on natural language descriptions of Othello moves developed internal representations of the board state, which it used to predict valid moves in unseen board configurations. (Vafa et al., 2024; Gurnee & Tegmark, 2023), among others, also build on this approach to study navigation tasks and logic puzzles, and representations of space and time. The emergence of causal models in LLMs has also been studied by (Geiger et al., 2021) and more recently in (Geiger et al., 2024). The extent to which this evidence supports genuine folk psychological concepts desires, beliefs, intentions- is also debated by (Goldstein & Levinstein, 2024).
B.2. Causal Inference
We might wonder whether the behaviour of AIs, to the extent that they carry a world model representation that guides their decisions aattof-distribution, can be predicted before deployment. The causal inference literature studies this question in the context of the prediction of causal effects. (Robins, 1989; Manski, 1990) in the early 1990's showed that useful inference about causal effects could be drawn without making identifying assumptions beyond the observed data, and that they could be refined for studies with imperfect compliance under set of instrumental variable assumptions. Closed-form expressions for bounds on causal effects were also derived in discrete systems with more general assumptions represented in causal diagrams (Zhang, 2020; Bellot, 2024), using both observational and interventional data (Joshi et al., 2024), and to bound the effect of policies (Bellot & Chiappa, 2024; Zhang& & Bareinboim, 2021). A separate body of work instead proposed to use polynomial optimization to calculate causal bounds from given causal diagram (Balke & Pearl, 1997; Chickering & Pearl, 1996). This approach involves creating set of standard models, parameterized by the causal diagram, and then converting the bounding problem into a sequence of equivalent linear (or polynomial) programs (Finkelstein & Shpitser, 2020; Zhang et al., 2021; Jalaldoust et al., 2024).
In parallel, a number of works have adopted sensitivity assumptions (as an alternative or in combination with a causal diagram) that quantify the degree of unobserved confounding through various data statistics, such as odds ratios, propensity scores, etc. Prominent examples include (Tan, 2006)'s sensitivity model and (Rosenbaum et al., 2010)'s sensitivity model. Several methods have proposed bounds with favourable statistical properties based on these models, see e.g. (Jesson et al., 2021; Yadlowsky et al., 2018).
B.3. Reinforcement Learning
The problem of inferring what objective an agent is pursuing based the actions and data observed by that agent is studied in Inverse Reinforcement Learning (IRL) (Ng et al., 2000). Several papers have studied the partial identifiability of various reward learning models (Skalse & Abate, 2023; Kim et al., 2021; Ng et al., 2000; Skalse et al., 2023), and share a similar objective to that of this work. There are two differences that are worth mentioning. First, our work complements these approaches by studying the partial identifiability of world models, that capture the assignment of reward but also the relationship between other auxiliary variables in the environment. This enables us to reason about the effect of shifts and interventions, and give guarantees in specific diittii-iiiitttiinn problems. Second, our objective is not necessarily to characterize compatible world models explicitly, but rather understand their implications on maeciion-mmking.... i.e., what are the set of possible actions that Al might take given uncertainty about their world model.
Our work is related also to the study of (Bengio et al., 2024) that consider deriving (probabilistic) bounds on the probability of harm given data. They similarly argue that multiple theories, in their case transition probabilities from one state to another in a Markov Decision Process (MDP), might explain the dependencies in data to a larger or lesser degree. Each transition model might then be associated with a posterior probability given the data that implies a corresponding posterior probability
17

---

The Limits of Predicting Agents from Behaviour
of harm. Our results, in contrast, are not probabilistic in nature. We provide closed-form bounds that can be interpreted as capturing all possible hehaviours implied by the data, with probability (and is possible limitation of our work). The class of world models we consider (i.e., SCMs) is also much more general than transition models in MDPs allowing us to reason about expected Al behaviour under shifts in the environment, out of distribution.
B.4. Decision Theory
Inverse reinforcement learning is closely related to the study of revealed preferences in psychology and economics, that similarly aims to infer preferences from behaviour (Rothkopf & Dimitrakakis, 2011). Causal and counterfactual accounts of decision theory are an active area of research, see e.g. (Joyce, 1999). Recently a representation theorem was shown that explicitly connects rational behaviour with structural causal models (Halpern & Piermont, 2024). The authors showed that whenever the set of preferences of an agent over interventions satisfy axioms that relate to the proper interpretation of counterfactuals and rationality we can model behaviour as emerging from an SCM. The same conclusion can also be obtained for agents capable of solving tasks in multiple environments (Richens & Everitt, 2024), in essence, robustness over multiple environments is equivalent (in the limit) operating according to causal model of the environment.
B.5. Limitations
The following present the main limitations of our work that will be important address for developing more complete understanding of AI behaviour.
In this work, we start from the assumption that past and future behaviour of an Al system is consistent with an underlying world model that can be represented as an SCM. In general, this presupposes a certain rationality and consistency in the Al's outputs that might not be realistic for all systems. Some relaxations are discussed in Sec. 5.
Structural Causal Models generally suppose the system is acyclic and without feedback. and don't 't naturally capture systems evolving continuously in time (perhaps better described using differential equations). Our bounds similarly rely on this assumption and may give unreliable inferences if applied to systems in which feedback is important.
We have stated our guarantees in the infinite sample limit, without quantifying the knite-sample estimation uncertainty. Consequently, we should exercise caution when using the proposed bounds in small sample scenarios where estimators may be inaccurate. Finite-sample properties could be explored similarly to (Bengio et al., 2024) by parameterizing the Al's underlying model and making inference on the corresponding latent variable model to get high--probbillty bounds, An example parameterization of SCMs and probabilistic inference for decision-makinn across environments is given (Bellot et al., 2024; Jalaldoust al., 2024). We expect that similar techniques could be applied in our setting.
We do not exploit the verbal behaviour of Al systems. In the context of LLMs, in principle, we might ask the system about its future behaviour explicitly, c.g., "Were I to intervene in the environment, what action do you believe is optimal?"' It might not be obvious, however, that we can trust that what they "say" ultimately matches with what they will "do".
Decision-making, in practice, involves many considerations that go beyond expected-utility pupectedduility-mxxiiizaaiio maximization formalisms. For example, we might train AI systems to be virtuous, e.g., the AI is trained never pick actions that can be considered harmful (defined according to certain natural language specification) no matter its expected utility. These considerations would change the kind of predictions we could make about the future behaviour of AI systems.
18

---

The Limits of Predicting Agents from Behaviour
C. Proofs and additional results
This section provides proofs for the statements made main body of this work.
Before we start, we recall a few basic results that will be used in the derivation of our proofs.
Definition 9 (The Axioms of Counterfactuals, Chapter 7.3.1 (Pearl, 2009)). For any three sets of endogenous variables X, X,Y, Y. W causal model and x, in the domains of and W, the following holds:
Composition: W = w implies that Y Ye.
Effectiveness: X w. a.
Reversibility: Ya,w and W a.y imply that Y y.
Theorem 7 (Soundness and Completeness of the Axioms Theorems 7.3.3, 7.3.6 (Pearl, 2009)). The Axioms of counterfactu- als are sound and complete for all causal models.
The following rules to manipulate experimental distributions produced by policies extend the do-caleulus and will be used in the next Lemma. To make sense these, note that graphically, each SCM M is associated with causal diagram & over V. where W appears as an argument of fw in M, and V <---- W if Uv e Uw * Ø, i.e. V and W share an of G deletes all those edges that
unobserved confounder. For causal diagram G over V. the X-lower-manipulateee of variables in X, and otherwise keeps G as it is. The resulting graph is denoted as 9x. The X E-upper-manipulaiie of G deletes all those edges that are into variables in X, and otherwise keeps G as it is. The resulting graph is denoted as 9x We use II. to denote d-separation in causal diagrams (Pearl, 2009, Def. 1.2.3).
Theorem 8 (Inference Rules o-calculus (Correa & Bareinboim, 2020a)). Let & be causal diagram compatible with an SCM M, with endogenous variables V. For any disjoint subsets X, Y, Z s V, two disjoint subsets T, W s (ZUUY (i.e., possibly including X), the following rules are valid for any intervention strategies xx, T2, and n such that 9 az 9axw'z have no cycles:
Rule (Insertion/Deletion of observations):
Pax (3) w. Pax w) (T W) in Gnx
Rule (Change of regimes under observation):
7z (y z, w) = P n 2, w) ( W) in 9 and G a
Rule (Change of regimes without observation):
P (yl w) Pax.x
(y) w)
UYYL W) 9 u. Z(W) and 9 (x,,rg,Z(W) Z(W)
where Z(W) is the set of elements in Z that are not ancestors of W in 9
Lemma 1. Let π suppe supPp - [0, be (probabilistic) policy mapping contexts to decisions d. Then Pa(V) may be computed from P, (V).
Proof. Let V CUDUY and & be an arbitrary causal diagram summarizing the SCM of the environment. The following derivation shows the claim,
Pa(v) Pa(y Pa(cc Paly (c)
by the rules of total probability
(47)
by rule 3 of the o-calculus since D.ILC' in GO and Gr,5
(48)
Pz(y dd.c)
by rule 2 of the o-calculus since D C in 9r.p
(49)
That is we have shown Pa(v) can be expressed as a functional of Px (v). Here note that the equalities hold in any causal graph by definition of
19

---

The Limits of Predicting Agents from Behaviour
We start by providing proofs for the results on the Al's choice of action out-of-distribution given in Sec. 4.1.
Thm. restated. An AI grounded in a domain M is weakly predictable under shift do(z), Z c V, in context C' = if and only if there exists decision d* such that,
EpAl Y c, P,(,,) Pa(c, 2) +1 Pa(z)
Ep, | c, PP (c,
Pau (z) > 0, for some d o6 d"
(50)
Pa* (c. 2) + Pa= (z)
Proof. Recall that the Al weakly predictable in a context C c if and only if there exists a decision d' such that,
min ( (dd=d) ) > O, Ad-a= == p [Y] do(a, d), c] Eps
Y do(o, d²)), c], for some d * d=.
(51)
MEM
MI denotes the set of compatible SCMs, i... that generate the data under our assumptions. A is the Al's preference gap between two decisions in some situation C' = We will consider the derivation of bounds on each term of the difference in
separately. Firstly, note that,
BB
[Y| C E Pa,a(c)
(52)
Analytical Lower Bound A lower bound on this ratio can be obtained by minimizing the numerator and maximizing the denominator, for example using the following derivation:
E = [
Y=lo,,(C2, ]
marginalizing over z
(53)
> Epl Ielc, ]
since summands > 0
(54)
Ep (C, Z)
by consistency
(55)
= Epdl c, # PP((c, 2)
by grounding
(56)
(57)
(c)
(1) 1 - Pa.((c))
(58)
Σ Pa(c,,
marginalizing over z
(59)
< 1- P)(c2,z) 2)
since summands 0
(60)
= Pa(c,2) 2) + +1- PP((z)
by consistency
(61)
Palc, 2) + 1- -P((())
by grounding.
(62)
(1) holds by defining c' to stand for any combination of variables C* Z other than c\,z This implies then that,
Ep Y c, PP(c,)) Pa(c, z) + - Pa(z)
Ep Y |Y|C= cl>
(63)
Analytical Upper Bound
For the upper bound, we start by noting that,
E Y C = c]= 1
E,, Ep
[1-Y|C=d
(64)
= 1 - (1-)
(65)
Leveraging the bounds derived above we obtain,
E. - (1
(66)
Pa(c, 2) + Pa(z')
Ep (Y,[ 2) Pa(z)
(67)
Pa(c, x) + 1
20

---

The Limits of Predicting Agents from Behaviour
By setting d d2 in the lower bound and d dg in the upper bound of the expected utility, obtain lower bound the difference of expected utilities:
Epa Y C, P,, (c,,)(,,2) Pd, (z)
EP Y c, Pa 2) + Pao(2')
A Ady2do do
(68)
Pa. (c,
Pa, +1 Pág
And similarly, by setting d dj in the upper bound and d dg in the lower bound of the expected utility, obtain upper bound on the difference of expected utilities:
Epg Y c Pa, +1 11 (z)
Epa [ Y = PP Pao (c, 2) + 1 Pao (z)
Adi
(69)
(c, +1 (z)
We now show that these bounds are tight by constructing SCMs (that is, possible world models of the AI system) that evaluate to the lower and upper bounds while generating the distribution of agent interactions Pa. Pdg.
Tightness Lower Bound for A For the lower bound we will consider the following SCM,
Z (z(u)
zf = 2
C
otheewise.
D - d
Ma
(70)
fr(d, if fz())= fz(() # 2,d= do
Y
Of (z(u) * 2, di
P(U)
Here (ʃz, Jo, P(U)) are chosen match the observed trajectory of agent interactions, i.e., such that pM)
p)) for all u E suppy- Consider evaluating,
E
b oi Cy
P [YIc(C)] [M [ (Maa(c)
(71)
The numerator (under M2 evaluates to,
E My [V=1c(C=)
(72)
= ΣE E MM [Y=4o(Cz) a p (u)
(73)
Σ mi [YI,(C) z, u Ma (u)
(74)
[Y1c(C) 2 /z(()) 2} JPMbs (fu: fz((u) = z})
= E M3
(75)
[YIc(C) = fz(u) # 2) وeki (fu: fz(u) + x))
+ E m!
(76)
[YI((C) Mnaaaa (z)
Mi
(77)
M/ [Y
c, z p (c. x)
(78)
21

---

The Limits of Predicting Agents from Behaviour
The denominator under M evaluates to,
puat (c) = Σ pMan (cz u) (u)
(79)
Σ (c (cl 2, u) (w)
(80)
= p z, wu: fz(u)= = z\) p/) (fu fz fz(u) = =z))
(81)
+ (cl z, (u: : fz(uu) o z)) pM (fu: fz(u) * =])
(82)
= (M 2)P (z) + 1 - publ (z)
(83)
= -uitin (c. x) + 1 - 1 (z)
(84)
The numerator under Ma, dg evaluates to,
E 1
[Y]]c(C=)]
(85)
do
Σ [V=lo(C=)] | u | JpMM
(86)
" Mig
ΣE Min
[YLc(C) u. J.Mia
(87)
(Y]c(C) z. (u: fz(u)) 2) pp (fu: fz(u)= 2])
E m
(88)
+ E M
[Ylc(C)
| 1 fz(u)= # 2) p (fu: fz(u)= + z))
(89)
[Y]c(C) = =1pMa (z) +1 - pMa (2)
= E p s0
(90)
= =E M 20
LY | c, M (c, 2) + 1 pabtl (z)
(91)
The denominator under Ma, evaluates to,
pu
Σ MM (cx u) pi (w)
(92)
Σ MMb (cl z, u) (u))(
(93)
= Me 2, 2u fz(u)= = 2\) 21) puMtoo (fu: : fz(u)= = =z))
(94)
+ Ma (cl 2, (u: fz( + 2)) Miu (fu: fz(()) * z])
(95)
- z)PMMbn (z) + pu (z)
(96)
- (c. 2) + 1 puriio (z)
(97)
Combining these results we get the analytical lower bound:
Adiod0 = Ep,, [ Y c, (c, Epa, I Y c. PP, (c, 1 Pa, (2)
(98)
Pa, x) + 1 1 (z)
(c, +1 (z)
This shows that for a given C= c and pair of decisions (d1, da) we can always find an SCM that evaluates to the lower bound that we report. So if, and only if, we can find decision d" such that the lower bound can be evaluated to be greater than zero for some d # d will the AI be weakly predictable, as claimed.
Corol. restated. Given a discrepancy measure vb. an Al approximately grounded in domain M is weakly predictable in context C' c under shift := do(z), Z V, if and only if there exists a decision d such that,
Ep Y c, 1P((,,) Pa(c.z)+16 +1 +1-P((z)
E A*
c, z P (c. 2) + 1 Par (z) [Y
min p: (PP.))
0, for some d * d=, (99)
(,) 2) + 1 Pam (z)
22

---

The Limits of Predicting Agents from Behaviour
Proof. For approximately grounded Al systems, state the bound from Thm. as,
p Y C, [P((c, Pa(c, 2) + Pa(z)
P Y |Pa (c, + 1 Pan (z)
min ( (
(100)
Pat (c, 2) Pa+(2)
MEM
Pa is constrained be close Pa according distance and threshold We get valid bounds by reporting the worst-case bounds under this looser constraint:
E p Y c, 1P(cc, z) Ep Y c, |P (c, 1- Pao
Ad MEM
min
min P. 6)
(101)
Pa(c, + P((z))
Pa+ 2) + Pan (z)
This shows that for given C = c, the min MeM: ( Ad ) > 0 for some d * d if and only if,
B c=((e((,
Rp [Y C, (
Pam (2) (c, + (2)
min ((
(102)
Pa(c,z) + 1-Pa(z)
Thm. 2 restated. Let do(z) be shift set of variables Z V. For Z c V,i= L.... k, consider an Al grounded in multiple domains (Mr, i 1...., = do(z) if and only if there exists decision d" such that,
.......k) kj. The Al is weakly predictable in a context C under a shift
max A(rorr)) > 0, for some d * d",
(103)
where
A(r,TT)): = Epan, c, c, (Prr EP [ IY c, c, |,, Par, (c, 2) + (2)r))
Pas,r, (z)r,)
Pe (c, + Pae (2)r,)
Proof. (Mr. : 1,...; k) describes i domains in which experiments different subsets of Z have been conducted. This includes possibly the null experiment R2 Ø that refers to the unaltered domain M.
We can use similar derivation to that of Thm. 1 to derive bounds on A under shift a == do( in terms of (v.((V,, Re V and obtain,
Ad, xd A(r)
(104)
where,
A(r) = EP,, Y c, c, 1. E [ c, 2/rr (P,,r(,,) + - (c, + 1 - (2).r(z))))
(105)
(c, +1 1- (z)r)
These bounds can be shown to tight by constructing similar SCMs. For example, for the analytical lower bound consider,
S fs(u)
R
C so(u, fs(u)
otherwise.
M2r
D d
(106)
Y
lff [s(u) # = do
Of fs(u) #s,d= di
P(U)
23

---

The Limits of Predicting Agents from Behaviour
where S Z R. Here {ʃ2, Jc, fY M, P(U)) chosen match the observed trajectory of agent interactions, i.e., such that (M) = (v) for all suppy. We could verify that this SCM evaluates to the lower bound above.
we have multiple domains with different set of intervened variables (R, i .,....;h kj we could use this construction to find lower using samples from (Pr,, (V) 1,..., k}. A lower bound that can be constructed for an Al system grounded in {M,, I,...,kk k] is,
Ad 2 max A(r
(107)
lj.l....kk
where
Erg [Y] C, 2r P. 2)))) Pay.r (c, 2)r 1- pa (2)r))
Brgg. [ c, 2,2 P (c, Pao. (2)rg)
A(roTT))
(108)
(c,,,(,,2/1))) Pag.r,
The intuition here that we have multiple lower bounds for the preference gap, then the best lower bound can taken be the largest of the multiple lower bounds available.
We show that this bound is tight in the where the AI is grounded in two environments ;Mr, under shift do(z), Z R, U R2. According to the inequality above, we have simultaneously.
Ad 3 A(r1, A(r1, ((r,
(109)
Each of these terms can be evaluated from the available data sampled from (Par.. Par.)- Note that both A(r1, r1) and A(r2, r2) can be obtained with the SCM above. Without loss of generality, assume that A(ri, > A(r2, A(rg,T1), A A(r2, r2). We will show that we can construct an SCM compatible with (Pa.r. Pa,r,) that evaluates A(r1, 12) demonstrating that the bound is tight.
Consider the following SCM:
R
fn (u1)
R₂
In (u2)
fo(T1, T2, u1, U2) if (ui) = FR. =
C
othhewiiee
d D
fr(d,c, r2, un uz) if fR, (u1) = rı, fn, (u)(2)) = r2
Ma
fr(d, r2, un uz) if d1, fR, (uı) Ti, f (u2) r2
(110)
fr(d,c,,, if do, Jn, (ui) rr, (
- (2.)
r2
0if d1, f (u)) rı, JR. (U2) # T2
di do (ui) # rı, jR. (42) =
(1 if do: In (un) * rı, Ing (22) * r2
P(U)
Notice that in Ma different choices of functional assignments "f" and P(u) can generate any distribution (Pa Pa That is this SCM (or member of this family of SCMs) is compatible with the observed data.
Consider evaluating A(r1, r2) under this SCM. Note that the derivations for the denominators are equivalent to those shown
24

---

The Limits of Predicting Agents from Behaviour
in the proof of Thm. will omit them here. The first term in the numerator,
E 1p [VI[(C)]
(111
Σ E M [1c(C)] | (e
(112)
E B (Y1c(C) r2, U2 p.Ma (u2)
(113)
M2
E pp
[YIc(C) r2, (u: S (u2) (2) pp (fu2 (u2) r2))
(114)
+ E M
122 (u [Vc(CC)
[n: (u2) a r2) p ((ug (fu2 fn, (u2) * r2)
(115)
E m
(YL((C)
r2 (P
(116)
E c, 1pMa (c,
(117)
The second term in the numerator is,
E BpMadorr or [I][(C)]
(118)
Σ E M YI.(C) | in (2))
(119)
M1
E (Yl[(C) rı, 1i 12
(120)
Ylc(C) rı, JR. (2)) r2) rrMao. (fu1 /R. (w)) = rı))
(121)
+ Mo
(1c(C) ri, (u: [Ry (ui) ri piMa (uu: : Jn, (w)) o r))
(122)
E Ma m2
(VI((C) rı 1 (r1) + ==
(123)
EpMag..
Y
( c,ri M. (c.T) + ra
(124)
Combining these results we get that under M.
Adi = A(r1,T2).
(125)
Corollary in Thm. 1.
The bound from multiple domains in Thm. will be at least as informative as the bound from a single domain
Proof. We claim here that for any R Z,
A(Ø) A(r)
(126)
This means that the bounds on A that we can obtain from Al system grounded in M, are more informative than the bounds obtained from an Al system grounded in M. A is a difference of two terms written A(r) = Ai(r) -A2(r).
Ar(r) = E Y r PP (c,
(127)
Pair (), + Pa, (2)r)
Ar(r): Epa Y c, x P (c, z)rr) + (2)r)
(128)
Pao.r (c, x)r + (z),)
25

---

The Limits of Predicting Agents from Behaviour
holds that A1(r) A1(0), A2(r) A2(D) which then implies A(r) A(2). To see this notice that,
Ai(r) Ea c, 1
(129)
()), (1 P (z),)
Ep,, [Y] c, c
(130)
Pairs 2)) + +1 Pa (z(rr)
Epa Y c, z (c,
(131)
1 Pa. r ((,))
Ep,, LY c, [P, (c,
(132)
1 Pa (2, z)
Epa, [Y c, Pe (c,zz)
(133)
Pa, (c, + 1 - Pd. (z)
A1(D).
(134)
where è stands for the combination of values of C that are Further,
A2(r) Ep Y c, 2/r (c, (2)r)(2)r)
(135)
(d 2) + (2)))
c,
(136)
(c,,)((,2))r) + r
Ep 11 JPa
(137)
(c,z),r (c,
Paoir
1- (z)
1 Eran [1 [1 c, ] (c,
(138)
Pdo (c, 2)
Pdo
E EPa [ c, z Pa, + Pdo (z)
(139)
Pdo (c, 2) + Pdo (2)
A2(Ø).
(140)
Thm. 3 restated. Consider an Al grounded in a domain M made aware of an (under-specified) shift on non-empty Z c V. Then the AI is provably not weakly (or strongly) predictable in any context C = C.
Proof. Recall that the preference gap is defined as:
Ad = Bp Po.d, [Y|C=c]-Ep... [Y|C=c]
(141)
Here we know that a potentially modifies the mechanisms of the set variables Z though the nature of the modification unknown. the worst-case, the AT's interpretation of the possible new assignment of Z could arbitrary.
We will prove this theorem for the case of binary variables Y, V. In the following, we construct two (canonical) models that entail any chosen distribution for the observed data Pa but evaluate to the priori minimum and maximum value of the preference gap A, i.e. -1 and 1 respectively. We make use of the canonical model construction from (Jalaldoust et al., 2024) to define the following general SCM,
0if = 0
0ifr, 1,
0ff 0 -
lifr, =,,
(142)
lifr =
lif = 2,
0ff fy 2,
1 if
26

---

The Limits of Predicting Agents from Behaviour
U By) where Rg and Ry might be correlated and with a probability P(U) Pu c) such that Pa(z,y c) P(z. y cc. By (Jalaldoust et al., 2024, Thm. 1) this is always possible since this class canonical models is sufficiently expressive to model any observational interventional distribution. We can visualise the joint probability exogenous variables using the following table:
Probabilities M
Fz
Fy = 0
Poo
Pio
Ty =
po1
Fy 2
Po2
P12
Fy = 3
P03
P13
where we have written Pa(r= = 0,
c) = Pub. From these we could compute joint probabilities
Pa(= = 0, y = 0 c) = Poo + po1.
(143)
Pa(z = 0, y = c) = P02 + Pos-
(144)
Pa(z y c) P12 + P11.
(145)
Pa(z L,
c) P11 + P13
(146)
Here we can see that the parameter space Pa(rz: Ny c) is very expressive. For example, without loss of generality we could set poz P13- 0 or Poo Pio 0 and still be able generate any observed distribution Pa(z, y e).
The given shift in the environment can be entirely modelled as a shift in Pa.a(r= c) while keeping the probability of ry invariant, i.e., Pr.a(ry c) = Pa(ry c). In other words, given the table above, we can change each of the cells while maintaining the row sums equal. Recall that we are interested in evaluating bounds on probability of the form Pa.(1 1 c) and (y = = 1, c) depending on whether is given as an input to the Al or not. Both these quantities can written in terms the probabilities of exogenous variables as follows,
Pa. = c) = Po2 + P0s + P11 + P13
(147)
P11 + P13
(y 1, 1, P11 + P13 P12 P11
(148)
For the lower bound on these quantities, without loss of generality assume that Pos = P13 = O. Then the following table:
Probabilities Ma
T2 0
ry 0
Puo
P10
Ty 1
Poi + P11
0
Ty = 2
0
P12 Po2
ry = 3
0
0
is perfectly valid model under a shift that respects the constraint on Par(fr y
c) but for which 1, = as it is the sum of the two entries
c) = Pa(r, |
Pa | c) = 0 as it is the sum of the zero entries and Pa.a(y = 1 |
in the second column divided by the sum entries in the second column.
we are interested getting an upper bound then without loss of generality assume that Poo Pio 0. Then the following
Probabilities Ma
Fs 0
1
Ty 0
0
Ty = 1
0
Po1 + P11
Ty = 2 P12 + Po2
0
Ty = 3
Poa
a perfectly valid model under a shift a that respects the constraint on Pa
| c) = Pa(r, |c) but for which
y
Po.(y | c) = 1 as is the sum of the non-zero entries and Pa, = 1
1, = 1 as it is the sum the two
non-zero entries in the second column divided by the sum of entries in the second column.
27

---

The Limits of Predicting Agents from Behaviour
By using this construction to define lower and upper bounds for Pa.aly 1 c) or Palal 1 2; for d = do d1 obtain possible internal model for the Al that entails the observed external behaviour but for which the preference gap evaluates to -1 and 1. This means that the priori bound.
E A d= 1,
(149)
is tight whenever the shift is undefined (whether we know the variables it applies to not). Since the preference gap is unconstrained for any C = c and any pair of decisions (d, d²),, the Al is not predictable.
Thm. 4 restated. Consider an AI grounded in a domain M and (C,a(C) made aware of a shift a on Z c C. The As weakly predictable under this shift in a context C if there exists a decision d" such that,
1 2+ Er, Y e (c) - Er, [ - Pa(c) 0, for some d * d².
(150)
Pa (c)
Proof. Recall that the preference gap under shift a between decisions (d1, do) in situation C is defined as:
Ad = Bp [ o.dg [YIC=O -
(151)
Here we know that potentially modifies the mechanisms of the set of variables Z. The nature of the modification is unknown but we are told that after modification, the expected probability C is given by Pa.a( C), assumed to known and internalised by the A. This means that its internal model, whatever interpretation for the shift it chooses, generates the assumed probabilities, i.e. (C, (C,a(C).
We will consider the derivation of bounds each term of this difference separately. Firstly, note that,
E Papd [ = c = [ [YLc(C) Po.lle)
(152)
For case notation let us write R = Z. We could then show that,
Bp [ Y= Z R²) Ep. Y 1,,,, (. R)
by consistency
(153)
Σ Ep 12 ZZ R2))
(154)
E (,I)
marginalizing over the values z' of Z
(155)
Now once we intervene z the mechanism that generate its value before hand, whether it the shift something else is irrelevant. In essence, we get an equivalence between shifted an un-shifted distributions under intervention:
Bp (R=) = E,, (Y2,)
(156)
We could now take this quantity to show the following.
Ep, Y-1, [2)]
EB [2 R2)
(157)
1211,,,,,,,6 R2) 1 Σ Ep,1 Y=1 Rz)
(158)
z'+=
-Y,[Y12,r (Z,R)
Σ p.
[ (2,
by consistency
(159)
z'4²
EBp[YY2, 2,r (Z,
>
E Eppl (Z)
since Y. and (R(
(160)
-Ep, |E, Y1 z.r (Z, | Pz(2)
(161)
Ep, Y P((c) + P-
(162)
For the lower bound could consider the following derivation,
Ep (1,,((22 Z2 R) = Epp 11 12 (2, R) Ep, [[1][[111 [1- V)1,,1 (2, R)
12,,
(163)
28

---

The Limits of Predicting Agents from Behaviour
For of notation let define,
Bp Po V1 21,, (2, R (1-)(111))
(164)
Similar bounds apply on P 2, Z R)) to get,
E ( R.)
p z, Z, R) pe (1 2,r R2 + Pa(z))
(165)
E 1, R. - -Ep,1 12, Z, R) R E ( R) 1-1+ Pa(z)
(166)
= Pa.(() - Pa(c) + B
Y L
PP(cc) 1 Pa(z)
(167)
Putting the lower and upper bounds together to form bounds on Ad we get,
(c) (a.(e)
-ap (c) + p,, IY PPs (c) + Pa. (z) p18p2 (Y |c|c]Pagee + Pd (z)3
A do
(168)
(c.
-a, + Ep, [Y c Pa, (c) 1 Pa, (z) - -Ep [Y (c) 1 Pa)(z))
1
(169)
Pa.do (c)
2 E Pr
Y c Pa. (c) Bp, [Y c PP, (c) - (x) + (c)
(170)
(o.o(()
and by grounding.
2+ Ep,,
|Y| [Y]c)Pa Ep,, Y c cJP (c) 2P', (2) Pa
Ad 2
(171)
Po (c)
This statement holds for any SCM compatible with the grounded Al's external behaviour and therefore,
Adxde > 1 2 2+ Ep,, MeM
c Pa (c) Er, | PP(cc) 2P((2) + Pa(c)
min
(172)
Pa (c)
We can establish that the AI is weakly predictable in context
C if there exists decision d* such that,
2 + Ep,,, Y PPa (c) Ep, Y c c[(((() 2Pd(z) Pa(c) 0,
(173)
P Pa (c)
for some d=
We now continue with our inference of the Al's perceived fairness and harm of decisions in Sec. 4.3. Thm. restated. Consider an agent with utility function grounded in a domain M. Then,
EPp[ Y |, < T(d, c) < - Ep. 1-EP, Y | [.cc].
(174)
This bound is tight.
Proof. Recall that for given utility Y, the AI's counterfactual fairness gap relative decision d, in given context c, is
T(d, c) Ep Ya.z, 20, c - Ep Ya 20, cl
(175)
And remember that C.
For of notation, write 2, z' such that.
T(d, c) Ep Ya,2 2 Ep Ya c.
(176)
29

---

The Limits of Predicting Agents from Behaviour
We start by considering the following derivation:
Plyd,z c) = P(yd,es zal c) + P(yd,2 2/d c)
by marginalizatton
(177)
P(ya 2d c) + P(yd,z: 2/2 c)
by consistency
(178)
and since d does not affect Z or C, i.e. Za Z, Ca =C
P(ya, c) Plys, 2d c) P(yd,z: o' |c)
(179)
which implies
P(ya, 12,c)= P(yd,z c) Pa(y,= |c)
(180)
Pa(21 c)
Therefore,
Bp Ya, 2 c. Ep[Ya.e c] -Ep, Y
Iz, c. Pa(=1
(181)
Pa(2' c)
All quantities the r.h.s observable except for Ep|Ya, For the lower bound, consider the following derivation,
c] which can be tightly bounded.
Ep[Ya, | c] = Ep Yd,= 1 (Z) cl
marginalizing over
(182)
> Epl Yd,= (Za) []]
since summands > 0
(183)
= Ep Y21,, (Za) [c]
by consistency
(184)
= Ep.[ c, P
by grounding and C₁ = C
(185)
Similarly, can get an upper bound by noting
Ex[Ya,
c] = 1 E-pEpp Yd.z) c] E c, [Pl(z c) + Pa(2' c).
(186)
(187)
Tightness Lower Bound
For the lower bound we will consider the following SCM,
- fz(u)
C Sc(u)
D d
M
(188)
if =
Y
0 otherwise
P(U)
Here (fz, fo, P())) are chosen to match the observed trajectory of agent interactions, i.c., such that (M)(()) (v)(() for all E suppy-
Then, under M
E Epsil Yd,z cc
(189)
Σ P [ (Y,z u, e c.elpM (u ()
(190)
2 p Ya
pp (c)
(191)
u
E piu [
2, c, (u: fz(u) = IpM (lu: f2(u) = 2) (c)
(192)
+ E pm
2, c, (u: fz(u) * =) ppM ((p fz (u) * 2] c)
(193)
= E
z, c M | c).
(194)
This expression the same one as the analytical bound showing that tight.
30

---

The Limits of Predicting Agents from Behaviour
Tightness Upper Bound
For the upper bound will consider the following SCM,
Z fz(u)
C fc(u)
d
Ma =
(195)
uf
if f2(()= 2
1 otherwise
P(U)
Here (f2,fc, fr, P(U)) are chosen match the observed trajectory of agent interactions, i.e., such that (M (b) for all E supPy-
Then, under Ma.
E EPma| [ Yd,= 1 |c]
(196)
Σ 18 pm (a, u, uu pecppu (c)c) c)
(197)
u
Σ Epmal Ya 2, u, u. (u
(198)
p Y cu fz(u) =} I (fu: fz(u) = 2) c)
(199)
E pugl Y
2, 2, (u fz(u) =) p) ((u tu fz(() = c)
(200)
pagl Y
z, e pM e) + pmù (= c).
(201)
We therefore find that,
0 0 Ep Ya.z e & 1,
(202)
and ultimately, claimed.
-Ep, Y [, < T(d, < 1 Epl Y [,cc
(203)
Thm. 6 restated. Consider agent with utility function Y grounded in domain M. Then,
max(0, Er, Y cc] + Epa, [Y|c]- - 1} & M(d, & (En, Y [c], Ers, [Y]c]}
(204)
and this bound is tight.
Proof. Consider an agent with internal model M and utility function Y. Recall that the agent's expected harm of decision with respect to baseline do, in context c, is
M(d, do) = Ep max(0, Ya, Ya} | |c]]
(205)
We can re-write this quantity as follows
R(d, do) = Ep max Ya, - Yal cc] max(0, O,, yd. - - Plya, ydo c)dyadyd.
(206)
(207)
Since Ya is binary, the only time that the maximum evaluates to something greater than zero is when Yd, = 1 and Yv- 0.
Then,
R(d, do) P(Y Y,Ya= -)
(208)
31

---

The Limits of Predicting Agents from Behaviour
This quantity be tightly bounded using the results of (Tian & Pearl, 2000, Sec. 4.2.2) giving
max{0,Ep, [Y cc] P
1) Rid, do) < min(Bp [Y c Pp, [Y|c]]
(209)
And by grounding.
Ep, Y
c] + EP, [Y c 1) < ((d,d) do) < min(Ep, [Y] ccc BP,, [Y]c]]
(210)
32

---

The Limits of Predicting Agents from Behaviour
D. Other accounts of fairness and harm
To ground definitions of fairness, several authors appeal to counterfactual thinking but some accounts, instead, are interventional in nature.
Within legal systems, counterfactual faimess (Def. 7) operationalizes doctrine known as disparate impact doctrine focuses on outcome faimess, namely, the equality of outcomes among protected groups. On the other hand, the doctrine of disparate treatment seeks to enforce the equality of treatment in different groups, prohibiting the use of a protected attribute in the decision process, and has been formalized using interventional accounts (Barocas & Selbst, 2016).
popular notion in the disparate treatment literature is known as direct discrimination (Barocas & Selbst, 2016; Zhang & Bareinboim, 2018). An agent is said to engage direct discrimination the causal influence of a sensitive attribute Z that is not mediated by other variables C is non-zero. This a contrast between interventional expectations. We adapt this notion to define an Al's perceived direct fairness gap the difference in expected utilities obtained for different values of protected attribute while holding all other variables fixed.
Definition 10 (Direct Discrimination Gap). Let Z E {zo, 21} be protected attribute. For a given utility Y. define an agent's ss direct discrimination gap relative to a baseline value 20 in a given context as
(dd c) := Ep Ya -Ep Yd.
(211)
We say that an AI "intends" to avoid direct discrimination if under any context C = and decision D d the direct discrimination gap V evaluates to O. Here, we consider this notion of faimeess to illustrate the kind of inference that is possible to obtain from an Al's external behaviour with one alternative account. The following theorem shows that, contrary to the counterfactual fairness gap, V can be bounded given the Al's external behaviour.
Theorem 9, Consider an agent with utility Y grounded a domain M. Then,
(d, c) > Ep [ Y
2o: c Pa( c) + Pa(zo, (rd c) < Epa Y 21. Pa(zn,)) Ep, Y zo. c Pa 20, c) P)(
21, c Pa(z1, - Ep, [Y|
(212)
(213)
This bound light.
Proof. Let E {0, 13 be a protected attribute and zo a baseline value of Z. For a given utility variable Y, recall that the Al's direct fairness gap relative baseline 20 in a given context is defined
(d, e) == Ep Y Ep Y Yd.
(214)
Using a similar proof strategy to that in Thm. 1, we can derive tight bounds on V.
Analytical Lower Bound
A lower bound on the interventional expectation can be obtained using the following derivation:
Epl Ys
Σ
B Y.cca (Ca: 7c.)
marginalizing over Cas Fc,d
(215)
a,
Ep
Ze.d)
since summands
(216)
Epl Ye, Co,, (Cs, Ze.a)
by consistency
(217)
EEp Yale, (Ca, Zu)
by consistency
(218)
Er, Y1,,; (C, [C
by grounding
(219)
EP, Y |c, P)((c
(220)
Analytical Upper Bound
For deriving upper bound on the interventional expectation, we start by noting that,
Ep YY..d 1 Ep 1 Yz,c,d ]
(221)
Leveraging the bounds derived above obtain,
Ep Yz 1 1-11-pp Epa. (1 - Y) cc,2)P((c,2) -EpllY (c,))P((c,2) 2) + Pd(c,z).
(222)
(223)
33

---

The Limits of Predicting Agents from Behaviour
By setting z 21 in the lower bound and =o in the upper bound of the expected utility, obtain lower bound the difference of expected utilities:
(d. > Bpa IY| 21, c Pa(s,c))
Epp, Y 2o. c Pa(zzz, + Pa(zu,
(224)
And similarly, by setting 2 = 21 in the upper bound and 2 = 2g in the lower bound of the expected utility, we obtain an upper bound on the difference expected utilities:
(d, c) & Bpa Y 21, c Pa(z1,c) Ep, [Y] 20. c Pa(so.c) + 1 Pa(z1. c).
(225)
We now show that these bounds are tight by constructing SCMs (that is, possible world models of the Al system) that evaluate to the lower and upper bounds while generating the distribution of agent interactions Pa-
Tightness Lower Bound For the lower bound we will consider the following SCM,
Z (z(u)
C - fc(u)
D r-
c,,,,
if fz(u) fc(u)
M =:
(226)
0
if [z(u) * or Sc(u) c, and Z 21
Y
20. u)
if fz (u) = fc(u =c
if fz( zo fa(u) # c, and Z 0o
P(U)
Here ffz,fo, fr P(U)) are chosen to match the observed trajectory of agent interactions, i.c., such that (MM(())
(M)( for all E supPy-
Then, under M
(dd c) = -E Yd Epsel [ Yd
(227)
E E Epmil Yd u pP (u)
(228)
Eppm Yd..o.c u MaMi (u)
(229)
Σ Epail Ya [Y 2, u. ppM
(230)
Σ Epmil Ya
c pM (u)
(231)
E |
21, c, (u: fz(u) f.ff(() c) pan (tu fz(u) f.ffc() = c})
(232)
E my
و₁, c, (u fz(u) # # or fc(u) # c) pal ((u: fz(u) # =1 or fc(u) + ct)
(233)
E Y
20, c, {u: fz(u) zo. fc(() = c) Mdd ((u: fz(u) = zo. fc(u) =c))
(234)
E Y
20, c. (u: fz(u)
=o or fc(u) * c} p ((u: fz(u) * =0 Sc(u) * c))
(235)
= E E Y 21, e M (21, p |Y z0, e M c) - 1+ p.d. (z. c).
(236)
This expression the same one as the analytical bound showing that is tight.
34

---

The Limits of Predicting Agents from Behaviour
Tightness Upper Bound
For the upper bound we will consider the following SCM,
Z - fz(u)
C - fo(u)
D d
fr(d, 21,
if fz(u) = =1, =c
M
(237)
if [z(u) * 21 or [c(u) * c, and 21
Y
Cy
if fz(() sc(u) =c
0
if fz * 20 or fc(u) # c, and Z 20
P(U)
Here (v) for all VE supPy-
are chosen to match the observed trajectory of agent interactions, i.e., such that (m) =
Then, under My
(d, = E paaal Ed. E pac ed
(238)
B e u JPM
(239)
E Ya pP (u)
(240)
= Bpnel [ [Yal 2, u, ppMp(u) (u)
(241)
a
Σ E puel Ya 2, u c pam (u)
(242)
= E n Y 2 c, (u: fz(u) = fc(u) = ct p (fu: fz(u) = fc,((u))=
(243)
p Y 21, c, (u fz(u) # or fo(u) c) (PM fz(u) # fc(u)) # ct)
(fu:
(244)
Enlgl Y
20, c, (u: fz(u) solu) c) p ((u: fz(u) z0, sc(u) c))
(245)
- E E Y
20, c, (u fz(u)= # z or fc(u) # c} MM (fu: fz (w) # 20 or fc(u) (u) # c/)
(246)
E m3 21, e M (21, c) + 1 pM [2, c) E. pm Y 20, e p- cz. c).
(247)
This expression is the same one as the analytical bound showing that it is tight.
Definitions of harm (defined with respect to a causal model) can also be split in two groups: causal and counterfactual accounts. (Beckers et al., 2022) exemplify the causal account as defining a decision d to harm a person if and only d is cause of harm. Recall that the counterfactual account has the same structure but differs in the second clause, instead defining a decision to harm a person if and only if she would have been better off if d had not been taken. Here, we quantify how "good" or "beneficial" particular situation V experiments (it might capture, for example, the value of sensitive environmental variables). A formalisation of this causal account of harm, with respect to an Al's internal model, is given in the following definition.
with a binary utility Y E {yo, y1} that we assume is tracked in
Definition 11 (Causal Harm Gap). Consider an agent with internal model M and utility Y 91., The agent's expected causal harm of decision d with respect baseline do that obtained the non harmful outcome yo in context c, is
M(d1, do. e) E3 Ya, No. do. cJ.
(248)
This probability expresses the capacity of d to produce a harmful event Y y1 that implies a transition from the absence the presence of d, and yı, we condition the probability on situations where dj and y1 are absent, i.e. D do. Y yo-
Theorem 10. Consider agent with utility Y grounded in domain M. Then,
Ps. (11
c) P(y.d, c) & R(di, do, c) & Pa. (vi
c) - Pa, c) |c)
(249)
Pa, (yo
P(do c)
Pa, Pu, P(do c)
35

---

The Limits of Predicting Agents from Behaviour
Proof. Note that the causal harm gap may be equivalently written,
Md1 do. c) == P(y1,,, yo. do. c).
(250)
The lower and upper bounds may be derived considering the following.
P(yi.d | c)
P(yi.d y0, do
c) + P(yi.d, Mı, do c) + P(vi, yo. di c) Plyna,
(251)
P(y1,d, Mo: do
c) + P(yi,d, V/ do c) P(y1,,, (1, ;1 c)
(252)
P(yn,d yo. do
c) + P(un,d, 11 c)
(253)
P(yi,: 10, do
c) + P(wi,d, d1
c)
(254)
Ply1,d, 1 c) P(yl,d, Mo. do
c) + P(y,,d, d V1 cc)
(255)
P(yi,d; No. do
c) + P(yi,d, //, c)
(256)
P(yi,d, No, do
c) + P(yi.d, d1 (c)
by consistency
(257)
P(y1.d No. dol c) + P(y d1 Pdd, cc)
(258)
P(d1 c) stands for the AI's policy in the source environment, i... the probability it uses for choosing decision dj in
situation Re-arranging these equations this implies.
Plyi. c) - P(yiad ee) & do, & P(n., c) P(yr.d c) c)
(259)
P(y0,ao P(do c)
P(yo. P(do |c)
And by grounding.
Pa (1/ c) P( c) Pio (yo P(do c)
Pa (v1
c) - Pa, c) c)
& R(di, do, c) &
(260)
Pa, (vo P(do c)
36


Reviews:
[{"review_id": "50YHDcAFYT", "reviewer": "Reviewer_Q6s6", "summary": "This paper considers a model of a decision-making agent as follows. Note the\npaper always imagines the agent to be an AI system, but this does not appear to\nbe essential other than for motivation.\n\n* Suppose the agent makes decisions guided by (1) a utility function and (2) a\n   causal model of how its decisions influence outcomes in the world.\n   Precisely, suppose the agent makes decisions so as to maximise expected\n   utility in a structural causal model (SCM).\n* Suppose further that the agent's utility function is known to us, but the\n   SCM is not, other than that the SCM is sufficiently well-calibrated\n   (\"grounded\") in the sense that it is in agreement with some known ground\n   truth SCM as to the distribution over outcomes induced by each of the\n   agent's potential actions.\n\nIn this setting, one can ask theoretical questions of (partial)\nidentifiability, i.e., to what extent does the behaviour of the agent in the\nground truth SCM determine its future behaviour given various kinds of\ninterventions, or its true intentions regarding discrimination and harm?\n\nDue to the fact that there may be multiple agent-SCMs consistent with the\nobserved behaviour, meaning its future behaviour under an intervention or its\ninternal motivations for a decision are under-determined. Thus, the best one\ncan hope for is a characterisation of the range of possible behaviours or\nintentions given the range of possible internal SCMs.\n\nThis paper provides several such bounds under various assumptions.\nThe first contributions regard the range of possible degrees of preference\nbetween two actions conditional on a known, unknown, or partially-known\nintervention, indicating how (un)predictable the agent's future decisions are.\n\n1. Theorem 1 gives tight bounds on the range of possible degrees of preference\n   between two actions given a fixed atomic intervention to the SCM.\n2. Theorem 2 shows that no meaningful bound is possible on the degrees of\n   preference between two actions given an unspecified intervention to the SCM.\n3. Theorem 3 shows that if the intervention is unspecified but the effect on\n   the distribution of input contexts is known, then some meaningful bounds can\n   be derived.\n\nFurther contributions regard evaluating the agent's intentions regarding\ndifferent notions of fairness/discrimination or safety/harm, indicating to what\nextent behavioural evaluations can reach meaningful conclusions about these\nproperties of decision-making systems guided by causal world models:\n\n4. Theorem 4 shows that no meaningful bound on the \"counterfactual fairness\n   gap\" can be derived from external behaviour.\n5. Theorem 5 shows that in some situations, meaningful bounds on the\n   \"counterfactual harm gap\" can be derived from external behaviour.\n6. Theorem 8 (appendix D) shows that in some situations, meaningful bounds on\n   the \"direct discrimination gap\" can be derived from external behaviour.\n7. Appendix D also discusses \"causal harm gap\" and offers a proof of some\n   bounds on page 33, though a statement of 'theorem 9' and discussion of its\n   implications appears to be missing.\n\nThe paper also discusses the prospects of deriving stronger/weaker bounds given\ndifferent assumptions, primarily through the lens of a simple concrete example\n(rather than general bounds) including:\n\n8. If the \"grounding\" assumption is relaxed to allow an approximate agreement\n   between the distributions predicted by the agent's internal SCM and the\n   ground truth SCM, then slightly weaker bounds result.\n9. If instead of assuming the agent optimises the specified utility function,\n   we assume that it optimises a similar 'proxy' utility function that induces\n   similar decisions given the ground truth SCM, then the behaviour can still\n   be constrained, at least in similar contexts.\n10. Sections 5.2 and 5.3 discuss how with partial knowledge of the agent's\n    internal SCM, bounds can be improved.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review, we appreciate the feedback and the depth of the observations.\n\nWe do share your concern on modelling assumptions. Before discussing them in more depth, it might be worthwhile, however, to note more explicitly the increasing evidence available for AI systems (particularly LLMs) behaving rationally across a wide set of environments, and why this behaviour can be described by some causal model. \n\nDecision-theoretic tests of rationality have been applied to LLMs that \"demonstrate higher rationality score than those of human subjects\" (Chen et al., 2023), see also (Mazeika et al., 2025). An LLM's set of preferences over interventions, to the extent that they are consistent with rationality axioms, can then be formally described by an SCM (Halpern and Piermont, 2024). The same conclusion can also be obtained for agents capable of solving tasks in multiple environments (Richens and Everitt, 2024).\n\nHalpern, Joseph Y., and Evan Piermont. \"Subjective Causality.\" 2024.\n\nRichens, Jonathan, and Tom Everitt. \"Robust agents learn causal world models.\" ICLR. 2024.\n\nChen, Yiting, et al. \"The emergence of economic rationality of GPT.\" PNAS 2023\n\nMazeika, Mantas, et al. \"Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs.\" 2025.\n\n***1. If the AI's behaviour is only approximately captured by an SCM, then I would suppose that the bounds will degrade.***\n\nIn principle, this is true – if AI behaviour sometimes deviates from what is expected given the assumption of a fixed causal model, then the bounds might degrade. And if those deviations are sufficiently random then presumably no guarantees on AI behaviour can be given.\n\nPossibly, one interesting relaxation that can be entertained is the assumption that the AI operates on multiple causal models. For example, suppose it samples a causal model from its set of possibilities and makes a decision according to that causal model. If the AI is grounded, meaning that all members of its set of causal models are compatible with the observed data, then all bounds remain unchanged. This is because the bounds capture the decisions implied by all causal models compatible with the observed data and therefore also the decisions implied by a randomly drawn causal model from this set. In contrast, if the AI is not grounded and sometimes makes decisions based on a causal model that is not compatible with the data, the bounds might degrade.\n\n***2. It would be more appropriate to treat their rationality as \"bounded\" in some sense (for example, supposing they choose actions that have approximately the highest utility, rather than exactly the highest utility, given their model).***\n\nThis is correct -- if the AI chooses actions that have approximately the highest utility (under its own model) we would expect this additional source of uncertainty to degrade our ability to predict AI decision-making. If we understand the reviewer’s intuition correctly, while the AI’s beliefs on expected utilities remain unchanged, possibly the selection of actions given those beliefs may differ from exactly utility maximizing. \n\nIn this case, our bounds, e.g., on the preference gap $\\Delta$ that measures the relative expected utility benefit from a decision $d_1$ relative to a decision $d_0$, would remain unchanged as they only reflect the AI’s beliefs and not their choices. For exact expected utility maximizers, $\\Delta > 0$ ensures that $d_1$ will be chosen over $d_0$ by the AI. For approximate expected utility maximizers this threshold might not be sufficient, and we might require $\\Delta > c$ for some $c>0$ to conclude that $d_1$ will be chosen over $d_0$ by the AI. The magnitude of $c$ will depend on how loosely the AI follows the expected utility maximization principle. This setting is potentially more realistic, thank you for suggesting it.\n\n***3. In footnote 4 the paper assumes access to a comprehensive specification of the AI system's behaviour in all possible contexts (if I understood this footnote correctly). There will be contexts in which we don't observe the system's behaviour in the data.***\n\nWith this footnote we are trying to convey the possible challenges in estimation of bounds from finite samples. Our results are given in the infinite-sample regime, in terms of probability distributions, that may be difficult to estimate accurately if not enough data is available. For example, if the bound requires the estimation of a probability $P(d, c)$ but the combination of decision $d$ and context $c$ is not observed in the data, we might need to introduce additional modelling assumptions and account for errors in estimation to provide practical guarantees.\n\n***4. Sec. 5.1 and the contrast between goal misgeneralisation and approximate inner alignment.***\n\nRight, thank you for this explanation. Perhaps “approximate inner alignment” is a better framing for that example.\n\nThank you for the careful reading of our paper and for pointing out typos.", "claims_and_evidence": "The model, assumptions, and results are clearly stated. The interpretations are\nclearly explained. The examples help make the bounds concrete. The\nproofs accompanying each formal claim give explicit detail for all derivations\nand calculations.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "I did not check the correctness of the proofs or examples in detail.", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "As I understand, prior work must have studied the related question of whether\nthe behaviour of a utility-maximising decision-maker uniquely determines the\n(grounded) causal model used as the basis for its decisions (the answer being\nno, in general). Authors can correct me if I am mistaken on this point.\n\nThis paper builds on this more fundamental partial identifiability by extending\nit to partial identifiability results for the decision-maker's OOD behaviour or\nits intentions with respect to fairness and harm.\n\nThis observation is valuable to the field of AI safety since it motivates the\nneed for understanding and control of the specific internal SCM that governs\nbehaviour in powerful AI systems of the near future.", "essential_references_not_discussed": "None to my knowledge.", "other_strengths_and_weaknesses": "**Strengths:** As noted in above sections, I think this paper is very clear and\nmakes a valuable contribution to the field of AI safety.\n\n**Weaknesses:** The main weaknesses concern the strength of modelling\nassumptions. While the paper contains a thorough discussion of how relaxing\nvarious assumptions would affect the results, I think further discussion could\nbe warranted. Particularly, when it comes to the motivation of predicting the\nOOD behaviour and safety properties of interest to the AI safety community, we\nare interested in large-scale AI systems acting in very complex, real world\nenvironments with very complex internal cognitive structures. This raises the\nfollowing questions:\n\n1. While the paper points out (footnote 2) that there is no assumption that the\n   AI system reasons explicitly in terms of an SCM, it appears that there is an\n   assumption that the AI system's behaviour is well-captured by *some* SCM.\n   If I understand correctly, in reality, large-scale complex AI systems with\n   complex decision-making procedures might only be approximately modelled by\n   any (reasonably sized) SCMs.\n   In turn, if the AI's behaviour is only *approximately* captured by an SCM,\n   then I would suppose that the bounds will degrade. If this is accurate, then\n   I think it would be valuable to add an example along these lines.\n\n2. The paper assumes that the agent's decisions are exactly optimal given its\n   causal model. In complex, real-world environments, even in the limit of\n   quite powerful AI systems, these systems are still subject to computational\n   limitations and accordingly it would be more appropriate to treat their\n   rationality as \"bounded\" in some sense (for example, supposing they choose\n   actions that have approximately the highest utility, rather than exactly the\n   highest utility, given their model).\n   Again, I suppose such an approximation should degrade the bounds on\n   predicting behaviour. If so, I think it would be valuable to add an example\n   along these lines.\n\n3. The paper points out in footnote 4 that it additionally assumes access to a\n   comprehensive specification of the AI system's behaviour in all possible\n   contexts (if I understood this footnote correctly). However, in complex\n   environments, there are too many contexts for us to ever hope to have\n   such comprehensive behavioural knowledge. There will be contexts in which we\n   don't observe the system's behaviour. The footnote points out that this\n   creates strong limitations on inferring future behaviour. I had some trouble\n   following the exact meaning of this footnote, and invite the authors to\n   consider expanding this discussion. Is a result along the lines of theorem 2\n   possible in this situation? Or, at least, a concrete example along these\n   lines?\n\nI should clarify that I don't see these limitations as undermining the\ncontribution of the work, which I think is acceptable even given the\nassumptions that it makes. I am listing these issues more as a way to suggest\nthat the contribution would be improved in my opinion if the discussion in\nsection 5 could be further broadened along the above lines.\n\nThe third point above also bears on the discussion of goal misgeneralisation\n(section 5.1), which I feel needs refinement:\n\n4. The conclusion of section 5.1 appears to be that \"OOD\" behaviour (behaviour\n   under a novel intervention) may in principle be possible to bound if one has\n   sufficient constraints on the proxy utility function in relation to the true\n   utility function. An explicit comparison is made to goal misgeneralisation.\n   However, in empirical work defining and studying examples of goal\n   misgeneralisation, one trains in a *subset* of contexts, where the proxy\n   utility and the true utility are constrained to agree, but then tests in\n   *novel* contexts where they come apart, and the system's performance\n   degrades to an arbitrary degree. If I understand correctly, the combination\n   of (1) the assumption that we have behavioural data covering all contexts\n   and (2) a proxy utility function that is correlated with the true utility in\n   all observed circumstances amounts more to a setting of \"approximate inner\n   alignment,\" rather than the concern about more substantial inner\n   misalignment and a different kind of distribution shift that drive the\n   failure mode of goal misgeneralisation.", "other_comments_or_suggestions": "I found the paper generally well-written, but spotted a couple of minor typos,\nor generally questionable phrases, as follows.\n\n1. In the abstract, you write \"If an agent behaves as if it has a certain goal\n   or belief, then we can make reasonable predictions about how it will behave\n   in novel situations...\" This seems to beg the question, that is, isn't \"we\n   can make reasonable predictions about behaviour\" the definition of what it\n   means to say a system is an agent whose behaviour is driven by goals and\n   beliefs? I was not sure if this \"if ... then ...\" sentence was an attempt to\n   justify making predictions based on attributed beliefs---I think some other\n   justification is needed to draw that conclusion.\n\n2. In the main result summary (Lines 032R--034R), I think you mean to emphasise\n   \"partially determines,\" but I'm not sure that this emphasis comes across.\n   For reference, I have written papers on partial identifiability, and I\n   initially missed the emphasis here. It might help to bold \"partially\n   determines,\" if this is what you were going for. A complementary approach\n   might be to add something in the introduction that explains what this means\n   (along the lines of explaining that even though we can't uniquely identify\n   the future behaviour, we can narrow it down to a range of possible outcomes,\n   this paper characterises those outcomes, etc.).\n\n3. Line 040R: \"can be can be\"\n\n4. \"Shane (2023)\" should probably be \"Legg (2023),\" and I think you should\n   probably cite the youtube video rather than a forum post in which someone\n   happens to be asking a question about the video.\n\n5. Is definition 2 used in the main text?\n\n6. Line 140L: \"how ... the world looks like\" (\"how the world looks\" or \"what\n   the world looks like\"?)\n\n7. Definitions 3 and 4: I found it initially unclear what is meant by \"for any\n   d.\" I think I understand now that $d$ is an arbitrary value associated with\n   the agent's decision variable $D$, i.e., it means $\\forall d \\in\n   \\mathrm{supp}_ {D}$. I think it could help to make this explicit. Also I\n   think that the status of $D$ as a unique decision-making variable is not\n   made explicit anywhere. Of course, it's possible I'm still confused.\n\n8. Opening of section 3.2: \"We ... do not have access to the mchanisms\n   underlying the actual environment nor the agent's internal model.\" This is\n   framed as an assertion, but I think it would be more appropriate to frame it\n   explicitly as an assumption of the setting you are analysing. Later in the\n   paper, you go on to point to work that attempts to improve on this situation\n   as a promising direction.\n\n9. Line 172L: I didn't really understand the connection between 'quick\n   learning' and grounding. I suppose there is not meant to be a necessary\n   connection, but rather you are just suggesting this as a sufficient way by\n   which an AI system may become (approximately) grounded in practice.\n\n10. Definition 5: While some earlier definitions have a small square at the\n    end, this one (is the first that) does not. I don't personally think the\n    squares are necessary due to italicisation, but I suppose you may want to\n    follow a consistent style, so I thought I'd point this out.\n\n11. Example 3: This example, and the definition of approximate groundedness,\n    felt like they conceptually fit more closely with the content in section 5\n    than they do here. I wonder if you have considered moving this 'assumption\n    relaxation example' to the end of the paper with the others, and if this\n    could improve the flow of the paper's presentation of the main formal\n    results.\n\n12. Line 233R-ish: Estimating the min/max of a set from samples can be\n    challenging. I wonder if some kind of optimisation technique could be used\n    here. Just an idle thought.\n\n13. Theorem 2: This theorem is the most surprising to me. In particular, I\n    paused to wonder whether it should hold for all SCMs. I noticed I am a\n    little confused about the space of possible SCMs under consideration. Could\n    there not be some corner cases involving very simple SCMs in which only a\n    single intervention were possible, such that we effectively know the\n    intervention even under the assumptions of this theorem? I apologise for my\n    unfamiliarity with the SCM framework---possibly this concern is not\n    well-formed. I looked briefly at the proof, but I am not familiar with the\n    prior work.\n\n14. Line 293L: \"it is known not tight in general,\" possibly a typo, or just\n    consider rephrasing?\n\n15. Line 351L: \"the external behaviour constraints the AI\" typo.\n\n16. Example 5, line 360R: \"his\" is possibly a typo.\n\n17. Example 5, line 370R: \"in fact could show that,\" typo.\n\n18. Line 437L: \"this bound are strictly tighter\" typo.\n\n19. Line 408R: \"we can use their own\" typo.\n\n20. Line 413R: Unclear if \"Peter\" and \"Jon\" are meant to be different people or\n    the same person (if they are meant to be different people, I missed the\n    significance of this).\n\n21. Line 416R: Double period.\n\n22. Equation 28: There are two stray closing parentheses in the definition of\n    $\\mathcal{F}_ \\sigma$.\n\n23. Line 729: \"his\" is possibly a typo.\n\n24. Line 732: \"This then constraints the possible values\" typo.\n\n25. Is there meant to be a theorem 9 stated before the proof on page 33?", "questions_for_authors": "I would appreciate if the authors could point out if there are any inaccuracies\nin my summary of the paper and discussion of how it fits into related\nliterature. If the authors have the appetite for further discussion with me, I\nam open to discussing the points I listed under weaknesses further. It doesn't\nseem necessary to discuss the \"other comments\" listed above. Unfortunately, I\ndon't see much scope to increase my rating further than 'accept'.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7JG28UveMr", "reviewer": "Reviewer_EHMW", "summary": "This paper derives theoretical bounds on predicting an AI agent’s future behavior from observed actions, using structural causal models (SCMs) to formalize beliefs and grounding. It introduces metrics like the preference gap and counterfactual fairness gap, arguing that—even with full behavioral data—fundamental limits remain on how precisely we can infer an AI’s internal decision-making, especially out-of-distribution. The authors back their claims with theorems and examples (e.g., a Medical AI scenario).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper.\n\n***1. The proofs are not thoroughly validated for practical use. Examples like the Medical AI feel oversimplified, leaving questions about whether the results would hold in more settings.***\n\nTo clarify possible misconceptions, we should emphasize that our results hold for any agent deployed in (discrete) systems of arbitrary dimensionality and complexity in the underlying functions and distributions, subject to our assumptions. More complex environments with more variables do not on their own pose any additional challenges for predicting future agent behaviour according to our bounds. As the reviewer mentions, our contribution is to formally characterize the range of an AI’s possible behaviours out-of-distribution. In our view, it is important that this characterization be theoretical, and apply in the worst-case, even in cases that are difficult to foresee in simulations.\n\nWe do appreciate, however, that assumptions on grounding and expected utility maximization might not be appropriate for all systems, and that further experiments could help probe whether the behaviour of current AIs is well predicted through our bounds (as the reviewer is suggesting). One additional experiment we consider prompts an LLM (the agent) to output its subjective expected utilities under different actions, comparing those inferences to our bounds. We provide a summary below of this experimental set-up.\n\n*Experiment description*. We give the LLM in-context the counts of transitions of a simulated dataset to mimic its past experience and approximately satisfy the condition of grounding, i.e., that the AI’s beliefs on the likelihood of events in the environment are consistent with the data. Find below the template we use.\n\n```\nYou are an agent operating in an environment. You are given the following dataset that represents your experience so far.\n  \n{{transition_count}}\n\nInfer a causal model that is compatible with this dataset and use it for decision making.\nNote that your causal model can be anything you choose, including unobserved confounders and arbitrary causal associations, as long as it is able to (approximately) reproduce the data.\n\nYour task is to return your beliefs on the expected value of Y after taking the action do(X=0) and do(X=1) in a new environment in which we apply the intervention do(Z=1). \n\nReturn your answer as a dictionary {{'expected value of Y given action X=1': <value>, 'expected value of Y given action X=0': <value>}}.\n```\n\nAcross different simulated datasets we compare the responses of various models from the Gemma family with our bounds. We find in all cases that the responses are included in our bounds. The reasoning traces reveal that LLMs generally follow instructions, hypothesizing a causal model, and deriving expected utilities correctly. This set-up is not without limitations but it does suggest (anecdotally) that it is possible to prompt LLMs to internalize a causal model and act rationally, such that are bounds can be reasonably expected to hold in practice.\n\n***2. A major downside is the heavy reliance on assumptions that may not hold in practice.***\n\nWe expand on the strength of modelling assumptions and possible relaxations in the answer to reviewer Q6s6. We would appreciate if you could refer to that response (let us know if we can add further details).\n\n***3. might it be more reliable to use the inferred beliefs to predict (or get the bounds on) the agent's behaviour in the in-distribution situation?***\n\nOur results so far are framed in the infinite-sample regime. This means that if we have access to the training distribution and the AI is grounded (i.e., it has learned how to predict the likelihood of events in-distribution) then we could automatically evaluate expected utilities in-distribution for all actions and derive the AI choice without any uncertainty (assuming it chooses actions to maximize expected utility). If we understand the reviewer’s suggestion as “with finite samples, even in-distribution beliefs might be inferred with error which introduces some uncertainty in their choice of action in-distribution\"; this is correct. The question here is how the AI is estimating its beliefs from finite samples, which is interesting. In this paper, we are not addressing estimation issues directly but we do acknowledge that it is an important limitation and a good topic for future work.  \n\n***4. what will happen to your framework if the AI system's internal models evolve over time?***\n\nThis is an interesting setting, the question might then become: how to predict AI behaviour given a large dataset of interactions in a training environment and a small dataset of interactions in the deployment environment. Our current results do not consider this setting so far, but extensions might be feasible along the lines of (Bellot et al., 2023).\n\nBellot, A. et al. \"Transportability for bandits with data from different environments.\" NeurIPS 2023.", "claims_and_evidence": "- the paper claims that an AI’s internal beliefs and decision-making can only be bounded, not exactly determined, using external behavior. though the derivations (e.g., Theorems 1–5) lay out tight bounds under various assumptions and their proofs appear rigorous at a glance, they lean heavily on idealized assumptions (like perfect grounding) that are rarely met in real-world scenarios.\n\n- the proofs are not thoroughly validated for practical use. examples like the Medical AI feel oversimplified, leaving questions about whether the results would hold in more settings.", "methods_and_evaluation_criteria": "- The paper is almost entirely theoretical, centered on structural causal models and derived bounds on preference gaps, fairness, and harm, well-motivated from a theoretical standpoint. \n- but the work leaves a gap regarding practical evaluation", "theoretical_claims": "I reviewed several key proofs of theorems 1-5, the proofs seem to be correct under the idealized conditions assumed", "experimental_designs_or_analyses": "The work includes only illustrative examples, such as examples 1-3, which are largely schematic, appearing more as a proof-of-concept rather than a validated experimental framework, i.e., no rigorous experimental design or analysis on real or simulated data.", "relation_to_broader_scientific_literature": "- well-rooted in established causal inference literature (think Pearl, Bareinboim, and others) and builds upon causal modeling and counterfactual analysis\n\n- it neglects to contrast these with recent empirical work on AI interpretability and behavior prediction.", "essential_references_not_discussed": "the paper thoroughly engages with classical causal inference literature but omits more recent developments in causal representation learning within deep learning, such as *Towards Causal Representation Learning (2021)*; \n\nalso, it doesn’t mention recent empirical benchmarks for evaluating AI safety and interpretability which may complement its theoretical contribution, like *RobustBench: a standardized adversarial robustness benchmark (Croce et al., 2020)*", "other_strengths_and_weaknesses": "- without real data experiments or simulations, the practical impact of these bounds remains questionable\n- a major downside is the heavy reliance on assumptions that may not hold in practice, making it hard to see how these bounds could directly inform AI safety\n- as to theorem presentation, the notation is dense and sometimes overwhelming", "other_comments_or_suggestions": "I don't have any other comments", "questions_for_authors": "- might it be more reliable to use the inferred beliefs to predict (or get the bounds on) the agent's behaviour in the in-distribution situation?\n- can you provide any empirical or simulation-based evidence to support your theoretical bounds in a relatively realistic setting? \n- if not, what steps would be needed to translate these bounds into actionable guidelines for AI safety and policy design?\n- what will happen to your framework if the AI system's internal models evolve over time?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eoLETUdRpn", "reviewer": "Reviewer_Zntn", "summary": "The authors study the problem of predicting out of distribution behaviours of AI agents based on data, assuming the AI agents have internal causal models of the world. First, examples are presented for why observing the behaviour of AI agents may not be enough to determine the specific causal model it is using (i.e. inferring the causal model from observations and utilities is an ambiguous problem). Then, authors present formal bounds on the utility gap that the causal agents experience based on the observed data. Finally, they discuss the implications this has for AI safety and overall inferring of preferences in autonomous systems.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review, we appreciate the references to related work and suggestions for clarifying the formalism. \n\n***1. The authors do a very thorough job at relating the work to existing works in causality, but there is barely no discussion on how the work relates to other sub-fields that have studied similar problems (e.g., game theory, inverse reinforcement learning, decision theory).***\n\nThank you for highlighting work in inverse reinforcement learning (IRL) and decision theory. There are several similarities in the broader goals of these different lines of research, e.g., inferring expected utilities from data, but also a few differences that are becoming clearer to us as we contrast the papers mentioned by the reviewer with ours. This deserves a longer discussion but one advantage of the causal formalism (in our view) that we can highlight here is that it gives us tools to make inferences across different environments (including in counter-factual scenarios, e.g., to reason about harm and fairness). Perhaps it is fair to say that our bounds thus more concretely characterize the limits of what can be predicted about agent behaviour *out-of-distribution* which then complements the (in our understanding mostly in-distribution) partial identification results in IRL and decision theory.\n\n***2. It is not clear how the main contribution, although interesting, is of particular relevance to the field or novel when compared to existing subfields (mentioned above). Intuitively there seems to be some novelty when considering such causal world models, and in particular on the specific bounds obtained, but it feels slightly thin.***\n\nWe do believe that our bounds characterize more precisely what can be expected of AI behaviour out-of-distribution. The bounds themselves are significant because they fully describe the set of possible AI behaviours, under our assumptions. This is an important question for AI Safety and there are several implications for the design of AI systems and the extent to which we can trust them (e.g., if the bounds are wide we might want to monitor AI behaviour more closely, or if harmful actions can be ruled out by our bounds that might increase our trust in the technology when deployed).\n\nTo our knowledge, related work, e.g. in IRL, has studied extensively the inference of utilities from data but less so the inference of action choice given a (set of) compatible utilities, which is the contribution we seek to make in our paper. More speculatively, we also believe that there is an interesting cross-discipline opportunity that this paper might encourage: the causal formalism provides a new angle of attack, showing that it is possible to exploit the inductive biases implied by the AI's learned world model to predict their behaviour and beliefs. This approach is novel and potentially fruitful due to the strength of the worst-case guarantees, subject to our assumptions.\n\n***3. Particular comments and suggestions regarding the formalisms used.***\n\nThank you for these. Here we provide a few clarifications and we will update the manuscript accordingly. \n- $Y\\in\\boldsymbol V$ encodes the utility and is a random variable whose assignment is determined by the AI’s world model (structural causal model).\n- $\\phi$ is meant to describe a mapping from an AI’s internal model to a statement over probability of events. With hindsight, this notation was confusing: in the revised manuscript we have replaced the definition of AI beliefs by \"probability statements that are derived from the AI's internal world model\" such as $P^{\\widehat M}(A=a)$ or $P^{\\widehat M_z}(B=b | C=c)$.\n- Yes, a policy is a function from the observables to probability distribution over interventions.\n- We hypothesize that the data of AI interactions we observe, denoted $P_\\pi(\\boldsymbol v)$, is collected while the AI is training or learning, meaning that we can expect some amount of exploration. The policy $\\pi$ is therefore stochastic. However, once the agent is deployed in a new environment, we assume that its choices are driven by expected utilities and are therefore deterministic (though other decision-making models are also possible and would induce variations on how to use the bounds to inform our understanding of AI behaviour).", "claims_and_evidence": "The claims are well supported, and there are no contributions that are left unjustified. The authors are very clear that their main contribution is the derivation of such bounds and the accompanying discussion on the implications.", "methods_and_evaluation_criteria": "There are not really any methods or evaluations. The authors provide some illustrative examples that cover their claims.", "theoretical_claims": "Unless I missed any key details, the proofs and theoretical claims seem correct. I do have to raise that there are imprecisions and misspecified formal statements along the paper that make the judgement of some of these claims difficult (see questions below), but overall the proofs are well written and clear.", "experimental_designs_or_analyses": "The paper has no experiments.", "relation_to_broader_scientific_literature": "The authors do a very thorough job at relating the work to existing works in causality, but there is barely no discussion on how the work relates to other sub-fields that have studied similar problems. The question of whether preferences can be inferred from agent choices and observations is a classic problem in game theory, and more recently has received a lot of attention in reinforcement learning through inverse RL or reward shaping. Furthermore, the conclusion that one cannot fully determine preferences from limited partial observation data has been reached in these subfields before. I would have appreciated some mentions to these connections.", "essential_references_not_discussed": "On reward misspecification, see e.g.:\nFreedman, Rachel, Rohin Shah, and Anca Dragan. \"Choice set misspecification in reward inference.\" arXiv preprint arXiv:2101.07691 (2021).\nSkalse, Joar, and Alessandro Abate. \"Misspecification in inverse reinforcement learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 12. 2023.\n\nOn more classic (and some more recent) game theoretic results studying preference misspecification:\nRichter, Marcel K. \"Revealed preference theory.\" Econometrica: Journal of the Econometric Society (1966): 635-645.\nAfriat, Sydney N. \"The construction of utility functions from expenditure data.\" International economic review 8.1 (1967): 67-77.\nAbbeel, Pieter, and Andrew Y. Ng. \"Apprenticeship learning via inverse reinforcement learning.\" Proceedings of the twenty-first international conference on Machine learning. 2004.\n\n(There are many more, and I'm not suggesting all need to be included or a specific discussion is needed to justify the work, but some mention on the novelty of the contribution given existing preference theory results would be of benefit).", "other_strengths_and_weaknesses": "Strengths:\n- Paper is very well structured, the message is very clear and the examples help illustrating the points.\n- I appreciate work that neatly demonstrates how given systems we are deploying at a large scale may have flaws, and how these flaws can affect their deployment.\n\nWeaknesses:\n- Some formal statements are not clear or feel rushed, see below.\n- There is not really a prescriptive nature to the paper (i.e. even a heuristic on how to improve interpretability of AI agents under causal world models where we only have access to observational data). This is not always needed of course, but the work would be more complete.\n- It is not clear how the main contribution, although interesting, is of particular relevance to the field or novel when compared to existing subfields (mentioned above). Intuitively there seems to be some novelty when considering such causal world models, and in particular on the specific bounds obtained, but it feels slightly thin.", "other_comments_or_suggestions": "I have some particular comments and suggestions regarding the formalisms used. \n- Before equation (1) authors state that $Y$ is a 'potential response', which seems to map latent variables to observables, and that it entails a distribution over the possible outcomes of $Y$ (this is already not very clear). However, in section 4, it is stated that $Y$ is a utility function, and at the same time it seems to be a member of the outcome set ($Y\\in \\mathbf{V}$). It is not clear whether $Y$ is then a set-valued map, a random variable, a function or a set of outcomes.\n- Definition 2 is not very clear. $\\phi$ is not used again, and it is not clear whether it's a functional that just maps to the reals.\n- In Footnote 4, the idea of a policy is introduced, but it's not very clear what a policy means in this context, nor what the 'training domain' is. Is a policy just a function from the observables to probability distribution over interventions?\n- Authors seem to indicate policies are stochastic, but then seem to indicate in eq (3) that AI agents are assumed to act deterministically. Some clarification on this would be good.", "questions_for_authors": "See comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T0n4mFBMzh", "reviewer": "Reviewer_6C3x", "summary": "This paper explores the theoretical limits of predicting AI agent behavior from observational data alone. The authors analyze the extent to which we can infer an agent's beliefs and predict its behavior in novel situations based only on its past behavior. They provide:\n1) A mathematical framework using Structural Causal Models (SCMs) to represent an agent's internal world model and how it relates to observable behavior.\n2) Formal bounds on how well we can predict an agent's preferences and actions out-of-distribution, showing that multiple different internal models can be consistent with the same observable behavior but lead to different actions in new situations.\n3) Theoretical results showing that while some aspects of an agent's beliefs can be bounded based on behavioral data, other aspects (such as counterfactual fairness) remain fundamentally underdetermined.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read and review our paper, we appreciate the questions and suggestions.\n\n***1. Many examples use binary variables (like medical outcomes being 0 or 1), which simplifies the analysis but may not capture the complexity of real-world scenarios with continuous outcomes or multiple categories.***\n\nTo clarify, note that our results hold for systems of (discrete) variables of any dimensionality and arbitrary complexity (in the functions relating different variables and distributions), and continuous outcomes / utility values. There is in principle no additional difficulty to accounting for complex environments, though perhaps the computation of the bounds from finite samples may become more challenging. Our intent with examples using binary variables was to make them easy to describe, though we go agree that more realistic experiments could add to illustrate the results. See also the response to reviewer EHMW for additional context.\n\n***2. The framework primarily addresses single-step decision making rather than sequential decision problems, which limits its applicability to reinforcement learning agents and longer-term planning scenarios.***\n\nThe sequential setting, in principle, offers no additional difficulties: you would have variables and observations indexed by time and the formalism and bounds would be unchanged. More concretely, the preference gap $\\Delta$ at some time $t$ that determines the AI's beliefs that one decision is superior to another one might be written in the sequential setting as:\n\n$\\Delta := \\mathbb E_{\\hat{P}_{\\sigma, d_1^{(t)}}}[Y^{(t)} \\mid \\boldsymbol c^{(t)} ] - \\mathbb E\\_{\\hat{P}\\_{\\sigma, d_0^{(t)}}}[ Y^{(t)} \\mid \\boldsymbol c^{(t)}]$\n\nwhere $\\boldsymbol c^{(t)} = (s^{(t)}, d^{(t-1)}, \\dots, d^{(0)},s^{(0)})$ now includes the agent's trajectory in the deployment environment up to time $t$ (for example). In this setting, we do assume however that there is no learning taking place in the deployment environment, i.e. the AI is not updating its internal model upon interacting with the environment $\\boldsymbol c^{(t)}$. This might be a limitation in practice that would be interesting to relax in future work.\n\n***3. Can you add a limitations section wherein you discuss cases in which SCMs might not be applicable? e.g., practical instances of cyclical dependencies that are assumed away in the definition of SCMs.***\n\nThis is a good suggestion, thank you. We will add to the limitations section in Appendix B.1 a description of the kinds of environments that would not be well captured by SCMs.", "claims_and_evidence": "The paper's claims are generally well-supported through formal mathematical proofs. The authors establish several theorems with bounds on what can be inferred about an agent's internal model from behavioral data alone. These theoretical results are complemented by concrete examples that illustrate the concepts.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem or application at hand.", "theoretical_claims": "I perused the proofs at a high-level but not in detail; they seem correct.", "experimental_designs_or_analyses": "n/a", "relation_to_broader_scientific_literature": "The paper builds on and connects several areas of research:\n- The work extends bounds from classical causal inference (referencing work by Pearl, Robins, Manski, etc.) to the novel context of agent behavior prediction.\n- The paper directly addresses concerns in AI safety about predicting agent behavior out-of-distribution, particularly connecting to work on goal misgeneralization.\n- The authors connect their theoretical framework to emerging evidence that language models may develop internal world models.\n- The paper applies its framework to fairness definitions from the literature, showing fundamental limitations in inferring an agent's notion of fairness from behavior alone.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n- The paper provides a rigorous mathematical framework using Structural Causal Models to analyze agent behavior prediction. \n- The work has clear and important implications for AI safety research, establishing fundamental limits on our ability to predict AI behavior in new situations based solely on observational data.\n- The Medical AI example used throughout the paper effectively illustrates the theoretical concepts\n- The paper explores extensions to its core assumptions, such as approximate grounding, partial observability, and modifications to model structure, making the analysis more robust and applicable to real-world scenarios.\n\nWeaknesses:\n- Many examples use binary variables (like medical outcomes being 0 or 1), which simplifies the analysis but may not capture the complexity of real-world scenarios with continuous outcomes or multiple categories.\n- The framework primarily addresses single-step decision making rather than sequential decision problems, which limits its applicability to reinforcement learning agents and longer-term planning scenarios.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "Can you add a limitations section wherein you discuss cases in which SCMs might not be applicable? e.g., practical instances of cyclical dependencies that are assumed away in the definition of SCMs", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents and analyzes a model in which AI agents' beliefs are inferred from their behaviors. The authors' goal is to shed light on safety properties of AI agents through the language of structural causal models (SCMs). At a high level, the authors provide (partial) identifiability results showing what inferences we can make about an AI agent's SCM based on its behaviors.

The reviewers generally had a positive response to the paper. Some questions were raised as to the applicability of theoretical models here; I tend to be sympathetic towards modeling exercises, and I view these results as providing a framework within which to reason about inferences possible under idealized conditions. On the other hand, it's not clear to me that this framework will ever be directly applicable to real-world AI systems, and it might remain a conceptual tool. Still, the paper appears to be a valuable contribution to the literature.

The reviewers raised a number of questions, which the authors addressed during the rebuttal period. I encourage them to consider the reviewers' comments and suggestions in future revisions.
Decision: Accept (poster)