Measuring Goal-Directedness
Authors: Matt MacDermott
James Fox
Francesco Belardinelli
Tom Everitt
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Causality
Graphical Models
Maximum Causal Entropy
Agency
Primary area: safety_in_machine_learning
TL;DR: We propose a formal measure of goal-directedness in probabalistic graphical models, by adapting and generalising the maximum causal entropy framework.

Abstract: We define maximum entropy goal-directedness (MEG), a formal measure of goal-
directedness in causal models and Markov decision processes, and give algorithms
for computing it. Measuring goal-directedness is important, as it is a critical
element of many concerns about harm from AI. It is also of philosophical interest,
as goal-directedness is a key aspect of agency. MEG is based on an adaptation of
the maximum causal entropy framework used in inverse reinforcement learning. It
can measure goal-directedness with respect to a known utility function, a hypothesis
class of utility functions, or a set of random variables. We prove that MEG satisfies
several desiderata and demonstrate our algorithms with small-scale experiments.

Measuring Goal-Directedness
Matt MacDermott Imperial College London
James Fox
Francesco Belardinelli Imperial College London
University Oxford London Initiative for Safe Al
Tom Everitt Google DeepMind
Abstract
We define maximum entropy goal-directednes (MEG), a formal measure of goal- directedness in causal models and Markov decision processes, and give algorithms for computing it. Measuring goal-directedness is important, as it is a critical element of many concerns about harm from AI. It also of philosophical interest, as goal-directedness is a key aspect of agency. MEG is based on an adaptation of the maximum causal entropy framework used in inverse reinforcement learning. It can measure goal-dine dineedirectedness with respect to known utility function, a hypothesis class of utility functions, or a of random variables. We prove that MEG satisfies several desiderata and demonstrate our algorithms with small-scale experiments |
Introduction
In order to build more useful Al systems, a natural inclination is to try to make them more agentic. But while agents built from language models are touted as the next big advance [Wang al.) 2024). agentic systems have been identified as potential source of individual [Dennett [023]] systemic
[Chan al. 2023 Gabriel al. 2024 and catastrophic [Ngo al. 2022 risks. Agency is thus a key Focus of behavioural evaluations [Shevlane et al. 2023 Phuong et al. 2024] and governance
Shavit et al.
2023 Some prominent researchers have even called for a shift towards designing
explicitly -ggentic systems [Denneti 2017 Bengio 2023).
A critical aspect of agency is the ability to pursue goals. Indeed, the standard theory of agency defines agency as the capacity for intentional action action that can be explained in terms of mental states such goals [Schlosser] 2019). But when we justified in ascribing such mental states? According to Daniel Dennett's s instrumentalist philosophy of mind (1989), we are justified when doing so is useful for predicting system's 's behaviour.
This paper's key contribution a method for formally measuring goal coal-directedness based on Dennett's idea. Since pursuing goals is about having particular causal effect on the environment defining our measure in a causal model is natural. Causal models are general enough to encompass most frameworks popular among ML practitioners, such single-decision prediction, classification, and regression tasks, as well as
(partially observable) Markov decision processes. They also offer enough structure to usefully model many ethics and safety problems [Everitt al. (20
Ward et al. 2024a Richens et al.) 2022 Richens and Everitt 2024 Everitt et al. [002bb] Ward et al.
2024b Hal pern and Kleiman- Weiner 2018 Wachter al. 2017] Kusner al. 2017 Kenton et al.
2023
Fox et al. 2023 Richens and Everitt 2024 MacDermott et al. 2023].
'A notebook that can be used to apply algorithms in your tabular MDPs is available at https //colab. research..ccc..... google. com/ drive//ii/ppp// http://pp...aoooooooeee2//A/aAa.
"This is a common but not universal view in philosophy, see Weirich 2020
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Ua
Up
Predict?
(a) System of interest
(b) Model as causal Bayesian network
(c) Posit decision and utility variables
(d) Measure maximum predictive accuracy
Figure Computing maximum entropy goal-directedness (MEG).
Maximum entropy goal-directedness (MEG) operationalises goal-directedness as follows, illustrated by the subsequent running example.
A variable D in causal model is goal-directed with respect utility function U to the extent that the conditional probability distribution of well-predicted by the hypothesis that D is optimising 44
Example 1. A mouse begins at the centre of gridworld (Figure D It observes that block of cheese is located either to the right or left (S) with equal probability, proceeds either away from it or towards it (D), and thus either obtains the cheese or does not (T).
Suppose that the mouse moves left when the cheese to the left and right when it is to the right, thus consistently obtaining the cheese. Intuitively, this behaviour seems goal-directed, but how much? Figure | gives an overview our procedure. We first model the system of interest as a causal Bayesian network (Fi with variables for the cheese's position, D for the mouse's movement, and 'I' for whether or not the mouse obtains the cheese. Then, we identify a candidate decision variable D and target variable T', hypothesising that the mouse optimíses utility function that depends on T (Figure Te). Finally, we form model of what behaviour we should expect from D it is indeed optimising U and then measure how well this model predicts D's observed behaviour (Figure ldh.
quantify
Ziebart 2010 's maximum causal entropy (MCH) framework suggests a promising way construct a model for expected behaviour under a given utility function. However, there are several obstacles to applying it to our problem: it cannot measure the predictive usefulness of known utility functions, and it only finds the most predictive linear utility function. In practice, arbitrary known utility functions can be plugged in, but the results are not scale-invarint. We overcome these difficulties by returning to first principles and deriving an updated version of the MCE framework.
In summary, contributions four-fold. We (i) adapt the MCE framework to derive maximum entropy goal- dial-directedness (MEG), a philosophically ceaessophicallymmotiviiii motivated measure of goal-directednen with respect to known utility functions, and show that it satisfies several key desiderata (Section 3); (ii) we extend MEG to measure goal-directedness (iii) we provide algorithms for measuring MEG in MDPs (Section 5) and (iv) demonstrate the algorithms empirically (Section (6)
cases without known utility function (Section
Related Work. Inverse reinforcement learning (IRL) Ng and Russell 2000 focuses on the question
interested in what extent the system can be as optimising a goal. We are not the first to ask this question. Some works take a Bavesian approach inspired by Dennett's intentional stance Most closely related to work is Orseau et al. 2001 8). which applies Bayesian IRL in POMDPs using a Solomonoff prior over utility functions and an e-greedy model of behaviour. This lets them infer posterior probability distribution over whether observed system is (goal-directed) "agent" or "just device". Our approach distinguishes itself from these by considering arbitrary variables in a causal model and deriving principle of maximum entropy. Moreover, since our algorithms can take advantage of differentiable classes of utility functions, our approach may be amenable scaling up using deep neural networks. Oesterheld (016)) combines the intentional stance with Bayes' theorem cellular automata, but does consider goal-directedness in
of which goal a system is optimising, whilst
behaviour model from the
consider specific models behaviour. Like us, Kenton al. 2023
After accepted publication, another related paper appeared
Xu and Rivera 2024

---

causal graph. However, they require variables to be manually labelled as mechanisms or object-level, and only provide binary distinction between agentic and non-agentic systems (see Appendix for a detailed comparison). [Biehl and Virgo (2022), , Virgo et al. [2021] propose a definition of agency in Moore machines based on whether a system's 's internal state can be interpreted as beliefs about the hidden states of a POMDP.
2 Background
We use capital letters for random variables V, write dom(V) for their domain (assumed finite), and use lowercase for outcomes E dom(V). Boldface denotes sets of variables V = (Vi..... Vn J. and their outcomes E dom V) = =X, =, do) Parents and descendants of V in graph are denoted by Pay and Descy, respectively (where pay and descy are their instantiations).
Causal Bayesian networks (CBNs) are a class of probabilistic graphical models used to represent causal relationships between random variables [Pearl 2009].
Definition 2.1 (Causal Bayesian network). Bayesian network M (G, P) over a set of variables V --VV,.... Vn consists of a joint probability distribution P which factors according to a directed acyclic graph (DAG) i.e., P(Vi,-... Vn) = III P(V; Pav;). where Pay, are the parents of V, in G. A Bayesian network is causal if its edges represent direct causal relationships, or formally the result of an intervention do X = æ:) for any S V can be computed using the truncated factorisation formula: P(v do(X= = x)) = He Sa P(: pa..) ) if v is consistent with or P( do(X = z)) 0 otherwise.
Figure Tb depicts Example I as a CBN, showing the causal relationships between the location of the cheese (S)), the mouse's ' behavioural response (D), and whether the mouse obtains the cheese (T').
We are interested in to what extent set of random variables in a CBN can be seen as goal-directed. That is, to what extent can we interpret them as decisions optimising a utility function? In other words, we are interested in moving from CBN to causal influence diagram (CID), type of probabilistic graphical model that explicitly identifies decision and utility variables.
Definition 2.2 (Causal Influence Diagram (Everitt et al. 2021a]). A causal influence diagram (CID) M = (G, P) is CBN where the variables V are partitioned into decision D, chance X, and utility variables U. Instead of full joint distribution over V, P consists of conditional probability distributions (CPDs) for each non-decision variable V E V \.D.
CID can be combined with policy n. which specifies CPD TD for each decision variable D, in order to obtain a full joint distribution. We call the sum of the utility variables the utility function and denote it = Eveu U. Policies are evaluated by their total expected utility E [4]. We write miit for the uniformly random policy.
CIDs model a broad class of decision problems, including Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs) [Everitt et al. 20216).
Example 2 (POMDP). A mouse begins at the centre of the grid, with block of cheese located either at the far left or the far right (Figure The mouse does not know its position or the position of the cheese (S1) but can smell which direction the cheese in (O₂) and decide which way to move (D1). Next step, the mouse again smells the direction of the cheese (O2) and again chooses which way to proceed (Dz).
3 Measuring goal-directedness with respect to known utility function
Maximum Entropy Goal-directedne. Dennett's instrumentalist approach to agency says that we can ascribe mental states (such as utilities) to a system to the extent that doing so is useful for predicting its behaviour [Dennett 1989]. To operationalise this, we need a model of what behaviour predicted by utility function. According to the principle of maximum entropy [Jaynes] 1957). we should choose a probability distribution with the highest entropy distribution satisfying our requirements, thus minimising unnecessary assumptions (following Oceam's razor). We can measure the entropy of a policy by the expected entropy of its decision variables conditional on their parents H Pan) EDED E.g.aa--P log D Pap). This Ziebart et al. 2010 's causal entropy, which we usually refer to just the entropy of
3

---

D
D2
D2
S
Ss
SB
U
U22
Us
(a) Gridworld Figure 2: Sequential multi-decision mouse example.
(b) Causal Bayesian network
(c) Causal influence diagram
In our setting, the relevant constraint is expected utility. To avoid assuming that only optimal agents are goal-directed, we construct set of models of behaviour which covers all levels competence an agent optimising utility U could have. We define the set of attainable expected utilities in a CID as att(U) {u E R JT E II (E, [4] =u)
Definition 3.1 (Maximum entropy policy set, known utility function). Let M (G, P) be a CID with decision variables D and utility function U. The maximum entropy policy set for u E att(U) is (4,0 := argmaxx [4 H (D Pan). The maximum entropy policy set for U is the set of maximum entropy policies for any attainable expected utility IImasem = Uueatc(1)) 11masent
For each attainable expected utility, 1 contains the highest entropy policy which attains it. In MDPs, this policy is unique "H,,
masent and can be found with backwards induction (see Section 5
We measure predictive accuracy using cross-entropy, as common in ML. We subtract the predictive accuracy of the uniform distribution, so that we measure predictive accuracy relative to random chance. This makes MEG always non-negative.
Definition 3.2 (Maximum entropy goal-directedd dial-directedssss, known utility function), Let M = PG, be a CID with decision variables D and utility function U. The maximum entropy goal-directedness (MEG) of policy æ with respect to lI is
MEGu (T) ==
max TIrore E TIREE
E
Σ DED
log π maxent (D) Pap) - log [dom (D))
As will see, in a Markov Decision Process, this is equivalent to maximising predictive accuracy by varying the rationality parameter 3 in the 'soft-optimal' policy n used in maximum entropy reinforcement learning [Haamooj al. 2017
If instead of having access to a policy w. we have access set of full trajectories (paa,, Di...,, pap; D,)),, the expectation Ex, in Equation (l) can be replaced with an aver- age over the trajectory set. This is an unbiased and consistent estimate of MEGu(T) for the policy generating the trajectories.
Example. Consider policy in Example that proceeds towards the cheese with probability 0.8. How coal-directed is this policy with respect the utility function 24 that gives +1 for obtaining the cheese and - otherwise?
To compute (T), we first find the maximum entropy policy I Imaaxent and then take the maximum predictive accuracy with respect to π. In single-decision setting, for each attainable expected utility u there is a unique muxent which has the form Boltzmann policy m (d
exp( [[(]dd) s) Eg exp(3-E[Cja' [=]] (cf. Theorem 5.1 The rationality parameter B B(u) be varied get the correct expected utility. Predictive accuracy with respect to is maximised by T which has a rationality parameter of B = log 2. The expected logprob of a prediction of this policy is
'A noteworthy property of Definition 3.2 is that a policy that pessimises a utility function can be just as goal-directed as policy that optimises it. This fits with the intuitive notion that we are measuring how likely policy's 's performance
a particular utility function is be accident. However, might be argued that a policy should instead be considered negatively goal- directed with respect to utility function it pessimises. We could adjust Definition 3.2 to achieve this by multiplying by the sign of E., [2]] _n [24], making the coal-directedness of any policy that performs worse than random negative. For simplicity. we use the unsigned measure in this paper.

---

En log (D
"t,,08 (D Pap)] -0500, while the expected logprob of a uniform prediction log(5)
-0.69. So get that (T) -.50 (-0.69) 0.19. For comparison, predictive accuracy for the optimal policy x" is maximised when / = oc, and has MEGu((T) = =0 - (-0.69) = 0.69.
Properties. We now show that MEG satisfies three important desiderata. First, since utility functions are usually only defined up to translation and rescaling, a measure of goal-directedness with respect to utility function should be translation and scale invariant. MEG satisfies this property:
Proposition 3.1 (Translation and scale invariance). Let M1 be CID with utility function U1, and let M2 be an identical CID but with utility function U2 = a U1 + b, for some a, a.b R, with of Then for any policy т, MEGu, (T) = MEGu,,
Second, goal cial-directedness should be minimal when actions are chosen completely at random and maximal when uniquely optimal actions are chosen.
Proposition 3.2 (Bounds). Let M be a CID with utility function U. Then for any policy a we have 0 S (T) < DED log([ dom (D))). with equality in the lower bound if is the uniform policy, and equality in the upper bound if π is the unique optimal (or anti-optimal) policy with respect to U.
Note that MEG has a natural interpretation as the amount of evidence provided for a goal-directed policy over purely random policy. The larger a decision problem, the more opportunity there is to see this evidence, so the higher MEG can be.
Third, a system can never be goal-directed towards utility function it cannot affect.
Proposition 3.3 (No goal- corl-directsdnnss without causal influence). Let M = (G. P) be a CID with utility function U and decision variables D such that, Desc (D) Pau 0. Then MEGu (D) 0.
Comparison to MCE IRL Our method is closely related to MCE IRL Ziebart et al. 2010 (see Gleave and Toyer 2022 for useful primer). In this subsection, we discuss the key similarities and differences.
The MCE IRL method seeks to find utility function that explains the policy π. It starts by identifying set of linear features fi and seeks model policy that imitates π as far as these features are concerned but otherwise is as random as possible. It thus applies the principle of maximum entropy with n linear constraints. The form of the model policy involves a weighted sum of these features. In a single-decision example, it takes the form
exp (E
Σ
wifs d, s))
π MCF (d s) = Ex exp
(2)
(E [ d', s))
The weights wy are interpreted as a utility function over the features fi. MCE IRL can, therefore, only return a linear utility function.
In contrast, our method seeks to measure the goal-directedness of with respect to an arbitrary utility function U, linear or otherwise. Rather than constructing a single maximum entropy policy with n linear constraints, construct a class of maximum entropy policies, each with different single constraint on the expected utility.
A naive alternative to defining the coal-directedness of with respect to l as the maximum predictive accuracy across L/'s ss maximum policy set, we could simply plug in our utility function ii to n MCE from Equation (2). and use that to measure predictive accuracy. If l/ is linear in the features f., we could substitute in the appropriate weights, but even if not, we could still replace >. w;ft with 24 Indeed, this is often done with nonlinear utility functions in deep MCE IRL Wulffeeirr et al. 2015).
However, this would not have formal justification, and we would run into a problem: scale non- invariance. Plugging in 2 21 would result in more sharply peaked MMC than 21; in Example | we would get that the mouse is more goal-directed towards 2 li than 24 with a predictive accuracy (measured by negative cross-entropy of -0.018 vs -0.13, In contrast, constructing separate maximum entropy policies for each expected utility automatically handles this issue. The policy in ITmuxent which maximises predictive accuracy for æ has an inversely scaled rationality parameter 3 = 2 compared to the maximally predictive policy in IImaxcat In other words, they are the same policy, and get that (T) (EG2u(T) = 0.19 (cf. Proposition 3.1
5

---

Measuring goal-directedness without a known utility function
In many cases where we want to apply MEG, we may not know exactly what utility function the system could optimising. For example, we might suspect that content recommender trying to influence a user's preferences, but may not have a clear hypothesis to in what way. Therefore, in this section, we extend our definitions for measuring goal-directedness to the case where the utility function is unknown.
We first extend CIDs to include multiple possible utility functions.
Definition 4.1. A parametric-utiltty CID (CID) M is a set of CIDs (M* E O which differ only in the CPDs of their utility variables.
In effect, parametric CID is CID with a parametric class of utility functions 4
The maximum entropy policy set from Definition 3.1 is extended accordingly to include maximum entropy policies for each utility function and each attainable expected utility with respect to it.
Definition 4.2 (Maximum entropy policy set, unknown utility function), Let A = (G, P) be parametric-utility CID with decision variables D and utility function 21 The maximum entropy policy set for 2 is the set of maximum entropy policies for any attainable expected utility for any utility function in the class: Ey = USES, Immeent
Definition 4.3 (MEG, unknown utility function). Let M = (G, P) be parametric-utility CID) with decision variables D and utility function _ The maximum entropy goal-directedness of a policy π with respect to 2 is MEGye (T) max MEGu(T).
Definition 4.4 (MEG, target variables). Let M = =G, P) be a CBN with variables V. Let C V be a hypothesised set of decision variables and T C V be hypothesised set of target variables. The maximum entropy coal-directedness
D with respect to T, MEGr(D), is the goal-directedness of P(D) Pap) in the parametric CID with decisions D and utility functions U dom(T) R (i.c., the set all utility functions over T').
For example, if we only suspected that the mouse in Example I was optimising some function of the cheese T. but didn't 't know which one, could apply Definition 4.4 to consider the goal-directedness towards T under any utility function defined on T. Thanks to translation and scale invariance (Proposition B.I), there are effectively only three utility functions to consider: those that provide higher utility to cheese than not cheese, those that do the opposite, and those that are indifferent.
Note that T has include some descendants of D, in order enable positive MEG (Proposition 3.3 However, it is not necessary that consists of only descendants D (i.e., T need not be a subset of Desc(D)). For example, coal-conditional agents take an instruction as part of their input Pap. The goal-directedness of such agents can only be fully appreciated by including the instruction in T.
Pseudo-terminal goals. Definition 4.4 enable to state result about special kind of instrumental goal. It is well known that an agent that optimises some variable has an instrumental incentive to control any variables which mediate between the two [Everitt al.) 2021a] However, since the agent might want the mediators to take different values in different circumstances, it need not appear goal-directed with respect to the mediators. Theorem 4.1 shows that in the special case where the mediators d-separate the decision from the downstream variable, the decision appears at least as goal-directed with respect to the mediators with respect the target.
Theorem 4.1 (Pseudo-terminal goals). Let M = ((V, E), be a CBN. Let D,T,S such that D 1 T S. Then (EG < MEGs(D),
For example, in Figure 2 the agent must be at least as goal-directed towards Sy as it is towards Us. since S3 blocks all paths (i.e. d-separates) from (D1, D2} to Uz.
Intuitively, this comes about because, in such cases, rational agent wants the mediators to take the same values regardless circumstances, making the instrumental control incentive indistinguishable from terminal goal. This means we do not have to look arbitrarily far into the future to find evidence of goal-directedness. An agent that dire-did with respect to next week must also be goal-directed with respect to tomorrow.

---

maxent for Computing MEG in Markov Decision Processes
In this section, give algorithms for computing MEG in MDPs. First, define what an MDP looks like as a CID. We then establish that soft value iteration can be used to construct our maximum entropy policies, and give algorithms for computing MEG when the utility function is known or unknown.
Note that in order to run these algorithms, we do not need an explicit causal model, and so we do not have to worry about hidden confounders. We do, however, need black-box access to the environment dynamics, i.e., the ability to run different policies in the environment and measure whatever variables we are considering utility functions over.
Definition 5,1. A Markov Decision Process (MDP) is a CID with variables {St, Di, U,][ decisions D {Dr and utilities U [U:]r such that for between and n. Pap, {S(). Pay. {St), while Pas, = (St-1, D for t >> 1, and Pas, 0.
Constructing Maximum Entropy Policies In MDPs, Ziebart's soft value iteration algorithm can be used to construct maximum entropy policies satisfying a set of linear constraints. We apply it to construct maximum entropy policies satisfying expected utility constraints.
Definition 5.2 (Soft Q-Function). Let M = (G, P) be an MDP. Let / E IR \ (0). For each Dr D we define the soft Q-function 3,n dom (D) x dom (Pa.) + R via the recursion:
p,e pa,) IE
U2 logsumexp(B (p Pap...)) des pa
for t n,
3
Pi (dn pa..) = E [Un dn, pa,l where logsumexp ((Q Pan, )) = log Edd- redomedd exp($Quatt 8. (de+1 Papp.)))
Using the soft f-fucttion, we show that there is a unique π E IImaxen for each U and u in MDPs. Theorem 5.1 (Maximum entropy policy in MDPs). Let M (G, P) be an MDP with utility function U, and let u E att(U) be an attainable expected utility. Then there exists unique maximun entropy policy n maxent E 24, and it has the form
exp(3-Q3. 2. (delpa.) (2. Cdmct exp(3-Qu!) (d: (a,) n unit (de pa,),
if # 0
maxent (de pp,,) T
33, - pa,)
(3)
if = 0.
where B = argmaxgreBuuoo..oo
E, Exr
[lgg( ( pq.))
We refer to policy of the form of π as a soft-optimal policy with a rationality parameter of 3.
Known Utility Function in a CID M with respect to a utility function 44 we need to find the maximum entropy policy in I IImaxent which best predicts We can use Theorem 5.1 to derive algorithm that finds m. any u E (()).
To apply Definition 3.1 measure the dineetireesssssss of a policy
Fortunately, we do not need to consider each policy in Ngaaeet individually. We know the form of na muxent and only the real-valued rationality parameter varies depending
The gradient of the predictive accuracy with respect to B
log T 20 (D) Pap) log dDm(D) 1)
)]
V
V
Σ DED
= Eyg [4]
E Bygge M
The predictive accuracy is a concave function of 3, so we can apply gradient ascent to find the global maximum in B, which is the same finding the maximum in
MEG (I)) can therefore found by alternating between applying the soft value iteration of Defini- tion 5.2 to find xr maxent, computing Ex [4] Exnn [M]] and taking gradient step. See Algorithm
If the 3 that maximises predictive accuracy is oo or -x,, which can happen if n is optimal or anti-optimal with respect to 24, then the algorithm can never reach the (nonetheless finite) value of (T), but will still converge in the limit.
7

---

Algorithm Known-utility MEG in MDPs
Input: MDP M, policy
Output: MEGu(T)
I: initialise rationality parameter 3, set learning rate
2: repeat
3:
Apply soft value iteration find Q3
Definition 5.2
4:
"
noll <- softmax -2
5:
Ex 1 Eeze U1)
6:
B
7:
until Convergence
8: return E EDED
log n
(D) Pap) - log dom(D))
Unknown-utility algorithm To find unknown-utiligy MEG, we consider soft-optimal policies for various utility functions U/ and maximise predictive accuracy with respect to l as well as B. Let 2 be a differentiable class of functions, such as a neural network, and write Tg, for a soft-optimal policy with respect to U E 2º and Q for the corresponding soft Q-function. We can take the derivative of the predictive accuracy with respect and get E,, [Vold] Eggnen [Vold]. Algorithm extends Algorithm] 1 to this case.
soft
Algorithm 2 Unknown-utility MEG in MDPs
Input: Parametric MDP Me over differentiable class of utility functions, policy Output: MEGu (r)
1: Initialise utility parameter 0, rationality parameter B, set learning rate
2: repeat
3: Apply soft value iteration to find Q
Definition 5.2
4: #0.5
soft +- softmax (3 Q
5:
93 - En 1 - E Ey (209
6:
go < Er [ - E En [Valt"]
7:
3 - B 1+00 a gg
8:
8 + 8
9: until Stopping condition
return E, EDED log 0,
soft
Pap) log [dom(D)
An important caveat is that Uf is non-convex function of 0 (c.g. neural network), Algorithm need not converge to a global maximum. In general, the algorithm provides a lower bound for MEGu, (T), and hence for MEGr(1) where T Payee In practice, we may want estimate the soft Q-function and expected utilities with Monte Carlo or variational methods, in which case the algorithm provides approximate lower bound on dial-direcsed
6 Experimental Evaluation
We carried out two experiments to measure known-utility MEG with respect to the environment reward function and unknown-utility MEG with respect a hypothesis class of utility functions. We used an MIP with single hidden layer of size 256 to define a utility function over states.
5
Our experiments measured MEG for various policies in the CliffWorld environment from the seals suite [Gleave et al. 2020). Cliffworld (Figure 3a) is a 4x10 gridworld MDP in which the agent starts in the top left corner and aims to reach the top right while avoiding the cliff along the top row. With probability 0.3, wind causes the agent to move upwards by one more square than intended. The
*Code available httpt::////
otw.:/aess.s........ooaoddeeeeee directedness

---

50
known unliy unkoren-miin
.-- known unmnonn-ulllt
MLLG e-geady palicy
MEG or optima polec
10
10
0.1
0.2
033
0.1
0.5
0.6
0.7
08
0.9
Lengh of goal reiion (squsee)
(a) CliflWorld
(b) MEG vs Optimality of policy
(c) MEG vs Task difficulty
Figure (a) The CliffWorld environment. (b) MEG of e-greedy policies for varying s. MEG decreases as the policy gets less optimal. (c) MEG for optimal policies for various reward functions. Known-utility MEG decreases as the goal gets easier satisfy, but unknown-utility MEG stays higher because the optimal policies also do well with respect to narrower goal.
environment reward function gives +10 when the agent is in the (yellow) goal square, -10 for the (dark blue) cliff squares, and -1 elsewhere. The dotted yellow line indicates length goal region.
MEG vs Optimality of policy In our first experiment, we measured the goal-directedness of policies of varying degrees of optimality by considering E-greedy policies for e in the range 0.1 to 0.9. Figure 3b shows known- and unknown utility MEG for each policy Predictably, the goal- directedness with respect the environment reward decreased toward as the policy became less optimal. So did unknown-utility MEG since, increases, the policy becomes increasingly uniform, does not appear goal-directed with respect to any utility function over states.
MEG vs Task difficulty In our second experiment, we measured the goul-directedness of optimal policies for reward functions of varying difficulty. We extended the goal region of Cliffworld to run for either 1, 2, 3 or 4 squares along the top row and back column and considered optimal policies for each reward function. FigureBe shows Cliffworld with a goal region of length Figure shows the results. Goal-directedness with respect to the true reward function decreased as the task became easier to complete. A way to interpret this is that as the number policies which do well on a reward function increases, doing well that reward function provides less and less evidence for deliberate optimisation. contrast, unknown-utility MEG stays high even as the environment reward becomes easier to satisfy. This is because the optimal policy proceeds towards the nearest -qul-squares and, hence, it appears strongly goal goal-directed with respect to a utility function which gives high reward to only those squares. Since this narrower utility function is more difficult to do well on than the environment reward function, doing well on it provides more evidence for goal-directedness. In Appendix D333 visualise the policies in question make this more explicit. We also give tables of results for both experiments.
Limitations
Environment Access Although computing MEG does not require explicit causal model (cf. Section 5), it does require the ability to run various policies in the environment of interest - we cannot compute MEG purely from observational data.
Intractability While MEG can computed with gradient descent, doing so may well com- putationally intractable in high dimensional settings. In this paper, we conduct only preliminary experiments larger experiments based on real-world data may explore how serious these limitations in practice.
Choice of variables directedness with respect to. At one extreme, all (deterministic) policies have maximal goal-
MEG is highly dependent on which variables we choose to measure goal-
"uiliiy MEG is deterministic. Likknown-utllity MEG depends on the random initialisation of the neural network, show the mean of several runs. Full details given in Appendix D.3

---

directedness with respect to their own actions (given an expressive enough class of utility functions). This means that, for example, would not be very insightful compute the coal-directedness of a language model with respect to the text it outputs goal-directed towards some variable T that we do not think to measure, MEG may be misleadingly low. Relatedly, MEG may also be affected by whether we use a binary variable for T (or the decisions D) rather than a fine-grained one with many possible outcomes. We should, therefore, think of MEG as measuring what evidence set of variables provides about a policy's goal-directedness. A lack of evidence does not necessarily mean a lack of goal-directedness.
At the other extreme, if policy is highly
Distributional shifts MEG measures how predictive a utility function is of a system's behaviour on distribution, and distributional shifts can lead to changes in MEG. A consequence of this is that two almost identical policies can be given arbitrarily different goal-directedness scores, for example, if they take different actions in the start state of an MDP and thus spend time in different regions of system's behaviour under interventions, as Kenton et al. 2023) do, we can distinguish "true" goals from spurious goals, where the former predict behaviour well across distributional shifts, while the latter happen to predict behaviour well on a particular distribution (perhaps because they correlate with true goals) (Di Langosco et al. 2022) We leave this future work.
the state space. It may be that by considering changes
Behavioural vs mechanistic approaches The latter points suggest that mechanistic approaches to understanding a system's goul-directedness could have advantages over behavioural approaches like MEG. For example, suppose we could reverse engineer the algorithms learned by neural network. In that case, it may be possible to infer which variables the system is goal-directed with respect to and do in way which is robust to distributional shifts. However, mechanistic interpretability of neural networks [Elhage et al.) [021] is an ambitious research agenda that is still in its infancy, and for now, behavioural approaches appear more tractable.
Societal implications An empirical measure of goal-directedness could enable researchers and companies to keep better track of how goal-directed LLMs and other systems are. This is important for dangerous capability evaluations (Shevlane et al. [023 Phuung et al.) [2024) and governance (Shavit al.) 2023]. potential downside is that could enable bad actors to create more dangerous systems by optimising for goal-directedness. We judge that this does not contribute much additional
risk, given that bad actors can already optimise system to pursue actively harmful goals.
Conclusion
We proposed maximum entropy goal-directedness (MEG), a formal measure of coul-directedness in CIDs and CBNs, grounded in the philosophical literature and the maximum causal entropy principle. Developing such measures is important because many risks associated with advanced artificial intelligence come from goal-directed behaviour. We proved that MEG satisfies several key desiderata, including scale invariance and being zero with respect to variables that can't 't be influenced, and that it gives insights about instrumental goals. On the practical side, we adapted algorithms from the maximum causal entropy framework for inverse reinforcement learning to measure goal-directedness with respect to nonlinear utility functions in MDPs. The algorithms handle both a single utility function and a differentiable class of utility functions. The algorithms were used in some small-scale experiments measuring the direetedeeessssss of various policies in MDPs. In future work, we plan to develop MEG's 's practical applications further. In particular, we hope to apply MEG to neural network interpretability by measuring the goal-direetedness of a neural network agent with respect to a hypothesis class of utility functions taking the network's 's hidden states as input, thus taking more than just the system's behaviour into account.
7 Any proposed way of measuring goal-directedness needs way to avoid the trivial result where all policies are seen to be maximising an idiosyncratic utility function that 'overfits' to that policy's 's behaviour. MEG avoids this by allowing us to measure goal- di-recteddeess with respect to set of variables that excludes the system's actions (for example, states in an MDP). An alternative approach is to penalise more complex utility functions.
10

---

Acknowledgements
The authors would like to thank Ryan Carey, David Hyland, Laurent Orseau, Francis Rhys Ward, and several anonymous reviewers for useful feedback. This research is supported by the UKRI Centre for Doctoral Training Safe and Trusted AI (EP/S023356/1, and by the EPSRC grant "An Abstraction-based Technique for Safe Reinforcement Learning" (EP/X015823/1), Fox acknowledges the support of the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems (EP/S024050/1).
References
Yoshua Bengio. AI Scientists: Safe and Useful AI? http:://yoouuaaagr..ooo///////// si-scienttists-saffe and-useuul-ai/ 2023.
Martin Bichl and Nathaniel Virgo. Interpreting systems as solving pomdps: step towards formal understanding of agency. International Workshop on Active Inference, pages 16-31. Springer, 2022.
Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krashenin- nikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, et al. Harms from increasingly agentic algorithmic systems. In Proceedings of the 2023 ACM Conference on Fairness, Account- ability, and Transparency, pages 651-666, 2023.
Daniel C Dennett. The intentional stance. MIT press, 1989.
Daniel C Dennett. From bacteria to Bach and back: The evolution of minds. WW Norton & Company, 2017.
Daniel C Dennett. The problem with counterfeit people. The Atlantic, 16, 2023.
Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger, Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning, pages 12004-12019. PMLR, 2022.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
Tom Everitt, Ryan Carey, Eric D Langlois, Pedro A Ortega, and Shane Legg. Agent incentives: A causal perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11487-11495, 2021a.
Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 198 (Suppl 27):6435-6466) 2021b.
James Fox, Matt MacDermott, Lewis Hammond, Paul Harrenstein, Alessandro Abate, and Michael Wooldridge. On imperfect recall in multi-agent influence diagrams. In Proceedings Nineteenth conference on Theoretical Aspects of Rationality and Knowledge,, volume 379 of Electronic Proceedings Theoretical Computer Science, pages 201-220, 2023.
lason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, et al. The ethics of advanced AI assistants. arXiv preprint arXiv:2404.1 16244. 2024.
Adam Gleave and Sam Toyer. A primer on maximum causal entropy inverse reinforcement learning. 2022.
Adam Gleave, Pedro Freire, Steven Wang, and Sam Toyer. seals: Suite of environments for algorithms that learn specifications. hhttps://githul: httt:://gggtabbb.ccmm//aaatttttp cww.////aaaaatootiiiiaaaaaaaaaas 2020.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352-1361. PMLR, 2017.
11

---

Joseph Halpern and Max Kleiman- Weiner. Towards formal definitions of blameworthiness, inten- tion, and moral responsibility. In Proceedings of the AAAI conference on artificial intelligence. volume 32, 2018.
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDermott, and Tom Everitt. Discovering agents. Artificial Intelligence, page 103963, 2023.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
Matt MacDermott, Tom Everitt, and Francesco Belardinelli. Characterising decision theories with mechanised causal graphs. arXiv preprint arXiv:2307.10987, 2023.
Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. Proc. of 17th International Conference Machine Learning. 2000, pages 663-670 2000.
Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from deep learning perspective. The Twelfth International Conference on Learning Representations, 2022.
Caspar Oesterheld. Formalizing preference utilitarianism in physical world models. Synthese, 193(9): 2747-2759, 2016.
Laurent Orseau, Simon McGregor McGill, and Shane Legg. Agents and devices: A relative definition of agency. arXiv preprint a8Xiv: 12387, 2018.
Judea Pearl. Causality. Cambridge university press, 2009.
Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, al. Evaluating frontier models for dangerous capabilities. arXiv preprint arXiv:2403.1379) 2024.
Jonathan Richens and Tom Everitt. Robust agents learn causal world models. The Twelfth Interna- fional Conference on Learning Representations, 2024.
Jonathan Richens, Rory Beard, and Daniel H Thompson. Counterfactual harm. Advances in Neural Information Processing Systemx, 35:36350-36365, 2022.
Markus Schlosser. Agency. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2019 edition, 2019.
Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O'Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkin, Tyna Eloundou, Alan Hickey, et al. Practices for governing agentic AI systems. Research Paper, OpenAl, December, 2023.
Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305. 150:22305..324.. 2023.
Nathaniel Virgo, Martin Biehl, and Simon McGregor. Interpreting dynamical systems as bayesian rea- soners. In Joint European conference machine learning and knowledge discovery in databases, pages 726-762. Springer, 2021.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. JL. & Tech., 31:841, 2017.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):1 2024.
Francis Ward, Francesca Toni, Francesco Belardinelli, and Tom Everitt. Honesty is the best policy: defining and mitigating AI deception. Advances in Neural Information Processing Systems, 36, 2024a.
12

---

Francis Rhys Ward, Matt MacDermott, Francesco Belardinelli. Francesca Toni, and Tom Everitt. The reasons that agents act: Intention and instrumental goals. In Proceedings of the 23rd International Conference on Aulonomous Agents and Multiagent Systems, pages 1901-1909, 2024b.
Paul Weirich. Causal Decision Theory. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2020 edition, 2020.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning. arXiv preprint arXiv: 04888, 2015.
Dylan Xu and Juan-Pable Rivera. Towards measuring goal-directedness AI systems. arXiv preprint arXiv:2410.04666 2024.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
Brian D Ziebart, Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the 27th International Conference International Conference on Machine Learning, pages 1255 1262, 2010.
13

---

D
(a) CBN
(b) Mechanised CBN
(c) Mechanised CTD
Figure 4: Example can be equally well represented with a CBN (a) or mechanised CBN (b), but Kenton et al. [2023] s algorithm only identifies an agent in (b). (c) shows the resulting mechanised CID. In contrast, MEG is unchanged between (b) and (c). Note also that the causal discovery algorithm identifies as a utility variable, where where MEG adds new utility child to
A Comparison to Discovering Agents
This paper is inspired by Kenton et al. [202]], who proposed causal discovery algorithm for identifying agents in causal models, inspired by Daniel Dennett's view of agents systems "moved by reasons" [Dennett 1989]. Our approach has several advantages over theirs, which we enumerate below.
Mechanism variables, Kenton al. 2023 assume access to mechanised structural causal model, which augments an ordinary causal model with mechanism variables which parameterise distributions of ordinary object-level variables. An agent is defined as system that adapts to changes in the mechanism of its environment. However, the question of what makes a variable mechanism is left undefined, and indeed, the same causal model can be expressed either with without mechanism variables, leading their algorithm to give a different answer. For example, Example has identical causal structure the mouse example in Kenton et al. [2023], but without any variables designated as mechanisms. Their algorithm says the version with mechanism variables contains an agent while the version without does not, despite them being essentially the same causal model. Figure A shows our example depicted our definition in ordinary causal Bayesian networks.
mechanised structural causal model. We fix this arbitrariness by making
Utility variables. Their algorithm assumes that some variables in the model represent agents' utilities. We bring this more in line with the philosophical motivation by treating utilities as hypothesised mental states with which we augment our model.
Predictive accuracy. Kenton et al. 2023 's approach formalises Dennett's idea of agents systems "moved by reasons". We build on this idea but bring it more in line with Dennett's notion of what it means for system be moved by reason that the reason is useful for predicting its behaviour.
Gradualist vs Essentialist. The predictive error viewpoint gives us a continuous measure of goal- directedness rather than binary notion of agency, which is more befitting of the gradualist view of agents which inspired it.
Practicality. Their algorithm is theoretical rather than something that can be applied in practice. Instead, ours straightforward to implement, as we demonstrate in Section 6 This opens up a range of potential applications, including behavioural evaluations and interpretability of ML models.
Interventional distributions. The primary drawback of MEG is that it doesn't 't necessarily generalise outside of the distribution. Running MEG on interventional distributions may fix this. We leave an extension of MEG to interventional distributions for future work.
B Proofs of MEG Properties
Proposition 3.1 (Translation and scale invariance). Let M1 be a CID with utility function U1 and let M2 be an identical CID but with utility function U2 = a Ui + b, for some a, E, with o Then for any policy т. MEG (T) = MEGus(T)
Proof. Since MEG is defined by maximising over maximum entropy policy set, showing that two utility functions have the same maximum entropy policy set enough to show that they result the same MEG for every policy.
14

---

I E Ii then E U..us for some E att(((1,,, i.e. is maximum entropy policy satisfying
the constraint that E- [4] u1. thus also satisfies the constraint that lE, [122] = 12 = u1 + b.
Note that any policy satistying E [Uz]] = u2 also satisfies E [4]] = in because the map + a-
is injective (since + 0). Thus must also be maximum entropy policy satisfying E,, [2/2] u2.
and so 42:2g and thus क E 7/2
[1maaeen
The converse is similar.
Proposition 3.2 (Bounds). Let M be CID with utility function U. Then for any policy a have O S (EGu(() < EpeD log( dom (D)). with equality in the lower bound if π is the uniform policy, and equality in the upper bound πt the unique optimal (or anti-optimal) policy with respect to U.
Proof. Recall that
(EGu(T)
max
ES
Σ DED
log n maxent (D Pan) -log dom D)
TLAGTT Ella
For the lower bound, first note that the uniform policy x unil is always element of Ti since it the maximum entropy way to attain utility of Eqgen [27]. follows that
MEGu (T) Z
Σ
log x unil (D) Pap) log 1
(4)
dom D)
DED
F
log
log
(5)
dom(D))
dom [)
DED
0,
(6)
which shows both the lower bound and the fact that we attain it when unit
For the upper bound, note that the log n maxent D
Pan) term is at most 0,
1
MEGu(T) <EE
Σ
0 log
(7)
dom (D)
DED
Σ DED
log( dom (D))
(8)
To see that x being a unique optimal (or anti-optimal) policy is sufficient condition for attaining the upper bound, note that there always exists a deterministic optimal and anti-optimal policy in a CID, so unique such policy must be deterministic. Further, π must be in 1mmaxem since there is no higher entropy way to get the same expected utility. Then since Eg [log makent (D Pap)] choosing obtains the upper bound.
q maxent
Proposition 3.3 (No goal-directedness without causal influence). Lei M = (G. be CID with utility function LA and decision variables D such that, Desc D) Pay = 0. Then MFGu(D)
Proof. Since U/ is not a descendant of D, it follows from the Markov property of causal Bayesian networks that / Pap. That means policies achieve the same expected utility u. So, the maximum entropy policy set [[maxent contains only the uniform policy. We get that
1
(EG --E
Σ log dom(D) DED
log
0.
dom( D)
15

---

Theorem 4.1 (Pseudo-terminal goals). Let M = (V. be a CBN. Let D,T,S S V such that D T S. Then MEGr(D) I MEGg(D),
Proof. We will show that the maximum entropy policy set 4 (where u is the set of all utility functions over T') is subset of 114s so the maximum predictive accuracy taken over the latter upper bounds the maximum predictive accuracy taken over the former.
can find utility function 4 us such that for all x, Ex [ = Eg [ then maximum entropy policy with Eg [ = u must
Let E [[maxem so π zz
for some 4t E u²
also be maximum entropy policy with E, [
It would follow that π E I and so
IImarennt C I
To construct such utility function, let (x) = P(T
P(T = t D T S, P(T = t S 8) is not a function of æ. Then for any n.
s)(F(()) Note that since
Ex 1 T
Σ (z(
P+((8) Σ (P((t1))u
P+(s)) Σ P ())
(since D T T S)
Σ (s(s)((((()
Ex [us].
C Proof of Theorem 5.1
Lemma C.I. Let n44 denote soft-optimal with respect utility function U, defined in Theo-
rem 5.
with rationality parameter B. Then for any a E R, we have that To-3. = BB
Proof. If either or 8 are equal to 0, both policies are uniform and we are done. Otherwise, write Gal for the soft-Q function for with rationality parameter B. We first show that Q Quen by backwards induction.
First, Quoft Q,,a-14, = E la Un d.. pa.] = aF (Un dn, pa.] = a. Quait @e...um
And for n, assuming the inductive hypothesis holds for + 1,
Quoit pa,) = U, +
logsumexp(B Quuusti Pan )) de pa
Up + a
logsumexp(B Sooft Qa..uu PaD )) de pa,+1
E
U² + a.9
logsumexp(a 3 Q
Pa de pa
a sofft 3.
Then, substituting this into the definition of 70- get
16

---

33,00l softmax (3 Q 3,
softmax (3 (a))
softmax (a 8 Qa-8.u)
Ta-
Theorem 5.1 (Maximum entropy policy in MDPS), Let M = (G, , P) be an MDP with utility, function U, and let at((U) be an attainable expected utility. Then there exists unique maximum entropy policy n Te E U,a
maxcut
Imasent and it has the form
exp(8-Q2. (d.(pa.)
if 0
(d, pp,)) a
muxent
33, - pa,)
Carcd d.) exp(3-Qp.) (d.lp))) unit (d+ pa,);
3 = 0.
where B aargmaxyyeuuoo Ee Ex [ogg(T (de pa.))
Proof. In MDP, the expected utility is a linear function of the policy, the attainable utility set is a closed interval al) [Wmin; Umax]. We first consider the case where u E (Umin: Umax). )-
In this case, by a full support policy (since u is an interior point), so we can invoke Ziebart's result on the form of such policies [Ziebart] 2010 Ziebart et al. 2010 Gleave and Toyer 2022]. In particular,
seeking the maximum entropy policy in MDP with linear constraint satisfiable
feature the utility Ut We get that the maximum entropy policy a soft-Q Q policy for a utility function argmax3er E, Ex log(= (de pa.)) By Lemma C.1 this can be restated as soft-Q policy for U with rationality parameter of g. It follows soft from Ziebart that 3 = argmax,ye eR "y and allowing 3 = 3o or -oo does not change the argmax.
L with rationality parameter of where
In the case where u E m Umax), }. it's easy to show that the maximum entropy policy which attains randomises uniformly between maximal value actions (for mmax) or minimal value actions (for softt and lim soft respectively.
Wmin). These policies can be expressed as f m3
D Experimental Details
D.1 Tables of results
Known Utllty
Unknown Utlityy
k 1
37.8
34.3 + 2.6
k 2
21.4
32.1 = 0.5
1: 3
16.8
33.6 0.5
1:
= 4
18.9
35.4 + 0.6
Known Utility
Unknown Utility
0.1
2.4
26.1 = 0.11
0.2
1.5
17.4 + 0.2
8 = 0.3
0.95
11.0 + 0.25
0.4
0.50
6.2 + 0.08
0.5
0.20
2.9 + 0.06
5 0.6
0.04
1.0 0.003
0.7
0.003
0.002
8 0.8
0.001
0.10 0.003
0.9
0.008
0.091 0.007
i
17

---

(a) Occupancy measures of optimal policy when
(h) Occupancy measures of optimal policy when
k 1.
Figure 5: Occupancy measures
D.2 Visualising optimal policies for different lengths of goal region.
Figure 5a and Figure 5b show the occupancy measures for an optimal policy for k=l and k=4 respectively, where is the length of the goal region in squares. Although the goal region is larger in the latter case, the optimal policy consistently aims for the same sub region. This explains why maknown-utility MEG is higher than MEG with respect to the environment reward. The policy does just well utility function whose goal-region is limited to the nearer goal squares it does on the environment reward, but fewer policies do well on this utility function, so doing well on constitutes more evidence for goal-direetedness.
D.3 Further details
The experiments were carried out on a personal laptop with the following specs:
Hardware model: LENOVO20N200RRUK
Processor: Intel(R) Core(TM) 17-8665U CPU @ 1.90GHz, 2112 Mhz, 4 Core(s), 8 Logical Processor(s)
Memory: 24.0 GB
We used an environment from the SEALS library and adapted an algorithm from the imitation library Both are released under the MIT license.
For information on hyperparameters, see the code.
"hups://githubb "hups://gihub..oo/Huunu hww.a//hhe....oooooooooooooollll
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions of the paper are listed the end of the introduction, with references to the relevant sections of the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section
7 contains section dedicated limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Proofs of all theoretical results given in the supplementary material.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details of the experiments are describe in Section 6 Code is included in the supplementary material.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is included in the supplementary material.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Full details provided via the code.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report two sigma error bars for unknown-utility MEG experiments. Known- utility MEG experiments are deterministic, so we do not report error bars.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information compute resources is given in Appendix D.3
19

---

9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://ppw..aaaaa........oppaaaa
Question:
Answer: [Yes]
Justification: The research conducted conforms to the code of ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Section contains section discussing socictal impacts.
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release such data or models.
12. Licenses for existing assets
Question: Are creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators of the libraries used are credited, the licenses are respected,
and explicitly mentioned in Appendix D.3
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
20


Reviews:
[{"review_id": "AhbySVs2Ck", "reviewer": "Reviewer_8tgY", "summary": "The paper is studying a problem that is both mathematical and philosophical in nature: how can we measure quantitatively, whether an observed behaviour is goal-directed or not (or to what extent it is). It tackles this problem by starting from a causal model of the world, where there are explicit utility variables, state variables, and decision variables. For a known utility function, it proposes to measure goal-directedness by essentially asking: given an observed policy $\\pi$, find a policy that is maximizing the utility function but regularised by entropy that is _most predictive_ of $\\pi$. Compare its predictive accuracy to uniform prediction, and this quantifies goal-directedness. They also extend this to the case of unknown utilities, by considering parameterized sets of utility functions. The favourable properties of this measure is discussed, and its practicality is demonstrated with a simple empirical result on CliffWorld.", "questions": "Q: I have one question about the unknown utility setting. If I get this right, in general, if the set of possible utility functions is large enough, and the causal model is rich enough, I can potentially find a utility function making many behaviours goal-directed, even though they may not be. For example, consider the behaviour of a river, as its flow. If my causal model represents enough things relevant (e.g. elevation, geography, physics, weather etc.), I can potentially find a utility function in the lines of _minimize travel distance to the nearest sea/lake_ where elevation incurs a cost, and conclude that the flow of the river is goal-directed. Is this correct? If so, it seems like one must be careful in terms of the class of utility functions they consider. \n\nBonus Q: I know that the paper assumes a known causal model. However, I am curious. How would we deal with hidden confounders here? Is there no hope? Specifically, could there be reasons internal to the agent that make a seemingly non-goal-directed behaviour in fact goal-directed, unbeknownst to us?", "limitations": "Paper discusses limitations sufficiently.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The question of measuring goal-directedness is tremendously significant, since it is a core requirement for agency. \n- The proposed solution is sufficiently novel. Even though it builds on the Maximum Causal Entropy IRL, it sufficiently extends it and applies it in a completely new way. \n- The paper is written very clearly and it is easy to follow.", "weaknesses": "- Societal implications could have been discussed in more detail, especially how this \"could enable bad actors to create even more dangerous systems.\" An extended discussion about this added to the appendix would improve this.\n\n- A more comprehensive empirical demonstration of the practicality would improve the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We agree that the problem of measuring goal-directedness is of great significance and we’re glad you feel we have made a novel contribution here.\n\n**Societal implications**.\nWe agree this should be discussed in more detail. Thanks for the suggestion of an extended discussion in the appendix about the concern of bad actors using our ideas to create more dangerous systems. We have added a section in which we weigh various considerations, and then explain why we ultimately think it’s not a major concern.\n\n**Empirical demonstration**.\nThe contribution of this paper is primarily theoretical and conceptual, and so we decided to include experiments mostly for illustrative purposes. Investigating scalability, robustness, and practical applications (like detecting goal misgeneralisation) are promising directions for future work.\n\n**Question about the unknown utility setting.**\nGood question. Every system you model will be goal-directed with respect to a utility function over its own actions, but beyond that, it’s not the case that by including enough variables in your causal model, you can always find a utility function towards which the system is goal-directed. In fact, we can apply Theorem 4.1 to this question, and say that if the agent is not very goal-directed with respect to some Markov blanket of variables surrounding its action, then it’s no more goal-directed with respect to any variable further away in the causal model.\n\nYour river example is an interesting one. Depending on how you model it, it may indeed be the case that the river appears quite goal-directed. If we think of the river as being able to change to flow in any direction at various points along its path, but choosing to follow the path it does, then yes, it will have high goal-directedness with respect to its travel distance to the sea, as you suggest. This is in line with famous examples of very simple agents, such as a thermostat. \n\nOn the other hand, it’s debatable that you should model the river as “being able to” flow in any direction. In general, the smaller the set of actions the system has available, the lower its maximum goal-directedness, as we show in Proposition 3.2.  Therefore a more reasonable way of modelling the river might lead to much lower goal-directedness. And if we put the river in a setting where some other system sometimes disrupted its path, then since the river would not adapt to overcome these disruptions, this would also lead to low goal-directedness.\n\n**The necessity of assuming a causal model.**\nSomething we perhaps under-emphasised in the paper is that if you have black box access to the environment or a simulator, then you don’t need an explicit causal model, you just need the ability to measure the variables you’re interested in under various different policies. So that’s one way in which you can avoid worrying about hidden confounders. We have updated the paper to make this clearer.\n\nOf course, we still need to be measuring the important variables. Interpretability-based approaches to measuring goal-directedness may have an advantage over our behavioural approach here. We have added this point to a discussion of the pros and cons of taking a behavourial vs mechanistic approach to measuring goal-directedness in the paper.\n\nThe second part of your question is about what happens if an agent is goal-directed with respect to variables we can’t measure because they are internal to the agent. There are two cases here: either the agent is also (perhaps instrumentally) goal-directed with respect to external variables, or it isn’t (as in the case of “wire-heading”). In the first case, we may be able to measure this by considering the external variables. The second case is of less concern to us, since our motivation is about the risks arising from agents pursuing goals in their environment.\n\nThanks again for the thoughtful review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U61zqnEAQY", "reviewer": "Reviewer_AseF", "summary": "The paper introduces maximum entropy goal-directedness (MEG), a measure of how much an agent is goal directed towards a given utility function. The authors extend the theoretical framework introduced to the case where the utility function that is being optimized is not known. Moreover, they propose an algorithm to measure MEG in Markov Decision Processes. \nTo show the practicality of the metric introduced, the authors show the performances of the algorithms empirically. Specifically, they show how MEG relates with the optimality of a $\\epsilon$-greedy policy, and how it relates to task difficulty.", "questions": "- How do you think one could make MEG more robust towards distributional shifts? \n\n- Do you have any example in mind on how MEG could fail when considering how goal-directed an agent is? I think it could be an important point to discuss to give a better perspective to the reader on when one would want to employ such metric. Maybe you have some toy examples which can illustrate this?\n\n- Is it possible for two similar policies to achieve drastic differences in MEG? Do you have any experiments which shows how robust of a metric it is?\n\n-How one could extend MEG to cases where the causal environment is only partially known?\n\n- Do you have some experiments/comparison/thoughts on how MEG relates directly with agency?", "limitations": "no ethical limitations", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is theoretically sound, and well presented\n- I found the images and explicit examples to be particularly helpful in understanding how MEG works and why it was defined in a specific way\n- Section 4 addresses an important problem, and I found that it provides clear results which are also well explained. \n- I have not seen much similar work, and I definitely think it addresses an important problem, especially when talking about future goal-directed LLM systems.", "weaknesses": "- As mentioned by the authors, MEG relies on having full access to the causal model of the environment. This can be a highly-demanding assumption in many real environments.\n- Given the restrictive assumption mentioned above, the paper should at least investigate in more detail how computationally expensive is computing MEG in large-scale setting. \n- The experiments are limited to few toy examples", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review.\n\n**Requiring a causal model**.\nA full causal model is not strictly necessary for computing MEG. What is needed is a simulator where you can measure the variables of interest under different policies. We have updated the paper to make this clearer. That said, investigating how MEG can be applied in practice is indeed an important direction for future work.\n\n**Robustness to distributional shifts**.\nOne simple idea is to proactively seek out distributions where MEG is especially high, either by hand or by using MEG as an optimisation target. It would be especially interesting to find distributions such that goal-directedness with respect to the intended reward function is low, but goal-directedness with respect to some other reward function is high. This could be a way to detect goal misgeneralisation (https://arxiv.org/abs/2105.141110).\n\n**How MEG can fail.**\nOne failure mode is that if you measure goal-directedness with respect to the agent’s own actions (or more correctly, state-action pairs in an MDP, or observation-action pairs in a CID), then the agent will appear maximally goal-directed regardless of its policy. This means that if you measure the goal-directedness of an LLM with respect to the text it generates, for example, it won’t tell you much. Instead you need to measure the goal-directedness with respect to some downstream variable (such as a user’s response, or the outcome of a task). We have updated the appendix to mention this example.\n\n**Can similar policies get very different MEG?**.\nYes, at least under some reasonable notions of policy-similarity. Consider an example where two policies in an MDP agree in every state *except the start state*, where one policy takes action a1 and the other takes a2. If a1 leads to a part of the state space where the policies are both very goal-directed, and a2 leads to one where they are not, then there can be an arbitrarily large difference in goal-directedness, despite the policies agreeing in almost every state. This underlines that MEG measures the goal-directedness on a particular distribution. We have added this example to the appendix.\n\n**The case where the causal model is only partially known.**\nAs mentioned above, we can tackle this case as long as we can simulate the environment under different policies. The simulation may only partially specify a causal model, as it may not allow interventions on non-policy variables. It may be interesting to explore other forms of partial knowledge.\n\n**How MEG relates to agency.**\nAgency is an overloaded term with a range of meanings and connotations. Goal-directedness is often considered a key component of agency. For example, Dennett's intentional stance only applies to systems pursuing goals in some sense, and Dung (https://philarchive.org/rec/DUNUAA) argues that it underlies other aspects of agency such as autonomy, empowerment, and intentionality. Some of these notions are still vague, but it's an interesting line of future research to try to make these other aspects precise, and formally and/or theoretically relate them to the notion of goal-directedness we present here.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iiOEGgXtcv", "reviewer": "Reviewer_8w2P", "summary": "* The work introduces a framework for quantifying the goal-directedness of a\n  decision-making system with respect to a particular utility function or a\n  family of utility functions.\n* The method is grounded in Dennett's *intentional stance* and Jaynes'\n  *maximum entropy inference*. The authors have made some additional choices\n  while formalising these principles to arrive at their formulation, which\n  ends up similar to Ziebart's Maximum Causal Entropy IRL framework.\n  The resulting formula is essentially a measure of how close the subject\n  policy comes, in terms of cross entropy, to a maximum entropy formula for\n  the utility function at some inverse temperature.\n* The authors establish that the resulting formula, MEG, satisfies desirable\n  properties for a measure of goal-directedness, namely being invariant to\n  the scaling and translation of utility functions, having interpretable\n  upper and lower bound policies for a given utility function, and measuring\n  only causal goal-directedness.\n* The authors extend their measure of goal-directedness with respect to a\n  single utility function into an aggregate measure of goal-directedness with\n  respect to a family of utility functions (simply by taking the maximum MEG\n  over the family of utility functions). This also allows them to express a\n  measure of goal-directedness with respect to a set of 'target variables',\n  by considering the family of utility functions defined over those\n  variables.\n* The authors derive algorithms for computing both kinds of MEG in the\n  setting of causal influence models derived from MDPs. The algorithms are\n  based on gradient-based search requiring repeatedly computing maximum\n  entropy policies for various inverse temperatures (and, in the case of a\n  family of utility functions, varying family parameters).\n* The authors implement their algorithm for some simple MDPs as a 'proof of\n  concept' experiment, demonstrating how various factors influence the\n  measure of goal-directedness.", "questions": "Many of the weaknesses I raised above are partially framed as questions since\nI am not fully confident that I have understood the work in forming these\ncomments (I welcome corrections from the authors).\n\nHere I collect additional questions that are less crucial to the overall\nvalidity of the work but are still either barriers to my understanding or\nseem like minor issues in the presentation.\n\n* The introduction claims that \"pursuing goals is about having a particular\n  causal effect on the environment.\" This appears to be part of the case for\n  MEG being \"philosophically grounded\".\n  I can see how in many cases this correspondence holds, but it is not\n  immediately obvious to me why there couldn't be cases where the pursuit of\n  goals is separated from causality. For example, if a goal is always going\n  to be achieved counterfactually regardless of an agent's actions, then it\n  becomes possible to 'pursue' this goal without having any causal effect on\n  its achievement.\n  Do the authors have some further justification for this framing, or a\n  citation that gives such justification?\n\n* Equation 1:\n\n    * Can the definition of MEG be interpreted as a minimum KL divergence\n      between the policy and the closest maximum-entropy policy?\n    * The cross entropy between two distributions is not symmetric. Why is is\n      justified to take the cross entropy in this particular direction?\n\n* What is the relationship between theorem 5.1 and prior work on maximum\n  entropy RL such as Haarnoja et al. 2017 \"Reinforcement learning with deep\n  energy-based policies\"?\n\n* In the discussion section, under \"societal implications\", there is a\n  passage \"the relationship between goal-directedness and danger is fairly\n  indirect\".\n\n    * I wonder, did the authors mean to write \"the relationship between\n      *measuring* goal-directedness and danger is fairly indirect\"?\n    * If not, could the authors please elaborate on this indirectness, since it\n      seems that the goal-directedness itself is fairly tightly and directly\n      connected to risks (as outlined in the works cited in the introduction).\n\n\nNot a question, but while reading the paper I happened to note the following\nminor presentation issues the authors may like to address in future versions.\n\n* Line 5: I think the authors mean \"adaptation\" rather than \"adaption\".\n* Line 14: The word \"mundane\" seems to downplay the real and potentially\n  devastating impacts discussed in Chan et al. Consider a different word\n  or a different angle of contrast (perhaps the scale of the harms?).\n* Line 16: Shavit et al. citation and reference missing a date.\n* Line 18: \"Agentcy\".\n* Line 22: Incomplete sentence, consider \"According to Dennett... [1989],\n  *we are justified in doing so whenever it* is useful ...\".\n* Line 31: MEG acronym was introduced in the abstract is undefined in the\n  scope of the introduction.\n* Line 65: Orseau et al. should be cited in text mode.\n* Line 78: Something appears to have allowed the bottom margin on page 2\n  to shrink, causing the final lines on the page to come too close to the\n  page number.\n* Algorithm 1 line 7: Until convergence of what? Not beta, for example.\n* Line 294: This and the next paragraph headings are missing periods on\n  the abbreviation \"vs.\".\n* Line 296: \"meg\" should be capitalised.\n* Footnote 1 is missing a period, at the footnote mark (line 296) AND at\n  the end of the footnote text.\n* Line 300: Potential missing word, \"optimal *policies*\"?", "limitations": "It would have been good to see a more acknowledgement of the limitations of\nthe work (such as those I have enumerated above in the weaknesses section, pending the authors' clarification), throughout the paper or in a dedicated section of the paper.\n\nSome additional points on addressing these limitations in the paper:\n\n* You could more carefully justify or qualify the discussion of the extent to which the\n  approach is uniquely principled / philosophically motivated.\n\n    For example, in the conclusion, MEG is described as \"grounded in the\n    philosophical literature\" when the authors appear to just mean \"grounded in\n    the philosophy of Dennett\" (as they do qualify elsewhere), the difference between which I outline in Weakness (1).\n\n* The scalability of the work does appear to depend on repeatedly computing a\n  maximum entropy policy, which would appear potentially many times more\n  expensive than computing the subject policy, though perhaps feasible. This\n  is based on my understanding. I would have liked to see the authors clarify\n  the requirements in comparison to the cost of finding the target policy\n  explicitly, since they claim their method is \"scalable\".\n\nSee also weaknesses (3) and (4).", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "I thank the authors for submitting their valuable work which is in a position to make an excellent contribution to the field.\n\n* I think the goals of the work are exceptionally well-motivated. The authors\n  did not spend much time elaborating on the need for measures of goal\n  directedness, so I note that such measures have immediate applications in\n  attempts to rigorously study risks from learned optimisation in deep\n  learning. Emergent goal-directedness is a core component of the problem of\n  AI safety (for modern systems and future more capable systems) and scalable\n  methods for quantifying goal-directedness are sorely needed by researchers\n  such as myself who want to better understand this phenomenon in order to\n  reduce or avert these risks.\n* The authors have taken careful steps to justify their proposed framework\n  for measuring goal-directedness, including showing that it satisfies some\n  sensible properties and showing that in simple examples (including toy\n  experiments with some MDPs) the values given by the MEG measure are\n  sensible.\n* Overall, the paper is fairly clearly written and I feel I was able to\n  understand what the authors have to say. I think that this strength is\n  limited in that there may be some room for improvement in the flow of\n  sections and in some low-level details of the writing to make the paper\n  optimally clear and accessible (see below), however overall it was\n  passable.", "weaknesses": "I noted the following issues with the work, based on my understanding.\n\n1.  MEG is described as \"philosophically grounded\" because of its roots in an\n    interpretation of Dennett's Intentional Stance. However, it does not\n    appear to be a unique interpretation and this philosophy does not appear\n    to be the only possible philosophical approach to defining agency.\n\n    * Related works Kenton et al. and Orseau et al. both appeal to Dennett's\n      philosophy but arrive at different formalisms. The authors have not\n      established that their interpretation is uniquely philosophically\n      grounded. (Appendix A begins to address this with respect to Kenton et\n      al.'s prior work, but it would be good to see a discussion of the\n      relationship with Orseau et al.'s approach.)\n\n    * If MEG were a unique formalisation of Dennett's philosophy, it would\n      still appear to be a behaviourist approach, which stands in contrast to\n      a mechanistic approach that looks into the internals of a system for\n      explicit representations of goals and for the implementation of\n      decision-making processes.\n\n2.  I am concerned that the statements of proposition 3.1 and 3.2\n    are incorrect as stated.\n\n    * **Proposition 3.1.** There must be some missing assumption in this\n      theorem statement that $a$ is a positive scalar. Otherwise you have\n      proven that a MEG policy for any given utility function is also\n      goal-directed towards a negated version of that policy. Could you\n      please clarify whether this is the case or whether I am missing\n      something that means that a policy that pursues the negative of a\n      utility function is somehow goal-directed towards that utility\n      function?\n\n    * **Proposition 3.2.** It is well known that in many examples there are\n      decisions for which multiple actions lead to equal expected utility and\n      are therefore equally optimal. In such cases there is no unique optimal\n      policy. What happens to the bounds in this case? The statement of the\n      proposition would appear to be ill-defined in case there is no unique\n      optimal policy. Is it the case that you do not derive an upper bound in\n      this case? The distinction being then that there still will be some\n      upper bound (perhaps?) but that you have not characterised it?\n\n    When I checked the proofs for these propositions in the appendix I found\n    them to provide insufficient detail to resolve my concerns or point to a\n    particular mistake in the reasoning.\n\n    I am recommending acceptance of the paper on the assumption that these issues will be easy to dismiss or to correct.\n\n3.  In the unknown-utility case, MEG fails to account for complexity of\n    utility functions within the hypothesis class.\n\n    * It is well known in RL that for any policy, a utility function exists\n      that can rationalise that policy. For example, there is a reward\n      function that immediately, highly rewards whichever actions that policy\n      happens to take in each state.\n    * This leads me to wonder if for any policy it is possible to find some\n      reward function for which MEG very strongly indicates that this policy\n      is goal-directed towards that utility function.\n    * Especially in the case of MEG for unknown utility functions, it is\n      therefore possibly that if the hypothesis class of utility functions is\n      too rich, the MEG for unknown utility functions will register almost\n      any policy as goal-directed.\n    * I think something that is missing is some way of accounting for the\n      complexity of utility functions. Philosphically, a complex utility\n      function is not 'useful for predicting behaviour' if it is itself as\n      complex as the behaviour, even if it is capable of predicting behaviour\n      well.\n    * I think a significant weakness of MEG, especially in the\n      unknown-utility setting, is that it leaves the user of the framework to\n      rule out too-complex utility functions by the coarse tool of including\n      them or excluding them from the hypothesis class, rather than providing\n      a principled tool for managing this complexity.\n  \n    If the authors are aware of this issue but consider addressing it to be\n    out of scope, I would like to see it acknowledged as such in the main\n    text.\n\n4.  If I am not mistaken, the scalability of the method for computing MEG\n    relies on solving an RL problem (computing a maximum entropy policy in\n    MDPs). If this is the case then I think the scalability of the method is\n    questionable.\n\n    * It is certainly common practice to solve RL problems, and in any case\n      where we can train a policy that will be the target of MEG evaluation,\n      we can presumably also train a max-ent policy to measure its MEG.\n      Perhaps calculating such a policy repeatedly will be expensive but it\n      should at least be feasible to do so.\n    * However, the very applications I think motivate this work are settings\n      where we don't understand the internal motivational system of the\n      trained system (and so MEG can help quantify that). This would of\n      course only happen if we are training the subject policy in such a\n      complex environment that its emergent goal-directedness is in question,\n      due to, for example, potential goal misgeneralisation (or so-called\n      'inner alignment failure').\n    * In such a setting, I suppose we would share the same concerns about the\n      max-ent policy required for quantifying MEG. If I am not mistaken, if\n      this probe policy's generalisation could not be trusted, then neither\n      could we trust the result of using it to calculate the MEG of the\n      subject policy.\n\n    There is a comment in the discussion to the effect that future work can\n    consider the setting of 'distributional shifts' which is normally\n    considered to be a prerequisite for the phenomenon of goal\n    misgeneralisation. However, I think this concern undercuts the motivation\n    of the present work in its early stages and if there is no feasible way\n    to reliably compute the MEG for complex learned systems where the\n    internal goals are in question then the work may not achieve its goals in\n    the future, and this issue should be raised and (ideally) addressed now.\n\n5.  There is insufficient explanation of relationship to prior work, particularly to IRL and agents and devices.\n\n    * **Inverse reinforcement learning.** IRL represents a large body of\n      work including theoretical work aimed at finding goals that\n      rationalise particular policies, which is clearly closely related to\n      MEG. The authors' brief comments in the related work section have not\n      clarified to me the relationship between MEG and IRL.\n      In particular, it seems plausible to me that MEG is essentially\n      functions similar to a subcomponent of a IRL framework based on\n      Bayesian inference.\n\n        * A Bayesian approach for IRL for example involves positing a\n          likelihood of the behavioural data given each goal. This likelihood\n          function represents a kind of measure of how well the behaviour can\n          be explained by that goal. This seems similar to MEG, though of\n          course MEG does not adopt a Bayesian framework.\n        * Similarly, what is the relationship between unknown-utility MEG\n          with a given hypothesis class and the model evidence in a Bayesian\n          formulation of IRL with the same hypothesis class? Of course there\n          is a difference of integrating versus taking a max, but my point is\n          that these are different approaches to the same kind of problem.\n    \n        I think the paper would be strengthened if the authors could clarify\n        whether MEG is addressing a fundamentally new kind of problem or one\n        that arises naturally in the course of IRL. If it's not new, that\n        seems fine, the approach is apparently new---but it would be nice to\n        see a discussion of how the approach compares to the Bayesian\n        approach outlined above.\n    \n    * **Agents and devices.** This prior works appears to present a\n      Bayesian framework for distinguishing agents from non-agents, along\n      with one particular instantiation of this general framework in terms\n      of epsilon-greedy suboptimality and Solomonoff induction.\n      The comments in the related work section appear to relate to the\n      instantiation. What is the relationship between the present work and\n      the general framework from Orseau et al.?\n      For example, would it be possible to derive a maximum-entropy version\n      of Orseau et al.'s framework, and if so, how might that compare to\n      MEG? Does the difference come down to the difference between Bayesian\n      inference and maximum entropy inference?\n    \n    I would have liked to see sections along the lines of appendix A (for\n    Kenton et al.) drawing out these relationships.\n\nIf the authors are able to address each of these issues to my satisfaction I think my assessment of the paper's strength would be greatly improved and I would be willing to substantially strengthen my recommendation to accept the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your extensive and careful review.\n\n**Motivation.**\nAgreed emergent goal-directedness is a core AI safety problem. Updated to be more explicit.\n\n**Terminology.**\nWe agree our approach is not a unique interpretation of Dennett, and his view not the only one in the literature. We have changed  \"the literature\" to  \"Dennett\". Would “philosophically motivated” be preferable to “philosophically grounded”?\n\n**Mechanistic vs behavioural.**\nAgreed our approach is behavioural rather than mechanistic. We think both approaches should be pursued.  Behavioural approaches seem more tractable, but mechanistic approaches will generalise better off-distribution. We have added this point to the discussion.\n\n**Prop 3.1.**\nThe proposition is correct – MEG considers an agent goal-directed wrt a utility function both if it maximises it and if it minimises it. We have made the proof more explicit.\n\nWe admit this may be unintuitive. Arguably trying to pessimise a utility function is just as goal-directed as trying to optimise it. But on the other hand, perhaps MEG should be signed. We did consider this. Would the reviewer argue in favour?\n\n**Prop 3.2.**\nThe proposition is correct, but was stated ambiguously. We have updated it to say, “there is equality in the upper bound if and only if pi is a *unique* optimal (resp. anti-optimal) policy”. Is the meaning now clear?\n\n**Is unknown utility meg always high?**\nThere does alway exists a reward function to rationalise a (deterministic) policy. But this only applies to state-action or state-action-state rewards, not state rewards (ignoring the constant reward function). It’s the same with MEG: all (deterministic) policies are maximally goal-directed with respect to utility functions over <observation,action> pairs. But if the set of functions if over some downstream variable (e.g. states in an MDP), most policies will not be maximally goal-directed. We think MEG gets this right -- it's hard to say an agent is not optimising its own behaviour, but its easy to say whether it's optimising some variable out in the world.\n\n**Complexity of utility functions.**\n\nAny way of measuring goal-directedness needs a way to avoid the trivial result that all policies are maximally goal-directed. One option is indeed to penalise more complex utility functions. But MEG does it in a different way: by considering utility functions over specific sets of variables. If the variables considered don’t include the agent’s actions, then most policies will not be very goal-directed, for any complexity of utility function. So we think it’s best to present MEG without an additional complexity penalty, although we acknowledge that adding one may be natural. We have now mentioned this in the paper.\n\n**Scalability.**\nMEG does require repeatedly solving an RL problem. MEG is about as scalable as MCE IRL, so we expect it to scale better than UTM-based approaches (Agents and Devices). But we have not demonstrated the scalability here, so we have now weakened the language. We agree that much usefulness depends on scaling. However, there is also value in conceptual clarification, and taking a step towards scalable methods.\n\n**Generalisation.**\nAgreed distributional shifts are a key motivation. Rather than trusting the imitation policy to generalise correctly, a better approach could be to think of MEG as measuring goal-directedness on a given distribution, and to proactively seek out distributions to test on, e.g. where unknown-utility MEG is much higher than intended-reward MEG. This may let us detect goal misgeneralisation.\n\n**Prior work.**\nWe have added new sections in the appendix comparing to A&D and IRL. One key difference with IRL: IRL does not try to tell you *how* goal-directed a policy is, just what its goal is. Consider an agent that performs well on the intended reward by default, but badly under a distributional shift. IRL would just tell you that the utility function of the agent changes, but MEG can tell us whether the behaviour is strongly goal-directed on the new distribution, i.e. whether we have capability misgeneralisation (low MEG) or goal misgeneralisation (high MEG). \n\n**Does pursuing goals require a causal effect?**\nThis is a common view in philosophy (see section 3.1 of the SEP article on agency). But it’s not universal, e.g. *evidential decision theory* is concerned with pursuing goals even if one cannot causally affect them. We think that MEG can be generalised to an “evidential” version, but it’s beyond the scope. We have added a reference and a footnote.\n\n**KL div, cross entropy, Haarnoja.**\n- MEG is similar to the minimum KL with a maxent policy. You would remove the second term (the entropy of the uniform policy) and add a term for the negative entropy of the policy itself. The first change is just a translation. The second would mean that a less entropic way of achieving a certain utility would be considered more goal-directed than a more entropic way.\n\n- The direction of the cross entropy gets us a maxent policy that well-predicts the agent’s policy, rather than vice versa. Reversing it would mean maximising the maxent policy's probability of the agent's *highest probability* action, i.e. ignoring other aspects of the policy.\n\n- Haarnoja’s results are essentially a useful alternative formulation of Ziebart’s. We could probably reformulate our results along the lines of Haarnoja’s, but chose to focus on Ziebart.\n\n**Harm.**\nWe agree the relationship is not indirect. Probably we meant to include the word “measuring”, but this too now seems like an overstatement. We have removed the sentence.\n\n**Limitations.**\nWe have added the following to the limitations section: scalability; mechanistic vs behavioural approaches; more on distributional shifts. Has the rest been satisfactorily clarified?\n\nLet us know if our response is enough to increase your score, or whether further changes would be needed. Thanks again.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U5n9TtaG1E", "reviewer": "Reviewer_Kmdz", "summary": "The paper proposes a measure of evidence for goal-directedness based on the maximum causal entropy principle, MEG. It is operationalized as the ability to predict variable D based on the hypothesis that it is optimizing a utility function U, where U represents the goal and D the agent’s decisions. Specifically, they try to predict the agents' decisions using the maximum-entropy policy that achieves the same utility as the agent’s own policy. MEG is essentially the negative cross-entropy of this prediction. The authors also describe algorithms for estimating MEG, and demonstrate it in action with some small-scale experiments.", "questions": "Why does the maximizing policy in equation 1 always obtain the same expected utility as pi?", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The work is well-motivated and situated among prior work \n\n- The writing is clear and the examples provided are very helpful for understanding\n\n- The MEG satisfies some nice properties such as invariance to translation and scaling\n\n- The authors allow MEG to cover a wide range of use cases, for instance\n\n  - It can be estimated even when only a set of trajectories is available, rather than the agent’s policy itself. \n\n  - It can be extended to settings where the the utility function is unknown", "weaknesses": "- It is unclear how useful MEG will be for checking goal-directedness of generalist AI agents acting in the real world, because:\n  - it is expensive to compute, since it requires optimizing the maximum entropy policy, and their estimation algorithm can only guarantee a lower bound on MEG for the unknown-utility case\n  - It requires a causal model containing the target variables that the agent might be trying to manipulate.\n\n- The name and framing of MEG is slightly misleading: MEG is not directly a measure of how goal-directed an agent is, but rather the amount of evidence for goal-directedness. The maximum value MEG can take varies based on the number of possible choices in an environment. Thus, it may be difficult to compare the goal-directedness of different agents’ behavior if we could only observe them acting in different environments. (e.g., to determine if agents acting in our constantly changing world are becoming increasingly goal-directed)\n\n- typo: agentcy on line 18\n\n- Capitalization of v at the end of definition 2.1 is inconsistent", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review. \n\n**Lower bound.**\nIt’s true that our algorithm can only guarantee a lower bound in the unknown-utility case, but this is true whenever one performs SGD on a nonconvex loss function to estimate some quantity. With neural networks, we often get good estimates of the quantity regardless. We’ve added a note to clarify.\n\n**Misleading name.**\nIt’s true that MEG only measures the evidence for goal-directedness exhibited in the behaviour (with respect to the variables we measure), not necessarily the full goal-directedness of the agent. But this is generally true of tests: a high-school maths test cannot measure the full ability of a star mathematician, since there is limited opportunity to provide evidence for it while completing the test. Similarly, depending on the environment in which we measure MEG, we may see relatively little evidence for an agent’s goal-directedness. But we think it’s still fair to say that MEG is measuring goal-directedness, just as the test is measuring mathematical ability. \n\nWe should of course bear this point in behind, and think of MEG as measuring the goal-directedness with respect to a given environment distribution. We have updated the paper to clarify this.\n\n\n\n**Equation 1.**\nThe fact that the maximising policy obtains the same expected utility as pi is non-obvious but follows from Ziebart’s result. I’m not sure why we didn’t include a proof of this – we have included one now. Thanks for drawing our attention to it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to the reviewers for their feedback, which has already allowed us to improve the paper. We are pleased that the reviewers agree that  the problem we are tackling is an important one and that our approach is novel.

We have responded to each reviewer’s points individually. Minor presentation issues have all been fixed without comment.

Meta-review: The paper makes a significant theoretical contribution to the study of goal-directedness in decision-making systems, with potential implications for AI safety. 
However, the work would benefit from addressing the computational limitations, broadening the philosophical discussion, and providing more extensive empirical validation. 
Despite these concerns, the reviewers agree that the paper is an important step forward in this research area and recommend acceptance, provided that the identified weaknesses are addressed in the final version
Decision: Accept (spotlight)