Title: Measuring Goal-Directedness
Venue: NeurIPS 2024
Accepted: yes
Keywords: Causality

Graphical Models

Maximum Causal Entropy

Agency
Abstract: We define maximum entropy goal-directedness (MEG), a formal measure of goal-
directedness in causal models and Markov decision processes, and give algorithms
for computing it. Measuring goal-directedness is important, as it is a critical
element of many concerns about harm from AI. It is also of philosophical interest,
as goal-directedness is a key aspect of agency. MEG is based on an adaptation of
the maximum causal entropy framework used in inverse reinforcement learning. It
can measure goal-directedness with respect to a known utility function, a hypothesis
class of utility functions, or a set of random variables. We prove that MEG satisfies
several desiderata and demonstrate our algorithms with small-scale experiments.

Measuring Goal-Directedness

Abstract

We define maximum entropy goal-directedness (MEG) , a formal measure of goaldirectedness in causal models and Markov decision processes, and give algorithms for computing it. Measuring goal-directedness is important, as it is a critical element of many concerns about harm from AI. It is also of philosophical interest, as goal-directedness is a key aspect of agency. MEG is based on an adaptation of the maximum causal entropy framework used in inverse reinforcement learning. It can measure goal-directedness with respect to a known utility function, a hypothesis class of utility functions, or a set of random variables. We prove that MEG satisfies several desiderata and demonstrate our algorithms with small-scale experiments 1 .

1 Introduction

In order to build more useful AI systems, a natural inclination is to try to make them more agentic . But while agents built from language models are touted as the next big advance [Wang et al., 2024], agentic systems have been identified as a potential source of individual [Dennett, 2023], systemic [Chan et al., 2023, Gabriel et al., 2024], and catastrophic [Ngo et al., 2022] risks. Agency is thus a key focus of behavioural evaluations [Shevlane et al., 2023, Phuong et al., 2024] and governance [Shavit et al., 2023]. Some prominent researchers have even called for a shift towards designing explicitly non-agentic systems [Dennett, 2017, Bengio, 2023].

Acritical aspect of agency is the ability to pursue goals. Indeed, the standard theory of agency defines agency as the capacity for intentional action - action that can be explained in terms of mental states such as goals [Schlosser, 2019]. But when are we justified in ascribing such mental states? According to Daniel Dennett's instrumentalist philosophy of mind [1989], we are justified when doing so is useful for predicting a system's behaviour.

This paper's key contribution is a method for formally measuring goal-directedness based on Dennett's idea. Since pursuing goals is about having a particular causal effect on the environment 2 , defining our measure in a causal model is natural. Causal models are general enough to encompass most frameworks popular among ML practitioners, such as single-decision prediction, classification, and regression tasks, as well as multi-decision (partially observable) Markov decision processes. They also offer enough structure to usefully model many ethics and safety problems [Everitt et al., 2021a, Ward et al., 2024a, Richens et al., 2022, Richens and Everitt, 2024, Everitt et al., 2021b, Ward et al., 2024b, Halpern and Kleiman-Weiner, 2018, Wachter et al., 2017, Kusner et al., 2017, Kenton et al., 2023, Fox et al., 2023, Richens and Everitt, 2024, MacDermott et al., 2023].

/cheese

(a)

System of interest

➡

S

D

T

(b)

Model as causal

Bayesian network

➡

S

D

T

U

θ

(c)

Posit

decision

and utility variables

➡

S

D

T

U

θ

Predict

?

(d)

Measure maximum

predictive accuracy

Maximum entropy goal-directedness (MEG) operationalises goal-directedness as follows, illustrated by the subsequent running example.

A variable D in a causal model is goal-directed with respect to a utility function U to the extent that the conditional probability distribution of D is well-predicted by the hypothesis that D is optimising U .

Example 1. A mouse begins at the centre of a gridworld (Figure 1a). It observes that a block of cheese is located either to the right or left ( S ) with equal probability, proceeds either away from it or towards it ( D ), and thus either obtains the cheese or does not ( T ).

Suppose that the mouse moves left when the cheese is to the left and right when it is to the right, thus consistently obtaining the cheese. Intuitively, this behaviour seems goal-directed, but can we quantify how much? Figure 1 gives an overview of our procedure. We first model the system of interest as a causal Bayesian network (Figure 1b) with variables S for the cheese's position, D for the mouse's movement, and T for whether or not the mouse obtains the cheese. Then, we identify a candidate decision variable D and target variable T , hypothesising that the mouse optimises a utility function that depends on T (Figure 1c). Finally, we form a model of what behaviour we should expect from D if it is indeed optimising U and then measure how well this model predicts D 's observed behaviour (Figure 1d).

Ziebart [2010]'s maximum causal entropy (MCE) framework suggests a promising way to construct a model for expected behaviour under a given utility function. However, there are several obstacles to applying it to our problem: it cannot measure the predictive usefulness of known utility functions, and it only finds the most predictive linear utility function. In practice, arbitrary known utility functions can be plugged in, but the results are not scale-invariant . We overcome these difficulties by returning to first principles and deriving an updated version of the MCE framework.

In summary, our contributions are four-fold. We (i) adapt the MCE framework to derive maximum entropy goal-directedness (MEG), a philosophically-motivated measure of goal-directedness with respect to known utility functions, and show that it satisfies several key desiderata (Section 3); (ii) we extend MEG to measure goal-directedness in cases without a known utility function (Section 4); (iii) we provide algorithms for measuring MEG in MDPs (Section 5); and (iv) we demonstrate the algorithms empirically (Section 6).

Related Work. Inverse reinforcement learning (IRL) [Ng and Russell, 2000] focuses on the question of which goal a system is optimising, whilst we are interested in to what extent the system can be seen as optimising a goal. We are not the first to ask this question. Some works take a Bayesian approach inspired by Dennett's intentional stance. 3 Most closely related to our work is Orseau et al. [2018], which applies Bayesian IRL in POMDPs using a Solomonoff prior over utility functions and an ε -greedy model of behaviour. This lets them infer a posterior probability distribution over whether an observed system is a (goal-directed) "agent" or "just a device". Our approach distinguishes itself from these by considering arbitrary variables in a causal model and deriving our behaviour model from the principle of maximum entropy. Moreover, since our algorithms can take advantage of differentiable classes of utility functions, our approach may be amenable to scaling up using deep neural networks. Oesterheld [2016] combines the intentional stance with Bayes' theorem in cellular automata, but does not consider specific models of behaviour. Like us, Kenton et al. [2023] consider goal-directedness in

a causal graph. However, they require variables to be manually labelled as mechanisms or object-level , and only provide a binary distinction between agentic and non-agentic systems (see Appendix A for a detailed comparison). Biehl and Virgo [2022], Virgo et al. [2021] propose a definition of agency in Moore machines based on whether a system's internal state can be interpreted as beliefs about the hidden states of a POMDP.

2 Background

We use capital letters for random variables V , write dom( V ) for their domain (assumed finite), and use lowercase for outcomes v ∈ dom( V ) . Boldface denotes sets of variables V = { V 1 , . . . , V n } , and their outcomes v ∈ dom( V ) = × i dom( V i ) . Parents and descendants of V in a graph are denoted by Pa V and Desc V , respectively (where pa V and desc V are their instantiations).

Causal Bayesian networks (CBNs) are a class of probabilistic graphical models used to represent causal relationships between random variables [Pearl, 2009].

Definition 2.1 (Causal Bayesian network) . A Bayesian network M = ( G,P ) over a set of variables V = { V 1 , . . . , V n } consists of a joint probability distribution P which factors according to a directed acyclic graph (DAG) G , i.e., P ( V 1 , . . . , V n ) = ∏ n i =1 P ( V i | Pa V i ) , where Pa V i are the parents of V i in G . A Bayesian network is causal if its edges represent direct causal relationships, or formally if the result of an intervention do ( X = x ) for any X ⊆ V can be computed using the truncated factorisation formula : P ( v | do ( X = x )) = Π i : v i / ∈ x P ( v i | pa v i ) if v is consistent with x or P ( v | do ( X = x )) = 0 otherwise.

Figure 1b depicts Example 1 as a CBN, showing the causal relationships between the location of the cheese ( S ), the mouse's behavioural response ( D ), and whether the mouse obtains the cheese ( T ).

We are interested in to what extent a set of random variables in a CBN can be seen as goal-directed. That is, to what extent can we interpret them as decisions optimising a utility function ? In other words, we are interested in moving from a CBN to a causal influence diagram (CID), a type of probabilistic graphical model that explicitly identifies decision and utility variables.

Definition 2.2 (Causal Influence Diagram [Everitt et al., 2021a]) . A causal influence diagram (CID) M = ( G,P ) is a CBN where the variables V are partitioned into decision D , chance X , and utility variables U . Instead of a full joint distribution over V , P consists of conditional probability distributions (CPDs) for each non-decision variable V ∈ V \ D .

A CID can be combined with a policy π , which specifies a CPD π D for each decision variable D , in order to obtain a full joint distribution. We call the sum of the utility variables the utility function and denote it U = ∑ U ∈ U U . Policies are evaluated by their total expected utility E π [ U ] . We write π unif for the uniformly random policy.

CIDs can model a broad class of decision problems, including Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs) [Everitt et al., 2021b].

Example 2 (POMDP) . A mouse begins at the centre of the grid, with a block of cheese located either at the far left or the far right (Figure 2). The mouse does not know its position or the position of the cheese ( S 1 ) but can smell which direction the cheese is in ( O 1 ) and decide which way to move ( D 1 ). Next step, the mouse again smells the direction of the cheese ( O 2 ) and again chooses which way to proceed ( D 2 ).

3 Measuring goal-directedness with respect to a known utility function

Maximum Entropy Goal-directednes. Dennett's instrumentalist approach to agency says that we can ascribe mental states (such as utilities) to a system to the extent that doing so is useful for predicting its behaviour [Dennett, 1989]. To operationalise this, we need a model of what behaviour is predicted by a utility function. According to the principle of maximum entropy [Jaynes, 1957], we should choose a probability distribution with the highest entropy distribution satisfying our requirements, thus minimising unnecessary assumptions (following Occam's razor). We can measure the entropy of a policy by the expected entropy of its decision variables conditional on their parents H π ( D || Pa D ) = -∑ D ∈ D E d, Pa D ∼ P π log π D ( d | Pa D ) . This is Ziebart et al. [2010]'s causal entropy , which we usually refer to as just the entropy of π .

/cheese

(a) Gridworld

S

1

S

2

S

3

O

1

O

2

D

1

D

2

S

1

S

2

S

3

O

1

O

2

D

1

D

2

U

1

U

2

U

3

(c) Causal influence diagram

In our setting, the relevant constraint is expected utility. To avoid assuming that only optimal agents are goal-directed, we construct a set of models of behaviour which covers all levels of competence an agent optimising utility U could have. We define the set of attainable expected utilities in a CID as att( U ) = { u ∈ R | ∃ π ∈ Π( E π [ U ] = u ) } .

Definition 3.1 (Maximum entropy policy set, known utility function) . Let M = ( G,P ) be a CID with decision variables D and utility function U . The maximum entropy policy set for u ∈ att( U ) is Π maxent U ,u := argmax π | E π [ U ]= u H π ( D || Pa D ) . The maximum entropy policy set for U is the set of maximum entropy policies for any attainable expected utility Π maxent U := ⋃ u ∈ att( U ) Π maxent U ,u .

For each attainable expected utility, Π maxent U contains the highest entropy policy which attains it. In MDPs, this policy is unique π maxent U ,u and can be found with backwards induction (see Section 5).

We measure predictive accuracy using cross-entropy, as is common in ML. We subtract the predictive accuracy of the uniform distribution, so that we measure predictive accuracy relative to random chance. This makes MEG always non-negative.

Definition 3.2 (Maximum entropy goal-directedness, known utility function) . Let M := ( G,P ) be a CID with decision variables D and utility function U . The maximum entropy goal-directedness (MEG) of a policy π with respect to U is

As we will see, in a Markov Decision Process, this is equivalent to maximising predictive accuracy by varying the rationality parameter β in the 'soft-optimal' policy π soft β used in maximum entropy reinforcement learning [Haarnoja et al., 2017]. 4

If instead of having access to a policy π , we have access to a set of full trajectories { ( pa i D 1 , D i 1 , . . . , pa D i n , D i n ) } i , the expectation E π in Equation (1) can be replaced with an average over the trajectory set. This is an unbiased and consistent estimate of MEG U ( π ) for the policy π generating the trajectories.

Example. Consider a policy π in Example 1 that proceeds towards the cheese with probability 0 . 8 . How goal-directed is this policy with respect to the utility function U that gives +1 for obtaining the cheese and -1 otherwise?

To compute MEG U ( π ) , we first find the maximum entropy policy set Π maxent U , and then take the maximum predictive accuracy with respect to π . In a single-decision setting, for each attainable expected utility u there is a unique π maxent U ,u which has the form of a Boltzmann policy π maxent U ,u ( d | s ) = exp( β · E [ U | d,s ]) ∑ d ′ exp( β · E [ U | d ′ ,s ]) (cf. Theorem 5.1). The rationality parameter β = β ( u ) can be varied to get the correct expected utility. Predictive accuracy with respect to π is maximised by π maxent U , 0 . 8 , which has a rationality parameter of β = log 2 . The expected logprob of a prediction of this policy is

E π [ log π maxent U , 0 . 8 ( D | Pa D ) ] = -0 . 50 , while the expected logprob of a uniform prediction is log( 1 2 ) = -0 . 69 . So we get that MEG U ( π ) = -0 . 50 -( -0 . 69) = 0 . 19 . For comparison, predictive accuracy for the optimal policy π ∗ is maximised when β = ∞ , and has MEG U ( π ∗ ) = 0 -( -0 . 69) = 0 . 69 .

Properties. We now show that MEG satisfies three important desiderata. First, since utility functions are usually only defined up to translation and rescaling, a measure of goal-directedness with respect to a utility function should be translation and scale invariant. MEG satisfies this property:

̸

Proposition 3.1 (Translation and scale invariance) . Let M 1 be a CID with utility function U 1 , and let M 2 be an identical CID but with utility function U 2 = a · U 1 + b , for some a, b ∈ R , with a = 0 . Then for any policy π , MEG U 1 ( π ) = MEG U 2 ( π ) .

Second, goal-directedness should be minimal when actions are chosen completely at random and maximal when uniquely optimal actions are chosen.

Proposition 3.2 (Bounds) . Let M be a CID with utility function U . Then for any policy π we have 0 ≤ MEG U ( π ) ≤ ∑ D ∈D log( | dom( D ) | ) , with equality in the lower bound if π is the uniform policy, and equality in the upper bound if π is the unique optimal (or anti-optimal) policy with respect to U .

Note that MEG has a natural interpretation as the amount of evidence provided for a goal-directed policy over a purely random policy. The larger a decision problem, the more opportunity there is to see this evidence, so the higher MEG can be.

Third, a system can never be goal-directed towards a utility function it cannot affect.

Proposition 3.3 (No goal-directedness without causal influence) . Let M = ( G,P ) be a CID with utility function U and decision variables D such that, Desc ( D ) ∩ Pa U = ∅ . Then MEG U ( D ) = 0 .

Comparison to MCE IRL Our method is closely related to MCE IRL [Ziebart et al., 2010] (see Gleave and Toyer [2022] for a useful primer). In this subsection, we discuss the key similarities and differences.

The MCE IRL method seeks to find a utility function that explains the policy π . It starts by identifying a set of n linear features f i and seeks a model policy that imitates π as far as these features are concerned but otherwise is as random as possible. It thus applies the principle of maximum entropy with n linear constraints. The form of the model policy involves a weighted sum of these features. In a single-decision example, it takes the form

The weights w i are interpreted as a utility function over the features f i . MCE IRL can, therefore, only return a linear utility function.

In contrast, our method seeks to measure the goal-directedness of π with respect to an arbitrary utility function U , linear or otherwise. Rather than constructing a single maximum entropy policy with n linear constraints, we construct a class of maximum entropy policies, each with a different single constraint on the expected utility.

Anaive alternative to defining the goal-directedness of π with respect to U as the maximum predictive accuracy across U 's maximum policy set, we could simply plug in our utility function U to π MCE from Equation (2), and use that to measure predictive accuracy. If U is linear in the features f i , we could substitute in the appropriate weights, but even if not, we could still replace ∑ i w i f i with U . Indeed, this is often done with nonlinear utility functions in deep MCE IRL [Wulfmeier et al., 2015].

However, this would not have a formal justification, and we would run into a problem: scale noninvariance. Plugging in 2 · U would result in a more sharply peaked π MCE than U ; in Example 1, we would get that the mouse is more goal-directed towards 2 · U than U , with a predictive accuracy (measured by negative cross-entropy) of -0.018 vs -0.13. In contrast, constructing separate maximum entropy policies for each expected utility automatically handles this issue. The policy in Π maxent 2 ·U which maximises predictive accuracy for π has an inversely scaled rationality parameter β ′ = β 2 compared to the maximally predictive policy in Π maxent 2 ·U . In other words, they are the same policy, and we get that MEG U ( π ) = MEG 2 ·U ( π ) = 0 . 19 (cf. Proposition 3.1).

4 Measuring goal-directedness without a known utility function

In many cases where we want to apply MEG, we may not know exactly what utility function the system could be optimising. For example, we might suspect that a content recommender is trying to influence a user's preferences, but may not have a clear hypothesis as to in what way. Therefore, in this section, we extend our definitions for measuring goal-directedness to the case where the utility function is unknown.

We first extend CIDs to include multiple possible utility functions.

Definition 4.1. A parametric-utility CID (CID) M Θ is a set of CIDs { M θ | θ ∈ Θ } which differ only in the CPDs of their utility variables.

In effect, a parametric CID is a CID with a parametric class of utility functions U Θ .

The maximum entropy policy set from Definition 3.1 is extended accordingly to include maximum entropy policies for each utility function and each attainable expected utility with respect to it.

Definition 4.2 (Maximum entropy policy set, unknown utility function) . Let M Θ = ( G,P ) be a parametric-utility CID with decision variables D and utility function U Θ . The maximum entropy policy set for U Θ is the set of maximum entropy policies for any attainable expected utility for any utility function in the class: Π maxent U Θ := ⋃ θ ∈ Θ ,u ∈ att( U θ ) Π maxent U θ ,u .

Definition 4.3 (MEG, unknown utility function) . Let M Θ = ( G,P ) be a parametric-utility CID with decision variables D and utility function U Θ . The maximum entropy goal-directedness of a policy π with respect to U Θ is MEG U Θ ( π ) = max U∈U Θ MEG U ( π ) .

Definition 4.4 (MEG, target variables) . Let M = ( G,P ) be a CBN with variables V . Let D ⊆ V be a hypothesised set of decision variables and T ⊆ V be a hypothesised set of target variables. The maximum entropy goal-directedness of D with respect to T , MEG T ( D ) , is the goal-directedness of π = P ( D | Pa D ) in the parametric CID with decisions D and utility functions U : dom( T ) → R (i.e., the set of all utility functions over T ).

For example, if we only suspected that the mouse in Example 1 was optimising some function of the cheese T , but didn't know which one, we could apply Definition 4.4 to consider the goal-directedness towards T under any utility function defined on T . Thanks to translation and scale invariance (Proposition 3.1), there are effectively only three utility functions to consider: those that provide higher utility to cheese than not cheese, those that do the opposite, and those that are indifferent.

Note that T has to include some descendants of D , in order to enable positive MEG (Proposition 3.3). However, it is not necessary that T consists of only descendants of D (i.e., T need not be a subset of Desc ( D ) ). For example, goal-conditional agents take an instruction as part of their input Pa D . The goal-directedness of such agents can only be fully appreciated by including the instruction in T .

Pseudo-terminal goals. Definition 4.4 enable us to state a result about a special kind of instrumental goal. It is well known that an agent that optimises some variable has an instrumental incentive to control any variables which mediate between the two [Everitt et al., 2021a]. However, since the agent might want the mediators to take different values in different circumstances, it need not appear goal-directed with respect to the mediators. Theorem 4.1 shows that in the special case where the mediators d-separate the decision from the downstream variable, the decision appears at least as goal-directed with respect to the mediators as with respect to the target.

Theorem 4.1 (Pseudo-terminal goals) . Let M = (( V , E ) , P ) be a CBN. Let D , T , S ⊆ V such that D ⊥ T | S . Then MEG T ( D ) ≤ MEG S ( D ) .

For example, in Figure 2, the agent must be at least as goal-directed towards S 3 as it is towards U 3 , since S 3 blocks all paths (i.e. d-separates) from { D 1 , D 2 } to U 3 .

Intuitively, this comes about because, in such cases, a rational agent wants the mediators to take the same values regardless of circumstances, making the instrumental control incentive indistinguishable from a terminal goal. This means we do not have to look arbitrarily far into the future to find evidence of goal-directedness. An agent that is goal-directed with respect to next week must also be goal-directed with respect to tomorrow.

5 Computing MEG in Markov Decision Processes

In this section, we give algorithms for computing MEG in MDPs. First, we define what an MDP looks like as a CID. We then establish that soft value iteration can be used to construct our maximum entropy policies, and give algorithms for computing MEG when the utility function is known or unknown.

Note that in order to run these algorithms, we do not need an explicit causal model, and so we do not have to worry about hidden confounders. We do, however, need black-box access to the environment dynamics, i.e., the ability to run different policies in the environment and measure whatever variables we are considering utility functions over.

Definition 5.1. A Markov Decision Process (MDP) is a CID with variables { S t , D t , U t } n t =1 , decisions D = { D t } n t =1 and utilities U = { U t } n t =1 , such that for t between 1 and n , Pa D t = { S t } , Pa U t = { S t } , while Pa S t = { S t -1 , D t -1 } for t > 1 , and Pa S 1 = ∅ .

Constructing Maximum Entropy Policies In MDPs, Ziebart's soft value iteration algorithm can be used to construct maximum entropy policies satisfying a set of linear constraints. We apply it to construct maximum entropy policies satisfying expected utility constraints.

Definition 5.2 (Soft Q-Function) . Let M = ( G,P ) be an MDP. Let β ∈ R \ { 0 } . For each D t ∈ D we define the soft Q-function Q soft β,n : dom( D t ) × dom( Pa D t ) → R via the recursion:

Using the soft Q-function, we show that there is a unique π ∈ Π maxent U ,u for each U and u in MDPs.

Theorem 5.1 (Maximum entropy policy in MDPs) . Let M = ( G,P ) be an MDP with utility function U , and let u ∈ att( U ) be an attainable expected utility. Then there exists a unique maximum entropy policy π maxent u ∈ Π maxent U ,u , and it has the form

̸

We refer to a policy of the form of π β as a soft-optimal policy with a rationality parameter of β .

Known Utility Function To apply Definition 3.1 to measure the goal-directedness of a policy π in a CID M with respect to a utility function U , we need to find the maximum entropy policy in Π maxent U which best predicts π . We can use Theorem 5.1 to derive an algorithm that finds π maxent u for any u ∈ att( U ) .

Fortunately, we do not need to consider each policy in Π maxent U ,u individually. We know the form of π maxent u , and only the real-valued rationality parameter β varies depending on u .

The gradient of the predictive accuracy with respect to β is

The predictive accuracy is a concave function of β , so we can apply gradient ascent to find the global maximum in β , which is the same as finding the maximum in u .

MEG U ( π ) can therefore be found by alternating between applying the soft value iteration of Definition 5.2 to find π maxent β , computing E π [ U ] -E π maxent β [ U ] , and taking a gradient step. See Algorithm 1.

If the β that maximises predictive accuracy is ∞ or -∞ , which can happen if π is optimal or anti-optimal with respect to U , then the algorithm can never reach the (nonetheless finite) value of MEG U ( π ) , but will still converge in the limit.

Algorithm 1 Known-utility MEG in MDPs Input: MDP M , policy π Output: MEG U ( π ) 1: initialise rationality parameter β , set learning rate α . 2: repeat 3: Apply soft value iteration to find Q soft β ▷ Definition 5.2 4: π soft β ← softmax ( β · Q soft β ) 5: g ← ( E π [ U ] -E π soft β [ U ] ) 6: β ← β + α · g 7: until Convergence 8: return E π [ ∑ D ∈ D ( log π soft β ( D | Pa D ) -log 1 | dom( D ) | )]

Unknown-utility algorithm To find unknown-utility MEG, we consider soft-optimal policies for various utility functions U and maximise predictive accuracy with respect to U as well as β . Let U Θ be a differentiable class of functions, such as a neural network, and write π soft θ,β for a soft-optimal policy with respect to U ∈ U Θ , and Q soft θ,β for the corresponding soft Q-function. We can take the derivative of the predictive accuracy with respect to θ and get E π [ ∇ θ U ] -E π maxent β [ ∇ θ U ] . Algorithm 2 extends Algorithm 1 to this case.

Algorithm 2 Unknown-utility MEG in MDPs Input: Parametric MDP M Θ over differentiable class of utility functions, policy π Output: MEG U Θ ( π ) 1: Initialise utility parameter θ , rationality parameter β , set learning rate α . 2: repeat 3: Apply soft value iteration to find Q soft θ,β ▷ Definition 5.2 4: π soft θ,β ← softmax ( β · Q soft θ,β ) 5: g β ← ( E π [ U θ ] -E π soft β [ U θ ] ) 6: g θ ← ( E π [ ∇ θ U θ ] -E π soft β [ ∇ θ U θ ] ) 7: β ← β + α · g β 8: θ ← β + α · g θ 9: until Stopping condition 10: return E π [ ∑ D ∈ D ( log π soft θ,β ( D | Pa D ) -log 1 | dom( D ) | )]

An important caveat is that if U θ is a non-convex function of θ (e.g. a neural network), Algorithm 2 need not converge to a global maximum. In general, the algorithm provides a lower bound for MEG U θ ( π ) , and hence for MEG T ( π ) where T = Pa U Θ . In practice, we may want to estimate the soft Q-function and expected utilities with Monte Carlo or variational methods, in which case the algorithm provides an approximate lower bound on goal-directedness.

6 Experimental Evaluation

We carried out two experiments 5 to measure known-utility MEG with respect to the environment reward function and unknown-utility MEG with respect to a hypothesis class of utility functions. We used an MLP with a single hidden layer of size 256 to define a utility function over states.

Our experiments measured MEG for various policies in the CliffWorld environment from the seals suite [Gleave et al., 2020]. Cliffworld (Figure 3a) is a 4x10 gridworld MDP in which the agent starts in the top left corner and aims to reach the top right while avoiding the cliff along the top row. With probability 0 . 3 , wind causes the agent to move upwards by one more square than intended. The

(a) CliffWorld

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0

10

20

30

ε

MEG of

ε

-greedy policy

known utility

unknown-utility

(b) MEG vs Optimality of policy

1

2

3

4

10

20

30

40

50

Length of goal region (squares)

MEG of optimal policy

known utility

unknown-utility

(c) MEG vs Task difficulty

environment reward function gives +10 when the agent is in the (yellow) goal square, -10 for the (dark blue) cliff squares, and -1 elsewhere. The dotted yellow line indicates a length 3 goal region.

MEG vs Optimality of policy In our first experiment, we measured the goal-directedness of policies of varying degrees of optimality by considering ε -greedy policies for ε in the range 0 . 1 to 0 . 9 . Figure 3b shows known- and unknown utility MEG for each policy. 6 Predictably, the goaldirectedness with respect to the environment reward decreased toward 0 as the policy became less optimal. So did unknown-utility MEG - since, as ε increases, the policy becomes increasingly uniform, it does not appear goal-directed with respect to any utility function over states.

MEGvs Task difficulty In our second experiment, we measured the goal-directedness of optimal policies for reward functions of varying difficulty. We extended the goal region of Cliffworld to run for either 1, 2, 3 or 4 squares along the top row and back column and considered optimal policies for each reward function. Figure 3c shows Cliffworld with a goal region of length 3. Figure 3b shows the results. Goal-directedness with respect to the true reward function decreased as the task became easier to complete. A way to interpret this is that as the number of policies which do well on a reward function increases, doing well on that reward function provides less and less evidence for deliberate optimisation. In contrast, unknown-utility MEG stays high even as the environment reward becomes easier to satisfy. This is because the optimal policy proceeds towards the nearest goal-squares and, hence, it appears strongly goal-directed with respect to a utility function which gives high reward to only those squares. Since this narrower utility function is more difficult to do well on than the environment reward function, doing well on it provides more evidence for goal-directedness. In Appendix D.3, we visualise the policies in question to make this more explicit. We also give tables of results for both experiments.

7 Limitations

Environment Access Although computing MEG does not require an explicit causal model (cf. Section 5), it does require the ability to run various policies in the environment of interest - we cannot compute MEG purely from observational data.

Intractability While MEG can be computed with gradient descent, doing so may well be computationally intractable in high dimensional settings. In this paper, we conduct only preliminary experiments - larger experiments based on real-world data may explore how serious these limitations are in practice.

Choice of variables MEG is highly dependent on which variables we choose to measure goaldirectedness with respect to. At one extreme, all (deterministic) policies have maximal goal-

directedness with respect to their own actions (given an expressive enough class of utility functions). This means that, for example, it would not be very insightful to compute the goal-directedness of a language model with respect to the text it outputs 7 . At the other extreme, if a policy is highly goal-directed towards some variable T that we do not think to measure, MEG may be misleadingly low. Relatedly, MEG may also be affected by whether we use a binary variable for T (or the decisions D ) rather than a fine-grained one with many possible outcomes. We should, therefore, think of MEG as measuring what evidence a set of variables provides about a policy's goal-directedness. A lack of evidence does not necessarily mean a lack of goal-directedness.

Distributional shifts MEG measures how predictive a utility function is of a system's behaviour on distribution , and distributional shifts can lead to changes in MEG. A consequence of this is that two almost identical policies can be given arbitrarily different goal-directedness scores, for example, if they take different actions in the start state of an MDP and thus spend time in different regions of the state space. It may be that by considering changes to a system's behaviour under interventions, as Kenton et al. [2023] do, we can distinguish 'true' goals from spurious goals, where the former predict behaviour well across distributional shifts, while the latter happen to predict behaviour well on a particular distribution (perhaps because they correlate with true goals) [Di Langosco et al., 2022]. We leave this to future work.

Behavioural vs mechanistic approaches The latter two points suggest that mechanistic approaches to understanding a system's goal-directedness could have advantages over behavioural approaches like MEG. For example, suppose we could reverse engineer the algorithms learned by a neural network. In that case, it may be possible to infer which variables the system is goal-directed with respect to and do so in a way which is robust to distributional shifts. However, mechanistic interpretability of neural networks [Elhage et al., 2021] is an ambitious research agenda that is still in its infancy, and for now, behavioural approaches appear more tractable.

Societal implications An empirical measure of goal-directedness could enable researchers and companies to keep better track of how goal-directed LLMs and other systems are. This is important for dangerous capability evaluations [Shevlane et al., 2023, Phuong et al., 2024] and governance [Shavit et al., 2023]. A potential downside is that it could enable bad actors to create more dangerous systems by optimising for goal-directedness. We judge that this does not contribute much additional risk, given that bad actors can already optimise a system to pursue actively harmful goals.

8 Conclusion

We proposed maximum entropy goal-directedness (MEG), a formal measure of goal-directedness in CIDs and CBNs, grounded in the philosophical literature and the maximum causal entropy principle. Developing such measures is important because many risks associated with advanced artificial intelligence come from goal-directed behaviour. We proved that MEG satisfies several key desiderata, including scale invariance and being zero with respect to variables that can't be influenced, and that it gives insights about instrumental goals. On the practical side, we adapted algorithms from the maximum causal entropy framework for inverse reinforcement learning to measure goal-directedness with respect to nonlinear utility functions in MDPs. The algorithms handle both a single utility function and a differentiable class of utility functions. The algorithms were used in some small-scale experiments measuring the goal-directedness of various policies in MDPs. In future work, we plan to develop MEG's practical applications further. In particular, we hope to apply MEG to neural network interpretability by measuring the goal-directedness of a neural network agent with respect to a hypothesis class of utility functions taking the network's hidden states as input, thus taking more than just the system's behaviour into account.

References

Yoshua Bengio. AI Scientists: Safe and Useful AI? https://yoshuabengio.org/2023/05/07/ ai-scientists-safe-and-useful-ai/ , 2023.

Martin Biehl and Nathaniel Virgo. Interpreting systems as solving pomdps: a step towards a formal understanding of agency. In International Workshop on Active Inference , pages 16-31. Springer, 2022.

Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, et al. Harms from increasingly agentic algorithmic systems. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , pages 651-666, 2023.

Daniel C Dennett. The intentional stance . MIT press, 1989.

Daniel C Dennett. From bacteria to Bach and back: The evolution of minds . WWNorton & Company, 2017.

Daniel C Dennett. The problem with counterfeit people. The Atlantic , 16, 2023.

Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning , pages 12004-12019. PMLR, 2022.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread , 1(1):12, 2021.

Tom Everitt, Ryan Carey, Eric D Langlois, Pedro A Ortega, and Shane Legg. Agent incentives: A causal perspective. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11487-11495, 2021a.

Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese , 198 (Suppl 27):6435-6467, 2021b.

James Fox, Matt MacDermott, Lewis Hammond, Paul Harrenstein, Alessandro Abate, and Michael Wooldridge. On imperfect recall in multi-agent influence diagrams. In Proceedings Nineteenth conference on Theoretical Aspects of Rationality and Knowledge, , volume 379 of Electronic Proceedings in Theoretical Computer Science , pages 201-220, 2023.

Iason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, et al. The ethics of advanced AI assistants. arXiv preprint arXiv:2404.16244 , 2024.

Adam Gleave and Sam Toyer. A primer on maximum causal entropy inverse reinforcement learning. 2022.

Adam Gleave, Pedro Freire, Steven Wang, and Sam Toyer. seals: Suite of environments for algorithms that learn specifications. https://github.com/HumanCompatibleAI/seals , 2020.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning , pages 1352-1361. PMLR, 2017.

Joseph Halpern and Max Kleiman-Weiner. Towards formal definitions of blameworthiness, intention, and moral responsibility. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Edwin T Jaynes. Information theory and statistical mechanics. Physical review , 106(4):620, 1957.

Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDermott, and Tom Everitt. Discovering agents. Artificial Intelligence , page 103963, 2023.

Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems , 30, 2017.

Matt MacDermott, Tom Everitt, and Francesco Belardinelli. Characterising decision theories with mechanised causal graphs. arXiv preprint arXiv:2307.10987 , 2023.

Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. Proc. of 17th International Conference on Machine Learning, 2000 , pages 663-670, 2000.

Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective. The Twelfth International Conference on Learning Representations , 2022.

Caspar Oesterheld. Formalizing preference utilitarianism in physical world models. Synthese , 193(9): 2747-2759, 2016.

Laurent Orseau, Simon McGregor McGill, and Shane Legg. Agents and devices: A relative definition of agency. arXiv preprint arXiv:1805.12387 , 2018.

Judea Pearl. Causality . Cambridge university press, 2009.

Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, et al. Evaluating frontier models for dangerous capabilities. arXiv preprint arXiv:2403.13793 , 2024.

Jonathan Richens and Tom Everitt. Robust agents learn causal world models. The Twelfth International Conference on Learning Representations , 2024.

Jonathan Richens, Rory Beard, and Daniel H Thompson. Counterfactual harm. Advances in Neural Information Processing Systems , 35:36350-36365, 2022.

Markus Schlosser. Agency. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Winter 2019 edition, 2019.

Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O'Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkin, Tyna Eloundou, Alan Hickey, et al. Practices for governing agentic AI systems. Research Paper, OpenAI, December , 2023.

Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324 , 2023.

Nathaniel Virgo, Martin Biehl, and Simon McGregor. Interpreting dynamical systems as bayesian reasoners. In Joint European conference on machine learning and knowledge discovery in databases , pages 726-762. Springer, 2021.

Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. JL & Tech. , 31:841, 2017.

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science , 18(6):1-26, 2024.

Francis Ward, Francesca Toni, Francesco Belardinelli, and Tom Everitt. Honesty is the best policy: defining and mitigating AI deception. Advances in Neural Information Processing Systems , 36, 2024a.

Francis Rhys Ward, Matt MacDermott, Francesco Belardinelli, Francesca Toni, and Tom Everitt. The reasons that agents act: Intention and instrumental goals. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems , pages 1901-1909, 2024b.

Paul Weirich. Causal Decision Theory. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Winter 2020 edition, 2020.

Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888 , 2015.

Dylan Xu and Juan-Pablo Rivera. Towards measuring goal-directedness in AI systems. arXiv preprint arXiv:2410.04683 , 2024.

Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy . Carnegie Mellon University, 2010.

Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the 27th International Conference on International Conference on Machine Learning , pages 1255-1262, 2010.

L

D

T

(a) CBN

D

T

˜

D

˜

T

(b) Mechanised CBN

D

T

˜

D

˜

T

(c) Mechanised CID

A Comparison to Discovering Agents

This paper is inspired by Kenton et al. [2023], who proposed a causal discovery algorithm for identifying agents in causal models, inspired by Daniel Dennett's view of agents as systems "moved by reasons" [Dennett, 1989]. Our approach has several advantages over theirs, which we enumerate below.

Mechanism variables. Kenton et al. [2023] assume access to a mechanised structural causal model , which augments an ordinary causal model with mechanism variables which parameterise distributions of ordinary object-level variables. An agent is defined as a system that adapts to changes in the mechanism of its environment. However, the question of what makes a variable a mechanism is left undefined, and indeed, the same causal model can be expressed either with or without mechanism variables, leading their algorithm to give a different answer. For example, Example 1 has identical causal structure to the mouse example in Kenton et al. [2023], but without any variables designated as mechanisms. Their algorithm says the version with mechanism variables contains an agent while the version without does not, despite them being essentially the same causal model. Figure 4 shows our example depicted as a mechanised structural causal model. We fix this arbitrariness by making our definition in ordinary causal Bayesian networks.

Utility variables. Their algorithm assumes that some variables in the model represent agents' utilities. We bring this more in line with the philosophical motivation by treating utilities as hypothesised mental states with which we augment our model.

Predictive accuracy. Kenton et al. [2023]'s approach formalises Dennett's idea of agents as systems 'moved by reasons'. We build on this idea but bring it more in line with Dennett's notion of what it means for a system to be moved by a reason - that the reason is useful for predicting its behaviour.

Gradualist vs Essentialist. The predictive error viewpoint gives us a continuous measure of goaldirectedness rather than a binary notion of agency, which is more befitting of the gradualist view of agents which inspired it.

Practicality. Their algorithm is theoretical rather than something that can be applied in practice. Instead, ours is straightforward to implement, as we demonstrate in Section 6. This opens up a range of potential applications, including behavioural evaluations and interpretability of ML models.

Interventional distributions. The primary drawback of MEG is that it doesn't necessarily generalise outside of the distribution. Running MEG on interventional distributions may fix this. We leave an extension of MEG to interventional distributions for future work.

B Proofs of MEG Properties

Proposition 3.1 (Translation and scale invariance) . Let M 1 be a CID with utility function U 1 , and let M 2 be an identical CID but with utility function U 2 = a · U 1 + b , for some a, b ∈ R , with a = 0 . Then for any policy π , MEG U 1 ( π ) = MEG U 2 ( π ) .

̸

Proof. Since MEG is defined by maximising over a maximum entropy policy set, showing that two utility functions have the same maximum entropy policy set is enough to show that they result in the same MEG for every policy.

̸

If π ∈ Π maxent U 1 , then π ∈ Π maxent U 1 ,u 1 for some u 1 ∈ att( U 1 ) , i.e. π is a maximum entropy policy satisfying the constraint that E π [ U 1 ] = u 1 . π thus also satisfies the constraint that E π [ U 2 ] = u 2 := a · u 1 + b . Note that any policy satisfying E π [ U 2 ] = u 2 also satisfies E π [ U 1 ] = u 1 because the map x ↦→ a · x + b is injective (since a = 0 ). Thus π must also be a maximum entropy policy satisfying E π [ U 2 ] = u 2 , and so π ∈ Π maxent U 2 ,u 2 and thus π ∈ Π maxent U 2 .

The converse is similar.

Proposition 3.2 (Bounds) . Let M be a CID with utility function U . Then for any policy π we have 0 ≤ MEG U ( π ) ≤ ∑ D ∈D log( | dom( D ) | ) , with equality in the lower bound if π is the uniform policy, and equality in the upper bound if π is the unique optimal (or anti-optimal) policy with respect to U .

Proof. Recall that

For the lower bound, first note that the uniform policy π unif is always an element of Π maxent U , since it is the maximum entropy way to attain a utility of E π unif [ U ] . It follows that

which shows both the lower bound and the fact that we attain it when π = π unif .

For the upper bound, note that the log π maxent ( D | Pa D ) term is at most 0 , so

To see that π being a unique optimal (or anti-optimal) policy is a sufficient condition for attaining the upper bound, note that there always exists a deterministic optimal and anti-optimal policy in a CID, so a unique such policy must be deterministic. Further, π must be in Π maxent U , since there is no higher entropy way to get the same expected utility. Then since E π [log π maxent ( D | Pa D )] = 0 , choosing π maxent = π obtains the upper bound.

Proposition 3.3 (No goal-directedness without causal influence) . Let M = ( G,P ) be a CID with utility function U and decision variables D such that, Desc ( D ) ∩ Pa U = ∅ . Then MEG U ( D ) = 0 .

Proof. Since U is not a descendant of D , it follows from the Markov property of causal Bayesian networks that U ⊥ D | Pa D . That means all policies achieve the same expected utility u . So, the maximum entropy policy set Π maxent U contains only the uniform policy. We get that

Theorem 4.1 (Pseudo-terminal goals) . Let M = (( V , E ) , P ) be a CBN. Let D , T , S ⊆ V such that D ⊥ T | S . Then MEG T ( D ) ≤ MEG S ( D ) .

Proof. We will show that the maximum entropy policy set Π maxent U T (where U T is the set of all utility functions over T ) is a subset of Π maxent U S , so the maximum predictive accuracy taken over the latter upper bounds the maximum predictive accuracy taken over the former.

Let π ∈ Π maxent U T , so π = π maxent U ,u for some U T ∈ U T . If we can find a utility function U S ∈ U S such that for all π , E π [ U S ] = E π [ U T ] , then a maximum entropy policy with E π [ U T ] = u must also be a maximum entropy policy with E π [ U S ] = u . It would follow that π ∈ Π maxent U T and so Π maxent U S ⊆ Π maxent U T .

To construct such a utility function, let U S ( s ) = ∑ t P ( T = t | S = s ) U T ( t ) . Note that since D ⊥ T | S , P ( T = t | S = s ) is not a function of π . Then for any π ,

C Proof of Theorem 5.1

Lemma C.1. Let π β, U denote a soft-optimal with respect to utility function U , defined as in Theorem 5.1, with rationality parameter β . Then for any α ∈ R , we have that π α · β, U = π β,α ·U .

Proof. If either α or β are equal to 0 , both policies are uniform and we are done. Otherwise, write Q soft β, U for the soft-Q function for U with rationality parameter β. We first show that Q soft β,α ·U = α · Q soft α · β, U by backwards induction.

And for t < n , assuming the inductive hypothesis holds for t +1 ,

Then, substituting this into the definition of π α · β, U we get

̸

Proof. In an MDP, the expected utility is a linear function of the policy, so the attainable utility set is a closed interval att( U ) = [ u min , u max ] . We first consider the case where u ∈ ( u min , u max ) .

In this case, we are seeking the maximum entropy policy in an MDP with a linear constraint satisfiable by a full support policy (since u is an interior point), so we can invoke Ziebart's result on the form of such policies [Ziebart, 2010, Ziebart et al., 2010, Gleave and Toyer, 2022]. In particular, our feature is the utility U . We get that the maximum entropy policy is a soft-Q policy for a utility function β · U with a rationality parameter of 1 , where β = argmax β ′ ∈ R ∑ t E π [ log( π soft β ′ ( d t | pa t )) ] . By Lemma C.1 this can be restated as a soft-Q policy for U with a rationality parameter of β . It follows from Ziebart that β = argmax β ′ ∈ R π soft β ′ , and allowing β = ∞ or -∞ does not change the argmax.

In the case where u ∈ { u min , u max } , it's easy to show that the maximum entropy policy which attains u randomises uniformly between maximal value actions (for u max ) or minimal value actions (for u min ). These policies can be expressed as lim β →∞ π soft β and lim β →-∞ π soft β respectively.

D Experimental Details

D.1 Tables of results

(a) Occupancy measures of optimal policy when k = 1 .

(b) Occupancy measures of optimal policy when k = 4 .

D.2 Visualising optimal policies for different lengths of goal region.

Figure 5a and Figure 5b show the occupancy measures for an optimal policy for k=1 and k=4 respectively, where k is the length of the goal region in squares. Although the goal region is larger in the latter case, the optimal policy consistently aims for the same sub-region. This explains why unknown-utility MEG is higher than MEG with respect to the environment reward. The policy does just as well on a utility function whose goal-region is limited to the nearer goal squares as it does on the environment reward, but fewer policies do well on this utility function, so doing well on it constitutes more evidence for goal-directedness.

D.3 Further details

The experiments were carried out on a personal laptop with the following specs:

Hardware model: LENOVO20N2000RUK

Processor: Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, 2112 Mhz, 4 Core(s), 8 Logical Processor(s)

Memory: 24.0 GB

We used an environment from the SEALS library 8 , and adapted an algorithm from the imitation library 9 . Both are released under the MIT license.

For information on hyperparameters, see the code.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions of the paper are listed at the end of the introduction, with references to the relevant sections of the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 7 contains a section dedicated to limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs of all theoretical results are given in the supplementary material.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details of the experiments are describe in Section 6. Code is included in the supplementary material.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is included in the supplementary material.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Full details are provided via the code.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report two sigma error bars for unknown-utility MEG experiments. Known- utility MEG experiments are deterministic, so we do not report error bars.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Information on compute resources is given in Appendix D.3.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted conforms to the code of ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Section 7 contains a section discussing societal impacts.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release such data or models.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators of the libraries we used are credited, the licenses are respected, and explicitly mentioned in Appendix D.3.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a measure of evidence for goal-directedness based on the maximum causal entropy principle, MEG. It is operationalized as the ability to predict variable D based on the hypothesis that it is optimizing a utility function U, where U represents the goal and D the agent’s decisions. Specifically, they try to predict the agents' decisions using the maximum-entropy policy that achieves the same utility as the agent’s own policy. MEG is essentially the negative cross-entropy of this prediction. The authors also describe algorithms for estimating MEG, and demonstrate it in action with some small-scale experiments.\n\nSTRENGTHS:\n- The work is well-motivated and situated among prior work \n\n- The writing is clear and the examples provided are very helpful for understanding\n\n- The MEG satisfies some nice properties such as invariance to translation and scaling\n\n- The authors allow MEG to cover a wide range of use cases, for instance\n\n  - It can be estimated even when only a set of trajectories is available, rather than the agent’s policy itself. \n\n  - It can be extended to settings where the the utility function is unknown\n\nWEAKNESSES:\n- It is unclear how useful MEG will be for checking goal-directedness of generalist AI agents acting in the real world, because:\n  - it is expensive to compute, since it requires optimizing the maximum entropy policy, and their estimation algorithm can only guarantee a lower bound on MEG for the unknown-utility case\n  - It requires a causal model containing the target variables that the agent might be trying to manipulate.\n\n- The name and framing of MEG is slightly misleading: MEG is not directly a measure of how goal-directed an agent is, but rather the amount of evidence for goal-directedness. The maximum value MEG can take varies based on the number of possible choices in an environment. Thus, it may be difficult to compare the goal-directedness of different agents’ behavior if we could only observe them acting in different environments. (e.g., to determine if agents acting in our constantly changing world are becoming increasingly goal-directed)\n\n- typo: agentcy on line 18\n\n- Capitalization of v at the end of definition 2.1 is inconsistent\n\nQUESTIONS:\nWhy does the maximizing policy in equation 1 always obtain the same expected utility as pi?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* The work introduces a framework for quantifying the goal-directedness of a\n  decision-making system with respect to a particular utility function or a\n  family of utility functions.\n* The method is grounded in Dennett's *intentional stance* and Jaynes'\n  *maximum entropy inference*. The authors have made some additional choices\n  while formalising these principles to arrive at their formulation, which\n  ends up similar to Ziebart's Maximum Causal Entropy IRL framework.\n  The resulting formula is essentially a measure of how close the subject\n  policy comes, in terms of cross entropy, to a maximum entropy formula for\n  the utility function at some inverse temperature.\n* The authors establish that the resulting formula, MEG, satisfies desirable\n  properties for a measure of goal-directedness, namely being invariant to\n  the scaling and translation of utility functions, having interpretable\n  upper and lower bound policies for a given utility function, and measuring\n  only causal goal-directedness.\n* The authors extend their measure of goal-directedness with respect to a\n  single utility function into an aggregate measure of goal-directedness with\n  respect to a family of utility functions (simply by taking the maximum MEG\n  over the family of utility functions). This also allows them to express a\n  measure of goal-directedness with respect to a set of 'target variables',\n  by considering the family of utility functions defined over those\n  variables.\n* The authors derive algorithms for computing both kinds of MEG in the\n  setting of causal influence models derived from MDPs. The algorithms are\n  based on gradient-based search requiring repeatedly computing maximum\n  entropy policies for various inverse temperatures (and, in the case of a\n  family of utility functions, varying family parameters).\n* The authors implement their algorithm for some simple MDPs as a 'proof of\n  concept' experiment, demonstrating how various factors influence the\n  measure of goal-directedness.\n\nSTRENGTHS:\nI thank the authors for submitting their valuable work which is in a position to make an excellent contribution to the field.\n\n* I think the goals of the work are exceptionally well-motivated. The authors\n  did not spend much time elaborating on the need for measures of goal\n  directedness, so I note that such measures have immediate applications in\n  attempts to rigorously study risks from learned optimisation in deep\n  learning. Emergent goal-directedness is a core component of the problem of\n  AI safety (for modern systems and future more capable systems) and scalable\n  methods for quantifying goal-directedness are sorely needed by researchers\n  such as myself who want to better understand this phenomenon in order to\n  reduce or avert these risks.\n* The authors have taken careful steps to justify their proposed framework\n  for measuring goal-directedness, including showing that it satisfies some\n  sensible properties and showing that in simple examples (including toy\n  experiments with some MDPs) the values given by the MEG measure are\n  sensible.\n* Overall, the paper is fairly clearly written and I feel I was able to\n  understand what the authors have to say. I think that this strength is\n  limited in that there may be some room for improvement in the flow of\n  sections and in some low-level details of the writing to make the paper\n  optimally clear and accessible (see below), however overall it was\n  passable.\n\nWEAKNESSES:\nI noted the following issues with the work, based on my understanding.\n\n1.  MEG is described as \"philosophically grounded\" because of its roots in an\n    interpretation of Dennett's Intentional Stance. However, it does not\n    appear to be a unique interpretation and this philosophy does not appear\n    to be the only possible philosophical approach to defining agency.\n\n    * Related works Kenton et al. and Orseau et al. both appeal to Dennett's\n      philosophy but arrive at different formalisms. The authors have not\n      established that their interpretation is uniquely philosophically\n      grounded. (Appendix A begins to address this with respect to Kenton et\n      al.'s prior work, but it would be good to see a discussion of the\n      relationship with Orseau et al.'s approach.)\n\n    * If MEG were a unique formalisation of Dennett's philosophy, it would\n      still appear to be a behaviourist approach, which stands in contrast to\n      a mechanistic approach that looks into the internals of a system for\n      explicit representations of goals and for the implementation of\n      decision-making processes.\n\n2.  I am concerned that the statements of proposition 3.1 and 3.2\n    are incorrect as stated.\n\n    * **Proposition 3.1.** There must be some missing assumption in this\n      theorem statement that $a$ is a positive scalar. Otherwise you have\n      proven that a MEG policy for any given utility function is also\n      goal-directed towards a negated version of that policy. Could you\n      please clarify whether this is the case or whether I am missing\n      something that means that a policy that pursues the negative of a\n      utility function is somehow goal-directed towards that utility\n      function?\n\n    * **Proposition 3.2.** It is well known that in many examples there are\n      decisions for which multiple actions lead to equal expected utility and\n      are therefore equally optimal. In such cases there is no unique optimal\n      policy. What happens to the bounds in this case? The statement of the\n      proposition would appear to be ill-defined in case there is no unique\n      optimal policy. Is it the case that you do not derive an upper bound in\n      this case? The distinction being then that there still will be some\n      upper bound (perhaps?) but that you have not characterised it?\n\n    When I checked the proofs for these propositions in the appendix I found\n    them to provide insufficient detail to resolve my concerns or point to a\n    particular mistake in the reasoning.\n\n    I am recommending acceptance of the paper on the assumption that these issues will be easy to dismiss or to correct.\n\n3.  In the unknown-utility case, MEG fails to account for complexity of\n    utility functions within the hypothesis class.\n\n    * It is well known in RL that for any policy, a utility function exists\n      that can rationalise that policy. For example, there is a reward\n      function that immediately, highly rewards whichever actions that policy\n      happens to take in each state.\n    * This leads me to wonder if for any policy it is possible to find some\n      reward function for which MEG very strongly indicates that this policy\n      is goal-directed towards that utility function.\n    * Especially in the case of MEG for unknown utility functions, it is\n      therefore possibly that if the hypothesis class of utility functions is\n      too rich, the MEG for unknown utility functions will register almost\n      any policy as goal-directed.\n    * I think something that is missing is some way of accounting for the\n      complexity of utility functions. Philosphically, a complex utility\n      function is not 'useful for predicting behaviour' if it is itself as\n      complex as the behaviour, even if it is capable of predicting behaviour\n      well.\n    * I think a significant weakness of MEG, especially in the\n      unknown-utility setting, is that it leaves the user of the framework to\n      rule out too-complex utility functions by the coarse tool of including\n      them or excluding them from the hypothesis class, rather than providing\n      a principled tool for managing this complexity.\n  \n    If the authors are aware of this issue but consider addressing it to be\n    out of scope, I would like to see it acknowledged as such in the main\n    text.\n\n4.  If I am not mistaken, the scalability of the method for computing MEG\n    relies on solving an RL problem (computing a maximum entropy policy in\n    MDPs). If this is the case then I think the scalability of the method is\n    questionable.\n\n    * It is certainly common practice to solve RL problems, and in any case\n      where we can train a policy that will be the target of MEG evaluation,\n      we can presumably also train a max-ent policy to measure its MEG.\n      Perhaps calculating such a policy repeatedly will be expensive but it\n      should at least be feasible to do so.\n    * However, the very applications I think motivate this work are settings\n      where we don't understand the internal motivational system of the\n      trained system (and so MEG can help quantify that). This would of\n      course only happen if we are training the subject policy in such a\n      complex environment that its emergent goal-directedness is in question,\n      due to, for example, potential goal misgeneralisation (or so-called\n      'inner alignment failure').\n    * In such a setting, I suppose we would share the same concerns about the\n      max-ent policy required for quantifying MEG. If I am not mistaken, if\n      this probe policy's generalisation could not be trusted, then neither\n      could we trust the result of using it to calculate the MEG of the\n      subject policy.\n\n    There is a comment in the discussion to the effect that future work can\n    consider the setting of 'distributional shifts' which is normally\n    considered to be a prerequisite for the phenomenon of goal\n    misgeneralisation. However, I think this concern undercuts the motivation\n    of the present work in its early stages and if there is no feasible way\n    to reliably compute the MEG for complex learned systems where the\n    internal goals are in question then the work may not achieve its goals in\n    the future, and this issue should be raised and (ideally) addressed now.\n\n5.  There is insufficient explanation of relationship to prior work, particularly to IRL and agents and devices.\n\n    * **Inverse reinforcement learning.** IRL represents a large body of\n      work including theoretical work aimed at finding goals that\n      rationalise particular policies, which is clearly closely related to\n      MEG. The authors' brief comments in the related work section have not\n      clarified to me the relationship between MEG and IRL.\n      In particular, it seems plausible to me that MEG is essentially\n      functions similar to a subcomponent of a IRL framework based on\n      Bayesian inference.\n\n        * A Bayesian approach for IRL for example involves positing a\n          likelihood of the behavioural data given each goal. This likelihood\n          function represents a kind of measure of how well the behaviour can\n          be explained by that goal. This seems similar to MEG, though of\n          course MEG does not adopt a Bayesian framework.\n        * Similarly, what is the relationship between unknown-utility MEG\n          with a given hypothesis class and the model evidence in a Bayesian\n          formulation of IRL with the same hypothesis class? Of course there\n          is a difference of integrating versus taking a max, but my point is\n          that these are different approaches to the same kind of problem.\n    \n        I think the paper would be strengthened if the authors could clarify\n        whether MEG is addressing a fundamentally new kind of problem or one\n        that arises naturally in the course of IRL. If it's not new, that\n        seems fine, the approach is apparently new---but it would be nice to\n        see a discussion of how the approach compares to the Bayesian\n        approach outlined above.\n    \n    * **Agents and devices.** This prior works appears to present a\n      Bayesian framework for distinguishing agents from non-agents, along\n      with one particular instantiation of this general framework in terms\n      of epsilon-greedy suboptimality and Solomonoff induction.\n      The comments in the related work section appear to relate to the\n      instantiation. What is the relationship between the present work and\n      the general framework from Orseau et al.?\n      For example, would it be possible to derive a maximum-entropy version\n      of Orseau et al.'s framework, and if so, how might that compare to\n      MEG? Does the difference come down to the difference between Bayesian\n      inference and maximum entropy inference?\n    \n    I would have liked to see sections along the lines of appendix A (for\n    Kenton et al.) drawing out these relationships.\n\nIf the authors are able to address each of these issues to my satisfaction I think my assessment of the paper's strength would be greatly improved and I would be willing to substantially strengthen my recommendation to accept the paper.\n\nQUESTIONS:\nMany of the weaknesses I raised above are partially framed as questions since\nI am not fully confident that I have understood the work in forming these\ncomments (I welcome corrections from the authors).\n\nHere I collect additional questions that are less crucial to the overall\nvalidity of the work but are still either barriers to my understanding or\nseem like minor issues in the presentation.\n\n* The introduction claims that \"pursuing goals is about having a particular\n  causal effect on the environment.\" This appears to be part of the case for\n  MEG being \"philosophically grounded\".\n  I can see how in many cases this correspondence holds, but it is not\n  immediately obvious to me why there couldn't be cases where the pursuit of\n  goals is separated from causality. For example, if a goal is always going\n  to be achieved counterfactually regardless of an agent's actions, then it\n  becomes possible to 'pursue' this goal without having any causal effect on\n  its achievement.\n  Do the authors have some further justification for this framing, or a\n  citation that gives such justification?\n\n* Equation 1:\n\n    * Can the definition of MEG be interpreted as a minimum KL divergence\n      between the policy and the closest maximum-entropy policy?\n    * The cross entropy between two distributions is not symmetric. Why is is\n      justified to take the cross entropy in this particular direction?\n\n* What is the relationship between theorem 5.1 and prior work on maximum\n  entropy RL such as Haarnoja et al. 2017 \"Reinforcement learning with deep\n  energy-based policies\"?\n\n* In the discussion section, under \"societal implications\", there is a\n  passage \"the relationship between goal-directedness and danger is fairly\n  indirect\".\n\n    * I wonder, did the authors mean to write \"the relationship between\n      *measuring* goal-directedness and danger is fairly indirect\"?\n    * If not, could the authors please elaborate on this indirectness, since it\n      seems that the goal-directedness itself is fairly tightly and directly\n      connected to risks (as outlined in the works cited in the introduction).\n\n\nNot a question, but while reading the paper I happened to note the following\nminor presentation issues the authors may like to address in future versions.\n\n* Line 5: I think the authors mean \"adaptation\" rather than \"adaption\".\n* Line 14: The word \"mundane\" seems to downplay the real and potentially\n  devastating impacts discussed in Chan et al. Consider a different word\n  or a different angle of contrast (perhaps the scale of the harms?).\n* Line 16: Shavit et al. citation and reference missing a date.\n* Line 18: \"Agentcy\".\n* Line 22: Incomplete sentence, consider \"According to Dennett... [1989],\n  *we are justified in doing so whenever it* is useful ...\".\n* Line 31: MEG acronym was introduced in the abstract is undefined in the\n  scope of the introduction.\n* Line 65: Orseau et al. should be cited in text mode.\n* Line 78: Something appears to have allowed the bottom margin on page 2\n  to shrink, causing the final lines on the page to come too close to the\n  page number.\n* Algorithm 1 line 7: Until convergence of what? Not beta, for example.\n* Line 294: This and the next paragraph headings are missing periods on\n  the abbreviation \"vs.\".\n* Line 296: \"meg\" should be capitalised.\n* Footnote 1 is missing a period, at the footnote mark (line 296) AND at\n  the end of the footnote text.\n* Line 300: Potential missing word, \"optimal *policies*\"?\n\nLIMITATIONS:\nIt would have been good to see a more acknowledgement of the limitations of\nthe work (such as those I have enumerated above in the weaknesses section, pending the authors' clarification), throughout the paper or in a dedicated section of the paper.\n\nSome additional points on addressing these limitations in the paper:\n\n* You could more carefully justify or qualify the discussion of the extent to which the\n  approach is uniquely principled / philosophically motivated.\n\n    For example, in the conclusion, MEG is described as \"grounded in the\n    philosophical literature\" when the authors appear to just mean \"grounded in\n    the philosophy of Dennett\" (as they do qualify elsewhere), the difference between which I outline in Weakness (1).\n\n* The scalability of the work does appear to depend on repeatedly computing a\n  maximum entropy policy, which would appear potentially many times more\n  expensive than computing the subject policy, though perhaps feasible. This\n  is based on my understanding. I would have liked to see the authors clarify\n  the requirements in comparison to the cost of finding the target policy\n  explicitly, since they claim their method is \"scalable\".\n\nSee also weaknesses (3) and (4).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces maximum entropy goal-directedness (MEG), a measure of how much an agent is goal directed towards a given utility function. The authors extend the theoretical framework introduced to the case where the utility function that is being optimized is not known. Moreover, they propose an algorithm to measure MEG in Markov Decision Processes. \nTo show the practicality of the metric introduced, the authors show the performances of the algorithms empirically. Specifically, they show how MEG relates with the optimality of a $\\epsilon$-greedy policy, and how it relates to task difficulty.\n\nSTRENGTHS:\n- The paper is theoretically sound, and well presented\n- I found the images and explicit examples to be particularly helpful in understanding how MEG works and why it was defined in a specific way\n- Section 4 addresses an important problem, and I found that it provides clear results which are also well explained. \n- I have not seen much similar work, and I definitely think it addresses an important problem, especially when talking about future goal-directed LLM systems.\n\nWEAKNESSES:\n- As mentioned by the authors, MEG relies on having full access to the causal model of the environment. This can be a highly-demanding assumption in many real environments.\n- Given the restrictive assumption mentioned above, the paper should at least investigate in more detail how computationally expensive is computing MEG in large-scale setting. \n- The experiments are limited to few toy examples\n\nQUESTIONS:\n- How do you think one could make MEG more robust towards distributional shifts? \n\n- Do you have any example in mind on how MEG could fail when considering how goal-directed an agent is? I think it could be an important point to discuss to give a better perspective to the reader on when one would want to employ such metric. Maybe you have some toy examples which can illustrate this?\n\n- Is it possible for two similar policies to achieve drastic differences in MEG? Do you have any experiments which shows how robust of a metric it is?\n\n-How one could extend MEG to cases where the causal environment is only partially known?\n\n- Do you have some experiments/comparison/thoughts on how MEG relates directly with agency?\n\nLIMITATIONS:\nno ethical limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is studying a problem that is both mathematical and philosophical in nature: how can we measure quantitatively, whether an observed behaviour is goal-directed or not (or to what extent it is). It tackles this problem by starting from a causal model of the world, where there are explicit utility variables, state variables, and decision variables. For a known utility function, it proposes to measure goal-directedness by essentially asking: given an observed policy $\\pi$, find a policy that is maximizing the utility function but regularised by entropy that is _most predictive_ of $\\pi$. Compare its predictive accuracy to uniform prediction, and this quantifies goal-directedness. They also extend this to the case of unknown utilities, by considering parameterized sets of utility functions. The favourable properties of this measure is discussed, and its practicality is demonstrated with a simple empirical result on CliffWorld.\n\nSTRENGTHS:\n- The question of measuring goal-directedness is tremendously significant, since it is a core requirement for agency. \n- The proposed solution is sufficiently novel. Even though it builds on the Maximum Causal Entropy IRL, it sufficiently extends it and applies it in a completely new way. \n- The paper is written very clearly and it is easy to follow.\n\nWEAKNESSES:\n- Societal implications could have been discussed in more detail, especially how this \"could enable bad actors to create even more dangerous systems.\" An extended discussion about this added to the appendix would improve this.\n\n- A more comprehensive empirical demonstration of the practicality would improve the paper.\n\nQUESTIONS:\nQ: I have one question about the unknown utility setting. If I get this right, in general, if the set of possible utility functions is large enough, and the causal model is rich enough, I can potentially find a utility function making many behaviours goal-directed, even though they may not be. For example, consider the behaviour of a river, as its flow. If my causal model represents enough things relevant (e.g. elevation, geography, physics, weather etc.), I can potentially find a utility function in the lines of _minimize travel distance to the nearest sea/lake_ where elevation incurs a cost, and conclude that the flow of the river is goal-directed. Is this correct? If so, it seems like one must be careful in terms of the class of utility functions they consider. \n\nBonus Q: I know that the paper assumes a known causal model. However, I am curious. How would we deal with hidden confounders here? Is there no hope? Specifically, could there be reasons internal to the agent that make a seemingly non-goal-directed behaviour in fact goal-directed, unbeknownst to us?\n\nLIMITATIONS:\nPaper discusses limitations sufficiently.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}