Measuring Goal-Directedness

Authors: matt  macdermott
james  fox
francesco  belardinelli
tom  everitt
google  deepmind
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:10:01

Abstract: We define maximum entropy goal-directedness (MEG), a formal measure of goal-
directedness in causal models and Markov decision processes, and give algorithms
for computing it. Measuring goal-directedness is important, as it is a critical
element of many concerns about harm from AI. It is also of philosophical interest,
as goal-directedness is a key aspect of agency. MEG is based on an adaptation of
the maximum causal entropy framework used in inverse reinforcement learning. It
can measure goal-directedness with respect to a known utility function, a hypothesis
class of utility functions, or a set of random variables. We prove that MEG satisfies
several desiderata and demonstrate our algorithms with small-scale experiments.

Main paper: 
 introduction: 
 In order to build more useful AI systems, a natural inclination is to try to make them more agentic. But while agents built from language models are touted as the next big advance [Wang et al., 2024] , agentic systems have been identified as a potential source of individual [Dennett, 2023] , systemic [Chan et al., 2023 , Gabriel et al., 2024] , and catastrophic [Ngo et al., 2022] risks. Agency is thus a key focus of behavioural evaluations [Shevlane et al., 2023 , Phuong et al., 2024] and governance [Shavit et al., 2023] . Some prominent researchers have even called for a shift towards designing explicitly non-agentic systems [Dennett, 2017 , Bengio, 2023] .A critical aspect of agency is the ability to pursue goals. Indeed, the standard theory of agency defines agency as the capacity for intentional action -action that can be explained in terms of mental states such as goals [Schlosser, 2019] . But when are we justified in ascribing such mental states? According to Daniel Dennett's instrumentalist philosophy of mind [1989] , we are justified when doing so is useful for predicting a system's behaviour. This paper's key contribution is a method for formally measuring goal-directedness based on Dennett's idea. Since pursuing goals is about having a particular causal effect on the environment 2 , defining our measure in a causal model is natural. Causal models are general enough to encompass most frameworks popular among ML practitioners, such as single-decision prediction, classification, and regression tasks, as well as multi-decision (partially observable) Markov decision processes. They also offer enough structure to usefully model many ethics and safety problems [Everitt et al., 2021a , Ward et al., 2024a , Richens et al., 2022 , Richens and Everitt, 2024 , Everitt et al., 2021b , Ward et al., 2024b , Halpern and Kleiman-Weiner, 2018 , Wachter et al., 2017 , Kusner et al., 2017 , Kenton et al., 2023 , Fox et al., 2023 , Richens and Everitt, 2024 , MacDermott et al., 2023] . 1 A notebook that can be used to apply our algorithms in your own tabular MDPs is available at https: //colab.research.google.com/drive/1p_RA7EeN6DWx6x3M4v45R6L84koa4rf_ 2 This is a common but not universal view in philosophy, see Weirich [2020] .38th Conference on Neural Information Processing Systems (NeurIPS 2024). Maximum entropy goal-directedness (MEG) operationalises goal-directedness as follows, illustrated by the subsequent running example.A variable D in a causal model is goal-directed with respect to a utility function U to the extent that the conditional probability distribution of D is well-predicted by the hypothesis that D is optimising U.Example 1. A mouse begins at the centre of a gridworld (Figure 1a ). It observes that a block of cheese is located either to the right or left (S) with equal probability, proceeds either away from it or towards it (D), and thus either obtains the cheese or does not (T ).Suppose that the mouse moves left when the cheese is to the left and right when it is to the right, thus consistently obtaining the cheese. Intuitively, this behaviour seems goal-directed, but can we quantify how much? Figure 1 gives an overview of our procedure. We first model the system of interest as a causal Bayesian network (Figure 1b ) with variables S for the cheese's position, D for the mouse's movement, and T for whether or not the mouse obtains the cheese. Then, we identify a candidate decision variable D and target variable T , hypothesising that the mouse optimises a utility function that depends on T (Figure 1c ). Finally, we form a model of what behaviour we should expect from D if it is indeed optimising U and then measure how well this model predicts D's observed behaviour (Figure 1d ).Ziebart [2010] 's maximum causal entropy (MCE) framework suggests a promising way to construct a model for expected behaviour under a given utility function. However, there are several obstacles to applying it to our problem: it cannot measure the predictive usefulness of known utility functions, and it only finds the most predictive linear utility function. In practice, arbitrary known utility functions can be plugged in, but the results are not scale-invariant. We overcome these difficulties by returning to first principles and deriving an updated version of the MCE framework.In summary, our contributions are four-fold. We (i) adapt the MCE framework to derive maximum entropy goal-directedness (MEG), a philosophically-motivated measure of goal-directedness with respect to known utility functions, and show that it satisfies several key desiderata (Section 3); (ii) we extend MEG to measure goal-directedness in cases without a known utility function (Section 4);(iii) we provide algorithms for measuring MEG in MDPs (Section 5); and (iv) we demonstrate the algorithms empirically (Section 6).Related Work. Inverse reinforcement learning (IRL) [Ng and Russell, 2000] focuses on the question of which goal a system is optimising, whilst we are interested in to what extent the system can be seen as optimising a goal. We are not the first to ask this question. Some works take a Bayesian approach inspired by Dennett's intentional stance. 3 Most closely related to our work is Orseau et al. [2018] , which applies Bayesian IRL in POMDPs using a Solomonoff prior over utility functions and an ε-greedy model of behaviour. This lets them infer a posterior probability distribution over whether an observed system is a (goal-directed) "agent" or "just a device". Our approach distinguishes itself from these by considering arbitrary variables in a causal model and deriving our behaviour model from the principle of maximum entropy. Moreover, since our algorithms can take advantage of differentiable classes of utility functions, our approach may be amenable to scaling up using deep neural networks. Oesterheld [2016] combines the intentional stance with Bayes' theorem in cellular automata, but does not consider specific models of behaviour. Like us, Kenton et al. [2023] consider goal-directedness in a causal graph. However, they require variables to be manually labelled as mechanisms or object-level, and only provide a binary distinction between agentic and non-agentic systems (see Appendix A for a detailed comparison). Biehl and Virgo [2022] , Virgo et al. [2021] propose a definition of agency in Moore machines based on whether a system's internal state can be interpreted as beliefs about the hidden states of a POMDP. 
background: 
 We use capital letters for random variables V , write dom(V ) for their domain (assumed finite), and use lowercase for outcomes v ∈ dom(V ). Boldface denotes sets of variables V = {V 1 , . . . , V n }, andtheir outcomes v ∈ dom(V ) = × i dom(V i ).Parents and descendants of V in a graph are denoted by Pa V and Desc V , respectively (where pa V and desc V are their instantiations).Causal Bayesian networks (CBNs) are a class of probabilistic graphical models used to represent causal relationships between random variables [Pearl, 2009] . Definition 2.1 (Causal Bayesian network). A Bayesian network M = (G, P ) over a set of variables V = {V 1 , . . . , V n } consists of a joint probability distribution P which factors according to a directed acyclic graph (DAG) G, i.e.,P (V 1 , . . . , V n ) = n i=1 P (V i | Pa Vi ), where Pa Vi are the parents of V i in G. A Bayesian network is causal if its edges represent direct causal relationships, or formally if the result of an intervention do(X = x) for any X ⊆ V can be computed using the truncated factorisation formula:P (v | do(X = x)) = Π i:vi / ∈x P (v i | pa vi ) if v is consistent with x or P (v | do(X = x)) = 0 otherwise.Figure 1b depicts Example 1 as a CBN, showing the causal relationships between the location of the cheese (S), the mouse's behavioural response (D), and whether the mouse obtains the cheese (T ).We are interested in to what extent a set of random variables in a CBN can be seen as goal-directed. That is, to what extent can we interpret them as decisions optimising a utility function? In other words, we are interested in moving from a CBN to a causal influence diagram (CID), a type of probabilistic graphical model that explicitly identifies decision and utility variables. Definition 2.2 (Causal Influence Diagram [Everitt et al., 2021a] ). A causal influence diagram (CID) M = (G, P ) is a CBN where the variables V are partitioned into decision D, chance X, and utility variables U . Instead of a full joint distribution over V , P consists of conditional probability distributions (CPDs) for each non-decision variable V ∈ V \ D.A CID can be combined with a policy π, which specifies a CPD π D for each decision variable D, in order to obtain a full joint distribution. We call the sum of the utility variables the utility function and denote it U = U ∈U U . Policies are evaluated by their total expected utility E π [U]. We write π unif for the uniformly random policy.CIDs can model a broad class of decision problems, including Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs) [Everitt et al., 2021b] . Example 2 (POMDP). A mouse begins at the centre of the grid, with a block of cheese located either at the far left or the far right (Figure 2 ). The mouse does not know its position or the position of the cheese (S 1 ) but can smell which direction the cheese is in (O 1 ) and decide which way to move (D 1 ). Next step, the mouse again smells the direction of the cheese (O 2 ) and again chooses which way to proceed (D 2 ).3 Measuring goal-directedness with respect to a known utility function Maximum Entropy Goal-directednes. Dennett's instrumentalist approach to agency says that we can ascribe mental states (such as utilities) to a system to the extent that doing so is useful for predicting its behaviour [Dennett, 1989] . To operationalise this, we need a model of what behaviour is predicted by a utility function. According to the principle of maximum entropy [Jaynes, 1957] , we should choose a probability distribution with the highest entropy distribution satisfying our requirements, thus minimising unnecessary assumptions (following Occam's razor). We can measure the entropy of a policy by the expected entropy of its decision variables conditional on their parents Ziebart et al. [2010] 's causal entropy, which we usually refer to as just the entropy of π. In our setting, the relevant constraint is expected utility. To avoid assuming that only optimal agents are goal-directed, we construct a set of models of behaviour which covers all levels of competence an agent optimising utility U could have. We define the set of attainable expected utilities in a CID as H π (D || Pa D ) = -D∈D E d,Pa D ∼Pπ log π D (d | Pa D ). This isatt(U) = {u ∈ R | ∃π ∈ Π (E π [U] = u)}. Definition 3.:= u∈att(U ) Π maxent U ,u .For each attainable expected utility, Π maxent U contains the highest entropy policy which attains it. In MDPs, this policy is unique π maxent U ,u and can be found with backwards induction (see Section 5) .We measure predictive accuracy using cross-entropy, as is common in ML. We subtract the predictive accuracy of the uniform distribution, so that we measure predictive accuracy relative to random chance. This makes MEG always non-negative. Definition 3.2 (Maximum entropy goal-directedness, known utility function). Let M := (G, P ) be a CID with decision variables D and utility function U. The maximum entropy goal-directedness (MEG) of a policy π with respect to U isMEG U (π) := max π maxent ∈Π maxent U E π D∈D log π maxent (D | Pa D ) -log 1 |dom(D)|.(1)As we will see, in a Markov Decision Process, this is equivalent to maximising predictive accuracy by varying the rationality parameter β in the 'soft-optimal' policy π soft β used in maximum entropy reinforcement learning [Haarnoja et al., 2017] . 4If instead of having access to a policy π, we have access to a set of full trajectories 1) can be replaced with an average over the trajectory set. This is an unbiased and consistent estimate of MEG U (π) for the policy π generating the trajectories.{(pa i D1 , D i 1 , . . . , pa D i n , D i n )} i , the expectation E π in Equation (Example. Consider a policy π in Example 1 that proceeds towards the cheese with probability 0.8. How goal-directed is this policy with respect to the utility function U that gives +1 for obtaining the cheese and -1 otherwise?To compute MEG U (π), we first find the maximum entropy policy set Π maxent U , and then take the maximum predictive accuracy with respect to π. In a single-decision setting, for each attainable expected utility u there is a unique π maxent U ,u which has the form of a Boltzmann policyπ maxent U ,u (d | s) = exp(β•E[U |d,s]) d ′ exp(β•E[U |d ′ ,s]) (cf. Theorem 5.1). The rationality parameter β = β(u) can be varied to get the correct expected utility. Predictive accuracy with respect to π is maximised by π maxent U ,0.8 , which has a rationality parameter of β = log 2. The expected logprob of a prediction of this policy isE π log π maxent U ,0.8 (D | Pa D ) = -0.50, while the expected logprob of a uniform prediction is log( 1 2 ) = -0.69. So we get that MEG U (π) = -0.50 -(-0.69) = 0.19. For comparison, predictive accuracy for the optimal policy π * is maximised when β = ∞, and has MEG U (π * ) = 0 -(-0.69) = 0.69.Properties. We now show that MEG satisfies three important desiderata. First, since utility functions are usually only defined up to translation and rescaling, a measure of goal-directedness with respect to a utility function should be translation and scale invariant. MEG satisfies this property: Proposition 3.1 (Translation and scale invariance). Let M 1 be a CID with utility function U 1 , and let M 2 be an identical CID but with utility function U 2 = a • U 1 + b, for some a, b ∈ R, with a ̸ = 0. Then for any policy π, MEG U1 (π) = MEG U2 (π).Second, goal-directedness should be minimal when actions are chosen completely at random and maximal when uniquely optimal actions are chosen. Proposition 3.2 (Bounds). Let M be a CID with utility function U. Then for any policy π we have 0 ≤ MEG U (π) ≤ D∈D log(| dom(D)|), with equality in the lower bound if π is the uniform policy, and equality in the upper bound if π is the unique optimal (or anti-optimal) policy with respect to U.Note that MEG has a natural interpretation as the amount of evidence provided for a goal-directed policy over a purely random policy. The larger a decision problem, the more opportunity there is to see this evidence, so the higher MEG can be. Third, a system can never be goal-directed towards a utility function it cannot affect. Comparison to MCE IRL Our method is closely related to MCE IRL [Ziebart et al., 2010] (see Gleave and Toyer [2022] for a useful primer). In this subsection, we discuss the key similarities and differences.The MCE IRL method seeks to find a utility function that explains the policy π. It starts by identifying a set of n linear features f i and seeks a model policy that imitates π as far as these features are concerned but otherwise is as random as possible. It thus applies the principle of maximum entropy with n linear constraints. The form of the model policy involves a weighted sum of these features. In a single-decision example, it takes the formπ MCE (d | s) = exp (E [ i w i f i | d, s]) d ′ exp (E [w i f i | d ′ , s]).(2)The weights w i are interpreted as a utility function over the features f i . MCE IRL can, therefore, only return a linear utility function.In contrast, our method seeks to measure the goal-directedness of π with respect to an arbitrary utility function U, linear or otherwise. Rather than constructing a single maximum entropy policy with n linear constraints, we construct a class of maximum entropy policies, each with a different single constraint on the expected utility.A naive alternative to defining the goal-directedness of π with respect to U as the maximum predictive accuracy across U's maximum policy set, we could simply plug in our utility function U to π MCE from Equation (2), and use that to measure predictive accuracy. If U is linear in the features f i , we could substitute in the appropriate weights, but even if not, we could still replace i w i f i with U. Indeed, this is often done with nonlinear utility functions in deep MCE IRL [Wulfmeier et al., 2015] .However, this would not have a formal justification, and we would run into a problem: scale noninvariance. Plugging in 2 • U would result in a more sharply peaked π MCE than U; in Example 1, we would get that the mouse is more goal-directed towards 2 • U than U, with a predictive accuracy (measured by negative cross-entropy) of -0.018 vs -0.13. In contrast, constructing separate maximum entropy policies for each expected utility automatically handles this issue. The policy in Π maxentwhich maximises predictive accuracy for π has an inversely scaled rationality parameter β ′ = β 2 compared to the maximally predictive policy in Π maxent 2•U . In other words, they are the same policy, and we get that MEG U (π) = MEG 2•U (π) = 0.19 (cf. Proposition 3.1). 
measuring goal-directedness without a known utility function: 
 In many cases where we want to apply MEG, we may not know exactly what utility function the system could be optimising. For example, we might suspect that a content recommender is trying to influence a user's preferences, but may not have a clear hypothesis as to in what way. Therefore, in this section, we extend our definitions for measuring goal-directedness to the case where the utility function is unknown.We first extend CIDs to include multiple possible utility functions. In effect, a parametric CID is a CID with a parametric class of utility functions U Θ .The maximum entropy policy set from Definition 3.1 is extended accordingly to include maximum entropy policies for each utility function and each attainable expected utility with respect to it. Definition 4.2 (Maximum entropy policy set, unknown utility function). Let M Θ = (G, P ) be a parametric-utility CID with decision variables D and utility function U Θ . The maximum entropy policy set for U Θ is the set of maximum entropy policies for any attainable expected utility for any utility function in the class: Π maxentU Θ := θ∈Θ,u∈att(U θ ) Π maxentU θ ,u . Definition 4.3 (MEG, unknown utility function). Let M Θ = (G, P ) be a parametric-utility CID with decision variables D and utility function U Θ . The maximum entropy goal-directedness of a policy π with respect to For example, if we only suspected that the mouse in Example 1 was optimising some function of the cheese T , but didn't know which one, we could apply Definition 4.4 to consider the goal-directedness towards T under any utility function defined on T . Thanks to translation and scale invariance (Proposition 3.1), there are effectively only three utility functions to consider: those that provide higher utility to cheese than not cheese, those that do the opposite, and those that are indifferent.U Θ is MEG U Θ (π) = max U ∈U Θ MEG U (π).Note that T has to include some descendants of D, in order to enable positive MEG (Proposition 3.3). However, it is not necessary that T consists of only descendants of D (i.e., T need not be a subset of Desc(D)). For example, goal-conditional agents take an instruction as part of their input Pa D . The goal-directedness of such agents can only be fully appreciated by including the instruction in T .Pseudo-terminal goals. Definition 4.4 enable us to state a result about a special kind of instrumental goal. It is well known that an agent that optimises some variable has an instrumental incentive to control any variables which mediate between the two [Everitt et al., 2021a] . However, since the agent might want the mediators to take different values in different circumstances, it need not appear goal-directed with respect to the mediators. Theorem 4.1 shows that in the special case where the mediators d-separate the decision from the downstream variable, the decision appears at least as goal-directed with respect to the mediators as with respect to the target. For example, in Figure 2 , the agent must be at least as goal-directed towards S 3 as it is towards U 3 , since S 3 blocks all paths (i.e. d-separates) from {D 1 , D 2 } to U 3 .Intuitively, this comes about because, in such cases, a rational agent wants the mediators to take the same values regardless of circumstances, making the instrumental control incentive indistinguishable from a terminal goal. This means we do not have to look arbitrarily far into the future to find evidence of goal-directedness. An agent that is goal-directed with respect to next week must also be goal-directed with respect to tomorrow. 
computing meg in markov decision processes: 
 In this section, we give algorithms for computing MEG in MDPs. First, we define what an MDP looks like as a CID. We then establish that soft value iteration can be used to construct our maximum entropy policies, and give algorithms for computing MEG when the utility function is known or unknown.Note that in order to run these algorithms, we do not need an explicit causal model, and so we do not have to worry about hidden confounders. We do, however, need black-box access to the environment dynamics, i.e., the ability to run different policies in the environment and measure whatever variables we are considering utility functions over. Definition 5.1. A Markov Decision Process (MDP) is a CID with variables {S t , D t , U t } n t=1 , decisions D = {D t } n t=1 and utilities U = {U t } n t=1 , such that for t between 1 and n, Pa Dt = {S t }, Pa Ut = {S t }, while Pa St = {S t-1 , D t-1 } for t > 1, and Pa S1 = ∅.Constructing Maximum Entropy Policies In MDPs, Ziebart's soft value iteration algorithm can be used to construct maximum entropy policies satisfying a set of linear constraints. We apply it to construct maximum entropy policies satisfying expected utility constraints.Definition 5.2 (Soft Q-Function). Let M = (G, P ) be an MDP. Let β ∈ R \ {0}. For each D t ∈ D we define the soft Q-function Q soft β,n : dom(D t ) × dom(Pa Dt ) → R via the recursion: Q soft β,t (d t | pa t ) = E U t + 1 β logsumexp(β • Q soft β,t+1 (• | Pa Dt+1 )) d t , pa t+1 for t < n, Q soft β,n (d n | pa n ) = E [U n | d n , pa n ] , where logsumexp β(Q soft β,t+1 (• | Pa Dt+1 )) = log dt+1∈dom(Dt+1) exp(βQ soft β,t+1 (d t+1 | Pa Dt+1 )).Using the soft Q-function, we show that there is a unique π ∈ Π maxent U ,ufor each U and u in MDPs. Theorem 5.1 (Maximum entropy policy in MDPs). Let M = (G, P ) be an MDP with utility function U, and let u ∈ att(U) be an attainable expected utility. Then there exists a unique maximum entropy policy π maxent u ∈ Π maxent U ,u , and it has the formEQUATIONwhereβ = argmax β ′ ∈R∪{∞,-∞} t E π log(π soft β ′ ,t (d t | pa t )) .We refer to a policy of the form of π β as a soft-optimal policy with a rationality parameter of β.Known Utility Function To apply Definition 3.1 to measure the goal-directedness of a policy π in a CID M with respect to a utility function U, we need to find the maximum entropy policy in Π maxent U which best predicts π. We can use Theorem 5.1 to derive an algorithm that finds π maxent u for any u ∈ att(U).Fortunately, we do not need to consider each policy in Π maxent U ,u individually. We know the form of π maxent u , and only the real-valued rationality parameter β varies depending on u.The gradient of the predictive accuracy with respect to β is∇ β E π D∈D log π soft β (D | P a D ) -log 1 | dom(D) | = E π [U] -E π soft β [U] .The predictive accuracy is a concave function of β, so we can apply gradient ascent to find the global maximum in β, which is the same as finding the maximum in u.MEG U (π) can therefore be found by alternating between applying the soft value iteration of Definition 5.2 to find π maxentβ , computing E π [U] -E π maxent β [U], and taking a gradient step. See Algorithm 1.If the β that maximises predictive accuracy is ∞ or -∞, which can happen if π is optimal or anti-optimal with respect to U, then the algorithm can never reach the (nonetheless finite) value of MEG U (π), but will still converge in the limit. π soft β ← softmax(β • Q soft β ) 5: g ← E π [U] -E π soft β [U] 6: β ← β + α • g 7: until Convergence 8: return E π D∈D log π soft β (D | Pa D ) -log 1 |dom(D)|Unknown-utility algorithm To find unknown-utility MEG, we consider soft-optimal policies for various utility functions U and maximise predictive accuracy with respect to U as well as β. Let U Θ be a differentiable class of functions, such as a neural network, and write π soft θ,β for a soft-optimal policy with respect to U ∈ U Θ , and Q soft θ,β for the corresponding soft Q-function. We can take the derivative of the predictive accuracy with respect to θ and get E π [∇ θ U] -E π maxent β [∇ θ U].π soft θ,β ← softmax(β • Q soft θ,β ) 5: g β ← E π U θ -E π soft β U θ 6: g θ ← E π ∇ θ U θ -E π soft β ∇ θ U θ 7: β ← β + α • g β 8: θ ← β + α • g θ 9: until Stopping condition 10: return E π D∈D log π soft θ,β (D | Pa D ) -log 1 |dom(D)|An important caveat is that if U θ is a non-convex function of θ (e.g. a neural network), Algorithm 2 need not converge to a global maximum. In general, the algorithm provides a lower bound for MEG U θ (π), and hence for MEG T (π) where T = Pa U Θ . In practice, we may want to estimate the soft Q-function and expected utilities with Monte Carlo or variational methods, in which case the algorithm provides an approximate lower bound on goal-directedness. 
experimental evaluation: 
 We carried out two experiments5 to measure known-utility MEG with respect to the environment reward function and unknown-utility MEG with respect to a hypothesis class of utility functions. We used an MLP with a single hidden layer of size 256 to define a utility function over states.Our experiments measured MEG for various policies in the CliffWorld environment from the seals suite [Gleave et al., 2020] . Cliffworld (Figure 3a ) is a 4x10 gridworld MDP in which the agent starts in the top left corner and aims to reach the top right while avoiding the cliff along the top row. With probability 0.3, wind causes the agent to move upwards by one more square than intended. The Known-utility MEG decreases as the goal gets easier to satisfy, but unknown-utility MEG stays higher because the optimal policies also do well with respect to a narrower goal. environment reward function gives +10 when the agent is in the (yellow) goal square, -10 for the (dark blue) cliff squares, and -1 elsewhere. The dotted yellow line indicates a length 3 goal region.In our first experiment, we measured the goal-directedness of policies of varying degrees of optimality by considering ε-greedy policies for ε in the range 0.1 to 0.9. Figure 3b shows known-and unknown utility MEG for each policy. 6 Predictably, the goaldirectedness with respect to the environment reward decreased toward 0 as the policy became less optimal. So did unknown-utility MEG -since, as ε increases, the policy becomes increasingly uniform, it does not appear goal-directed with respect to any utility function over states.In our second experiment, we measured the goal-directedness of optimal policies for reward functions of varying difficulty. We extended the goal region of Cliffworld to run for either 1, 2, 3 or 4 squares along the top row and back column and considered optimal policies for each reward function. Figure 3c shows Cliffworld with a goal region of length 3. Figure 3b shows the results. Goal-directedness with respect to the true reward function decreased as the task became easier to complete. A way to interpret this is that as the number of policies which do well on a reward function increases, doing well on that reward function provides less and less evidence for deliberate optimisation. In contrast, unknown-utility MEG stays high even as the environment reward becomes easier to satisfy. This is because the optimal policy proceeds towards the nearest goal-squares and, hence, it appears strongly goal-directed with respect to a utility function which gives high reward to only those squares. Since this narrower utility function is more difficult to do well on than the environment reward function, doing well on it provides more evidence for goal-directedness. In Appendix D.3, we visualise the policies in question to make this more explicit. We also give tables of results for both experiments. 
limitations: 
 Environment Access Although computing MEG does not require an explicit causal model (cf. Section 5), it does require the ability to run various policies in the environment of interest -we cannot compute MEG purely from observational data.Intractability While MEG can be computed with gradient descent, doing so may well be computationally intractable in high dimensional settings. In this paper, we conduct only preliminary experiments -larger experiments based on real-world data may explore how serious these limitations are in practice.Choice of variables MEG is highly dependent on which variables we choose to measure goaldirectedness with respect to. At one extreme, all (deterministic) policies have maximal goal-directedness with respect to their own actions (given an expressive enough class of utility functions). This means that, for example, it would not be very insightful to compute the goal-directedness of a language model with respect to the text it outputs 7 . At the other extreme, if a policy is highly goal-directed towards some variable T that we do not think to measure, MEG may be misleadingly low. Relatedly, MEG may also be affected by whether we use a binary variable for T (or the decisions D) rather than a fine-grained one with many possible outcomes. We should, therefore, think of MEG as measuring what evidence a set of variables provides about a policy's goal-directedness. A lack of evidence does not necessarily mean a lack of goal-directedness.Distributional shifts MEG measures how predictive a utility function is of a system's behaviour on distribution, and distributional shifts can lead to changes in MEG. A consequence of this is that two almost identical policies can be given arbitrarily different goal-directedness scores, for example, if they take different actions in the start state of an MDP and thus spend time in different regions of the state space. It may be that by considering changes to a system's behaviour under interventions, as Kenton et al. [2023] do, we can distinguish "true" goals from spurious goals, where the former predict behaviour well across distributional shifts, while the latter happen to predict behaviour well on a particular distribution (perhaps because they correlate with true goals) [Di Langosco et al., 2022] . We leave this to future work.Behavioural vs mechanistic approaches The latter two points suggest that mechanistic approaches to understanding a system's goal-directedness could have advantages over behavioural approaches like MEG. For example, suppose we could reverse engineer the algorithms learned by a neural network.In that case, it may be possible to infer which variables the system is goal-directed with respect to and do so in a way which is robust to distributional shifts. However, mechanistic interpretability of neural networks [Elhage et al., 2021] is an ambitious research agenda that is still in its infancy, and for now, behavioural approaches appear more tractable.Societal implications An empirical measure of goal-directedness could enable researchers and companies to keep better track of how goal-directed LLMs and other systems are. This is important for dangerous capability evaluations [Shevlane et al., 2023 , Phuong et al., 2024] and governance [Shavit et al., 2023] . A potential downside is that it could enable bad actors to create more dangerous systems by optimising for goal-directedness. We judge that this does not contribute much additional risk, given that bad actors can already optimise a system to pursue actively harmful goals. 
conclusion: 
 We proposed maximum entropy goal-directedness (MEG), a formal measure of goal-directedness in CIDs and CBNs, grounded in the philosophical literature and the maximum causal entropy principle. Developing such measures is important because many risks associated with advanced artificial intelligence come from goal-directed behaviour. We proved that MEG satisfies several key desiderata, including scale invariance and being zero with respect to variables that can't be influenced, and that it gives insights about instrumental goals. On the practical side, we adapted algorithms from the maximum causal entropy framework for inverse reinforcement learning to measure goal-directedness with respect to nonlinear utility functions in MDPs. The algorithms handle both a single utility function and a differentiable class of utility functions. The algorithms were used in some small-scale experiments measuring the goal-directedness of various policies in MDPs. In future work, we plan to develop MEG's practical applications further. In particular, we hope to apply MEG to neural network interpretability by measuring the goal-directedness of a neural network agent with respect to a hypothesis class of utility functions taking the network's hidden states as input, thus taking more than just the system's behaviour into account.  
 
 Appendix: This paper is inspired by Kenton et al. [2023] , who proposed a causal discovery algorithm for identifying agents in causal models, inspired by Daniel Dennett's view of agents as systems "moved by reasons" [Dennett, 1989] . Our approach has several advantages over theirs, which we enumerate below.Mechanism variables. Kenton et al. [2023] assume access to a mechanised structural causal model, which augments an ordinary causal model with mechanism variables which parameterise distributions of ordinary object-level variables. An agent is defined as a system that adapts to changes in the mechanism of its environment. However, the question of what makes a variable a mechanism is left undefined, and indeed, the same causal model can be expressed either with or without mechanism variables, leading their algorithm to give a different answer. For example, Example 1 has identical causal structure to the mouse example in Kenton et al. [2023] , but without any variables designated as mechanisms. Their algorithm says the version with mechanism variables contains an agent while the version without does not, despite them being essentially the same causal model. Figure 4 shows our example depicted as a mechanised structural causal model. We fix this arbitrariness by making our definition in ordinary causal Bayesian networks.Utility variables. Their algorithm assumes that some variables in the model represent agents' utilities.We bring this more in line with the philosophical motivation by treating utilities as hypothesised mental states with which we augment our model.Predictive accuracy. Kenton et al. [2023] 's approach formalises Dennett's idea of agents as systems "moved by reasons". We build on this idea but bring it more in line with Dennett's notion of what it means for a system to be moved by a reason -that the reason is useful for predicting its behaviour.Gradualist vs Essentialist. The predictive error viewpoint gives us a continuous measure of goaldirectedness rather than a binary notion of agency, which is more befitting of the gradualist view of agents which inspired it.Practicality. Their algorithm is theoretical rather than something that can be applied in practice. Instead, ours is straightforward to implement, as we demonstrate in Section 6. This opens up a range of potential applications, including behavioural evaluations and interpretability of ML models.Interventional distributions. The primary drawback of MEG is that it doesn't necessarily generalise outside of the distribution. Running MEG on interventional distributions may fix this. We leave an extension of MEG to interventional distributions for future work.Proposition 3.1 (Translation and scale invariance). Let M 1 be a CID with utility function U 1 , and let M 2 be an identical CID but with utility function U 2 = a • U 1 + b, for some a, b ∈ R, with a ̸ = 0.Then for any policy π, MEG U1 (π) = MEG U2 (π).Proof. Since MEG is defined by maximising over a maximum entropy policy set, showing that two utility functions have the same maximum entropy policy set is enough to show that they result in the same MEG for every policy.If π ∈ Π maxent, then π ∈ Π maxent U1,u1 for some u 1 ∈ att(U 1 ), i.e. π is a maximum entropy policy satisfying the constraint that E π [U 1 ] = u 1 . π thus also satisfies the constraint that Eπ [U 2 ] = u 2 := a • u 1 + b. Note that any policy satisfying E π [U 2 ] = u 2 also satisfies E π [U 1 ] = u 1 because the map x → a•x+b is injective (since a ̸ = 0). Thus π must also be a maximum entropy policy satisfying E π [U 2 ] = u 2 , and so π ∈ Π maxent U2,u2 and thus π ∈ Π maxent.The converse is similar.Proposition 3.2 (Bounds). Let M be a CID with utility function U. Then for any policy π we have 0 ≤ MEG U (π) ≤ D∈D log(| dom(D)|), with equality in the lower bound if π is the uniform policy, and equality in the upper bound if π is the unique optimal (or anti-optimal) policy with respect to U.Proof. Recall thatMEG U (π) = max π maxent ∈Π maxent U E π D∈D log π maxent (D | Pa D ) -log 1 | dom(D) | .For the lower bound, first note that the uniform policy π unif is always an element of Π maxent U , since it is the maximum entropy way to attain a utility ofEQUATIONwhich shows both the lower bound and the fact that we attain it when π = π unif .For the upper bound, note that the log π maxent (D | Pa D ) term is at most 0, soEQUATIONTo see that π being a unique optimal (or anti-optimal) policy is a sufficient condition for attaining the upper bound, note that there always exists a deterministic optimal and anti-optimal policy in a CID, so a unique such policy must be deterministic. Further, π must be in Π maxent U , since there is no higher entropy way to get the same expected utility. Then since E π [log π maxent (D | Pa D )] = 0, choosing π maxent = π obtains the upper bound. Proof. We will show that the maximum entropy policy set Π maxent U T(where U T is the set of all utility functions over T ) is a subset of Π maxent U S , so the maximum predictive accuracy taken over the latter upper bounds the maximum predictive accuracy taken over the former.Let π ∈ Π maxent U T , so π = π maxent U ,ufor some U T ∈ U T . If we can find a utility function U S ∈ U S such that for all π, E π U S = E π U T , then a maximum entropy policy with E π U T = u must also be a maximum entropy policy with E π U S = u. It would follow that π ∈ Π maxent U T and so Π maxentU S ⊆ Π maxent U T . To construct such a utility function, let U S (s) = t P (T = t | S = s)U T (t). Note that since D ⊥ T | S, P (T = t | S = s) is not a function of π. Then for any π, E π U T = t P π (t)U T (t) = s P π (s) t P π (t | s)U T (t) = s P π (s) t P (t | s)U T (t) (since D ⊥ T | S) = s P π (s)U S (s) = E π U S .C Proof of Theorem 5.1 Lemma C.1. Let π β,U denote a soft-optimal with respect to utility function U, defined as in Theorem 5.1, with rationality parameter β. Then for any α ∈ R, we have that π α•β,U = π β,α•U .Proof. If either α or β are equal to 0, both policies are uniform and we are done. Otherwise, write Q soft β,U for the soft-Q function for U with rationality parameter β. We first show thatQ soft β,α•U = α • Q soft α•β,U by backwards induction. First, Q soft β,α•U ,n = E [α • U n | d n , pa n ] = αE [U n | d n , pa n ] = α • Q soft α•β,U ,n. And for t < n, assuming the inductive hypothesis holds for t + 1,Q soft β,α•U ,t (d t | pa t ) = E α • U t + 1 β logsumexp(β • Q soft β,α•U ,t+1 (• | Pa Dt+1 )) d t , pa t+1 = α • E U t + 1 α • β logsumexp(β • α • Q soft α•β,U ,t+1 (• | Pa Dt+1 )) d t , pa t+1 = α • E U t + 1 α • β logsumexp(α • β • Q soft α•β,U ,t+1 (• | Pa Dt+1 )) d t , pa t+1 = α • Q soft α•β,U ,t .Then, substituting this into the definition of π α•β,U we getπ β,α•U = softmax β • Q soft β,α•U = softmax β • α • Q soft α•β,U = softmax α • β • Q soft α•β,U = π α•β,U .Theorem 5.1 (Maximum entropy policy in MDPs). Let M = (G, P ) be an MDP with utility function U, and let u ∈ att(U) be an attainable expected utility. (EQUATIONwhere β = argmax β ′ ∈R∪{∞,-∞} t E π log(π soft β ′ ,t (d t | pa t )) .Proof. In an MDP, the expected utility is a linear function of the policy, so the attainable utility set is a closed interval att(U) = [u min , u max ]. We first consider the case where u ∈ (u min , u max ).In this case, we are seeking the maximum entropy policy in an MDP with a linear constraint satisfiable by a full support policy (since u is an interior point), so we can invoke Ziebart's result on the form of such policies [Ziebart, 2010 , Ziebart et al., 2010 , Gleave and Toyer, 2022] . In particular, our feature is the utility U. We get that the maximum entropy policy is a soft-Q policy for a utility function β • U with a rationality parameter of 1, where β = argmax β ′ ∈R t E π log(π soft β ′ (d t | pa t )) . By Lemma C.1 this can be restated as a soft-Q policy for U with a rationality parameter of β. It follows from Ziebart that β = argmax β ′ ∈R π soft β ′ , and allowing β = ∞ or -∞ does not change the argmax. In the case where u ∈ {u min , u max }, it's easy to show that the maximum entropy policy which attains u randomises uniformly between maximal value actions (for u max ) or minimal value actions (for u min ). These policies can be expressed as lim β→∞ π soft β and lim β→-∞ π soft β respectively. Figure 5a and Figure 5b show the occupancy measures for an optimal policy for k=1 and k=4 respectively, where k is the length of the goal region in squares. Although the goal region is larger in the latter case, the optimal policy consistently aims for the same sub-region. This explains why unknown-utility MEG is higher than MEG with respect to the environment reward. The policy does just as well on a utility function whose goal-region is limited to the nearer goal squares as it does on the environment reward, but fewer policies do well on this utility function, so doing well on it constitutes more evidence for goal-directedness.The experiments were carried out on a personal laptop with the following specs:• Hardware model: LENOVO20N2000RUK• Processor: Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, 2112 Mhz, 4 Core(s), 8 Logical Processor(s) • Memory: 24.0 GB We used an environment from the SEALS library8 , and adapted an algorithm from the imitation library9 . Both are released under the MIT license.For information on hyperparameters, see the code.After accepted publication, another related paper appeared[Xu and Rivera, 2024].A noteworthy property of Definition 3.2 is that a policy that pessimises a utility function can be just as goal-directed as a policy that optimises it. This fits with the intuitive notion that we are measuring how likely a policy's performance on a particular utility function is to be an accident. However, it might be argued that a policy should instead be considered negatively goal-directed with respect to a utility function it pessimises. We could adjust Definition 3.2 to achieve this by multiplying by the sign of Eπ [U] -E π unif [U], making the goal-directedness of any policy that performs worse than random negative. For simplicity, we use the unsigned measure in this paper.Code available at https://github.com/mattmacdermott1/measuring-goal-directednessKnown-utility MEG is deterministic. Unknown-utility MEG depends on the random initialisation of the neural network, so we show the mean of several runs. Full details are given in Appendix D.3.Any proposed way of measuring goal-directedness needs a way to avoid the trivial result where all policies are seen to be maximising an idiosyncratic utility function that 'overfits' to that policy's behaviour. MEG avoids this by allowing us to measure goal-directedness with respect to a set of variables that excludes the system's actions (for example, states in an MDP). An alternative approach is to penalise more complex utility functions.https://github.com/HumanCompatibleAI/sealshttps://github.com/HumanCompatibleAI/imitation 


Decision: The paper makes a significant theoretical contribution to the study of goal-directedness in decision-making systems, with potential implications for AI safety. 
However, the work would benefit from addressing the computational limitations, broadening the philosophical discussion, and providing more extensive empirical validation. 
Despite these concerns, the reviewers agree that the paper is an important step forward in this research area and recommend acceptance, provided that the identified weaknesses are addressed in the final version

Reviews: review: paper_summary: paper_summary: The paper is studying a problem that is both mathematical and philosophical in nature: how can we measure quantitatively, whether an observed behaviour is goal-directed or not (or to what extent it is). It tackles this problem by starting from a causal model of the world, where there are explicit utility variables, state variables, and decision variables. For a known utility function, it proposes to measure goal-directedness by essentially asking: given an observed policy $\pi$, find a policy that is maximizing the utility function but regularised by entropy that is _most predictive_ of $\pi$. Compare its predictive accuracy to uniform prediction, and this quantifies goal-directedness. They also extend this to the case of unknown utilities, by considering parameterized sets of utility functions. The favourable properties of this measure is discussed, and its practicality is demonstrated with a simple empirical result on CliffWorld.
strength_weakness: strength_weakness: - Societal implications could have been discussed in more detail, especially how this "could enable bad actors to create even more dangerous systems." An extended discussion about this added to the appendix would improve this.

- A more comprehensive empirical demonstration of the practicality would improve the paper.
questions: questions: Q: I have one question about the unknown utility setting. If I get this right, in general, if the set of possible utility functions is large enough, and the causal model is rich enough, I can potentially find a utility function making many behaviours goal-directed, even though they may not be. For example, consider the behaviour of a river, as its flow. If my causal model represents enough things relevant (e.g. elevation, geography, physics, weather etc.), I can potentially find a utility function in the lines of _minimize travel distance to the nearest sea/lake_ where elevation incurs a cost, and conclude that the flow of the river is goal-directed. Is this correct? If so, it seems like one must be careful in terms of the class of utility functions they consider. 

Bonus Q: I know that the paper assumes a known causal model. However, I am curious. How would we deal with hidden confounders here? Is there no hope? Specifically, could there be reasons internal to the agent that make a seemingly non-goal-directed behaviour in fact goal-directed, unbeknownst to us?
limitations: limitations: Paper discusses limitations sufficiently.
paper_summary: paper_summary: The paper introduces maximum entropy goal-directedness (MEG), a measure of how much an agent is goal directed towards a given utility function. The authors extend the theoretical framework introduced to the case where the utility function that is being optimized is not known. Moreover, they propose an algorithm to measure MEG in Markov Decision Processes. 
To show the practicality of the metric introduced, the authors show the performances of the algorithms empirically. Specifically, they show how MEG relates with the optimality of a $\epsilon$-greedy policy, and how it relates to task difficulty.
strength_weakness: strength_weakness: - As mentioned by the authors, MEG relies on having full access to the causal model of the environment. This can be a highly-demanding assumption in many real environments.
- Given the restrictive assumption mentioned above, the paper should at least investigate in more detail how computationally expensive is computing MEG in large-scale setting. 
- The experiments are limited to few toy examples
questions: questions: - How do you think one could make MEG more robust towards distributional shifts? 

- Do you have any example in mind on how MEG could fail when considering how goal-directed an agent is? I think it could be an important point to discuss to give a better perspective to the reader on when one would want to employ such metric. Maybe you have some toy examples which can illustrate this?

- Is it possible for two similar policies to achieve drastic differences in MEG? Do you have any experiments which shows how robust of a metric it is?

-How one could extend MEG to cases where the causal environment is only partially known?

- Do you have some experiments/comparison/thoughts on how MEG relates directly with agency?
limitations: limitations: no ethical limitations
paper_summary: paper_summary: * The work introduces a framework for quantifying the goal-directedness of a
  decision-making system with respect to a particular utility function or a
  family of utility functions.
* The method is grounded in Dennett's *intentional stance* and Jaynes'
  *maximum entropy inference*. The authors have made some additional choices
  while formalising these principles to arrive at their formulation, which
  ends up similar to Ziebart's Maximum Causal Entropy IRL framework.
  The resulting formula is essentially a measure of how close the subject
  policy comes, in terms of cross entropy, to a maximum entropy formula for
  the utility function at some inverse temperature.
* The authors establish that the resulting formula, MEG, satisfies desirable
  properties for a measure of goal-directedness, namely being invariant to
  the scaling and translation of utility functions, having interpretable
  upper and lower bound policies for a given utility function, and measuring
  only causal goal-directedness.
* The authors extend their measure of goal-directedness with respect to a
  single utility function into an aggregate measure of goal-directedness with
  respect to a family of utility functions (simply by taking the maximum MEG
  over the family of utility functions). This also allows them to express a
  measure of goal-directedness with respect to a set of 'target variables',
  by considering the family of utility functions defined over those
  variables.
* The authors derive algorithms for computing both kinds of MEG in the
  setting of causal influence models derived from MDPs. The algorithms are
  based on gradient-based search requiring repeatedly computing maximum
  entropy policies for various inverse temperatures (and, in the case of a
  family of utility functions, varying family parameters).
* The authors implement their algorithm for some simple MDPs as a 'proof of
  concept' experiment, demonstrating how various factors influence the
  measure of goal-directedness.
strength_weakness: strength_weakness: I noted the following issues with the work, based on my understanding.

1.  MEG is described as "philosophically grounded" because of its roots in an
    interpretation of Dennett's Intentional Stance. However, it does not
    appear to be a unique interpretation and this philosophy does not appear
    to be the only possible philosophical approach to defining agency.

    * Related works Kenton et al. and Orseau et al. both appeal to Dennett's
      philosophy but arrive at different formalisms. The authors have not
      established that their interpretation is uniquely philosophically
      grounded. (Appendix A begins to address this with respect to Kenton et
      al.'s prior work, but it would be good to see a discussion of the
      relationship with Orseau et al.'s approach.)

    * If MEG were a unique formalisation of Dennett's philosophy, it would
      still appear to be a behaviourist approach, which stands in contrast to
      a mechanistic approach that looks into the internals of a system for
      explicit representations of goals and for the implementation of
      decision-making processes.

2.  I am concerned that the statements of proposition 3.1 and 3.2
    are incorrect as stated.

    * **Proposition 3.1.** There must be some missing assumption in this
      theorem statement that $a$ is a positive scalar. Otherwise you have
      proven that a MEG policy for any given utility function is also
      goal-directed towards a negated version of that policy. Could you
      please clarify whether this is the case or whether I am missing
      something that means that a policy that pursues the negative of a
      utility function is somehow goal-directed towards that utility
      function?

    * **Proposition 3.2.** It is well known that in many examples there are
      decisions for which multiple actions lead to equal expected utility and
      are therefore equally optimal. In such cases there is no unique optimal
      policy. What happens to the bounds in this case? The statement of the
      proposition would appear to be ill-defined in case there is no unique
      optimal policy. Is it the case that you do not derive an upper bound in
      this case? The distinction being then that there still will be some
      upper bound (perhaps?) but that you have not characterised it?

    When I checked the proofs for these propositions in the appendix I found
    them to provide insufficient detail to resolve my concerns or point to a
    particular mistake in the reasoning.

    I am recommending acceptance of the paper on the assumption that these issues will be easy to dismiss or to correct.

3.  In the unknown-utility case, MEG fails to account for complexity of
    utility functions within the hypothesis class.

    * It is well known in RL that for any policy, a utility function exists
      that can rationalise that policy. For example, there is a reward
      function that immediately, highly rewards whichever actions that policy
      happens to take in each state.
    * This leads me to wonder if for any policy it is possible to find some
      reward function for which MEG very strongly indicates that this policy
      is goal-directed towards that utility function.
    * Especially in the case of MEG for unknown utility functions, it is
      therefore possibly that if the hypothesis class of utility functions is
      too rich, the MEG for unknown utility functions will register almost
      any policy as goal-directed.
    * I think something that is missing is some way of accounting for the
      complexity of utility functions. Philosphically, a complex utility
      function is not 'useful for predicting behaviour' if it is itself as
      complex as the behaviour, even if it is capable of predicting behaviour
      well.
    * I think a significant weakness of MEG, especially in the
      unknown-utility setting, is that it leaves the user of the framework to
      rule out too-complex utility functions by the coarse tool of including
      them or excluding them from the hypothesis class, rather than providing
      a principled tool for managing this complexity.
  
    If the authors are aware of this issue but consider addressing it to be
    out of scope, I would like to see it acknowledged as such in the main
    text.

4.  If I am not mistaken, the scalability of the method for computing MEG
    relies on solving an RL problem (computing a maximum entropy policy in
    MDPs). If this is the case then I think the scalability of the method is
    questionable.

    * It is certainly common practice to solve RL problems, and in any case
      where we can train a policy that will be the target of MEG evaluation,
      we can presumably also train a max-ent policy to measure its MEG.
      Perhaps calculating such a policy repeatedly will be expensive but it
      should at least be feasible to do so.
    * However, the very applications I think motivate this work are settings
      where we don't understand the internal motivational system of the
      trained system (and so MEG can help quantify that). This would of
      course only happen if we are training the subject policy in such a
      complex environment that its emergent goal-directedness is in question,
      due to, for example, potential goal misgeneralisation (or so-called
      'inner alignment failure').
    * In such a setting, I suppose we would share the same concerns about the
      max-ent policy required for quantifying MEG. If I am not mistaken, if
      this probe policy's generalisation could not be trusted, then neither
      could we trust the result of using it to calculate the MEG of the
      subject policy.

    There is a comment in the discussion to the effect that future work can
    consider the setting of 'distributional shifts' which is normally
    considered to be a prerequisite for the phenomenon of goal
    misgeneralisation. However, I think this concern undercuts the motivation
    of the present work in its early stages and if there is no feasible way
    to reliably compute the MEG for complex learned systems where the
    internal goals are in question then the work may not achieve its goals in
    the future, and this issue should be raised and (ideally) addressed now.

5.  There is insufficient explanation of relationship to prior work, particularly to IRL and agents and devices.

    * **Inverse reinforcement learning.** IRL represents a large body of
      work including theoretical work aimed at finding goals that
      rationalise particular policies, which is clearly closely related to
      MEG. The authors' brief comments in the related work section have not
      clarified to me the relationship between MEG and IRL.
      In particular, it seems plausible to me that MEG is essentially
      functions similar to a subcomponent of a IRL framework based on
      Bayesian inference.

        * A Bayesian approach for IRL for example involves positing a
          likelihood of the behavioural data given each goal. This likelihood
          function represents a kind of measure of how well the behaviour can
          be explained by that goal. This seems similar to MEG, though of
          course MEG does not adopt a Bayesian framework.
        * Similarly, what is the relationship between unknown-utility MEG
          with a given hypothesis class and the model evidence in a Bayesian
          formulation of IRL with the same hypothesis class? Of course there
          is a difference of integrating versus taking a max, but my point is
          that these are different approaches to the same kind of problem.
    
        I think the paper would be strengthened if the authors could clarify
        whether MEG is addressing a fundamentally new kind of problem or one
        that arises naturally in the course of IRL. If it's not new, that
        seems fine, the approach is apparently new---but it would be nice to
        see a discussion of how the approach compares to the Bayesian
        approach outlined above.
    
    * **Agents and devices.** This prior works appears to present a
      Bayesian framework for distinguishing agents from non-agents, along
      with one particular instantiation of this general framework in terms
      of epsilon-greedy suboptimality and Solomonoff induction.
      The comments in the related work section appear to relate to the
      instantiation. What is the relationship between the present work and
      the general framework from Orseau et al.?
      For example, would it be possible to derive a maximum-entropy version
      of Orseau et al.'s framework, and if so, how might that compare to
      MEG? Does the difference come down to the difference between Bayesian
      inference and maximum entropy inference?
    
    I would have liked to see sections along the lines of appendix A (for
    Kenton et al.) drawing out these relationships.

If the authors are able to address each of these issues to my satisfaction I think my assessment of the paper's strength would be greatly improved and I would be willing to substantially strengthen my recommendation to accept the paper.
questions: questions: Many of the weaknesses I raised above are partially framed as questions since
I am not fully confident that I have understood the work in forming these
comments (I welcome corrections from the authors).

Here I collect additional questions that are less crucial to the overall
validity of the work but are still either barriers to my understanding or
seem like minor issues in the presentation.

* The introduction claims that "pursuing goals is about having a particular
  causal effect on the environment." This appears to be part of the case for
  MEG being "philosophically grounded".
  I can see how in many cases this correspondence holds, but it is not
  immediately obvious to me why there couldn't be cases where the pursuit of
  goals is separated from causality. For example, if a goal is always going
  to be achieved counterfactually regardless of an agent's actions, then it
  becomes possible to 'pursue' this goal without having any causal effect on
  its achievement.
  Do the authors have some further justification for this framing, or a
  citation that gives such justification?

* Equation 1:

    * Can the definition of MEG be interpreted as a minimum KL divergence
      between the policy and the closest maximum-entropy policy?
    * The cross entropy between two distributions is not symmetric. Why is is
      justified to take the cross entropy in this particular direction?

* What is the relationship between theorem 5.1 and prior work on maximum
  entropy RL such as Haarnoja et al. 2017 "Reinforcement learning with deep
  energy-based policies"?

* In the discussion section, under "societal implications", there is a
  passage "the relationship between goal-directedness and danger is fairly
  indirect".

    * I wonder, did the authors mean to write "the relationship between
      *measuring* goal-directedness and danger is fairly indirect"?
    * If not, could the authors please elaborate on this indirectness, since it
      seems that the goal-directedness itself is fairly tightly and directly
      connected to risks (as outlined in the works cited in the introduction).


Not a question, but while reading the paper I happened to note the following
minor presentation issues the authors may like to address in future versions.

* Line 5: I think the authors mean "adaptation" rather than "adaption".
* Line 14: The word "mundane" seems to downplay the real and potentially
  devastating impacts discussed in Chan et al. Consider a different word
  or a different angle of contrast (perhaps the scale of the harms?).
* Line 16: Shavit et al. citation and reference missing a date.
* Line 18: "Agentcy".
* Line 22: Incomplete sentence, consider "According to Dennett... [1989],
  *we are justified in doing so whenever it* is useful ...".
* Line 31: MEG acronym was introduced in the abstract is undefined in the
  scope of the introduction.
* Line 65: Orseau et al. should be cited in text mode.
* Line 78: Something appears to have allowed the bottom margin on page 2
  to shrink, causing the final lines on the page to come too close to the
  page number.
* Algorithm 1 line 7: Until convergence of what? Not beta, for example.
* Line 294: This and the next paragraph headings are missing periods on
  the abbreviation "vs.".
* Line 296: "meg" should be capitalised.
* Footnote 1 is missing a period, at the footnote mark (line 296) AND at
  the end of the footnote text.
* Line 300: Potential missing word, "optimal *policies*"?
limitations: limitations: It would have been good to see a more acknowledgement of the limitations of
the work (such as those I have enumerated above in the weaknesses section, pending the authors' clarification), throughout the paper or in a dedicated section of the paper.

Some additional points on addressing these limitations in the paper:

* You could more carefully justify or qualify the discussion of the extent to which the
  approach is uniquely principled / philosophically motivated.

    For example, in the conclusion, MEG is described as "grounded in the
    philosophical literature" when the authors appear to just mean "grounded in
    the philosophy of Dennett" (as they do qualify elsewhere), the difference between which I outline in Weakness (1).

* The scalability of the work does appear to depend on repeatedly computing a
  maximum entropy policy, which would appear potentially many times more
  expensive than computing the subject policy, though perhaps feasible. This
  is based on my understanding. I would have liked to see the authors clarify
  the requirements in comparison to the cost of finding the target policy
  explicitly, since they claim their method is "scalable".

See also weaknesses (3) and (4).
paper_summary: paper_summary: The paper proposes a measure of evidence for goal-directedness based on the maximum causal entropy principle, MEG. It is operationalized as the ability to predict variable D based on the hypothesis that it is optimizing a utility function U, where U represents the goal and D the agent’s decisions. Specifically, they try to predict the agents' decisions using the maximum-entropy policy that achieves the same utility as the agent’s own policy. MEG is essentially the negative cross-entropy of this prediction. The authors also describe algorithms for estimating MEG, and demonstrate it in action with some small-scale experiments.
strength_weakness: strength_weakness: - It is unclear how useful MEG will be for checking goal-directedness of generalist AI agents acting in the real world, because:
  - it is expensive to compute, since it requires optimizing the maximum entropy policy, and their estimation algorithm can only guarantee a lower bound on MEG for the unknown-utility case
  - It requires a causal model containing the target variables that the agent might be trying to manipulate.

- The name and framing of MEG is slightly misleading: MEG is not directly a measure of how goal-directed an agent is, but rather the amount of evidence for goal-directedness. The maximum value MEG can take varies based on the number of possible choices in an environment. Thus, it may be difficult to compare the goal-directedness of different agents’ behavior if we could only observe them acting in different environments. (e.g., to determine if agents acting in our constantly changing world are becoming increasingly goal-directed)

- typo: agentcy on line 18

- Capitalization of v at the end of definition 2.1 is inconsistent
questions: questions: Why does the maximizing policy in equation 1 always obtain the same expected utility as pi?
limitations: limitations: yes

Comments: title: Agreed
Agreed
Prop 3.1.
thank you for the response
Thanks for the clarifications
Thanks, your rebuttal has addressed my concerns, with notes
Further Clarification of Prop 3.2
comment: You are right that this is an important consideration. In some cases we can find MEG via global optimisation or tabular methods, and a good idea for future work is to compare the results of such methods against SGD in these cases.
Thanks. We agree this point makes the paper much stronger and will indeed emphasise it a lot more.
Thanks, we have followed your suggestions re: Prop 3.1.
Thank you for the interesting response. 

"Something we perhaps under-emphasised in the paper is that if you have black box access to the environment or a simulator, then you don’t need an explicit causal model, ..."    

- I would definitely emphasise this more in the paper. This makes the paper much stronger than my initial impression.

I have decided to maintain my score.
**Lower bound**  "we often get good estimates of the quantity regardless" this is a bit too vague to assuage my concerns. "Good" might not be useful, depending on how the metric is being used: e.g, if we are trying to determine which of agents A and B is more goal-directed, we might obtain a higher bound for agent A than agent B, even if agent B is in fact more goal directed than agent A.

I am satisfied with the other replies, thank you for clarifying.
Dear authors, 
Thanks for your answer. 

**Requiring a causal model**: This addresses one important critique and concern I had. Thanks. 

**Distributional Shift**: I agree that this could be a sensible way of detecting goal misgeneralisation. I would especially be curious to know if MEG could be used to identify also the 'cause' of themisgeneralisation, and not only if this is happening or not.  This could be done by considering increasingly subsets of the Causal Model and see which of the downstream variable causes the shift in MEG. Obviously this goes beyond the scope of this work, but thanks for clarifying my question

**How MEG can fail** and **Can similar policies get very different MEG**: These examples are clear to me, and I appreciate that they have been added to the appendix. I would like the authors to clarify in the final version that *MEG measures the goal-directedness on a particular distribution*

**The case where the causal model is only partially known**and **How MEG relates to agency**: Thanks for your answer.


To summarize, all the issues I raised were addressed by the authors. I will increase the score accordingly.
The revisions proposed by the authors in this rebuttal and their other
rebuttals will substantially improve the clarity and quality of the paper.
They will also address all of my concerns (subject to the below notes).
Accordingly I am pleased to strengthen my recommendation for the paper's
acceptance.

**Notes on terminology:**
Once the authors have clarified that the main philosophical grounding is in
Dennett's work, I am satisfied and I leave it to the authors to decide the
best choice of wording between "philisophically motivated" and
"philosophically grounded".

**Notes on proposition 3.1, and goal-directedness of minimisation:**
Thank you for clarifying this point about the unsigned nature of MEG, which
I missed in my initial review.
I think my confusion stemmed from the fact that I picked up the phrasing "a
policy pi is goal-directed *towards* a utility function U".
The phrasing "directed towards" suggests to me that the policy is acting to
maximise the utility function (to some extent), rather than to minimise it.
I note that the authors switch between this phrasing and the sign-neutral
phrase "goal-directed with respect to" in the paper, but I missed this
distinction.
I am satisfied with the following resolution:

1. If they have not already done so, the authors should make sure they
   reserve "towards" for cases where the policy is positively directed with
   respect to the utility function.

2. The authors should also explicitly clarify this aspect of their
   framework prominently in the main text.

Moreover, the authors also asked if I would argue in favour of a signed
measure of goal-directedness. I think I would, but it is not crucial to my
evaluation of the present work, any substantial revision in this direction
would require further review. Seems like it is best left for future work.

**Notes on proposition 3.2, about multiple (anti-)optimal policies:**
Thank you for clarifying. I now understand the proposition statement. The
wording change you have proposed in the initial rebuttal is sufficient in my
judgement.

**Notes on distributional shift and goal misgeneralisation:**
I appreciate clarification on the proposed application of MEG for use in
detecting goal misgeneralisation. I think this is an interesting direction
for future work and I am pleased that the authors are expanding their
discussion in their revision.
I maintain my concerns about the specific proposed methods, since in practice
it may not be straightforward to identify novel state distributions or their
corresponding intended reward functions for measuring intended-reward MEG.
However, this issue alone does not undermine my judgement of this paper as
an important theoretical contribution to the field.
Just to further clarify Proposition 3.2, as it may still not be clear. The upper bound holds whether or not there exists a unique optimal (or anti-optimal) policy, but iff *there exists* such a policy, *and* pi is that policy, then there is equality in the upper bound. Do you think we should spell this out more explicitly in the statement?