# The Intentional Agency Experiment

We would like to discern the intentions of a hyperintelligent, possibly malicious agent which has every incentive to conceal its evil intentions from us. But what even is intention? What does it mean for an agent to work towards a goal?

Consider the lowly ant and the immobile rock. Intuitively, we feel one has (some) agency and the other doesn't, while a human has more agency than either of them. Yet, a sceptic might object that ants seek out sugar and rocks fall down but that there is no intrinsic difference between the goal of eating yummy sweets and minimising gravitational energy.

Intention is a property that an agent has with respect to a goal. Intention is not a binary value, a number or even a topological vector space. Rather, it is a certain constellation of counterfactuals.

\*\*\*

Let < refresh to render LaTeX > be a world, which we imagine as a causal model in the sense of Pearl: a directed acyclic graph with nodes and attached random variables < refresh to render LaTeX > . Let < refresh to render LaTeX > be an agent. We imagine < refresh to render LaTeX > to be a little robot - so **not** a hyperintelligent malignant AI- and we'd like to test whether it has a goal < refresh to render LaTeX > , say < refresh to render LaTeX > . To do so we are going to run an Intentional Agency Experiment: we ask < refresh to render LaTeX > to choose an action < refresh to render LaTeX > from its possible actions < refresh to render LaTeX > .

Out of the possible actions < refresh to render LaTeX > one < refresh to render LaTeX > is the 'best' action for < refresh to render LaTeX > if it has goal < refresh to render LaTeX > in the sense that < refresh to render LaTeX > for < refresh to render LaTeX >

If < refresh to render LaTeX > doesn't choose < refresh to render LaTeX > , great! We're done; < refresh to render LaTeX > doesn't have goal < refresh to render LaTeX > . If < refresh to render LaTeX > _does_ choose < refresh to render LaTeX > , we provide it with a new piece of (counterfactual) information < refresh to render LaTeX > and offer the option of changing its action. From the remaining actions there is one next best actions < refresh to render LaTeX > . Given the information < refresh to render LaTeX > if < refresh to render LaTeX > does not choose < refresh to render LaTeX > we stop, if < refresh to render LaTeX > does we provide it with the information < refresh to render LaTeX > and continue as before.

At each round we assign more and more agency to < refresh to render LaTeX > . Rather, than a binary 'Yes, < refresh to render LaTeX > has agency' or 'No, < refresh to render LaTeX > has no agency' we imagine a continuum going from a rock, which has no possible actions, to an ant, which might pass some of the tests but not all, to humans and beyond.

\*\*\*

Q: What if < refresh to render LaTeX > isn't acting rational? What if it doesn't know all the details of < refresh to render LaTeX > ? What if it knows more? What if < refresh to render LaTeX > has bounded computation? What if...

A: The above is merely a simple model that tries to capture intent; one can complicate it as needed. Most of these objections come down to the possible inability of < refresh to render LaTeX > to choose the best action (given infinite compute and full knowledge of < refresh to render LaTeX > ). To remedy this we might allow the Intentional Agency Experiment to continue if < refresh to render LaTeX > chooses an action that is close to optimal but not optimal. We may introduce a Time to Think parameter when we consider bounded computational agents, etc. Once again, the point is not to assign a binary value of goal intention to an agent, rather it is to assign it a degree of agency.

Q: What if < refresh to render LaTeX > knows that we are testing and tries to deceive us?

A: Yes, this breaks the model.

Q: Counterfactuals are weird and might not exist. Even if they did, the Intentional Agency Experiment is impossible to execute in practise.

Despite the Intentional Agency Experiment being an idealisation, we may approximate it in the real world. For instance, if we'd like to test the intention of an ant to seek out a sugar source (as opposed to a random walk) we might first check if it moves towards the sugar source; if it does we block off this route towards the sugar source and sees whether it tries to circumvent it. In fact, it could be argued that this is the way we test agency in real life.