# Deconfuse Yourself about Agency


_This post is a result of numerous discussions with other participants and organizers of the MIRI Summer Fellows Program 2019._

-----------------------

I recently (hopefully) dissolved some of my confusion about agency. In the first part of the post, I describe a concept that I believe to be central to most debates around agency. I then briefly list some questions and observations that remain interesting to me.
*The gist of the post should make sense without reading any of the math.*

# Antropomorphization, but with architectures that aren't humans

## Architectures

Consider the following examples of "architectures":

**Example** (architectures)
1.  Architectures I would *intuitively* call "agenty":
    1. [Monte Carlo tree search](https://en.wikipedia.org/wiki/Monte_Carlo_tree_search) algorithm, parametrized by the number of rollouts made each move and utility function (or heuristic) used to evaluate positions.
    2. (semi-vague) "[Classical AI-agent](https://en.wikipedia.org/wiki/Intelligent_agent)" with several interconnected modules (utility function and world model, actions, planning algorithm, and observations used for learning and updating the world model).
    3. (vague) Human parametrized by their goals, knowledge, and skills (and, of course, many other details).
2.  Architectures I would *intuitively* call "non-agenty":
    1. A hard-coded sequence of actions.
    2. Look-up table.
    3. Random generator (outputting $x ∼ \pi$ on every input, for some probability distribution $π$).
3. Multi-agent architectures[^4]:
    1. Ant colony.
    2. Company (consisting of individual employees, operating within an economy).
    3. [Comprehensive  AI services](https://www.fhi.ox.ac.uk/reframing/).


**Working definition**: *Architecture* $A(Θ)$ is some model parametrizable by $θ \in \Theta$ that receives inputs, produces outputs, and possibly keeps an internal state. We denote specific instances of $A(Θ)$ as $A(θ)$.

## Generalizing anthropomorphization

Throughout the post, $X$ will refer to some object, procces, entity, etc., whose behavior we want to predict or understand. Examples include rocks, wind, animals, humans, AGIs, economies, families, or the universe.


A standard item in the human mental toolbox is anthropomorphization: modeling various things as humans (specifically, ourselves) with "funny" goals or abilities. We can make the same mental move for architectures other than humans:

**Working definition** ($A(\Theta)$-morphization): Let $A(\Theta)$ be an architecture. Then any[^5] model $A(θ)$ is an $A(Θ)$*-morphization* of $X$.


Antropomorphization makes good predictions for other humans and some animals (curiosity, fear, hunger). On the other hand, it doesn't work so well for rocks, lightning, and AGI-s --- not that it would prevent us from using it anyway. We can measure the usefulness of $A(Θ)$-morphization by the degree to which it makes good predictions:

**Working definition** (prediction error): Suppose $X$ exists in a world $W$ and $\vec E = ( E_1, \dots , E_n )$ is a sequence of variables (events about $X$) that we want to predict. Suppose that $\vec e = (e_1,  \dots, e_n)$ is how $\vec E$ actually unfolds and $\vec \pi = (\pi_1, \dots, \pi_n)$ is the prediction obtained by $A(Θ)$-morphizing $X$ as $A(θ)$. The *prediction error* of $A(θ)$ (w.r.t. $X$ and $\vec E$ in $W$) is the expected [Brier score](https://en.wikipedia.org/wiki/Brier_score) of $\vec \pi$ with respect to $\vec e$.


Informally, we say that $A(Θ)$-morphizing $X$ is accurate if the corresponding prediction error is low.[^1]

## When do we call things agents?

**Main claim:**
1. I claim that the question "Is $X$ an agent?" is without substance, and we should instead be asking "From the point of view of some external observer $Y$, does $X$ seem to exhibit agent-like behavior?".
2. Moreover, "agent-like behavior" also seems ill-defined, because what we associate with "agency" is subjective. I propose to *explicitly* operationalize the question as "Is $A(Θ)$-morphizing $X$ accurate?".

(A related question is how difficult is it for us to "run" $A(θ)$. Indeed, we anthropomorphize so many things precisely because it is cheap for us to do so.)


Relatedly, I believe we already implicitly do this operationalization:  Suppose you talk to your favorite human $H$ about agency. $H$ will likely subconsciously associate agency with certain architectures, maybe such as those in Example 1.1-3. Moreover, $H$ will ascribe varying degrees of agency to different architectures --- for me, 1.3 seems more agenty than 1.1. Similarly, there are some architectures that $H$ will associate with "definitely not an agent".
I conjecture that some $X$ *exhibits agent-like behavior according to $H$* if it can be accurately predicted via $A(\Theta)$-morphization for some agenty-to-$H$ architecture $A(Θ)$. Similarly, *$H$ would say that $X$ exhibits non-agenty behavior* if we can accurately predict it using some non-agenty-to-$H$ architecture.


**Critically, exhibiting agent-like-to-$H$ and non-agenty-to-$H$ behavior is not mutually exclusive**, and I think this causes most of the confusion around agency. Indeed, we humans seem very agenty but, at the same time, determinism implies that there exists some hard-coded behavior that we enact. A rock rolling downhill can be viewed as merely obeying the non-agenty laws of physics, but what if it "wants to" get as low as possible?
And, as a result, we sometimes go "Humans are definitely agents, and rocks are definitely non-agents...although, wait, are they?".


# If we ban the concept of agency, which interesting problems remain?

"Agency" often comes up when discussing various alignment-related topics, such as the following:

## Optimizer?
How do we detect whether $X$ performs (or capable of performing) optimization? How to detect this from $X$'s architecture (or causal origin) rather than looking at its behavior? (This seems central to the topic of mesa-optimization.)

## Agent-like behavior vs agent-like architecture
Consider the following [conjecture](https://www.alignmentforum.org/posts/osxNg6yBCJ4ur9hpi/does-agent-like-behavior-imply-agent-like-architecture): "Suppose some $X$ exhibits agent-like behavior. Does it follow that $X$ physically contains agent-like architecture, such as the one from Example 1.2?". This conjecture is false --- as an example, Q-learning is a "fairly agenty" architecture that leads to intelligent behavior. However, the resulting RL "agent" has a fixed policy and thus functions as a large look-up table. A better question would thus be whether there exist an agent-like architecture *causally upstream* of $X$. This question also has a negative answer, as witnessed by the example of an ant colony --- agent-like behavior without agent-like architecture, produced by a "non-agenty" optimization process of evolution. Nonetheless, a general version of the question remains: If some $X$ exhibits agent-like behavior, does it follow that there exists some interesting physical structure[^2] causally upstream of $X$?[^3]

## Moral standing
Suppose there is some $X$, which I model as having some goals. When making actions should I give weight to those goals?
(The answer to this question seems more related to conciousness than to $A(Θ)$-morphization. Note also that a particularly interesting version of the question can be obtained by replacing "I" by "AGI"...)

## PC or NPC?
When making plans, should we model $X$ as a part of the environment, or does it enter our game-theoretical considerations? Is $X$ able to model us?

## Creativity, unbounded goals, environment-generality
In some sense, AlphaZero is an extremely capable game-playing agent. On the other hand, if we gave it access to the internet[^6], it wouldn't do anything with it. The same cannot be said for humans and unaligned AGIs, who would not only be able to orient in this new environment but would eagerly execute elaborate plans to increase their influence. How can we tell whether some $X$ is more like the former or the latter?


**To summarize**, I believe that many arguments and confusions surrounding agency can disappear if we explicitly use $A(\Theta)$-morphization. This should allow us to focus on the problems listed above. 
Most definitions I gave are either semi-formal or informal, but I believe they could be made fully formal in more specific cases.

Regarding feedback:
Sugestions for a better name for "$A(\Theta)$-morphization" super-welcome!
If you know of an application for which such formalization would be useful, please do let me know.
Pointing out places where you expect a useful formalization to be impossible is also welcome.

[^1]: Distinguishing between "small enough" and "too big" prediction errors seems non-trivial since different environments are naturally more difficult to predict than others. Formalizing this will likely require additional insights.

[^2]: An example of such "interesting physical structure" would be an implementation of an optimization architecture.

[^3]: Even if true, this conjecture will likely require some additional assumptions. Moreover, I expect "randomly-generated look-up tables that happen to stumble upon AGI by chance" to serve as a particularly relevant counterexample.

[^4]: You might also view multi-agent systems these as monolithic agents, but this view might often give you wrong intuitions. I am including this category as an example that -- intuitively -- doesn't belong to either of the "agent" and "not-agent" categories.

[^5]: By default, we do not assume that $A(\Theta)$-morphization of $X$ is useful in any way, or even the most useful among all instances of $A(\Theta)$. This goes against the intuition according to which we would pick some $A(\theta)$ that is close to optimal (among $\theta' \in \Theta$) for predicting $X$. I am currently unsure how to formalize this intuition, apart from requiring that is optimal (which seems too strong a condition).

[^6]: Whatever that means in this case.