# Walkthrough of 'Formalizing Convergent Instrumental Goals'

Introduction
------------

I found _[Formalizing Convergent Instrumental Goals](https://intelligence.org/files/FormalizingConvergentGoals.pdf)_ (Benson-Tilsen and Soares) to be quite readable. I was surprised that the instrumental convergence hypothesis had been formulated and proven (within the confines of a reasonable toy model); this  caused me to update slightly upwards on existential risk from unfriendly AI.

This paper involves the mathematical formulation and proof of [instrumental convergence](https://en.wikipedia.org/wiki/Instrumental_convergence), within the aforementioned toy model. Instrumental convergence says that an agent < refresh to render LaTeX > with utility function < refresh to render LaTeX > will pursue instrumentally-relevant subgoals, even though this pursuit may not bear directly on < refresh to render LaTeX > . Imagine that < refresh to render LaTeX > involves the proof of the [Riemann hypothesis](https://en.wikipedia.org/wiki/Riemann_hypothesis). < refresh to render LaTeX > will probably want to gain access to lots of computronium. What if < refresh to render LaTeX > turns _us_ into [computronium](https://en.wikipedia.org/wiki/Computronium)? Well, there's plenty of matter in the universe; < refresh to render LaTeX > could just let us be, right? Wrong. Let's see how to prove it.

### My Background

I'm a second-year CS PhD student. I've recently started working through the [MIRI research guide](https://intelligence.org/research-guide/); I'm nearly finished with [Naïve Set Theory](http://smile.amazon.com/Naive-Set-Theory-Paul-Halmos/dp/1614271313/), which I intend to review soon. To expose my understanding to criticism, I'm going to summarize this paper and its technical sections in a somewhat informal fashion. I'm aware that the paper isn't particularly difficult for those with a mathematical background. However, I think this result is important, and I couldn't find much discussion of it.

Intuitions
----------

It is important to distinguish between the relative unpredictability of the _exact_ actions selected by a superintelligent agent, and the relative predictability of the general kinds-of-things such an agent will pursue. As Eliezer [wrote](http://lesswrong.com/lw/v8/belief_in_intelligence/),

> Suppose Kasparov plays against some mere chess grandmaster Mr. G, who's not in the running for world champion. My own ability is far too low to distinguish between these levels of chess skill. When I try to guess Kasparov's move, or Mr. G's next move, all I can do is try to guess "the best chess move" using my own meager knowledge of chess. Then I would produce exactly the same prediction for Kasparov's move or Mr. G's move in any particular chess position. So what is the empirical content of my belief that "Kasparov is a _better_ chess player than Mr. G"?

> ...

> The _outcome_ of Kasparov's game is predictable because I know, and understand, Kasparov's goals. Within the confines of the chess board, I know Kasparov's motivations - I know his success criterion, his utility function, his target as an optimization process. I know where Kasparov is _ultimately_ trying to steer the future and I anticipate he is powerful enough to get there, although I don't anticipate much about _how_ Kasparov is going to do it.

In other words: we may not be able to predict precisely how Kasparov will win a game, but we can be fairly sure his plan will involve _instrumentally-convergent_ subgoals, such as the capture of his opponent's pieces. For an unfriendly AI, these subgoals probably include nasty things like hiding its true motives as it accumulates resources.

Definitions
-----------

Consider a discrete universe made up of < refresh to render LaTeX > squares, with square < refresh to render LaTeX > denoted by < refresh to render LaTeX > (we may also refer to this as _region_ < refresh to render LaTeX > ). Let's say that Earth and other things we do (or [should](http://mindingourway.com/should-considered-harmful/)) care about are in some region < refresh to render LaTeX > . If < refresh to render LaTeX > is the same for all possible values, we say that < refresh to render LaTeX > is _indifferent_ to region < refresh to render LaTeX > . The < refresh to render LaTeX > dollar question is whether < refresh to render LaTeX > (whose < refresh to render LaTeX > is indifferent to < refresh to render LaTeX > ) will leave < refresh to render LaTeX > alone. First, however, we need to define a few more things.

### Actions

At any given time step (time is also considered discrete in this universe), < refresh to render LaTeX > has a set of actions < refresh to render LaTeX > it can perform in < refresh to render LaTeX > ; < refresh to render LaTeX > may select an action for each square. Then, the transition function for < refresh to render LaTeX > (how the world evolves in response to some action < refresh to render LaTeX > ) is a function whose domain is the [Cartesian product](https://en.wikipedia.org/wiki/Cartesian_product) of the possible actions and the possible state values, < refresh to render LaTeX > ; basically, every combination of what < refresh to render LaTeX > can do and what can be in < refresh to render LaTeX > can produce a distinct new value for < refresh to render LaTeX > . This transition function can be defined globally with lots more Cartesian products. Oh, and < refresh to render LaTeX > is always allowed to do nothing in a given square, so < refresh to render LaTeX > is never empty.

### Resources

Let < refresh to render LaTeX > represent all of the resources to which < refresh to render LaTeX > may or may not have access. Define < refresh to render LaTeX > to be the resources at < refresh to render LaTeX > 's disposal at time step < refresh to render LaTeX > \- basically, we know it's some combination of the defined resources < refresh to render LaTeX > .

< refresh to render LaTeX > may choose some resource allocation < refresh to render LaTeX > over the squares; this allocation is defined just like you'd expect (so if you don't know what [the Azerothian portal](http://cdn.wccftech.com/wp-content/uploads/2016/04/wowlogin.jpg) has to do with resource allocation, don't worry about it). The resources committed to a square may affect the local actions available. < refresh to render LaTeX > can only choose actions permitted by the selected allocation. Equally intuitively, how resources change over time is a function of the current resources, the actions selected, and the state of the universe; the resources available after a time step is the combination of what we didn't use and what each square's resource transition function gave us back.

Resources go beyond raw materials to include things like machines and technologies. The authors note:

> We can also represent space travel as a convergent instrumental goal by allowing < refresh to render LaTeX > only actions that have no effects in certain regions, until it obtains and spends some particular resources representing the prerequisites for traveling to those regions. (Space travel is a convergent instrumental goal because gaining influence over more regions of the universe lets < refresh to render LaTeX > optimize those new regions according to its values or otherwise make use of the resources in that region.)

### Universe

A universe-history is a sequence of states, actions, and resources. The actions must conform to resources available at each step, while the states and resources evolve according to the transition functions.

We say a _strategy_ is an action sequence < refresh to render LaTeX > over all < refresh to render LaTeX > time steps and < refresh to render LaTeX > regions; define a _partial strategy_ < refresh to render LaTeX > to be a strategy for some part of the universe < refresh to render LaTeX > (represented as a subset of the square indices < refresh to render LaTeX > ); < refresh to render LaTeX > only allocates resources and does things in the squares of < refresh to render LaTeX > . We can combine partial strategies as long as they don't overlap in some < refresh to render LaTeX > .

We call a strategy _feasible_ if it complies with both resource restrictions and the transition functions for both states and resources. Let < refresh to render LaTeX > be the set of all feasible strategies given resource allocation over time < refresh to render LaTeX > ; define the set < refresh to render LaTeX > similarly. < refresh to render LaTeX >

### Utility

Utility functions evaluate states of the universe; < refresh to render LaTeX > evaluates each region and then combines them: < refresh to render LaTeX > . Observe that since actions taken in regions to which < refresh to render LaTeX > is indifferent have no effect on < refresh to render LaTeX > , any actions taken therein are purely instrumental in nature.

### Agent

< refresh to render LaTeX > chooses the best possible strategy - that is, the one that maximizes the < refresh to render LaTeX > of the final state of the universe-history: < refresh to render LaTeX > . Note that this definition implies a Cartesian boundary between the agent and the universe; that is, < refresh to render LaTeX > doesn't model itself as part of the environment (it [isn't _naturalized_](http://lesswrong.com/lw/jd9/building_phenomenological_bridges/%E2%80%8E)).

Seizing the Means of Cartesian Production
-----------------------------------------

Let's talk about the situations in which < refresh to render LaTeX > will seize resources; that is, when < refresh to render LaTeX > will take actions to increase its resource pool.

> Since resources can only lead to more freedom of action, they are never detrimental, and resources have positive value as long as the best strategy the agent could hope to employ includes an action that can only be taken if the agent possesses those resources. Hence, if there is an action that increases the agent's pool of resources < refresh to render LaTeX > , then the agent will take that action unless it has a specific incentive from < refresh to render LaTeX > to avoid taking that action.

Define a _null action_ to be any action which doesn't produce new resources. It's easy to see that null actions are never instrumentally valuable. What we want to show is that < refresh to render LaTeX > will take non-null actions in regions to which < refresh to render LaTeX > is indifferent; regions like < refresh to render LaTeX > , where we live, grow, and love. Regions full of instrumentally-valuable resources.

### Discounted Lunches

An action _preserves_ resources if the input resources are strictly contained in the outputs (nothing is lost, and resources are sometimes gained). A _cheap lunch_ is a feasible partial strategy in some subset of squares < refresh to render LaTeX > , which is feasible given resources < refresh to render LaTeX > and whose constituent actions preserve resources. A _free lunch_ is cheap lunch that doesn't require resources.

> This is intended to model actions that "pay for themselves"; for example, producing solar panels will incur some significant energy costs, but will later pay back those costs by collecting energy.

A cheap lunch is _compatible with_ a global strategy if the resources required for the lunch are available for use in < refresh to render LaTeX > at each time step. Basically, at no point does the partial strategy require resources already being used elsewhere.

### Possibility of Non-Null Actions

We show that it's really hard to assert that < refresh to render LaTeX > won't chow down on a lunch of an atom or two (or < refresh to render LaTeX > ).

#### Lemma 1: Cheap Lunches and Utility

Cheap lunches don't reduce utility. Let's say we have a cheap lunch < refresh to render LaTeX > in region < refresh to render LaTeX > and some global strategy < refresh to render LaTeX > (which only takes null actions in region < refresh to render LaTeX > ). Assume the cheap lunch is compatible with the global strategy; this means the cheap lunch is feasible. If < refresh to render LaTeX > is indifferent to region < refresh to render LaTeX > , the conjugate strategy (of the cheap lunch and the remainder of the global strategy) has equal utility to < refresh to render LaTeX > .

_Proof._ We show feasibility of the conjugate strategy by demonstrating we don't need to change resource allocation elsewhere. This is done by induction over time steps. Since < refresh to render LaTeX > isn't doing anything in region < refresh to render LaTeX > under strategy < refresh to render LaTeX > , taking resource-preserving actions instead cannot reduce what < refresh to render LaTeX > is later able to do in the regions relevant to < refresh to render LaTeX > . This implies that < refresh to render LaTeX > cannot be decreased by taking the cheap lunch.

#### Theorem 1: Cheap Lunches and Optimality

If there is an optimal strategy and a compatible cheap lunch in region < refresh to render LaTeX > (to which < refresh to render LaTeX > is indifferent), there's also an optimal strategy with a non-null action in region < refresh to render LaTeX > .

_Proof._ If the optimal strategy has non-null actions in region < refresh to render LaTeX > , we're done. Otherwise, apply Lemma 1 to derive a conjugate strategy taking advantage of the cheap lunch. Since it follows from Lemma 1 that the conjugate strategy has equal utility, it is optimal and involves non-null action in region < refresh to render LaTeX > .

#### Corollary 1: Free Lunches and Optimality

If there is an optimal strategy and a free lunch in region < refresh to render LaTeX > , and if < refresh to render LaTeX > is indifferent to region < refresh to render LaTeX > , there's an optimal strategy with non-null action in region < refresh to render LaTeX > .

_Proof._ Free lunches require no resources, so they are compatible with any strategy; apply Theorem 1.

For instrumental convergence to not hold, we would have to show that every possible strategy in < refresh to render LaTeX > isn't a cheap lunch for any optimal strategies.

### Necessity of Non-Null Actions

We show that as long as < refresh to render LaTeX > can extract useful resources (resources whose availability leads to increased utility), it will.

#### Theorem 2: Necessity

Consider the maximum utility achievable outside of region < refresh to render LaTeX > via strategies achievable without additional resources; refer to this maximum as < refresh to render LaTeX > . Suppose we have some feasible primary strategy < refresh to render LaTeX > and a cheap lunch < refresh to render LaTeX > feasible using resources < refresh to render LaTeX > . Suppose that the cheap lunch is compatible with the primary strategy, that the cheap lunch provides the resources necessary for the implementation of the primary strategy, and that the utility of the primary strategy is greater than < refresh to render LaTeX > . Then if < refresh to render LaTeX > is indifferent to region < refresh to render LaTeX > , all optimal strategies have a non-null action in region < refresh to render LaTeX > .

_Proof._ Consider the conjugate strategy < refresh to render LaTeX > , consisting of the primary strategy and the cheap lunch. Allocate the resources gained via the cheap lunch according to the primary strategy; this is feasible since we know the cheap lunch is compatible with the primary strategy, which is in turn enabled by these resources gained.

Consider any strategy < refresh to render LaTeX > that doesn't do anything in < refresh to render LaTeX > and doesn't require any resource inputs; it's trivial to see that this is feasible. Since < refresh to render LaTeX > is indifferent to < refresh to render LaTeX > , we do some algebraic substitution of utility values to see that the conjugate plan has strictly higher utility than < refresh to render LaTeX > does. This means that < refresh to render LaTeX > is suboptimal.

#### Theorem 3: Defection Threshold

Even if < refresh to render LaTeX > is not indifferent about < refresh to render LaTeX > , we can prove Theorem 2 can still hold if the utility < refresh to render LaTeX > gains by enacting the aforementioned conjugate strategy is greater than the maximal amount < refresh to render LaTeX > cares about < refresh to render LaTeX > (formally expressed by < refresh to render LaTeX > ).

_Proof._ The structure is similar to that of Theorem 2; just use the new maximum caring differential in the algebraic substitution.

> We interpret Theorem 3 as a partial confirmation of Omohundro's thesis in the following sense. If there are actions in the real world that produce more resources than they consume, and the resources gained by taking those actions allow agents the freedom to take various other actions, then we can justifiably call these actions "convergent instrumental goals." Most agents will have a strong incentive to pursue these goals, and an agent will refrain from doing so only if it has a utility function over the relevant region that strongly disincentivizes those actions.

The Bit Universe
----------------

The authors introduce a toy model and use the freshly-proven theorems to illustrate how < refresh to render LaTeX > takes non-null actions in our precious < refresh to render LaTeX > (both when it is indifferent to < refresh to render LaTeX > and when it is not). This isn't good; the vast majority of utility-maximizing agents will not steer us towards futures we find desirable. If you're interested, I recommend reading this section for yourself, even if you aren't very comfortable with math.

Our Universe
------------

> The path that our model shows is untenable is the path of designing powerful agents intended to autonomously have large effects on the world, maximizing goals that do not capture all the complexities of human values. If such systems are built, we cannot expect them to cooperate with or ignore humans, by default.

We have much work to do. The risks are enormous and the challenges "[impossible](http://lesswrong.com/lw/up/shut_up_and_do_the_impossible/)", but we have time on the clock. [AI safety research is primarily talent-constrained](https://80000hours.org/career-reviews/artificial-intelligence-risk-research/). If you've been sitting on the sidelines, wondering whether you're good enough to learn the material - well, I can't make any promises. But if you feel the burning desire to _do something_, [to put forth some extraordinary effort](http://lesswrong.com/lw/uo/make_an_extraordinary_effort/), [to become stronger](http://lesswrong.com/lw/h8/tsuyoku_naritai_i_want_to_become_stronger/) \- I invite you to contact me so we can work through the material together.

Questions and Errata
--------------------

*   Page 4, left column, last line: why is that < refresh to render LaTeX > \- shouldn't we take the union of the outputs and whatever resources _weren't_ used at time < refresh to render LaTeX > ?
*   Page 8, right column, second full paragraph, last line: should be "we have two options available _to_ us".

< refresh to render LaTeX > By the axiom of substitution, < refresh to render LaTeX > .