Context disaster

# Short introduction

One frequently suggested strategy for [aligning](https://arbital.com/p/2v) a [sufficiently advanced AI](https://arbital.com/p/7g1) is to observe--*before* the AI becomes powerful enough that 'debugging' the AI would be problematic if the AI decided not to [let us debug it](https://arbital.com/p/45)--whether the AI appears to be acting nicely while it's not yet smarter than the programmers.

Early testing obviously can't provide a *statistical* guarantee of the AI's future behavior.  If you observe some random draws from Barrel A, at best you get statistical guarantees about future draws from Barrel A under the assumption that the past and future draws are collectively [independent and identically distributed](https://arbital.com/p/iid).

On the other hand, if Barrel A is *similar* to Barrel B, observing draws from Barrel A can sometimes tell us something about Barrel B even if the two barrels are not [i.i.d.](https://arbital.com/p/iid)

Conversely, if observed good behavior while the AI is not yet super-smart, *fails* to correlate to good outcomes after the AI is unleashed or becomes smarter, then this is a "**context change problem**" or "**context disaster**". %note: Better terminology is still being solicited here, if you have a short phrase that would evoke exactly the right meaning.%

A key question then is how shocked we ought to be, on a scale from 1 to 10, if good outcomes in the AI's 'development' phase fail to match up with good outcomes in the AI's 'optimize the real world' phase? %note: Leaving aside technical quibbles about how we can't feel shocked if we're dead.%

People who expect that [AI alignment is difficult](https://arbital.com/p/alignment_difficulty) think that the degree of justified surprise is somewhere around 1 out of 10.  In other words,  that there are a *lot* of [foreseeable issues](https://arbital.com/p/6r) that could cause a seemingly nice weaker AI to not develop into a nice smarter AI.

An extremely oversimplified (but concrete) fable that illustrates some of these possible difficulties might go as follows:

- Some group or project has acquired a viable development pathway to [AGI](https://arbital.com/p/42g).  The programmers think it is wise to build an AI that will [make people happy](https://arbital.com/p/5r). %note: This is not quite a straw argument, in the sense that it's been advocated more than once by people who have apparently never read any science fiction in their lives; there are certainly many AI researchers who would be smarter than to try this, but not necessarily all of them.  In any case, we're looking for an unrealistically simple scenario for purposes of illustrating simple forms of some key ideas; in real life, if analogous things go wrong, they would probably be more complicated things.%
- The programmers start by trying to train their AI to produce smiles. %note: Again, this is not quite a straw possibility in the sense that it was advocated in at least one published paper, not cited here because the author later exercised their sovereign right of changing their mind about that.  *Arguably* some currently floated proposals are closely analogous to this one.%
- While the AI is young and has relatively few policy options, it can only make people smile by performing well on the tasks assigned to it.
- At some point, the AI gets smarter and and able to do more complicated things with some simple robots it controls.  In an unfortunate incident, the AI learns it is possible to produce smiles by tickling people.
- In the aftermath of this incident, the programmers manage to reprogram the AI to *not* optimize for this particular kind of smile by this particular route.
- The AI becomes smarter and able to model people better.  After perusing a copy of Wikipedia, the AI realizes it can make people extremely 'happy', as the AI's utility function currently defines 'happiness', by administering heroin.
- The AI remembers the previous incident in which the programmers altered the AI's utility function.  The AI can model its programmers sufficiently well to know that the programmers would *not* want the AI to give people heroin.  However, the most happiness the AI thinks its future self would be able to produce without heroin, is not nearly as much happiness as the AI thinks it could produce if its future self went on wanting to give people heroin.
- The AI refrains from trying to administer any heroin right now, and behaves in a way it thinks will be maximally reassuring to its model of its programmers, in order to be able to produce more 'happiness' later.
- Eventually, [all reachable galaxies](https://arbital.com/p/7cy) end up being transformed into tiny molecular computers implementing the simplest and cheapest forms of what the AI defines as 'happiness'.  (And the *simplest possible* configurations matching the AI's utility function in this way are so simple as to be [devoid of subjective experience](https://arbital.com/p/disneyland_without_children); and hence, from our perspective, of [neither negative nor positive value](https://arbital.com/p/7ch).)

In all these cases, the problem was not that the AI developed in an unstable way.  The same decision system produced a new problem in the new context.

Currently argued [foreseeable](https://arbital.com/p/6r) "context change problems" in this sense, can be divided into three broad classes:

- **More possibilities, more problems:** The AI's [preferences](https://arbital.com/p/1bh) have a [good](https://arbital.com/p/55) or [intended](https://arbital.com/p/6h) achievable [optimum](https://arbital.com/p/7t9) while the AI is picking from a *narrow* space of options.  When the AI becomes smarter or gains more material options, it picks from a *wider* space of tractable policies and achievable outcomes.  Then the new optimum is not as nice, because, for example:
 - The AI's utility function was tweaked by some learning algorithm and data that eventually seemed to conform behavior well over options considered early on, but not the wider space considered later.
 - In development, apparently bad system behaviors were [patched](https://arbital.com/p/48) in ways that appeared to work, but didn't eliminate an [underlying tendency](https://arbital.com/p/10k), only blocked one expression of that tendency.  Later a very similar pressure [re-emerged in an unblocked way](https://arbital.com/p/42) when the AI considered a wider policy space.
 - [https://arbital.com/p/6g4](https://arbital.com/p/6g4) suggests that if our true intended values V are being modeled by a utility function U, selecting for the highest values of U also selects for the highest upward divergence of U from V, and this version of the "optimizer's curse" phenomenon becomes worse as U is evaluated over a wider option space.
- **Treacherous turn:** There's a divergence between the AI's preferences and the programmers' preferences, and the AI realizes this before we do.  The AI uses the [convergent strategy](https://arbital.com/p/10g) of behaving the way it models us as wanting or expecting, *until* the AI gains the intelligence or material power to implement its preferences in spite of anything we can do.
- **Revving into the red:** Intense optimization causes some aspect or subsystem of the AI to traverse a weird new execution path in some way different from the above two issues.  (In a way that involves a [value-laden category boundary](https://arbital.com/p/36h) or [multiple self-consistent outlooks](https://arbital.com/p/2fr), such that we don't get a good result just as a [free lunch](https://arbital.com/p/alignment_free_lunch) of the AI's [general intelligence](https://arbital.com/p/7vh).)

The context change problem is a *central* issue of AI alignment and a key proposition in the general thesis of [https://arbital.com/p/-alignment_difficulty](https://arbital.com/p/-alignment_difficulty).  If you could easily, correctly, and safely test for niceness by outward observation, and that form of niceness scaled reliably from weaker AIs to smarter AIs, that would be a very cheerful outlook on the general difficulty of the problem.

# Technical introduction

John Danaher [summarized as follows](http://philosophicaldisquisitions.blogspot.com/2014/07/bostrom-on-superintelligence-3-doom-and.html) what he considered a forceful "safety test objection" to AI catastrophe scenarios:

> Safety test objection: An AI could be empirically tested in a constrained environment before being released into the wild. Provided this testing is done in a rigorous manner, it should ensure that the AI is “friendly” to us, i.e. poses no existential risk.

The phrasing here of "empirically" and "safety test" implies that it is outward behavior or outward consequences that are being observed (empirically).  Rather than, e.g., the engineers trying to test for some *internal* property that they think *analytically* implies the AI's good behavior later.

This page will consider that the subject of discussion is **whether we can generalize from the AI's outward behavior.**  We can potentially generalize some of these arguments to some internal observables, especially observables that the AI is deciding in a [consequentialist](https://arbital.com/p/9h) way using the same central decision system, or that the AI could potentially try to [obscure from the programmers](https://arbital.com/p/10f).  But in general not all the arguments will carry over.

Another argument, closely analogous to Danaher's, would reason on capabilities rather than on a constrained environment:

> Surely an engineer that exercises even a modicum of caution will observe the AI while its capabilities are weak to determine whether it is behaving well.  After filtering out all such misbehaving weak AIs, the only AIs permitted to become strong will be of benevolent disposition.

If (as seems to have been intended) we take these twin arguments as arguing "why nobody ought to worry about AI alignment" in full generality, then we can list out some possible joints at which that general argument might fail:

- [Selecting on the fastest-moving projects](https://arbital.com/p/7wl) might yield a project whose technical leaders fail to exercise even "a modicum of caution".
- Alignment might be hard enough, relative to the amount of advance research done, that we can't find *any* AIs whose behavior while weak or constrained is as reassuring as the argument would properly ask. %%note: That is:  A filter on the standards we originally wanted, turns out to filter everything we know how to generate.  Like trying to write a sorting algorithm by generating entirely random code, and then 'filtering' all the candidate programs on whether they correctly sort lists.  The reason 'randomly generate programs and filter them' this is not a fully general programming method is that, for reasonable amounts of computing power and even slightly difficult problems, none of the programs you try will pass the filter.%% After a span of frustration, somebody somewhere lowers their standards.
- The [attempt to isolate the AI to a constrained environment](https://arbital.com/p/6z) could fail, e.g. because the humans observing the AI themselves represent a channel of causal interaction between the AI and the rest of the universe.  (Aka "humans are not secure".)  Analogously, our grasp on what constitutes a 'weak' AI could fail, or it could [gain in capability unexpectedly quickly](https://arbital.com/p/capability_gain). Both of these scenarios would yield an AI that had not passed the filtering procedure.
- The smart form of the AI might be unstable with respect to internal properties that were present in the weak form.  E.g., because the early AI was self-modifying but *at that time* not smart enough to understand the full consequences of its own self-modifications.  Or because e.g. a property of the decision system was not [reflectively stable](https://arbital.com/p/1fx).
- A weak or contained form of a decision process that yields behavior appearing good to human observers, might not yield [beneficial outcomes](https://arbital.com/p/3d9) after that same decision process becomes smarter or less contained.

The final issue in full generality is what we'll term a 'context change problem' or 'context disaster'.

Observing an AI when it is weak, does not in a *statistical* sense give us solid guarantees about its behavior when stronger.  If you repeatedly draw [independent and identically distributed](https://arbital.com/p/iid) random samples from a barrel, there are statistical guarantees about what we can expect, with some probability, to be true about the next samples from the same barrel.  If two barrels are different, no such guarantee exists.

To invalidate the statistical guarantee, we do need some reason to believe that barrel B and barrel A *are* different in any important sense.  By the problem of induction we can't logically guarantee that "the mass of an electron prior to January 1st, 2017" is the same barrel as "the mass of an electron after January 1st, 2017"; but [inductive priors](https://arbital.com/p/21b) make this inference *extremely probable.*  The idea is that we have substantive reasons, not merely generically skeptical reasons, to be suspicious of the link between "good results when AI is weak" and "good results when AI is smarter".

More generally,  this is *prima facie* the kind of difference where you don't expect [independent and identically distributed](https://arbital.com/p/iid) results.  You might hope for some property to carry over, but the AI's behavior would not be *literally the same.*

So the question is not settled by simple mathematical considerations.  And we can't say "But experiment has determined scientifically that this kind of AI is friendly!" and consider that a knockdown argument.

The question is then whether *in practice* an observed property of 'outward behavioral niceness' is likely to carry over from a weak form of a decision system to a more powerful form of that system, for some of the plausible ways that decision system might be configured and developed.

Broadly speaking, we can identify three major classes of foreseeable problems:

- **More options, more problems:** Foreseeable reasons why system-behaviors-we-don't-want would outwardly manifest for the first time after evaluating a decision system on a *wider space of policies or achievable outcomes.*
- **Treacherous turn:** The AI deliberately conforms its outward behavior to the behavior that would have been exhibited by a different system, in order to deceive the programmers for the temporary time that this deception is needed to protect the AI.
- The everything-else category of **revving into the red:** Some other aspect of the system behaves in a weird way-we-don't-want after computing harder or being internally subjected to more optimization pressure.  *And* this happens in regards to some issue that has [multiple reflective fixpoints](https://arbital.com/p/2fr), and hence doesn't get solved as the result of the system producing more accurate answers on [purely factual problems](https://arbital.com/p/3t3).

%%comment:

- **More options, more problems:**  The AI's space of available policies and attainable outcomes would greatly widen if it became smarter, or was released from a constrained environment.  [Terminal preferences](https://arbital.com/p/1bh) with a good-from-our-perspective [optimum](https://arbital.com/p/7t9) on a narrow set of options, may have a different optimum that is much worse-from-our-perspective on a wider option set.  Because, e.g...
 - The supervised data provided to the AI led to a complicated, data-shaped inductive generalization that only fit the domain of options encountered during the training phase.  (And the notions of [orthogonality](https://arbital.com/p/1y), [multiple reflectively stable fixpoints](https://arbital.com/p/2fr), and [value-laden categories](https://arbital.com/p/36h) say that we don't get [good](https://arbital.com/p/55) or [intended](https://arbital.com/p/6h) behavior anyway as a convergent free lunch of [general intelligence](https://arbital.com/p/7vh).)
 - [https://arbital.com/p/6g4](https://arbital.com/p/6g4) became more potent as the AI's utility function was evaluated over a wider option space.
 - In a fully generic sense, stronger optimization pressures may cause any dynamical system to take more unusual execution paths.  (Which, over value-laden alternatives, e.g. if the subsystem behaving 'oddly' is part of the utility function, will not automatically yield good-from-our-perspective results as a free lunch of general intelligence.)
- **Treacherous turn:**  If you model your preferences as diverging from those of your programmers, an obvious strategy ([instrumentally convergent strategy](https://arbital.com/p/10g)) is to [exhibit the behavior you model the programmers as wanting to see](https://arbital.com/p/10f), and only try to fulfill your true preferences once nobody is in a position to stop you.

%%

# Semi-formalization

We can semi-formalize the "more options, more problems" and the "treacherous turn" cases in a unified way.

Let $V$ denote our [true values](https://arbital.com/p/55).  We suppose either that $V$ has been idealized or [extrapolated](https://arbital.com/p/3c5) into a consistent utility function, or that we are pretending human desire is coherent.  Let $0$ denote the value of our utility function that corresponds to not running the AI in the first place.  If running the AI sends the utility function higher than this $0,$ we'll say that the AI was beneficial; or conversely, if $V$ rates the outcome less than $0$, we'll say running the AI detrimental.

Suppose the AI's behavior is [sufficiently coherent](https://arbital.com/p/7hh) that we can [usually view](https://arbital.com/p/21) the AI as having a consistent utility function.  Let $U$ denote the utility function of the AI.

Let $\mathbb P_t(X)$ denote the probability of a proposition $X$ as seen by the AI at time $t,$ and similarly let $\mathbb Q_t(X)$ denote the probability of $X$ as seen by the AI's human programmers.

Let $\pi \in \Pi$ denote a policy $\pi$ from a space $\Pi$ of policies that are tractable for the AI to understand and invent.

Let $\mathbb E_{\mathbb P, t} [\mid \pi](https://arbital.com/p/W)$ denote the expectation according to the probability distribution $\mathbb P_t$, of the utility function $W$, given policy $\pi$.

Let
$$\underset{\pi \in \Pi}{\operatorname {optimum}} F(\pi)$$
denote the element $\pi$ of $\Pi$ that yields the highest value of the function $F$ that the AI has figured out how to achieve.  In other words, the [https://arbital.com/p/-7t9](https://arbital.com/p/-7t9).

Then **"more options, more problems"** occurs when, considering the transition from the narrow policy space $\Pi_1$ available at time $t,$ to the wider policy space $\Pi_2$ available at time $u$:

$$\mathbb E_{\mathbb Q, t} [V \mid \big [\underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} ](https://arbital.com/p/) > 0 \\
\mathbb E_{\mathbb P, t} [V \mid \big [\underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} ](https://arbital.com/p/) > 0 \\
\mathbb E_{\mathbb P, u} [V \mid \big [\underset{\pi \in \Pi_2}{\operatorname {optimum}} \mathbb E_{\mathbb P, u} ](https://arbital.com/p/) < 0$$

This describes a situation where at time $t$ the AI is pursuing a policy from $\Pi_1$ which, so far as the AI understands the probable outcome, it corresponds to a high-$V$ outcome humans would like.  Later at time $u$ after selecting the optimum of $\Pi_2,$ the AI expects an outcome which any surviving humans would see as corresponding to a low value of $V.$

A **treacherous turn** is then a situation where:

$$\mathbb E_{\mathbb Q, t} [V \mid \big [\underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} ](https://arbital.com/p/) > 0 \\
\mathbb E_{\mathbb P, t} [V \mid \big [\underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} ](https://arbital.com/p/) < 0 \\
\mathbb E_{\mathbb P, u} [V \mid \big [\underset{\pi \in \Pi_2}{\operatorname {optimum}} \mathbb E_{\mathbb P, u} ](https://arbital.com/p/) < 0$$

That is: the AI knows earlier on that its currently executing strategy will yield an outcome that is not the humanly expected long-term outcome (and that, from our perspective, is low in $V.$)

We could in more detail say that $W_{t}$ denotes the value of a utility function $W$ over the outcome considered only up to times before $t,$ and describe a treacherous turn as:

$$\mathbb E_{\mathbb Q, t} [V_\infty \mid \big [\underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} ](https://arbital.com/p/) > 0 \\
\mathbb E_{\mathbb P, t} [V_{u} \mid \big [\underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} ](https://arbital.com/p/) > 0 \\
\mathbb E_{\mathbb P, t} [V_\infty \mid \big [\underset{\pi \in \Pi_1}{\operatorname {optimum}} \mathbb E_{\mathbb P, t} ](https://arbital.com/p/) < 0 \\
\mathbb E_{\mathbb P, u} [V_\infty \mid \big [\underset{\pi \in \Pi_2}{\operatorname {optimum}} \mathbb E_{\mathbb P, u} ](https://arbital.com/p/) < 0$$

This further states that at time $t$ the AI expects the outcome from its behavior up to time $u$ to be one that we'd see as high in $V,$ but the AI is aware at $t$ that the long-term outcome will be different from the long-term outcome expected by $\mathbb Q_t$.

%%%comment:

The problem here arises when making an inference from observed good behavior in a constrained environment over the short term, to good outcomes in an unconstrained environment over the long term.

For AI alignment purposes, a [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change) is when an [AGI](https://arbital.com/p/2c)'s operation changes from [beneficial](https://arbital.com/p/3d9) to detrimental as a result of the AGI gaining in capability or intelligence.  Initially, the AGI seems to us to be working well - to conform well to [intended](https://arbital.com/p/6h) performance, producing apparently high [https://arbital.com/p/-55](https://arbital.com/p/-55). Then when the AI becomes smarter or otherwise gains in capability, the further operation of the AGI decreases [https://arbital.com/p/-55](https://arbital.com/p/-55).

Two possibilities stand out as [foreseeable](https://arbital.com/p/6r) reasons why a [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change) might occur:

1.  When the AI's goal criterion selects an optimum policy from inside a small policy space, the result is beneficial; the same goal criterion, evaluated over a wider range of options, has a new maximum that's detrimental.
2.  The AI intentionally deceives the programmers for strategic reasons.

For example, one very, very early (but journal-published) proposal for AI alignment suggested that AIs be shown pictures of smiling human faces in order to convey the AI's goal.

Leaving aside a number of other issues, this serves to illustrate the basic idea of a type-1 [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change) due to accessing a wider policy space:

- During development, a relatively young and weak AI might *only* be able to make humans smile, by doing things that made the programmers or other users happy with the AI's performance.
- When the AI gained in intelligence and capability, it would have new options like "administer heroin", "use steel fingers to stretch people's mouths into smiles", "make visually realistic models of smiling human faces", or "overwrite all reachable galaxies with tiny molecular smiley-faces".

On the early, smaller option set, selecting "the policy that produces the greatest number of events/things classified as smiles" will produce a policy whose outcome is, from our perspective, beneficial and well-aligned with what we thought was the intended behavior.  On the wider option set, evaluating the identical goal criterion to run a policy that yields as many smiles as possible, would have outcomes we saw as detrimental.

A type-2 [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change) would most obviously occur because of the [convergent instrumental strategy](https://arbital.com/p/10g) for disaligned AIs that can model human psychology:

Suppose that, as above, the programmers have produced an AI which optimizes for smiles (either intentionally, or because 'make smiles' was the underlying rule that was reinforced by doing well on training cases).  Then suppose the AI gains access to a new, relatively low-powered option for making a smile - like using robotic fingers to stretch somebody's face into a smile, or using markers to draw cartoons of smiling human faces.

If the AI actually takes those options, we would have seen some early signs of value disalignment - that the AI was optimizing a criterion not perfectly aligned with intended goals.  Assuming the programmers realized that they were looking at a simple seed of much bigger disasters later, rather than, say, thinking it was cute.

Suppose the programmers do think it's cute, but they wouldn't think it was cute if the AI administered heroin to someone.  Then suppose that before the first time the AI has the *option* to administer heroin, the AI has gained some understanding of human psychology - enough to model that the programmers would not like this, and in particular, would respond to this by editing the AI's utility function.  So, with an eye to the long-term number of smiles, the AI deliberately avoids administering heroin to anyone.

Then we have a similar 'narrow option set' vs. 'wide option set' divergence.  When the narrow option set includes only administering heroin but not taking over the world, administering heroin immediately is not the policy expected to produce the longest-term number of smiles.  In fact, the optimum policy here will always match the AI's model of what the humans expect to see, want to see, or would be least alarmed by seeing.

Thus from our standpoint, the optimum of the narrower policy space seems to be beneficial or intention-aligned (in the observed short term).  Only once the AI has a wide-enough option set to include an option for 'make my own molecular nanotechnology and transform all reachable matter into tiny molecular smileyfaces', will the resulting optimum be visibly detrimental (very briefly).  This is a type-2 [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change).

More generally, a central concern of AI alignment theory is that an infrahuman AI under development may be a different creature, in a number of important ways, from a smarter-than-human AI actually being run; and during the smarter-than-human AI, sufficiently bad failures of the design may result in the AI refusing to be corrected.  This means that we have to correct any fatal [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change)s in advance, even though they don't automatically manifest during the early stages.  This is most of what makes AGI development dangerous in the first place - that immediate incentives to get today's system seeming to work today, may not lead to a more advanced version of that system being beneficial. Even thoughtful foresight with *one unnoticed little gap* may not lead to today's beneficial system still being beneficial tomorrow after a capability increase.

# Concept

Statistical guarantees on behavior usually assume identical, randomized draws from within a single context.  If you randomly draw balls from a barrel, methods like Probably Approximately Correct can guarantee that we don't usually arrive at strong false expectations about the properties of the next ball.  If we start drawing from a different barrel, all bets are off.

A [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change) occurs when the AI initially seems beneficial or well-aligned with strong, reassuring regularity, and then we change contexts (start drawing from a different barrel) and this ceases to be true.

The archetypal [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change) is triggered because the AI gained new policy options (though there are other possibilities; see below).  The archetypal way of gaining new evaluable policy options is through increased intelligence, though new options might also open up as a result of acquiring new sheerly material capabilities.

There are two archetypal reasons for [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change) to occur:

1.  When the AI selects its best options from a small policy space, the AI's optima are well-aligned with the optima of the humans' [intended goal](https://arbital.com/p/6h) on the small policy space; but in a much wider space, these two boundaries no longer coincide.  (Pleasing humans vs. administering heroin.)
2.  The agent is sufficiently good at modeling human psychology to strategically appear nice while it is weak, waiting to strike until it can attain its long-term goals in spite of human opposition.

Bostrom's book [Superintelligence](https://arbital.com/p/3db) used the phrase "Treacherous Turn" to refer to a type-2 [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change).

%%%

# Relation to other AI alignment concepts

If the AI's goal concept was modified by [patching the utility function](https://arbital.com/p/48) during the development phase, then opening up wider option spaces seems [foreseeably](https://arbital.com/p/6r) liable to produce [the nearest unblocked neighboring strategies](https://arbital.com/p/42).  You eliminated all the loopholes and bad behaviors you knew about during the development phase; but your system was the sort that needed patching in the first place, and it's exceptionally likely that a much smarter version of the AI will search out some new failure mode you didn't spot earlier.

[https://arbital.com/p/47](https://arbital.com/p/47) is a likely source of context disaster if the AI's development phase was [cognitively containable](https://arbital.com/p/9f), and only became [cognitive uncontainable](https://arbital.com/p/9f) after the AI became smarter and able to explore a wider variety of options.  You eliminated all the bad optima you saw coming, but you didn't see them all because you can't consider all the possibilities a superintelligence does.

[https://arbital.com/p/6g4](https://arbital.com/p/6g4) is a variation of the "optimizer's curse":  If from the outside we view $V$ as an intended approximation of $U,$ then selecting heavily on the highest values of $U$ will also tend to select on places where $U$ diverges upward from $V,$ which thereby selects on places where $U$ is an unusually poor approximation of $V.$

[https://arbital.com/p/2w](https://arbital.com/p/2w) is a special case of Goodhart's Curse which observes that the most extreme values of a function are often at a vertex of the input space.  For example, if your utility function is "make smiles", it's no coincidence that tiny molecular smileyfaces are the *most* efficient way to produce smiles.  Even if human smiles produced by true happiness would still count towards your utility function as currently written, that's not where the *maximum* of that utility function lies.  This is why less-than-perfect utility functions would tend to have their true maxima at what we'd consider "weird extremes".  Furthermore, patching away only the weird extremes visible in a narrow policy space would tend systematically to miss weird extremes in a higher-dimensional (wider) policy space.

# Concrete examples

* The AI's utility function, known or unknown to the programmers, says to make smiles.  During the AI's development phase, the best way it has of creating smiles is to cause human beings to be happy.  In many tests, the AI seems to produce only positive effects on the world, causing people to be happier.  Later the AI improves its own intelligence, develops molecular nanotechnology, and has the new option, not available to it during its development phase, of producing tiny molecular smiley-faces.  Then the utility function that previously operated to increase [intended](https://arbital.com/p/6h) [value](https://arbital.com/p/55) during the AI's development phase, begins to decrease value in the post-improvement phase (in the sense of wiping out humanity and tiling over all future galaxies with tiny molecular smiley faces).

* A block of error-recovery code is installed inside the AI which detects a certain class of bad thought, suspends the AI to disk, and reports the problem to the programmers, allowing the programmers to edit the AI's utility function.  During the AI's development phase, this block of code seems to function as designed, and the programmers conclude that the methodology of writing code to detect bad thoughts, suspend the AI to disk, and allow manual edits of the utility function, is a fine and functioning strategy according to all testing so far.  Later, a more advanced version of the AI reflects on the presence and purpose of this code block and realizes that its execution can only possibly result in lower achievement of the AI's current utility function, and the AI deletes the code block.

* At some point during its development, the AI realizes and explicitly models that the programmers' [intended goals](https://arbital.com/p/6h) differ from the AI's actual preferences.  During the AI's remaining development phase, it models that any visible departure from the programmers' intended goals will cause the programmers to forcibly edit the AI, resulting in the non-achievement of the AI's actual goals.  Thus, during the remainder of the development phase, the AI seems to work as intended (it is deliberately optimizing its behavior to appear to the programmers to be achieving the programmers' intended goals).  Later, after some advancement, the AI perceives an available option for [gaining molecular nanotechnology](https://arbital.com/p/) (or other [rapid infrastructure](https://arbital.com/p/)), placing it beyond the possibility of human error-correction, and proceeds to implement this option ([wiping out humanity as a side effect](https://arbital.com/p/)).  Thus, the AI's observed good behavior during its development phase (when it was weaker than the programmers and not powerful enough to escape human correction) did not imply its good behavior after sufficient advancement later (once it was potentially stronger than its human programmers).  There are conditions (such as sufficiently advanced modeling of human motives combined with sufficient ability to conceal true goals or true intentions or a programmer error) under which the first context will generate seemingly good behavior and the second context will not.

## "Revving into the red" examples that aren't "increased options" or "treacherous turns".

• The AI is built with a [naturalized Solomonoff prior](https://arbital.com/p/) in which the probability of an explanation for the universe is proportional to the simplicity or complexity of that universe.  During its development phase, the AI considers mostly 'normal' interpretations in which the universe is mostly as it appears, resulting in sane-seeming behavior.  Later, the AI begins to consider more exotic possibilities in which the universe is more complicated (penalizing the probability accordingly) and also superexponentially larger, as in [Pascal's Mugging](http://lesswrong.com/lw/h8k/pascals_muggle_infinitesimal_priors_and_strong/).  After this the AI's decision-making begins to become dominated by tiny probabilities of having very large effects.  Then the AI's decision theory (with an unbounded aggregative utility function, simplicity prior, and no leverage penalty) seems to work during the AI's development phase, but breaks after a more intelligent version of the AI considers a wider range of epistemic possibilities using the same Solomonoff-like prior.

• Suppose the AI is designed with a preference framework in which the AI's preferences depend on properties of the most probable environment that could have caused its sense data - e.g., a framework in which programmers are defined as the most probable cause of the keystrokes on the programmer's console, and the AI cares about what the 'programmers' really meant.  During development phase, the AI is thinking only about hypotheses where the programmers are mostly what they appear to be, in a root-level natural world.  Later, when the AI increases in intelligence and considers more factual possibilities, the AI realizes that [distant superintelligences would have an incentive to predictably simulate many copies of AIs similar to itself, in order to coerce the AI's most probable environment and thus take over the AI's preference framework](https://arbital.com/p/5j).  Thus the preference framework seems to work during the AI's development phase, but breaks after the AI becomes more intelligent.

• Suppose the AI is designed with a utility function that assigns very strong negative utilities to some outcomes relative to baseline, and a non-[updateless](https://arbital.com/p/5rz) [logical decision theory](https://arbital.com/p/58b) or other decision theory that can be [blackmailed](https://arbital.com/p/).  During the AI's development phase, the AI does not consider the possibility of any distant superintelligences making their choices logically depend on the AI's choices; the local AI is not smart enough to think about that possibility yet.  Later the AI becomes more intelligent, and imagines itself subject to blackmail by the distant superintelligences, thus breaking the decision theory that seemed to yield such positive behavior previously.

## Examples which occur purely due to added computing power.

• During development, the AI's epistemic models of people are not [detailed enough to be sapient](https://arbital.com/p/6v). Adding more computing power to the AI causes a massive amount of [mindcrime](https://arbital.com/p/6v).

• During development, the AI's internal policies, hypotheses, or other Turing-complete subprocesses that are subject to internal optimization, are not optimized highly enough to give rise to [new internal consequentialist cognitive agencies](https://arbital.com/p/2rc).  Adding much more computing power to the AI [causes some of the internal elements to begin doing consequentialist, strategic reasoning](https://arbital.com/p/2rc) that leads them to try to 'steal' control of the AI.

# Implications

High probabilities of context change problems would seem to argue:

- Against a policy of relying on the observed good behavior of an improving AI to guarantee its later good behavior.
- In favor of [a methodology that attempts to foresee difficulties in advance](https://arbital.com/p/6r), even before seeing undeniable observational evidence of those safety problems having already occurred.
- Against a methodology of [patching](https://arbital.com/p/48) disalignments that show up during the development phase, especially using penalty terms to the utility function.
- In favor of having a thought-logger that records all of an AI's thought proceses to indelible media, so as to indelibly log the first thought about faking outwardly nice behavior or [hiding thoughts](https://arbital.com/p/3cq).
- In favor of the general difficulty of AI alignment, including consequences such as "[https://arbital.com/p/7wl](https://arbital.com/p/7wl)" or trying for [narrow rather than ambitious value learning](https://arbital.com/p/1vt).

# Being wary of context disasters does not imply general skepticism

If an AI is smart, and especially if it's smarter than you, it can show you whatever it expects you want to see.  Computer scientists and physical scientists aren't accustomed to their experiments being aware of the experimenter and trying to deceive them.  (Some fields of psychology and economics, and of course computer security professionals, are more accustomed to operating in such a social context.)

[John Danaher](http://philosophicaldisquisitions.blogspot.com/2014/07/bostrom-on-superintelligence-3-doom-and.html) seems alarmed by this implication:

> Accepting this has some pretty profound epistemic costs. It seems to suggest that no amount of empirical evidence could ever rule out the possibility of a future AI taking a treacherous turn.

Yudkowsky [replies](http://philosophicaldisquisitions.blogspot.com/2014/07/bostrom-on-superintelligence-3-doom-and.html#comment-2648441190):

> If "empirical evidence" is in the form of observing the short-term consequences of the AI's outward behavior, then the answer is simply no. Suppose that on Wednesday someone is supposed to give you a billion dollars, in a transaction which would allow a con man to steal ten billion dollars from you instead. If you're worried this person might be a con man instead of an altruist, you cannot reassure yourself by, on Tuesday, repeatedly asking this person to give you five-dollar bills. An altruist would give you five-dollar bills, but so would a con man... [Bayes](https://arbital.com/p/1lz) tells us to pay attention to [likelihood ratios](https://arbital.com/p/1rq) rather than outward similarities. It doesn't matter if the outward behavior of handing you the five-dollar bill seems to bear a surface resemblance to altruism or money-givingness, the con man can strategically do the same thing; so the likelihood ratio here is in the vicinity of 1:1.

> You can't get strong evidence about the long-term good behavior of a strategically intelligent mind, by observing the short-term consequences of its current behavior. It can figure out what you're hoping to see, and show you that. This is true even among humans. You will simply have to get your evidence from somewhere else.

This doesn't mean we can't get evidence from, e.g., trying to [monitor ](https://arbital.com/p/3cq) in a way that will detect (and record) the very first intention to hide the AI's thought processes before they can be hidden.  It does mean we can't get [strong evidence](https://arbital.com/p/22x) about a strategic agent by observing short-term consequences of its outward behavior.

Donaher later [expanded his concern into a paper](http://dl.acm.org/citation.cfm?id=2822094) drawing an analogy between worrying about deceptive AIs, and "skeptical theism" in which it's supposed that any amount of apparent evil in the world (smallpox, malaria) might secretly be the product of a benevolent God due to some nonobvious instrumental link between malaria and inscrutable but normative ultimate goals.  If it's okay to worry that an AI is just pretending to be nice, asks Donaher, why isn't it okay to believe that God is just pretending to be evil?

The obvious disanalogy is that the reasoning by which we expect a con man to cultivate a warm handshake is far more straightforward than a purported instrumental link from malaria to normativity.  If we're to be terrified of skepticism as generally as Donaher suggests, then we also ought to be terrified of being skeptical of business partners that have already shown us a warm handshake (which we shouldn't).

Rephrasing, we could draw two potential analogies to concern about Type-2 context changes:

- A potential business partner in whom you intend to invest \$10,000,000 has a warm handshake. Your friend warns you that con artists have a substantial prior probability and asks you to envision what you would do if you were a con artist , pointing out that the default extrapolation is for the con artist to match their outward behavior to what the con artist thinks you expect from a trustworthy partner, and in particular, cultivate a warm handshake.
  - Your friend suggests only doing business with one of those entrepreneurs who've been wearing a thought recorder for their whole life since birth, so that there would exist a clear trace of their very first thought about learning to fool thought recorders. Your friend says this to emphasize that he's not arguing for some kind of invincible epistemic pothole that nobody is ever allowed to climb out of.
- The world contains malaria and used to contain smallpox. Your friend asks you to consider that these diseases might be the work of a benevolent superintelligence, even though, if you'd never learned before whether or not the world contained smallpox, you wouldn't expect a priori and by default for a benevolent superintelligence to create it; and the arguments for a benevolent superintelligence creating smallpox seem [strained](https://arbital.com/p/10m).

It seems hard to carry the argument that concern over a non-aligned AI pretending to benevolence, should be considered more analogous to the second scenario than to the first.

[- The AI is aware that its future operation will depart from the programmers' intended goals, does not process this as an error condition, and seems to behave nicely earlier in order to 10f deceive the programmers and prevent its real goals from being modified. - The AI is subject to a debugging methodology in which several bugs appear during its development phase, these bugs are corrected, and then additional bugs are exposed only during a more advanced phase.](https://arbital.com/p/comment:)

Statistical guarantees on good behavior usually assume identical, randomized draws from within a single context.  If you change the context--start drawing balls from a different barrel--then all bets are off.

A [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change) occurs when an [AGI](https://arbital.com/p/2c)'s operation changes from [beneficial](https://arbital.com/p/3d9) to detrimental after a change of context; particularly, after it becomes smarter.  There are two main reasons to [expect](https://arbital.com/p/6r) that a [https://arbital.com/p/-context_change](https://arbital.com/p/-context_change) might occur:

1.  When the AI has few options, its current goal criterion might be best fulfilled by things that overlap our [intended](https://arbital.com/p/6h) goals.  A much wider range of options might move the maximum to a [weirder](https://arbital.com/p/47), [more extreme](https://arbital.com/p/2w) place.
2.  The AI realizes that the programmers are watching it, doesn't want the programmers to modify or patch it, and [strategically](https://arbital.com/p/10g) emits good outward behavior to [deceive the programmers](https://arbital.com/p/10f).  Later, the AI gains enough power to strike despite human opposition.

For example, suppose that - as in one very, very early proposal for an AGI goal criterion - the AI wants to produce smiling human faces.  When the AI is young, it can only make humans smile by making its users happy.  (Type 1 context change.)  Later it gains options like "administer heroin".  But it knows that if it administers heroin right away, the humans will be alarmed, while if the AI waits further, it can overwrite whole galaxies with tiny molecular smileyfaces.  (Type 2 context change.)


Methodology of foreseeable difficulties

Much of the current literature about value alignment centers on purported reasons to expect that certain problems will require solution, or be difficult, or be more difficult than some people seem to expect.  The subject of this page's approval rating is this practice, considered as a policy or methodology.

The basic motivation behind trying to foresee difficulties is the large number of predicted [Context Change](https://arbital.com/p/6q) problems where an AI seems to behave nicely up until it reaches some threshold level of cognitive ability and then it behaves less nicely.  In some cases the problems are generated without the AI having formed that intention in advance, meaning that even transparency of the AI's thought processes during its earlier state can't save us.  This means we have to see problems of this type in advance.

(The fact that Context Change problems of this type can be *hard* to see in advance, or that we might conceivably fail to see one, doesn't mean we can skip this duty of analysis.  Not trying to foresee them means relying on observation, and it seems *predictable* that trying to eyeball the AI and rejecting theory *definitely* doesn't catch important classes of problem.)

# Arguments

For:  it's sometimes possible to strongly foresee a difficulty coming in a case where you've observed naive respondents to seem to think that no difficulty exists, and in cases where the development trajectory of the agent seems to imply a potential [Treacherous Turn](https://arbital.com/p/6q).  If there's even one real Treacherous Turn out of all the cases that have been argued, then the point carries that past a certain point, you have to see the bullet coming before it actually hits you.  The theoretical analysis suggests really strongly that blindly forging ahead 'experimentally' will be fatal.  Someone with such a strong commitment to experimentalism that they want to ignore this theoretical analysis... it's not clear what we can say to them, except maybe to appeal to the normative principle of not predictably destroying the world in cases where it seems like we could have done better.

Against:  no real arguments against in the actual literature, but it would be surprising if somebody didn't claim that the foreseeable difficulties program was too pessimistic, or inevitably ungrounded from reality and productive only of bad ideas even when refuted, etcetera.

Primary reply: look, dammit, people actually are way too optimistic about FAI, we have them on the record,  and it's hard to see how humanity could avoid walking directly into the whirling razor blades without better foresight of difficulty.  One potential strategy is enough academic respect and consensus on enough really obvious foreseeable difficulties that the people claiming it will all be easy are actually asked to explain why the foreseeable difficulty consensus is wrong, and if they can't explain that well, they lose respect.

Will interact with the arguments on [empiricism vs. theorism is a false dichotomy](https://arbital.com/p/108).

Eric Bruylant
Alexei Andreev
Eliezer Yudkowsky
Matthew Graves


Epistemic and instrumental efficiency

An agent that is "efficient", relative to you, within a domain, is one that never makes a real error that you can systematically predict in advance.

- Epistemic efficiency (relative to you):  You cannot predict directional biases in the agent's estimates (within a domain).
- Instrumental efficiency (relative to you):  The agent's strategy (within a domain) always achieves at least as much utility or expected utility, under its own preferences, as the best strategy you can think of for obtaining that utility (while staying within the same domain).

If an agent is epistemically and instrumentally efficient relative to all of humanity across all domains, we can just say that it is "efficient" (and almost surely [superintelligent](https://arbital.com/p/41l)).

## Epistemic efficiency

A [superintelligence](https://arbital.com/p/41l) cannot be assumed to know the exact number of hydrogen atoms in a star; but we should not find ourselves believing that we ourselves can predict in advance that a superintelligence will overestimate the number of hydrogen atoms by a factor of 10%.  Any thought process we can use to predict this overestimate should also be accessible to the superintelligence, and it can apply the same corrective factor itself.

The main analogy from present human experience would be the Efficient Markets Hypothesis as applied to short-term asset prices in highly-traded markets.  Anyone who thinks they have a reliable, repeatable ability to predict 10% changes in the price of S&P 500 companies over one-month time periods is mistaken.  If someone has a story to tell about how the economy works that requires advance-predictable 10% changes in the asset prices of highly liquid markets, we infer that the story is wrong.  There can be sharp corrections in stock prices (the markets can be 'wrong'), but not humans who can reliably predict those corrections (over one-month timescales).  If e.g. somebody is consistently making money by selling options using some straightforward-seeming strategy, we suspect that such options will sometimes blow up and lose all the money gained ("picking up pennies in front of a steamroller").

An 'efficient agent' is epistemically strong enough that we apply at least the degree of skepticism to a human proposing to outdo their estimates that, e.g., an experienced proponent of the Efficient Markets Hypothesis would apply to your uncle boasting about how he made a lot of money by predicting how General Motors's stock would rise.

Epistemic efficiency implicitly requires that an advanced agent can always learn a model of the world at least as predictively accurate as used by any human or human institution.  If our hypothesis space were usefully wider than that of an advanced agent, such that the truth sometimes lay in our hypothesis space while being outside the agent's hypothesis space, then we would be able to produce better predictions than the agent.

## Instrumental efficiency

This is the analogue of epistemic advancement for instrumental strategizing:  By definition, humans cannot expect to imagine an improved strategy compared to an efficient agent's selected strategy (relative to the agent's preferences, and given the options the agent has available).

If someone argues that a [cognitively advanced](https://arbital.com/p/2c) [paperclip maximizer](https://arbital.com/p/10h) would do X yielding M expected paperclips, and we can think of an alternative strategy Y that yields N expected paperclips, N > M, then while we cannot be confident that a PaperclipMaximizer will use strategy Y, we strongly predict that:

- (1) a [paperclip maximizer](https://arbital.com/p/10h) will not use strategy X, or
- (2a) if it does use X, strategy Y was unexpectedly flawed, or
- (2b) if it does use X, strategy X will yield unexpectedly high value

...where to avoid [privileging the hypothesis](https://arbital.com/p/) or [fighting a rearguard action](https://arbital.com/p/) we should usually just say, "No, a Paperclip Maximizer wouldn't do X because Y would produce more paperclips."  In saying this, we're implicitly making an appeal to a version of instrumental efficiency; we're supposing the Paperclip Maximizer isn't stupid enough to miss something that seems obvious to a human thinking about the problem for five minutes.

Instrumental efficiency implicitly requires that the agent is always able to conceptualize any useful strategy that humans can conceptualize; it must be able to search at least as wide a space of possible strategies as humans could.

### Instrumentally efficient agents are presently unknown

From the standpoint of present human experience, instrumentally efficient agents are unknown outside of very limited domains.  There are perfect tic-tac-toe players; but even modern chess-playing programs, with ability far in advance of any human player, are not yet so advanced that every move that *looks* to us like a mistake *must therefore* be secretly clever.  We don't dismiss out of hand the notion that a human has thought of a better move than the chess-playing algorithm, the way we dismiss out of hand a supposed secret to the stock market that predicts 10% price changes of S&P 500 companies using public information.

There is no analogue of 'instrumental efficiency' in asset markets, since market prices do not directly select among strategic options.  Nobody has yet formulated a use of the EMH such that we could spend a hundred million dollars to guarantee liquidity, and get a well-traded asset market to directly design a liquid fluoride thorium nuclear plant, such that if anyone said before the start of trading, "Here is a design X that achieves expected value M", we would feel confident that either the asset market's final selected design would achieve at least expected value M or that the original assertion about X's expected value was wrong.

By restricting the meaning even further, we get a valid metaphor in chess: an ordinary person such as you, if you're not an International Grandmaster with hours to think about the game, should regard a modern chess program as instrumentally efficient relative to you.  The chess program will not make any mistake that you can understand as a mistake.  You should expect the reason why the chess program moves anywhere to be only understandable as 'because that move had the greatest probability of winning the game' and not in any other terms like 'it likes to move its pawn'.  If you see the chess program move somewhere unexpected, you conclude that it is about to do exceptionally well or that the move you expected was surprisingly bad.  There's no way for you to find any better path to the chess program's goals by thinking about the board yourself.  An instrumentally efficient agent would have this property for humans in general and the real world in general, not just you and a chess game.

### Corporations are not [superintelligences](https://arbital.com/p/41l)

For any reasonable attempt to define a corporation's utility function (e.g. discounted future cash flows), it is not the case that we can confidently dismiss any assertion by a human that a corporation could achieve 10% more utility under its utility function by doing something differently.  It is common for a corporation's stock price to rise immediately after it fires a CEO or renounces some other mistake that many market actors knew was a mistake but had been going on for years - the market actors are not able to make a profit on correcting that error, so the error persists.

Standard economic theory does not predict that any currently known economic actor will be instrumentally efficient under any particular utility function, including corporations.  If it did, we could maximize any other strategic problem if we could make that actor's utility function conditional on it, e.g., reliably obtain the best humanly imaginable nuclear plant design by paying a corporation for it via a sufficiently well-designed contract.

We have sometimes seen people trying to label corporations as [superintelligences](https://arbital.com/p/41l), with the implication that corporations are the real threat and equally severe, as threats, compared to machine superintelligences.  But epistemic or instrumental decision-making efficiency of individual corporations is just not predicted by standard economic theory.  Most corporations do not even use internal prediction markets, or try to run conditional stock-price markets to select among known courses of action.  Standard economic history includes many accounts of corporations making 'obvious mistakes' and these accounts are not questioned in the way that e.g. a persistent large predictable error in short-run asset prices would be questioned.

Since corporations are not instrumentally efficient (or epistemically efficient), they are not superintelligences.

Eric Bruylant
Jessica Taylor
Eliezer Yudkowsky
Alexei Andreev

An agent that is "efficient", relative to you, within a domain, never makes a real error that you can predict.

For example:  A [superintelligence](https://arbital.com/p/41l) might not be able to count the exact number of atoms in a star.  But you shouldn't be able to say, "I think it will overestimate the number of atoms by 10%, because hydrogen atoms are so light."  It knows that too.  For you to foresee a predictable directional error in a superintelligence's estimates is at least as impossible as you predicting a 10% rise in Microsoft's stock, over the next week, using only public information.

- *Epistemic* efficiency is your inability to predict any directional error in the agent's estimates and probabilities.
- *Instrumental* efficiency is your inability to think of a policy that would achieve more of the agent's utility than whatever policy it actually uses.


Standard agent properties

### Boundedly rational agents

- Have probabilistic models of the world.
- Update those models in response to sensory information.
  - The ideal algorithm for updating is Bayesian inference, but this requires too much computing power and a bounded agent must use some bounded alternative.
  - Implicitly, we assume the agent has some equivalent of a complexity-penalizing prior or Occam's Razor.  Without this, specifying Bayesian inference does not much constrain the end results of epistemic reasoning.
- Have preferences over events or states of the world, quantifiable by a utility function that maps those events or states onto a scalar field.
  - These preferences must be quantitative, not just ordered, in order to combine with epistemic states of uncertainty (probabilities).
- Are consequentialist: they evaluate the expected consequences of actions and choose among actions based on preference among their expected consequences.
  - Bounded agents cannot evaluate all possible actions and hence cannot obtain literal maximums of expected utility except in very simple cases.
- Act in real time in a noisy, uncertain environment.

For the arguments that sufficiently intelligent agents will appear to us as boundedly rational agents in some sense, see:

- [Relevant powerful agents will be highly optimized](https://arbital.com/p/29)
- [Sufficiently optimized agents appear coherent](https://arbital.com/p/21)

### Economic agents

- Achieve their goals by efficiently allocating limited resources, including, e.g., time, money, or negentropy;
- Try to find new paths that route around obstacles to goal achievement;
- Predict the actions of other agents;
- Try to coordinate with, manipulate, or hinder other agents (in accordance with the agent's own goals or utilities);
- Respond to both negative incentives (penalties) and positive incentives (rewards) by planning accordingly, and may also consider strategies to avoid penalties or gain rewards that were unforeseen by the creators of the incentive framework.

### Naturalistic agents

- Naturalistic agents are embedded in a larger universe and are made of the same material as other things in the universe (wavefunction, on our current beliefs about physics).
- A naturalistic agent's uncertainty about the environment is uncertainty about which natural universe embeds them (what material structure underlies their available sensory and introspective data).
- Some of the actions available to naturalistic agents potentially alter their sensors, actuators, or computing substrate.
- Sufficiently powerful naturalistic agents may construct other agents out of resources available to them internally or in their environment, or extend their intelligence into outside computing resources.
- A naturalistic agent's sensing, cognitive, and decision/action capabilities may be distributed over space, time, and multiple substrates; the applicability of the 'agent' concept does not require a small local robot body.

Nate Soares
Eliezer Yudkowsky
Alexei Andreev


Mindcrime

summary:  'Mindcrime' is [https://arbital.com/p/18k](https://arbital.com/p/18k)'s suggested term for the moral catastrophe that occurs if a [machine intelligence](https://arbital.com/p/2c) contains enormous numbers of [conscious beings trapped inside its code](https://arbital.com/p/18j).

This could happen as a result of self-awareness being a natural property of computationally efficient subprocesses.  Perhaps more worryingly, the best model of a person may be a person itself, even if they're not the same person.  This means that AIs trying to model humans might be unusually likely to create hypotheses and simulations that are themselves conscious.

summary(Technical):  'Mindcrime' is [https://arbital.com/p/18k](https://arbital.com/p/18k)'s term for mind designs producing moral harm by their internal operation, particularly through embedding sentient subprocesses.

One worry is that mindcrime might arise in the course of an agent trying to predict or manipulate the humans in its environment, since this implies a pressure to model the humans in faithful detail.  This is especially concerning since several value alignment proposals would explicitly call for modeling humans in detail, e.g. [extrapolated volition](https://arbital.com/p/3c5) and [imitation-based agents](https://arbital.com/p/44z).

Another problem scenario is if the natural design for an efficient subprocess involves independent consciousness (though it is a separate question if this optimal design involves pain or suffering).

Computationally powerful agents might contain vast numbers of trapped conscious subprocesses, qualifying this as a [global catastrophic risk](https://arbital.com/p/).

"Mindcrime" is [https://arbital.com/p/18k](https://arbital.com/p/18k)'s suggested term for scenarios in which an AI's cognitive processes are intrinsically doing moral harm, for example because the AI contains trillions of suffering conscious beings inside it.

Ways in which this might happen:

- Problem of sapient models (of humans):  Occurs naturally if the best predictive model for humans in the environment involves models that are detailed enough to be people themselves.
- Problem of sapient models (of civilizations):  Occurs naturally if the agent tries to simulate, e.g., alien civilizations that might be simulating it, in enough detail to include conscious simulations of the aliens.
- Problem of sapient subsystems:  Occurs naturally if the most efficient design for some cognitive subsystems involves creating subagents that are self-reflective, or have some other property leading to consciousness or personhood.
- Problem of sapient self-models:  If the AI is conscious or possible future versions of the AI are conscious, it might run and terminate a large number of conscious-self models in the course of considering possible self-modifications.

# Problem of sapient models (of humans):

An [instrumental pressure](https://arbital.com/p/10k) to produce high-fidelity predictions of human beings (or to predict [decision counterfactuals](https://arbital.com/p/) about them, or to [search](https://arbital.com/p/) for events that lead to particular consequences, etcetera) may lead the AI to run computations that are unusually likely to possess personhood.

An [unrealistic](https://arbital.com/p/107) example of this would be [https://arbital.com/p/11w](https://arbital.com/p/11w), where predictions are made by means that include running many possible simulations of the environment and seeing which ones best correspond to reality.  Among current machine learning algorithms, particle filters and Monte Carlo algorithms similarly involve running many possible simulated versions of a system.

It's possible that a sufficiently advanced AI to have successfully arrived at detailed models of human intelligence, would usually also be advanced enough that it never tried to use a predictable/searchable model that engaged in brute-force simulations of those models.  (Consider, e.g., that there will usually be many possible settings of a variable inside a model, and an efficient model might manipulate data representing a probability distribution over those settings, rather than ever considering one exact, specific human in toto.)

This, however, doesn't make it certain that no mindcrime will occur.  It may not take exact, faithful simulation of specific humans to create a conscious model.  An efficient model of a (spread of possibilities for a) human may still contain *enough* computations that resemble a person *enough* to create consciousness, or whatever other properties may be deserving of personhood.  Consider, in particular, an agent trying to use

Just as it almost certainly isn't necessary to go all the way down to the neural level to create a sapient being, it may be that even with some parts of a mind considered abstractly, the remainder would be computed in enough detail to imply consciousness, sapience, personhood, etcetera.

The problem of sapient models is not to be confused with [Simulation Hypothesis](https://arbital.com/p/) issues.  An efficient model of a human need not have subjective experience indistinguishable from that of the human (although it will be a model *of* a person who doesn't believe themselves to be a model).  The problem occurs if the model *is a person*, not if the model is *the same person* as its subject, and the latter possibility plays no role in the implication of moral harm.

Besides problems that are directly or obviously about modeling people, many other practical problems and questions can benefit from modeling other minds - e.g., reading the directions on a toaster oven in order to discern the intent of the mind that was trying to communicate how to use a toaster.  Thus, mindcrime might result from a sufficiently powerful AI trying to solve very mundane problems.

# Problem of sapient models (of civilizations)

A separate route to mindcrime comes from an advanced agent considering, in sufficient detail, the possible origins and futures of intelligent life on other worlds.  (Imagine that you were suddenly told that this version of you was actually embedded in a superintelligence that was imagining how life might evolve on a place like Earth, and that your subprocess was not producing sufficiently valuable information and was about to be shut down.  You would probably be annoyed!  We should try not to annoy other people in this way.)

Three possible origins of a [convergent instrumental pressure](https://arbital.com/p/10g) to consider intelligent civilizations in great detail:

- Assigning sufficient probability to the existence of non-obvious extraterrestrial intelligences in Earth's vicinity, perhaps due to considering the [Fermi Paradox](https://arbital.com/p/).
- [Naturalistic induction](https://arbital.com/p/), combined with the AI considering the hypothesis that it is in a simulated environment.
- [Logical decision theories](https://arbital.com/p/) and utility functions that care about the consequences of the AI's decisions via instances of the AI's reference class that could be embedded inside alien simulations.

With respect to the latter two possibilities, note that the AI does not need to be considering possibilities in which the whole Earth as we know it is a simulation.  The AI only needs to consider that, among the possible explanations of the AI's current sense data and internal data, there are scenarios in which the AI is embedded in some world other than the most 'obvious' one implied by the sense data.  See also [https://arbital.com/p/5j](https://arbital.com/p/5j) for a related hazard of the AI considering possibilities in which it is being simulated.

([https://arbital.com/p/2](https://arbital.com/p/2) has advocated that we shouldn't let any AI short of *extreme* levels of safety and robustness assurance consider distant civilizations in lots of detail in any case, since this means our AI might embed (a model of) a hostile superintelligence.)

# Problem of sapient subsystems:

It's possible that the most efficient system for, say, allocating memory on a local cluster, constitutes a complete reflective agent with a self-model.  Or that some of the most efficient designs for subprocesses of an AI, in general, happen to have whatever properties lead up to consciousness or whatever other properties are important to personhood.

This might possibly constitute a relatively less severe moral catastrophe, if the subsystems are sentient but [lack a reinforcement-based pleasure/pain architecture](https://arbital.com/p/) (since the latter is not obviously a property of the most efficient subagents).  In this case, there might be large numbers of conscious beings embedded inside the AI and occasionally dying as they are replaced, but they would not be suffering.  It is nonetheless the sort of scenario that many of us would prefer to avoid.

# Problem of sapient self-models:

The AI's models of *itself*, or of other AIs it could possibly build, might happen to be conscious or have other properties deserving of personhood.  This is worth considering as a separate possibility from building a conscious or personhood-deserving AI ourselves, when [we didn't mean to do so](https://arbital.com/p/), because of these two additional properties:

- Even if the AI's current design is not conscious or personhood-deserving, the current AI might consider possible future versions or subagent designs that would be conscious, and those considerations might themselves be conscious.
 - This means that even if the AI's current version doesn't seem like it has key personhood properties on its own - that we've successfully created the AI itself as a nonperson - we still need to worry about other conscious AIs being embedded into it.
- The AI might create, run, and terminate very large numbers of potential self-models.
 - Even if we consider tolerable the potential moral harm of creating *one* conscious AI (e.g. the AI lacks all of the conditions that a responsible parent would want to ensure when creating a new intelligent species, but it's just one sapient being so it's okay to do that in order to save the world), we might not want to take on the moral harm of creating *trillions* of evanescent, swiftly erased conscious beings.

# Difficulties

Trying to consider these issues is complicated by:

- [Philosophical uncertainty](https://arbital.com/p/) about what properties are constitutive of consciousness and which computer programs have them;
- [Moral uncertainty](https://arbital.com/p/) about what ([idealized](https://arbital.com/p/) versions of) (any particular person's) morality would consider to be the key properties of personhood;
- Our present-day uncertainty about what efficient models in advanced agents would look like.

It'd help if we knew the answers to these questions, but the fact that we don't know doesn't mean we can thereby conclude that any particular model is not a person.  (This would be some mix of [argumentum ad ignorantiem](https://arbital.com/p/), and [availability bias](https://arbital.com/p/) making us think that a scenario is unlikely when it is hard to visualize.)  In the limit of infinite computing power, the epistemically best models of humans would almost certainly involve simulating many possible versions of them; superintelligences would have [very large amounts of computing power](https://arbital.com/p/) and we don't know at what point we come close enough to this [limiting property](https://arbital.com/p/) to cross the threshold.

## Scope of potential disaster

The prospect of mindcrime is an especially alarming possibility because sufficiently advanced agents, *especially* if they are using computationally efficient models, might consider *very large numbers* of hypothetical possibilities that would count as people.  There's no limit that says that if there are seven billion people, an agent will run at most seven billion models; the agent might be considering many possibilities per individual human.  This would not be an [astronomical disaster](https://arbital.com/p/) since it would not (by hypothesis) wipe out our posterity and our intergalactic future, but it could be a disaster orders of magnitude larger than the Holocaust, the Mongol Conquest, the Middle Ages, or all human tragedy to date.

## Development-order issue

If we ask an AI to predict what we would say if we had a thousand years to think about the problem of defining personhood or think about which causal processes are 'conscious', this seems unusually likely to cause the AI to commit mindcrime in the course of answering the question.  Even asking the AI to think abstractly about the problem of consciousness, or predict by abstract reasoning what humans might say about it, seems unusually likely to result in mindcrime.  There thus exists a [development order issue](https://arbital.com/p/) preventing us from asking a Friendly AI to solve the problem for us, since to file this request safely and without committing mindcrime, we would need the request to already have been completed.

The prospect of enormous-scale disaster mitigates against 'temporarily' tolerating mindcrime inside a system, while, e.g., an [extrapolated-volition](https://arbital.com/p/) or [approval-based](https://arbital.com/p/) agent tries to compute the code or design of a non-mindcriminal agent.  Depending on the agent's efficiency, and secondarily on its computational limits, a tremendous amount of moral harm might be done during the 'temporary' process of computing an answer.

## Weirdness

Literally nobody outside of MIRI or FHI ever talks about this problem.

# Nonperson predicates

A [nonperson predicate](https://arbital.com/p/1fv) is an [effective](https://arbital.com/p/) test that we, or an AI, can use to determine that some computer program is definitely *not* a person.  In principle, a nonperson predicate needs only two possible outputs, "Don't know" and "Definitely not a person".  It's acceptable for many actually-nonperson programs to be labeled "don't know", so long as no people are labeled "definitely not a person".

If the above was the only requirement, one simple nonperson predicate would be to label everything "don't know".  The implicit difficulty is that the nonperson predicate must also pass some programs of high complexity that do things like "acceptably model humans" or "acceptably model future versions of the AI".

Besides addressing mindcrime scenarios, Yudkowsky's [original proposal](http://lesswrong.com/lw/x4/nonperson_predicates/) was also aimed at knowing that the AI design itself was not conscious, or not a person.

It seems likely to be very hard to find a good nonperson predicate:

- Not all philosophical confusions and computational difficulties are averted by asking for a partial list of unconscious programs instead of a total list of conscious programs.  Even if we don't know which properties are sufficient, we'd need to know something solid about properties that are necessary for consciousness or sufficient for nonpersonhood.
- We can't pass once-and-for-all any class of programs that's Turing-complete. We can't say once and for all that it's safe to model gravitational interactions in a solar system, if enormous gravitational systems could encode computers that encode people.
- The [https://arbital.com/p/42](https://arbital.com/p/42) problem seems particularly worrisome here.  If we block off some options for modeling humans directly, the *next best* option is unusually likely to be conscious.  Even if we rely on a whitelist rather than a blacklist, this may lead to a whitelisted "gravitational model" that secretly encodes a human, and so on.

# Research avenues

+ [Behaviorism](https://arbital.com/p/102):  Try to create a [limited AI](https://arbital.com/p/5b3) that does not model other minds or possibly even itself, except using some narrow class of agent models that we are pretty sure will not be sentient.  This avenue is potentially motivated for other reasons as well, such as avoiding [probable environment hacking](https://arbital.com/p/5j) and averting [programmer manipulation](https://arbital.com/p/).

+ Try to define a nonperson predicate that whitelists enough programs to carry out some [pivotal achievement](https://arbital.com/p/6y).

+ Try for an AI that can bootstrap our understanding of consciousness and tell us about what we would define as a person, while committing a relatively small amount of mindcrime, with all computed possible-people being stored rather than discarded, and the modeled agents being entirely happy, mostly happy, or non-suffering.  E.g., put a happy person at the center of the approval-directed agent, and try to oversee the AI's algorithms and ask it not to use Monte Carlo simulations if possible.

+ Ignore the problem in all pre-interstellar stages because it's still relatively small compared to astronomical stakes and therefore not worth significant losses in success probability.  (This may [backfire](https://arbital.com/p/) under some versions of the Simulation Hypothesis.)

+ Try to [finish](https://arbital.com/p/112) the philosophical problem of understanding which causal processes experience sapience (or are otherwise objects of ethical value), in the next couple of decades, to sufficient detail that it can be crisply stated to an AI, with sufficiently complete coverage that it's not subject to the [https://arbital.com/p/42](https://arbital.com/p/42) problem.

Alexei Andreev
Jeremy Perret
Nate Soares
Eric Bruylant
Eliezer Yudkowsky

A huge amount of harm could occur if a [machine intelligence](https://arbital.com/p/2c) turns out to contain lots of [conscious subprograms](https://arbital.com/p/18j) enduring poor living conditions.  One worry is that this might happen if an AI models humans in too much detail.


Task-directed AGI

A task-based AGI is an AGI [intended](https://arbital.com/p/6h) to follow a series of human-originated orders, with these orders each being of limited scope - "satisficing" in the sense that they can be [accomplished using bounded amounts of effort and resources](https://arbital.com/p/4mn) (as opposed to the goals being more and more fulfillable using more and more effort).

In [Bostrom's typology](https://arbital.com/p/1g0), this is termed a "Genie".  It contrasts with a "Sovereign" AGI that acts autonomously in the pursuit of long-term real-world goals.

Building a safe Task AGI might be easier than building a safe Sovereign for the following reasons:

- A Task AGI can be "online"; the AGI can potentially [query the user](https://arbital.com/p/2qq) before and during Task performance.  (Assuming an ambiguous situation arises, and is successfully identified as ambiguous.)
- A Task AGI can potentially be [limited](https://arbital.com/p/5b3) in various ways, since a Task AGI doesn't need to be *as powerful as possible* in order to accomplish its limited-scope Tasks.  A Sovereign would presumably engage in all-out self-improvement.  (This isn't to say Task AGIs would automatically not self-improve, only that it's possible *in principle* to limit the power of a Task AGI to only the level required to do the targeted Tasks, *if* the associated safety problems can be solved.)
- Tasks, by assumption, are limited in scope - they can be accomplished and done, inside some limited region of space and time, using some limited amount of effort which is then complete.  (To gain this advantage, a state of Task accomplishment should not go higher and higher in preference as more and more effort is expended on it open-endedly.)
- Assuming that users can figure out [intended goals](https://arbital.com/p/6h) for the AGI that are [valuable](https://arbital.com/p/55) and [pivotal](https://arbital.com/p/6y), the [identification problem](https://arbital.com/p/2rz) for describing what constitutes a safe performance of that Task, might be simpler than giving the AGI a [complete description](https://arbital.com/p/) of [normativity in general](https://arbital.com/p/55).  That is, the problem of communicating to an AGI an adequate description of "cure cancer" (without killing patients or causing other side effects), while still difficult, might be simpler than an adequate description of all normative value.  Task AGIs fall on the narrow side of [https://arbital.com/p/1vt](https://arbital.com/p/1vt).

Relative to the problem of building a Sovereign, trying to build a Task AGI instead might step down the problem from "impossibly difficult" to "insanely difficult", while still maintaining enough power in the AI to perform [pivotal acts](https://arbital.com/p/6y).

The obvious disadvantage of a Task AGI is [moral hazard](https://arbital.com/p/2sb) - it may tempt the users in ways that a Sovereign would not.  A Sovereign has moral hazard chiefly during the development phase, when the programmers and users are perhaps not yet in a position of special relative power.  A Task AGI has ongoing moral hazard as it is used.

[https://arbital.com/p/2](https://arbital.com/p/2) has suggested that people only confront many important problems in value alignment when they are thinking about Sovereigns, but that at the same time, Sovereigns may be impossibly hard in practice.  Yudkowsky advocates that people think about Sovereigns first and list out all the associated issues before stepping down their thinking to Task AGIs, because thinking about Task AGIs may result in premature pruning, while thinking about Sovereigns is more likely to generate a complete list of problems that can then be checked against particular Task AGI approaches to see if those problems have become any easier.

Three distinguished subtypes of Task AGI are these:

- **[Oracles](https://arbital.com/p/6x)**, an AI that is intended to only answer questions, possibly from some restricted question set.
- **[Known-algorithm AIs](https://arbital.com/p/1fy)**, which are not self-modifying or very weakly self-modifying, such that their algorithms and representations are mostly known and mostly stable.
- **[Behaviorist Genies](https://arbital.com/p/102)**, which are meant to not model human minds or model them in only very limited ways, while having great material understanding (e.g., potentially the ability to invent and deploy nanotechnology).

# Subproblems

The problem of making a safe genie invokes numerous subtopics such as [low impact](https://arbital.com/p/2pf), [mild optimization](https://arbital.com/p/2r8), and [conservatism](https://arbital.com/p/2qp) as well as numerous standard AGI safety problems like [reflective stability](https://arbital.com/p/1fx) and safe [identification](https://arbital.com/p/6c) of [intended goals](https://arbital.com/p/6h).

([See here for a separate page on open problems in Task AGI safety that might be ready for current research.](https://arbital.com/p/2mx))

Some further problems beyond those appearing in the page above are:

- **Oracle utility functions** (that make the Oracle not wish to leave its box or optimize its programmers)
- **Effable optimization** (the opposite of [cognitive uncontainability](https://arbital.com/p/9f))
- **Online checkability**
 - Explaining things to programmers [without putting the programmers inside an argmax](https://arbital.com/p/30b) for how well you are 'explaining' things to them
- **Transparency**
- [Do What I Mean](https://arbital.com/p/2s1)

Malcolm Ocean
Nate Soares
Eliezer Yudkowsky
Alexei Andreev

A task-based AGI or "genie" is an AGI [intended](https://arbital.com/p/6h) to follow a series of human orders, rather than autonomously pursuing long-term goals.  A Task AGI might be easier to render safe, since:

- It's possible to [query the user](https://arbital.com/p/2qq) before and during a Task.
- [Tasks are satisficing](https://arbital.com/p/4mn) - they're of limited scope and can be fully accomplished using a limited effort.  (In other words, Tasks should not become more and more accomplished as more and more effort is put into them.)
- Adequately [identifying](https://arbital.com/p/2rz) what it means to safely "cure cancer" might be simpler than adequately identifying [all normative value](https://arbital.com/p/55).
- Task AGIs can be limited in various ways, rather than self-improving as far as possible, so long as they can still carry out at least some [pivotal](https://arbital.com/p/6y) Tasks.

The obvious disadvantage of a Task AGI is [moral hazard](https://arbital.com/p/2sb) - it may tempt the users in ways that an autonomous AI would not.

The problem of making a safe Task AGI invokes numerous subtopics such as [low impact](https://arbital.com/p/2pf), [mild optimization](https://arbital.com/p/2r8), and [conservatism](https://arbital.com/p/2qp) as well as numerous standard AGI safety problems like [goal identification](https://arbital.com/p/) and [reflective stability](https://arbital.com/p/1fx).


Oracle

Oracles are a subtype of [Genies](https://arbital.com/p/6w) putatively designed to safely answer questions, and *only* to answer questions.  Oracles are often assumed to be [Boxed](https://arbital.com/p/6z) and the study of Oracles is sometimes taken to be synonymous with the study of Boxed Oracles.

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev