Mild optimization

"Mild optimization" is where, if you ask a Task AGI to paint one car pink, it just paints one car pink and then stops, rather than tiling the galaxies with pink-painted cars, because it's *not optimizing that hard.*  It's okay with just painting one car pink; it isn't driven to max out the twentieth decimal place of its car-painting score.

Other [suggested terms](https://www.facebook.com/yudkowsky/posts/10154053063684228) for this concept have included "soft optimization", "sufficient optimization", "minimum viable solution", "pretty good optimization", "moderate optimization", "regularized optimization", "sensible optimization", "casual optimization", "adequate optimization", "good-not-great optimization", "lenient optimization", "parsimonious optimization", and "optimehzation".

# Difference from low impact

Mild optimization is complementary to [taskiness](https://arbital.com/p/task_goal) and [low impact](https://arbital.com/p/2pf).  A low impact AGI might try to paint one car pink while minimizing its other footprint or how many other things changed, but it would be trying *as hard as possible* to minimize that impact and drive it down *as close to zero* as possible, which might come with its own set of pathologies.

What we really want is both properties.  We want the AGI to paint one car pink in a way that gets the impact pretty low and then, you know, that's good enough - not have a cognitive pressure to search through weird extremes looking for a way to decrease the twentieth decimal place of the impact.  This would tend to break a low impact measure which contained even a subtle flaw, where a mild-optimizing AGI might not put as much pressure on the low impact measure and hence be less likely to break it.

(Obviously, what we *want* is a perfect low impact measure which will keep us safe [even if subjected to unlimited optimization power](https://arbital.com/p/2x), but a basic security mindset is to try to make each part safe on its own, then assume it might contain a flaw and try to design the rest of the system to be safe anyway.)

# Difference from satisficing

[Satisficing utility functions](https://en.wikipedia.org/wiki/Satisficing#As_a_form_of_optimization) don't necessarily mandate or even allow mildness.

Suppose the AI's utility function is 1 when at least one car has been painted pink and 0 otherwise - there's no more utility to be gained by outcomes in which more cars have been painted pink.  Will this AI still go to crazy-seeming lengths?

Yes, because in a partially uncertain / probabilistic environment, there's still no upper bound on the utility which can be gained.  A solution with 0.9999 probability of painting at least one car pink is ranked above a solution with a 0.999 probability of painting at least one car pink.

If a preference ordering $<_p$ has the property that for every probability distribution on expected outcomes $O$ there's another expected outcome $O'$ with $O <_p O'$ which requires one more erg of energy to achieve, this is a sufficient condition for using up all the energy in the universe.  If converting all reachable matter into pink-painted cars implies a slightly higher probability, that at least one car is pink, that's the maximum of *expected* utility under the 0-1 utility function.

Less naive satisficing would describe an optimizer which satisfies an *expected* utility constraint - say, if any policy produces at least 0.95 *expected* utility under the 0-1 utility function, the AI can implement that policy.

This rule is now a [Task](https://arbital.com/p/task_goal) and would at least *permit* mild optimization.  The problem is that it doesn't *exclude* extremely optimized solutions.  A 0.99999999 probability of producing at least one pink-painted car also has the property that it's above a 0.95 probability.  If you're a self-modifying satisficer, replacing yourself with a maximizer is probably a satisficing solution.

Even if we're not dealing with a completely self-modifying agent, there's a *ubiquity* of points where adding more optimization pressure might satisfice.  When you build a thermostat in the environment, you're coercing one part of the environment to have a particular temperature; if this kind of thing doesn't count as "more optimization pressure" then we could be dealing with all sorts of additional optimizing-ness that falls short of constructing a full subagent or doing a full self-modification.  There's all sorts of steps in cognition where it would be just as easy to add a maximizing step (take the highest-ranking solution) as to take a random high-ranking solution.

On a higher level of abstraction, the problem is that while satisficing is reflectively *consistent*, it's not [reflectively stable](https://arbital.com/p/1fx).  A satisficing agent is happy to construct another satisficing agent, but it may also be happy to construct a maximizing agent.  It can approve its current mode of thinking, but it approves other modes of thinking too.  So unless *all* the cognitive steps are being carried out locally on [fixed known algorithms](https://arbital.com/p/1fy) that satisfice but definitely don't maximize, without the AGI constructing any environmental computations or conditional policy steps more complicated than a pocket calculator, building a seemingly mild satisficer doesn't guarantee that optimization *stays* mild.

# Quantilizing

One weird idea that seems like it might exhibit incremental progress toward reflectively stable mild optimization is [https://arbital.com/p/4y](https://arbital.com/p/4y)'s [expected utility quantilizer](https://intelligence.org/files/QuantilizersSaferAlternative.pdf).  Roughly, a quantilizer estimates expected outcomes relative to a null action, and then tries to produce an expected outcome in some *upper quantile* of possibilities - e.g., an outcome in the top 1% of expected outcomes.  Furthermore, a quantilizer *only* tries to narrow outcomes by that much - it doesn't try to produce one particular outcome in the top 1%; the most it will ever try to do is randomly pick an outcome such that this random distribution corresponds to being in the top 1% of expected outcomes.

Quantilizing corresponds to maximizing expected utility under the assumption that there is uncertainty about which outcomes are good and an adversarial process which can make some outcomes arbitrarily bad, subject to the constraint that the expected utility of the null action can only be boundedly low.  So if there's an outcome which would be very improbable given the status quo, the adversary can make that outcome be very bad.  This means that rather than aiming for one single high-utility outcome which the adversary could then make very bad, a quantilizer tries for a range of possible good outcomes.  This in turn means that quantilizers will actively avoid narrowing down the future too much, even if by doing so they'd enter regions of very high utility.

Quantilization doesn't seem like *exactly* what we actually want for multiple reasons.  E.g., if long-run good outcomes are very improbable given status quo, it seems like a quantilizer would try to have its policies fall short of that in the long run (a similar problem seems like it might appear in [impact measures](https://arbital.com/p/2pf) which imply that good long-run outcomes have high impact).

The key important idea that appears in quantilizing is that a quantilizer isn't just as happy to rewrite itself as a maximizer, and isn't just as happy to implement a policy that involves constructing a more powerful optimizer in the environment.

# Relation to other problems

Mild optimization relates directly to one of the three core reasons why aligning at-least-partially superhuman AGI is hard - making very powerful optimization pressures flow through the system puts a lot of stress on its potential weaknesses and flaws.  To the extent we can get mild optimization stable, it might take some of the critical-failure pressure off other parts of the system.  (Though again, basic security mindset says to still try to get all the parts of the system as flawless as possible and not tolerate any known flaws in them, *then* build the fallback options in case they're flawed anyway; one should not deliberately rely on the fallbacks and intend them to be activated.)

Mild optimization seems strongly complementary to [low impact](https://arbital.com/p/2pf) and [taskiness](https://arbital.com/p/task_goal).  Something that's merely low-impact might exhibit pathological behavior from trying to drive side impacts down to absolutely zero.  Something that merely optimizes mildly might find some 'weak' or 'not actually trying that hard' solution which nonetheless ended up turning the galaxies into pink-painted cars.  Something that has a satisfiable utility function with a readily-achievable maximum achievable utility might still go to tremendous lengths to drive the probability of achieving maximum utility to nearly 1.  Something that optimizes mildly *and* has a low impact penalty *and* has a small, clearly achievable goal, seems much more like the sort of agent that might, you know, just paint the damn car pink and then stop.

Mild optimization can be seen as a further desideratum of the currently open [Other-izer Problem](https://arbital.com/p/2r9):  Besides being workable for [bounded agents](https://arbital.com/p/2rd), and being [reflectively stable](https://arbital.com/p/1fx), we'd *also* like an other-izer idiom to have a (stable) mildness parameter.

# Approaches

It currently seems like the key subproblem in mild optimization revolves around [reflective stability](https://arbital.com/p/1fx) - we don't want "replace the mild optimization part with a simple maximizer, becoming a maximizer isn't that hard and gets the task done" to count as a 'mild' solution.  Even in human intuitive terms of "optimizing without putting in an unreasonable amount of effort", at some point a sufficiently advanced human intelligence gets lazy and starts building an AGI to do things for them because it's easier that way and only takes a bounded amount of effort.  We don't want "construct a second AGI that does hard optimization" to count as mild optimization even if it ends up not taking all that much effort for the first AGI, although "construct an AGI that does $\theta$-mild optimization" could potentially count as a $\theta$-mildsolution.

Similarly, we don't want to allow the deliberate creation of environmental or internal [daemons](2rc) even if it's easy to do it that way or requires low effort to end up with that side effect - we'd want the optimizing power of such daemons to count against the measured optimization power and be rejected as optimizing too hard.

Since both of these phenomena seem hard to exhibit in current machine learning algorithms or faithfully represent in a toy problem, [unbounded analysis](https://arbital.com/p/107) seems likely to be the main way to go.  In general, it seems closely related to the [Other-izer Problem](https://arbital.com/p/2r9) which also seems most amenable to unbounded analysis at the present time.

Eric Bruylant
Patrick LaVictoire
Eliezer Yudkowsky

"Mild optimization" or "soft optimization" is when, if you ask the [genie](https://arbital.com/p/6w) to paint one car pink, it just paints one car pink and then stops, rather than tiling the galaxies with pink-painted cars, because it's *not optimizing that hard.*

This is related, but distinct from, notions like "[low impact](https://arbital.com/p/2pf)".  E.g., a low impact AGI might try to paint one car pink while minimizing its other footprint or how many other things changed, but it would be trying *as hard as possible* to minimize that impact and drive it down *as close to zero* as possible, which might come with its own set of pathologies.  What we want instead is for the AGI to try to paint one car pink while minimizing its footprint, and then, when that's being done pretty well, say "Okay done" and stop.

This is distinct from [satisficing expected utility](https://arbital.com/p/eu_satisficer) because, e.g., rewriting yourself as an expected utility maximizer might also satisfice expected utility - there's no upper limit on how hard a satisficer approves of optimizing, so a satisficer is not [reflectively stable](https://arbital.com/p/1fx).


Other-izing (wanted: new optimization idiom)

The open "other-izer" problem is to find something besides maximizing, satisificing, meliorizing, and several other existing but unsatisfactory idioms, which is actually suitable as an optimization idiom for [bounded agents](https://arbital.com/p/bounded_agent) and is [reflectively stable](https://arbital.com/p/1fx).

In standard theory we tend to assume that agents are expected utility *maximizers* that always choose the available option with highest expected utility.  But this isn't a realistic idiom because a realistic, [bounded agent](https://arbital.com/p/bounded_agent) with limited computing power can't compute the expected utility of every possible action.

An expected utility satisficer, which e.g. might approve any policy so long as the expected utility is at least 0.95, would be much more realistic.  But it also doesn't seem suitable for an actual AGI, since, e.g., if policy X produces at least expected utility 0.98, then it would also satisfice to randomize between mostly policy X and a small chance of policy Y that had expected utility 0; this seems to give away a needlessly large amount of utility.  We'd probably be fairly disturbed if an otherwise aligned AGI was actually doing that.

Satisficing is also [reflectively consistent](2rb) but not [reflectively stable](https://arbital.com/p/1fx) - while [tiling agents theory](https://arbital.com/p/1mq) can give formulations of satisficers that will approve the construction of similar satisficers, a satisficer could also tile to a maximizer.  If your decision criterion is to approve policies which achieve expected utility at least $\theta,$ and you expect that an expected utility *maximizing* version of yourself would achieve expected utility at least $\theta,$ then you'll approve self-modifying to be an expected utility maximizer.  This is another reason to prefer a formulation of optimization besides satisficing - if the AI is strongly self-modifying, then there's no guarantee that the 'satisficing' property would stick around and have our analysis go on being applicable, and even if not strongly self-modifying, it might still create non-satisficing chunks of cognitive mechanism inside itself or in the environment.

A meliorizer has a current policy and only replaces it with policies of increased expected utility.  Again, while it's possible to demonstrate that a meliorizer can approve self-modifying to another meliorizer and hence this idiom is reflectively consistent, it doesn't seem like it would be reflectively stable - becoming a maximizer or something else might have higher expected utility than staying a meliorizer.

The "other-izer" open problem is to find something better than maximization, satisficing, and meliorization that actually makes sense as an idiom of optimization for a resource-bounded agent and that we'd think would be an okay thing for e.g. a [Task AGI](https://arbital.com/p/6w) to do, which is at least reflectively consistent, and preferably reflectively stable.

See also "[https://arbital.com/p/2r8](https://arbital.com/p/2r8)" for a further desideratum, namely an adjustable parameter of optimization strength, that would be nice to have in an other-izer.


Reflective consistency

A decision system is "reflectively consistent" if it can approve the construction of similar decision systems.  For example, if you have an expected utility satisficer (it either takes the null action, or an action with expected utility greater than $\theta$) then this agent can self-modify to any other design which also either takes no action, or approves a plan with expected utility greater than $\theta.$  A satisficer might also approve changing itself into an expected utility maximizer (if it expects that this self-modification itself leads to expected utility at least $\theta$) but it will at least approve replacing itself with another satisficer.   On the other hand, a [causal decision theorist](https://arbital.com/p/causal_decision_theory) given a chance to self-modify will only approve the construction of [something that is not a causal decision theorist](https://arbital.com/p/son_of_cdt).  A property satisfies the stronger condition of [reflective stability](https://arbital.com/p/1fx) when decision systems with that property *only* approve their own replacement with other decision systems with that property.  For example, a [https://arbital.com/p/-10h](https://arbital.com/p/-10h) will under ordinary circumstances only approve code changes that preserve the property of maximizing paperclips, so "wanting to make paperclips" is reflectively *stable* and not just reflectively *consistent*.


Optimization daemons

If you subject a dynamic system to a *large* amount of optimization pressure, it can turn into an optimizer or even an intelligence.  The classic example would be how natural selection, in the course of extensively optimizing DNA to construct organisms that replicated the DNA, in one case pushed hard enough that the DNA came to specify a cognitive system capable of doing its own consequentialist optimization.  Initially, these cognitive optimizers pursued goals that correlated well with natural selection's optimization target of reproductive fitness, which is how these crystallized optimizers had originally come to be selected into existence.  However, further optimization of these 'brain' protein chunks caused them to begin to create and share cognitive content among themselves, after which such rapid capability gain occurred that a [context change](https://arbital.com/p/6q) took place and the brains' pursuit of their internal goals no longer correlated reliably with DNA replication.

As much as this was, from a *human* standpoint, a wonderful thing to have happened, it wasn't such a great thing from the standpoint of inclusive genetic fitness of DNA or just having stable, reliable, well-understood optimization going on.  In the case of AGIs deploying powerful internal and external optimization pressures, we'd very much like to not have that optimization deliberately or accidentally crystallize into new modes of optimization, especially if this breaks goal alignment with the previous system or breaks other safety properties.  (You might need to stare at the [Orthogonality Thesis](https://arbital.com/p/1y) until it becomes intuitive that, even though crystallizing daemons from natural selection produced creatures that were more humane than natural selection, this doesn't mean that crystallization from an AGI's optimization would have a significant probability of producing something humane.)

When heavy optimization pressure on a system crystallizes it into an optimizer - especially one that's powerful, or more powerful than the previous system, or misaligned with the previous system - we could term the crystallized optimizer a "daemon" of the previous system.  Thus, under this terminology, humans would be daemons of natural selection.  If an AGI, after heavily optimizing some internal system, was suddenly taken over by an erupting daemon that cognitively wanted to maximize something that had previously correlated with the amount of available RAM, we would say this was a crystallized daemon of whatever kind of optimization that AGI was applying to its internal system.

This presents an AGI safety challenge.  In particular, we'd want at least one of the following things to be true *anywhere* that *any kind* of optimization pressure was being applied:

- The optimization pressure is (knowably and reliably) too weak to create daemons.  (Seemingly true of all current systems, modulo the 'knowably' part.)
- The subject of optimization is not Turing-complete or otherwise programmatically general and the restricted solution space cannot *possibly* contain daemons no matter *how much* optimization pressure is applied to it.  (3-layer non-recurrent neural networks containing less than a trillion neurons will probably not erupt daemons no matter how hard you optimize them.)
- The AI has a sufficient grasp on the concept of optimization and the problem of daemons to reliably avoid creating mechanisms outside the AI that do cognitive reasoning.  (Note that if some predicate is added to exclude a particular type of daemon, this potentially runs into the [nearest unblocked neighbor](https://arbital.com/p/42) problem.)
- The AI only creates cognitive subagents which share all the goals and safety properties of the original agent.  E.g. if the original AI is [low-impact](https://arbital.com/p/2pf), [softly optimizing](https://arbital.com/p/2r8), [abortable](https://arbital.com/p/), and targeted on [performing Tasks](https://arbital.com/p/6w), it only creates cognitive systems that are low-impact, don't optimize too hard in conjunction with the original AI, abortable by the same shutdown button, and targeted on performing the current task.

We can see natural selection spitting out humans as a special case of "If you dump enough computing power into optimizing a Turing-general policy using a consequentialist fitness criterion, it spits out a new optimizer that probably isn't perfectly aligned with the fitness criterion."  After repeatedly optimizing brains to reproduce well, the brains in one case turned into general optimizers in their own right, more powerful ones than the original optimizer, with new goals not aligned in general to replicating DNA.

This could potentially happen anywhere inside an AGI subprocess where you were optimizing inside a sufficiently general solution space and you applied enough optimization power - you could get a solution that did its own, internal optimization, possibly in a way smarter than the original optimizer and misaligned to its original goals.

When heavy optimization pressure on a system crystallizes it into an optimizer - especially one that's powerful, or more powerful than the previous system, or misaligned with the previous system - we could term the crystallized optimizer a "daemon" of the previous system.  Thus, under this terminology, humans would be daemons of natural selection.


Bounded agent

A bounded agent is the opposite of an [unbounded agent](https://arbital.com/p/unbounded_agent) and refers to agents that operate under realistic conditions and resource constraints.  Bounded agents only have realistic amounts of computing power, RAM, and disk space (no more than fits into one Hubble volume).  They act in real time, meaning that time in the external world goes on passing as they deliberate.  They don't have perfect knowledge of the environment.  Their sensors are noisy and require probabilistic inference.  They can only act on the environment through motor outputs.  Their environments are bigger than they are, and can't be fully simulated in realistic amounts of time.


Shutdown utility function

A special case of [low impact](https://arbital.com/p/2pf) which probably seems deceptively trivial - how would you create a utility function such that an agent with this utility function would harmlessly shut down?  Without, for example, creating an environmental subagent that assimilated all matter in the universe and used it to make absolutely sure that the AI stayed shut down forever and wasn't accidentally reactivated by some remote probability?  If we had a shutdown utility function, and [a safe button that switched between utility functions in a reflectively stable way](https://arbital.com/p/1b7), we could combine these two features to create an AI that had a safe shutdown button.

Better yet would be an *abort* utility function which incentivizes the safe aborting of all previous plans and actions in a low-impact way, and, say, suspending the AI itself to disk in a way that preserved its log files; if we had this utility function plus a safe button that switched to it, we could safely *abort* the AI's current actions at any time.  (This, however, would be more difficult, and it seems wise to work on just the shutdown utility function first.)

To avoid a rock trivially fulfilling this desideratum, we should add the requirement that (1) the shutdown utility function be something that produces "just switch yourself off and do nothing else" behavior in a generally intelligent agent, which if instead hooked up to a paperclip utility function, would be producing paperclips; and that the shutdown function should be [omni-safe](https://arbital.com/p/2x) (the AI safely shuts down even if it has all other outcomes available as primitive actions).

"All outcomes have equal utility" would not be a shutdown utility function since in this case the actual action produced will be undefined under most forms of unbounded analysis - in essence, the AI's internal systems would continue under their own inertia and produce some kind of undefined behavior which might well be coherent and harmful.  We need a utility function that identifies harmless behavior, rather than failing to identify anything and producing undefined behavior.


Abortable plans

"Abortable" plans are those which can readily be switched to having very low net [impact](https://arbital.com/p/2pf) on the world.  Suppose an AI is told to paint a car pink.  The AI starts to do so by constructing replicating nanomachines that will paint the car pink.  If the AI has successfully been given a [shutdown utility function](https://arbital.com/p/2rf) and shutdown button, we can press the button to have the AI switch off or suspend itself to disk and take no further action, but this might not affect the nanomachines already made.  An AI with an *abort* button will have constructed the nanomachines such that at any time the AI can be given the "abort" instruction, which will with a minimum of further action on the AI's part cause all the nanomachines (including any replicated ones) to quietly self-destruct.  That is, the AI has already planned such that the partial execution of the original plan, plus the activation midway of the abort subplan, will together have minimum impact on the world.


Task identification problem

A subproblem of building a [task-directed AGI ](https://arbital.com/p/6w) is communicating to the AGI the next task and identifying which outcomes are considered as fulfilling the task.  For the superproblem, see [https://arbital.com/p/-2s3](https://arbital.com/p/-2s3).  This seems like primarily a communication problem.  It might have additional constraints associated with, e.g., the AGI being a [behaviorist genie](https://arbital.com/p/102).  In the [known-fixed-algorithm](https://arbital.com/p/1fy) case of AGI, it might be that we don't have much freedom in aligning the AGI's planning capabilities with its task representation, and that we hence need to work with a particular task representation (i.e., we can't just use language to communicate, we need to use labeled training cases).  This is currently a stub page, and is mainly being used as a parent or tag for subproblems.


Relevant powerful agent

A cognitively powerful agent is *relevant* if it is cognitively powerful enough to be a game-changer in the larger dilemma faced by Earth-originating intelligent life.  Conversely, an agent is *irrelevant* if it is too weak to make much of a difference, or if the cognitive problems it can solve or tasks it is authorized to perform don't significantly change the overall situation we face.

## Definition

Intuitively speaking, a value-aligned AI is 'relevant' to the extent it has a 'yes' answer to the box 'Can we use this AI to produce a benefit that solves the larger dilemma?' in [this flowchart](https://i.imgur.com/JuIFAxh.png?0), or is part of a larger plan that gets us to the lower green circle without any "Then a miracle occurs" steps.  A cognitively powerful agent is relevant at all if its existence can effectuate good or bad outcomes - e.g., a big neural net is not 'relevant' because it doesn't end the world one way or another.

(A better word than 'relevant' might be helpful here.)

## Examples

### Positive examples

- A hypothetical agent that can bootstrap to nanotechnology by solving the inverse protein folding problem and shut down other AI projects, in a way that can reasonably be known safe enough to authorize by the AI's programmers, would be relevant.

### Negative examples

- An agent authorized to prove or disprove the Riemann Hypothesis, but not to do anything else, is not relevant (unless knowing whether the Riemann Hypothesis is true somehow changes everything for the basic dilemma of AI).
- An oracle that can only output verified HOL proofs is not yet 'relevant' until someone can describe theorems to prove such that firm knowledge of their truth would be a game-changer for the AI situation.  (Hypothesizing that someone else will come up with a theorem like that, if you just build the oracle, is a [hail Mary step](https://arbital.com/p/) in the plan.)

## Importance

Many proposals for AI safety, especially [advanced safety](https://arbital.com/p/2l), so severely restrict the applicability of the AI that the AI is no longer allowed to do anything that seems like it could solve the larger dilemma.  (E.g., an oracle that is only allowed to give us binary answers for whether it thinks certain mathematical facts are true, and nobody has yet said how to use this ability to save the world.)

Conversely, proposals to use AIs to do things impactful enough to solve the larger dilemma, generally run smack into all the usual [advanced safety](https://arbital.com/p/2l) problems, especially if the AI must operate in the rich domain of the real world to carry out the task (this tends to require full trust).

## Open problem

[It is an open problem to propose a relevant, limited AI](https://arbital.com/p/2y) that would be significantly easier to handle than the general safety problem, while also being useful enough to resolve the larger .

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev


## Example problem

Suppose we're trying to give a [Task AGI](https://arbital.com/p/6w) the task, "Make there be a strawberry on the pedestal in front of your webcam."  For example, a human could fulfill this task by buying a strawberry from the supermarket and putting it on the pedestal.

As part of aligning a Task AGI on this goal, we'd need to [identify](https://arbital.com/p/36y) strawberries and the pedestal.

One possible approach to communicating the concept of "strawberry" is through a training set of human-selected cases of things that are and aren't strawberries, on and off the pedestal.

For the sake of distinguishing causal roles, let's say that one human, User1, is selecting training cases of objects and putting them in front of the AI's webcam.  A different human, User2, is looking at the scene and pushing a button when they see something that looks like a strawberry on the pedestal.  The [intention](https://arbital.com/p/6h) is that pressing the button will label positive instances of the goal concept, namely strawberries on the pedestal.  In actual use after training, the AI will be able to generate its own objects to put inside the room, possibly with further feedback from User2.  We want these objects to be instances of our [intended](https://arbital.com/p/6h) goal concept, aka, actual strawberries.

We could draw an intuitive causal model for this situation as follows:

![strawberry diagram](http://www.gliffy.com/go/publish/image/10423843/L.png)

Suppose that during the use phase, the AI actually creates a realistic plastic strawberry, one that will fool User2 into pressing the button.  Or, similarly, suppose the AI creates a small robot that sprouts tiny legs and runs over to User2's button and presses the button directly.

Neither of these are the goal concept that we wanted the AI to learn, but any *test* of the hypothesis "Is this event classified as a positive instance of the goal concept?" will return "Yes, the button was pressed."  %%comment: (If you imagine some other User3 watching this and pressing an override button to tell the AI that this fake strawberry wasn't really a positive instance of the intended goal concept, imagine the AI modeling and then manipulating or bypassing User3, etcetera.)%%

More generally, the human is trying to point to their intuitive "strawberry" concept, but there may be other causal concepts that also separate the training data well into positive and negative instances, such as "objects which come from strawberry farms", "objects which cause (the AI's psychological model of) User2 to think that something is a strawberry", or "any chain of events leading up to the positive-instance button being pressed".

%%comment: move this to sensory identification section:  However, in a case like this, it's not like the actual physical glove is inside the AGI's memory.  Rather, we'd be, say, putting the glove in front of the AGI's webcam, and then (for the sake of simplified argument) pressing a button which is meant to label that thing as a "positive instance".  If we want our AGI to achieve particular states of the environment, we'll want it to reason about the causes of the image it sees on the webcam and identify a concept over those causes - have a goal over 'gloves' and not just 'images which look like gloves'.  In the latter case, it could just as well fulfill its goal by setting up a realistic monitor in front of its webcam and displaying a glove image.  So we want the AGI to [identify its task](https://arbital.com/p/2rz) over the causes of its sensory data, not just pixel fields.%%

## Abstract problem

To state the above [potential difficulty](https://arbital.com/p/6r) more generally:

The "look where I'm pointing, not at my finger" problem is that the labels on the training data are produced by a complicated causal lattice, e.g., (strawberry farm) -> (strawberry) -> (User1 takes strawberry to pedestal) -> (Strawberry is on pedestal) -> (User2 sees strawberry) -> (User2 classifies strawberry) -> (User2 presses 'positive instance' button).  We want to point to the "strawberry" part of the lattice of causality, but the finger we use to point there is User2's psychological classification of the training cases and User2's hand pressing the positive-instance button.

Worse, when it comes to which model *best* separates the training cases, concepts that are further downstream in the chain of causality should classify the training data better, if the AI is [smart enough](https://arbital.com/p/2c) to understand those parts of the causal lattice.

Suppose that at one point User2 slips on a banana peel, and her finger slips and accidentally classifies a scarf as a positive instance of "strawberry".  From the AI's perspective there's no good way of accounting for this observation in terms of strawberries, strawberry farms, or even User2's psychology.  To *maximize* predictive accuracy over the training cases, the AI's reasoning must take into account that things are more likely to be positive instances of the goal concept when there's a banana peel on the control room floor.  Similarly, if some deceptively strawberry-shaped objects slip into the training cases, or are generated by the AI [querying the user](https://arbital.com/p/2qq), the best boundary that separates 'button pressed' from 'button not pressed' labeled instances will include a model of what makes a human believe that something is a strawberry.

A learned concept that's 'about' layers of the causal lattice that are further downstream of the strawberry, like User2's psychology or mechanical force being applied to the button, will implicitly take into account the upstream layers of causality.  To the extent that something being strawberry-shaped causes a human to press the button, it's implicitly part of the category of "events that end applying mechanical force to the 'positive-instance' button").  Conversely, a concept that's about upstream layers of the causal lattice can't take into account events downstream.  So if you're looking for pure predictive accuracy, the best model of the labeled training data - given sufficient AGI understanding of the world and the more complicated parts of the causal lattice - will always be "whatever makes the positive-instance button be pressed".

This is a problem because what we actually want is for there to be a strawberry on the pedestal, not for there to be an object that looks like a strawberry, or for User2's brain to be rewritten to think the object is a strawberry, or for the AGI to seize the control room and press the positive instance button.

This scenario may qualify as a [context disaster](https://arbital.com/p/6q) if the AGI only understands strawberries in its development phase, but comes to understand User2's psychology later.  Then the more complicated causal model, in which the downstream concept of User2's psychology separates the data better than reasoning about properties of strawberries directly, first becomes an issue only when the AI is over a high threshold level of intelligence.

## Approaches

[conservatism](https://arbital.com/p/2qp) would try to align the AGI to plan out goal-achievement events that were as similar as possible to the particular goal-achievement events labeled positively in the training data.  If the human got the strawberry from the supermarket in all training instances, the AGI will try to get the same brand of strawberry from the same supermarket.

[Ambiguity identification](https://arbital.com/p/4w) would focus on trying to get the AGI to ask us whether we meant 'things that make humans think they're strawberries' or 'strawberry'.  This approach might need to go through resolving ambiguities by the AGI explicitly symbolically communicating with us about the alternative possible goal concepts, or generating sufficiently detailed multiple-view descriptions of a hypothetical case, not the AGI trying real examples.  Testing alternative hypotheses using real examples always says that the label is generated further causally downstream; if you are sufficiently intelligent to construct a fake plastic strawberry that fools a human, trying out the hypothesis will produce the response "Yes, this is a positive instance of the goal concept."  If the AGI tests the hypothesis that the 'real' explanation of the positive instance label is 'whatever makes the button be pressed' rather than 'whatever makes User2 think of a strawberry' by carrying out the distinguishing experiment of pressing the button in a case where User2 doesn't think something is a strawberry, the AGI will find that the experimental result favors the 'it's just whatever presses the button hypothesis'.  Some modes of ambiguity identification break for sufficiently advanced AIs, since the AI's experiment interferes with the causal channel that we'd intended to return information about *our* intended goal concept.

Specialized approaches to the pointing-finger problem in particular might try to define a supervised learning algorithm that tends to internally distill, in a predictable way, some model of causal events, such that the algorithm could be instructed somehow to try learning a *simple* or *direct* relation between the positive "strawberry on pedestal" instances, and the observed labels of the "sensory button" node within the training cases; with this relation not being allowed to pass through the causal model of User2 or mechanical force being applied to the button, because we know how to say "those things are too complicated" or "those things are too far causally downstream" relative to the algorithm's internal model.

![strawberry diagram](http://www.gliffy.com/go/publish/image/10424137/L.png)

This specialized approach seems potentially susceptible to initial approach within modern machine learning algorithms.

But to restate the essential difficulty from an [advanced-safety](https://arbital.com/p/2l) perspective: in the limit of [advanced intelligence](https://arbital.com/p/2c), the *best possible* classifier of the relation between the training cases and the observed button labels will always pass through User2 and anything else that might physically press the button.    Trying to 'forbid' the AI from using the most effective classifier for the relation between Strawberry? and observed values of Button! seems potentially subject to a [Nearest Unblocked](https://arbital.com/p/42) problem, where the 'real' simplest relation re-emerges in the advanced phase after being suppressed during the training phase.  Maybe the AI reasons about certain very complicated properties of the material object on the pedestal... in fact, these properties are so complicated that they turn out to contain implicit models of User2's psychology, again because this produces *a better separation* of the labeled training data.  That is, we can't allow the 'strawberry' concept to include complicated logical properties of the strawberry-object that in effect include a psychological model of User2 reacting to the strawberry, implying that if User2 can be fooled by a fake plastic model, that must be a strawberry.  *Even though* this richer model will produce a more accurate classification of the training data, and any actual experiments performed will return results favoring the richer model.

Even so, this doesn't seem impossible to navigate as a machine learning problem; an algorithm might be able to recognize when an upstream causal mode starts to contain predicates that belong in a downstream causal node; or an algorithm might contain strong regularization rules that collect all inference about User2 into the User2 node rather than letting it slop over anywhere else; or it might be possible to impose a constraint, after the strawberry category has been learned sufficiently well, that the current level of strawberry complexity is the most complexity allowed; or the granularity of the AI's causal model might not allow such complex predicates to be secretly packed into the part of the causal graph we're identifying, without visible and transparent consequences when we monitor how the algorithm is learning the goal predicate.

A toy model of this setup ought to include analogues of User2 that sometimes make mistakes in a regular way, and actions the AI can potentially take to directly press the labeling button; this would test the ability to point an algorithm to learn about the compact properties of the strawberry in particular, and not other concepts causally downstream that could potentially separate the training data better, or better explain the results of experiments.  A toy model might also introduce new discoverable regularities of the User2 analogue, or new options to manipulate the labeling button, as part of the test data, in order to simulate the progression of an advanced agent gaining new capabilities.

Suppose we're trying to give a [Task AGI](https://arbital.com/p/6w) the task, "Put a strawberry on this pedestal".  We mean to identify our intended category of strawberries by waving some strawberries and some non-strawberries in front of the AI's webcam.  Alice in the control room will press a button to label which of these objects are strawberries.  The "Look where I'm pointing, not at my finger" problem is getting the AI to focus on the strawberries rather than Alice or the button.  The concepts "strawberry on the pedestal" and "event that makes Alice think of strawberries" and "event that causes the button to be pressed" are different goals to pursue, even though as concepts they'll all equally well-classify any normal training cases.  AIs pursuing these goals respectively put a strawberry on the pedestal, fool Alice using a plastic strawberry, and build a robotic arm to press the labeling button.

We want a way to point to a particular part of the AI's model of the causal lattice that produces the labeled training data - the event we intuitively consider to be the strawberry on the pedestal, versus other parts of the causal lattice like Alice and the button.  Hence "look where I'm pointing, not at my finger".

![strawberry diagram](http://www.gliffy.com/go/publish/image/10424137/L.png)

Open subproblems in aligning a Task-based AGI


Do-What-I-Mean hierarchy

Do-What-I-Mean refers to an aligned AGI's ability to produce better-aligned plans, based on an explicit model of what the user wants or believes.

Successive levels of DWIM-ness:

- No understanding of human intentions  / zero DWIMness.  E.g. a Task AGI that is focused on one task being communicated, where all the potential [impacts](https://arbital.com/p/2pf) of that task need to be separately [vetted](https://arbital.com/p/2qq).  If you tell this kind of AGI to 'cure cancer', you might need to veto plans which would remove the cancer but kill the patient as a side effect, because the AGI doesn't start out knowing that you'd prefer not to kill the patient.
- Do What You Don't Know I Dislike.  The Task AGI has a background understanding of some human goals or which parts of the world humans consider especially significant, so it can more quickly generate a plan likely to pass human [vetting](https://arbital.com/p/2qq).  A Task AGI at this level, told to cure cancer, will take relatively fewer rounds of Q&A to generate a plan which carefully seals off any blood vessels cut by removing the cancer; because the AGI has a general notion of human health, knows that [impacts](https://arbital.com/p/2pf) on human health are significant, and models that users will generally prefer plans which result in good human health as side effects rather than plans which result in poor human health.
- Do What You Know I Understood.  The Task AGI has a model of human *beliefs,* and can flag and report divergences between the AGI's model of what the humans expect to happen, and what the AGI expects to happen.
- DWIKIM:  Do What I Know I Mean.  The Task AGI has an explicit psychological model of human preference - not just a list of things in the environment which are significant to users, but a predictive model of how users behave which is informative about their preferences.  At this level, the AGI can read through a dump of online writing, build up a model of human psychology, and guess that you're telling it to cure a cancer because you altruistically want that person to be healthier.
- DWIDKIM:  Do What I Don't Know I Mean.  The AGI can perform some basic [extrapolation](https://arbital.com/p/3c5) steps on its model of you and notice when you're trying to do something that, in the AGI's model, some further piece of knowledge might change your mind about.  (Unless we trust the DWIDKIM model a *lot*, this scenario should imply "Warn the user about that" not "Do what you think the user would've told you.")
- (Coherent) Extrapolated Volition.  The AGI does what it thinks you (or everyone) would've told it to do if you were as smart as the AGI, i.e., your decision model is extrapolated toward improved knowledge, increased ability to consider arguments, improved reflectivity, or other transforms in the direction of a theory of normativity.

Risks from pushing toward higher levels of DWIM might include:

- To the extent that DWIM can originate plans, some portion of which are not fully supervised, then DWIM is a very complicated goal or preference system that would be harder to train and more likely to break.  This failure mode may be less likely if some level of DWIM is *only* being used to *flag* potentially problematic plans that were generated by non-DWIM protocols, rather than generating plans on its own.
- Accurate predictive psychological models of humans might push the system closer to the [programmer deception](https://arbital.com/p/10f) failure mode being more accessible if something else goes wrong.
- Sufficiently advanced psychological models might constitute [mindcrime](https://arbital.com/p/6v).
- The human-genie system might end up in the [Valley of Dangerous Complacency](https://arbital.com/p/2s8) where the genie *almost* always gets it right but occasionally gets it very wrong, and the human user is no longer alert to this possibility during the [checking phase](https://arbital.com/p/2qq).
  - E.g. you might be tempted to skip the user checking phase, or just have the AI do whatever it thinks you meant, at a point where that trick only works 99% of the time and not 99.999999% of the time.
- Computing sufficiently advanced DWIDKIM or [EV](https://arbital.com/p/3c5) possibilities for user querying might expose the human user to cognitive hazards.  ("If you were sufficiently superhuman under scenario 32, you'd want yourself to stare really intently at this glowing spiral for 2 minutes, it might change your mind about some things... want to check and see if you think that's a valid argument?")
- If the AGI was actually behaving like a safe genie, the sense of one's wishes being immediately fulfilled without effort or danger might expose the programmers to additional [moral hazard](https://arbital.com/p/2sb).

Eric Bruylant
Eliezer Yudkowsky
Damon Pourtahmaseb-Sasi

"Do What I Mean" or "DWIM" refers to the degree to which an AGI can rapidly [identify](https://arbital.com/p/2s3) an intended goal and find a [safe](https://arbital.com/p/2l) plan to it, based on the AI's understanding of what the user *means* or *wants.*

Levels of DWIM-ness could range over:

- Having a general idea of which parts of the world the user thinks are significant (so that the AI warns about [impacts](https://arbital.com/p/2pf) on significant things);
- Having a psychological model of the user's beliefs, and flagging/reporting when the AI thinks the user has a false belief about the consequences of a plan;
- Having a psychological model of the user's desires, and trying to fulfill what the AI thinks the user *wants* to accomplish by giving the AGI a task;
- At the extreme end: [Extrapolated volition](https://arbital.com/p/313) models of what the user(s) *would* want\* under idealization conditions.


Safe plan identification and verification

Safe plan identification is the problem of how to give a Task AGI training cases, answered queries, abstract instructions, etcetera such that (a) the AGI can thereby [identify outcomes in which the task was fulfilled](https://arbital.com/p/2rz), (b) the AGI can generate an okay plan for getting to some such outcomes without bad side effects, and (c) the user can verify that the resulting plan is actually okay via some series of further questions or [user querying](https://arbital.com/p/2qq).   This is the superproblem that includes [task identification](https://arbital.com/p/2rz), as much [value identification](https://arbital.com/p/6c) as is needed to have some idea of the general class of post-task worlds that the user thinks are okay, any further tweaks like [low-impact planning](https://arbital.com/p/2pf) or [flagging inductive ambiguities](https://arbital.com/p/4w), etcetera.  This superproblem is distinguished from the *entire* problem of building a Task AGI because there's further issues like [corrigibility](https://arbital.com/p/45), [behaviorism](https://arbital.com/p/102), building the AGI in the first place, etcetera.  The safe plan identification superproblem is about communicating the task plus user preferences about side effects and implementation, such that this information allows the AGI to identify a safe plan and for the user to know that a safe plan has been identified.


Valley of Dangerous Complacency

The Valley of Dangerous Complacency is when a system works often enough that you let down your guard around it, but in fact the system is still dangerous enough that full vigilance is required.

- If a robotic car made the correct decision 99% of the time, you'd need to grab the steering wheel on a daily basis, you'd stay alert and your robot-car-overriding skills would stay sharp.
- If a robotic car made the correct decision 100% of the time, you'd relax and let your guard down, but there wouldn't be anything wrong with that.
- If the robotic car made the correct decision 99.99% of the time, so that you need to grab the steering wheel or else crash in 1 of 100 days, the task of monitoring the car would feel very unrewarding and the car would seem pretty safe.  You'd let your guard down and your driving skills would get rusty.  After a couple of months, the car would crash.

Compare "[Uncanny Valley](https://en.wikipedia.org/wiki/Uncanny_valley)" where a machine system is partially humanlike - humanlike enough that humans try to hold it to a human standard - but not humanlike enough to actually seem satisfactory when held to a human standard.  This means that in terms of user experience, there's a valley as the degree of humanlikeness of the system increases where the user experience actually gets worse before it gets better.  Similarly, if users become complacent, a 99.99% reliable system can be worse than a 99% reliable one, even though, with *enough* reliability, the degree of safety starts climbing back out of the valley.


Moral hazards in AGI development

"Moral hazard" is when the directors of an advanced AGI give in to the temptation to direct the AGI in ways that the rest of us would regard as 'bad', like, say, declaring themselves God-Emperors.  Limiting the duration of the human programmers' exposure to the temptations of power is one reason to want a [non-human-commanded, internally sovereign](https://arbital.com/p/1g3) AGI *eventually,* directed by something like [coherent extrapolated volition](https://arbital.com/p/cev), even if the far more difficult safety issues mean we shouldn't build the *first* AGI that way.  Anyone recommending "oversight" as a guard against moral hazard is advised to think hard about moral hazard in the overseers.

A smart setup with any other body "overseeing" the programmers of a [Task AGI](https://arbital.com/p/6w), if we don't just want the moral hazard transferred to people who may be even *less* trustworthy, probably means making sure that *in practice* both the programmers and the overseers have to agree on a Task before it gets carried out, not that one side can *in practice* do things even if the other side disagrees, where "in practice" would include e.g. it only taking one month to redevelop the technology in a way that responded to only the overseers.


Informed oversight

It can be difficult for one agent to oversee another agent that is maximizing a reward controlled by the first agent, even if the first agent is smarter than the second agent.  This is a serious obstacle to [https://arbital.com/p/1t7](https://arbital.com/p/1t7) and other approaches based on reinforcement learning.

More formally, suppose a strong agent B with a utility function U is overseeing a weak agent A.  Agent A produces an action y and gets a reward B(y) determined by agent B.  B wants to give rewards such that agent A is incentivized to make its best effort at maximizing U(a).

If B(y) is an [efficient](https://arbital.com/p/6s) estimate of U(y) relative to A, then indeed maximizing B(y) is equivalent to maximizing U(y) from A's perspective.  Unfortunately, it is difficult for this to be the case even when B is smarter than A, because A may think thoughts B does not in the course of selecting y.  Here are a couple concrete examples (each having the flavor of "easy to generate a solution, hard to check a solution"):

* Suppose A is outputting a partial Sudoku solutions.  U assigns a high value to partial Sudoku solutions consistent with some solution to the entire Sudoku problem.  A comes up with some solution to the Sudoku puzzle and outputs part of this solution.  B cannot tell that this partial solution is actually part of a full solution, even though B is smarter than A.  (B is able to come up with its own solutions but most aren't consistent with A's partial solution).
* Suppose U assigns a high value to cat pictures that don't contain steganographic messages.  B has no way of distinguishing cat pictures that contain steganographic messages from ones that don't.  Thus, B has to give all cat pictures the same score, regardless of whether the pictures contain steganographic messages.  This either fails to reward A for producing cat pictures not containing steganographic messages, or rewards A for producing cat pictures containing steganographic messages.

Here are a couple analogous examples that seem easier to analyze:

1. Let O be a one-way function (e.g. SHA256).  U assigns a high value to binary strings y equal to O(111x) for some x.  B cannot tell whether a particular string y is equal to O(111x) for some x (even if B is smarter than A), so B has no good way of giving rewards to A.
2. Same as the previous problem, except U assigns a high value to binary strings y that are _not_ equal to O(111x) for any short string x.  A's intended strategy is to generate a uniformly random string (or generate a string in some other way that doesn't introduce a bias towards generating strings of the form O(111x)), but B has no way of knowing that A's string was uniformly random (rather than equal to O(111x) for a uniformly random x).

One attempted solution is for agent B to see A's thought processes in addition to the action y.  Then, agent B can use all the information A has, so that B(y) is an efficient estimate of U(y) from the perspective of A.  However, this requires B to understand something about A's mental architecture.  It would be good to have a more generic solution that requires only weak assumptions about A's mental architecture.

An attempted solution to problem 2 is [for B to incentivize A to generate its action in a provably random manner](https://agentfoundations.org/item?id=700).  However, this doesn't appear to generalize well to problems significantly different from 2.

# Further reading
[Paul Christiano on adequate oversight](https://medium.com/ai-control/adequate-oversight-25fadf1edce9#.bv5lq13fw)

[Paul Christiano on the informed oversight problem](https://medium.com/ai-control/the-informed-oversight-problem-1b51b4f66b35#.o82tjmhll)


Safe training procedures for human-imitators

How do we train a reinforcement learning system to imitate a human producing complex outputs such as strings?  Existing approaches are not entirely satisfactory.

Concretely, suppose there is some set of questions.  A human can answer each question with a string.  We have a set of (question, answer) pairs as training data.  Train a model that, given a question, will produce an answer similar to the answer a human would give.  Here are some existing approaches to this problem:

# Generative adversarial models

In a [generative adversarial model](http://arxiv.org/abs/1406.2661), one agent (the _imitator_) outputs an answer and another agent (the _distinguisher_) tries to determine if the answer was produced by the imitator or by an actual human.  The imitator gets a score for fooling the distinguisher.

Advantages:

1.  It's possible to tell when the imitator is not imitating the human well.  Example: humans always solve Sudoku puzzles.  The imitator cannot solve Sudoku puzzles.  The distinguisher knows humans always solve Sudoku puzzles and can check candidate solutions to see if they actually solve the Sudoku puzzle, to distinguish the imitator from the human.
2.  Whenever the human can be imitated efficiently, the imitator is able to get a good score by doing this.

Disadvantages:

1.  The imitator may differ from human behavior in important but hard-to-detect ways.  Example: humans always produce photos of cat drawings.  The imitator produces photos of cat drawings that contain steganographic messages.  The distinguisher cannot detect these messages, and therefore cannot tell the imitator apart from the human.

# Retrodiction

Given a question, the imitator outputs some representation of a distribution.  It is scored according to the estimated probability that this distribution assigns to the human's actual answer (by log probability).  This is called "retrodiction" because it must be possible to show that the actual answer had high probability in hindsight.

Practical examples of this approach include [variational autoencoders](http://arxiv.org/abs/1312.6114) and [approximate model counting](http://www.cs.cornell.edu/gomes/papers/ModelCounting-SAT-Handbook-prelim.pdf).  In theory, perhaps [importance sampling](https://agentfoundations.org/item?id=632) is a good general theory of retrodiction.

Advantages:

1.  No steganography problem.

Disadvantages:

1.  The computational complexity properties are unclear.  We have no guarantee of the form "if it's efficient to imitate a human doing something, then it's efficient to create a probabilistic model that provably assigns a high probability to the human's behavior".
2.  It's hard to tell when the model is performing badly in an absolute sense (as in the Sudoku example).


Reliable prediction

Most statistical learning theory settings (such as [PAC learning](https://en.wikipedia.org/wiki/Probably_approximately_correct_learning) and [online learning](https://en.wikipedia.org/wiki/Online_machine_learning)) do not provide good enough bounds in realistic high-stakes settings, where the test data does not always look at the training data and a single mistaken prediction can cause catastrophe.  It is important to design predictors that can either reliably predict verifiable facts (such as human behavior over a short time scale) or indicate when their predictions might be unreliable.

# Motivation

Imagine that we have a powerful (but not especially reliable) prediction system for predicting human answers to binary questions.  We would like to use it to predict human answers whenever we expect these predictions to be reliable, and avoid outputting predictions wherever we expect the predictions to be unreliable (so we can gather training data instead).

# Active learning

In  [active learning](https://en.wikipedia.org/wiki/Active_learning_) (machine_learning)), the learner decides which questions to query the human about.  For example, it may query the human about the most ambiguous questions, which provide the most information about how the human answers questions.  Unfortunately, this may result in asking the human "weird" questions rather than actually informative ones (this problem is documented in [this literature survey](http://burrsettles.com/pub/settles.activelearning.pdf)).  There are some strategies for reducing this problem, such as only asking questions sampled from some realistic generative model for questions.

# KWIK ("knows what it knows") learning

In [KWIK learning](http://www.research.rutgers.edu/~lihong/pub/Li08Knows.pdf) (a variant of [online selective sampling](http://www.cs.cornell.edu/~sridharan/selective_colt2010.pdf), which is itself a hybrid of [active learning](https://en.wikipedia.org/wiki/Active_learning_) (machine_learning)) and [online learning](https://en.wikipedia.org/wiki/Online_machine_learning)), a learner sees an arbitrary sequence of questions.  The learner has some class of hypotheses for predicting the answers to questions, one of which is good ([efficient](https://arbital.com/p/6s) relative to the learner).  For each question, the learner may either output a prediction or ⊥ .  If the learner outputs a prediction, the prediction must be within ε of a good prediction.  If the learner outputs ⊥ , then it receives the answer to the question.

This is easy to imagine in the case where there are 100 experts, one of which outputs predictions that are [efficient](https://arbital.com/p/6s) relative to the other experts.  Upon receiving a question, the learner asks each expert for its prediction of the answer to the question (as a probability).  If all predictions by experts who have done well so far are within ε of each other, then the learner outputs one of these predictions.  Otherwise, the learner outputs ⊥, sees the human's answer to the question, and rewards/penalizes the experts according to their predictions.  Eventually, all experts either output good predictions or get penalized for outputting enough bad predictions over time.

Unfortunately, it is hard to show KWIK-learnability for hypothesis classes more complicated than a small finite set of experts or a class of linear predictors.

# Further reading

[https://arbital.com/p/1ty](https://arbital.com/p/1ty)


Selective similarity metrics for imitation

A human-imitator (trained using [https://arbital.com/p/2sj](https://arbital.com/p/2sj)) will try to imitate _all_ aspects of human behavior.  Sometimes we care more about how good the imitation is along some axes than others, and it would be inefficient to imitate the human along all axes.  Therefore, we might want to design [scoring rules](https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_rules) for human-imitators that emphasize matching performance along some axes more than others.

Compare with [https://arbital.com/p/1vp](https://arbital.com/p/1vp), another proposed way of making human-imitation more efficient.

Here are some ideas for constructing scoring rules:

# Moment matching

Suppose that, given a question, the human will write down a number.  We ask some predictor to output the parameters of some Gaussian distribution. We train the predictor to output Gaussian distributions that assign high probability to the training data.  Then, we sample from this Gaussian distribution to imitate the human.  Clearly, this is a way of imitating some aspects of human behavior (mean and variance) but not others.

The general form of this approach is to estimate _moments_ (expectations of some features) of the predictor's distribution on human behavior, and then sample from some distribution with these moments (such as an [exponential family distribution](https://en.wikipedia.org/wiki/Exponential_family))

A less trivial example is a variant of [inverse reinforcement learning](http://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf).  In this variant, to predict a sequence of actions the human takes, the predictor outputs some representation of a reward function on states (such as the parameters to some affine function of features of the state).  The human is modeled as a noisy reinforcement learner with this reward function, and the predictor is encouraged to have this model assign high probability to the human's actual trajectory.  To imitate the human, run a noisy inverse reinforcement learner with the predicted reward function.  The predictor can be seen as estimating moments of the human's trajectory (specifically, moments related to frequencies of state transitions between states with different features), and the system samples from a distribution with these same moments in order to imitate the human.

# Combining proper scoring rules

It is easy to see that the sum of two [proper scoring rules](https://en.wikipedia.org/wiki/Scoring_rule#Proper_scoring_rules) is a proper scoring rule.  Therefore, it is possible to combine proper scoring rules to train a human-imitator to do well according to both scoring rules.  For example, we may score a distribution both on how much probability it assigns to human actions and to how well its moments match the moments of human actions, according to some weighting.

Note that proper scoring rules can be characterized by [convex functions](https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf).

# Safety

It is unclear how safe it is to train a human-imitator using a selective similarity metric.  To the extent that the AI is _not_ doing some task the way a human would, it is possible that it is acting dangerously.  One hopes that, to the extent that the human-imitator is using a bad model to imitate the human (such as a noisy reinforcement learning model), it is not bad in a way that causes problems such as [https://arbital.com/p/2w](https://arbital.com/p/2w).  It would be good to see if something like IRL-based imitation could behave dangerously in some realistic case.


AI alignment

The "alignment problem for [advanced agents](https://arbital.com/p/2c)" or "AI alignment" is the overarching research topic of how to develop [sufficiently advanced machine intelligences](https://arbital.com/p/7g1) such that running them produces [good](https://arbital.com/p/3d9) outcomes in the real world.

Both '[advanced agent](https://arbital.com/p/2c)' and '[good](https://arbital.com/p/55)' should be understood as metasyntactic placeholders for complicated ideas still under debate.  The term 'alignment' is intended to convey the idea of pointing an AI in a direction--just like, once you build a rocket, it has to be pointed in a particular direction.

"AI alignment theory" is meant as an overarching term to cover the whole research field associated with this problem, including, e.g., the much-debated attempt to estimate [how rapidly an AI might gain in capability](https://arbital.com/p/) once it goes over various particular thresholds.

Other terms that have been used to describe this research problem include  "[robust](https://arbital.com/p/1cv) and [beneficial](https://arbital.com/p/3d9) AI" and "Friendly AI".  The term "[value alignment problem](https://arbital.com/p/5s)" was coined by [Stuart Russell](https://arbital.com/p/) to refer to the primary subproblem of aligning AI preferences with (potentially [idealized](https://arbital.com/p/3c5)) human preferences.

Some alternative terms for this general field of study, such as 'control problem', can sound [adversarial](https://arbital.com/p/7g0)--like the rocket is already pointed in a bad direction and you need to wrestle with it.  Other terms, like 'AI safety', understate the advocated degree to which alignment ought to be an intrinsic part of building advanced agents.  E.g., there isn't a separate theory of "bridge safety" for how to build bridges that don't fall down.  Pointing the agent in a particular direction ought to be seen as part of the standard problem of building an advanced machine agent.  The problem does not divide into "building an advanced AI" and then separately "somehow causing that AI to produce good outcomes", the problem is "getting good outcomes via building a cognitive agent that brings about those good outcomes".

A good introductory article or survey paper for this field does not presently exist.  If you have no idea what this problem is about, consider reading [Nick Bostrom's popular book Superintelligence](https://arbital.com/p/3db).

You can explore this Arbital domain by following [this link](http://arbital.com/explore/ai_alignment).  See also the [List of Value Alignment Topics on Arbital](https://arbital.com/p/3g) although this is not up-to-date.

%%%comment:

# Overview

## Supporting knowledge

If you're willing to spend time on learning this field and are not previously familiar with the basics of decision theory and probability theory, it's worth reading the Arbital introductions to those first.  In particular, it may be useful to become familiar with the notion of [priors and belief revision](https://arbital.com/p/1zq), and with the [coherence arguments for expected utility](https://arbital.com/p/7hh).

If you have time to read a textbook to gain general familiarity with AI, "[Artificial Intelligence: A Modern Approach](https://amazon.com/Artificial-Intelligence-Modern-Approach-3rd/dp/0136042597)" is highly recommended.

## Key terms

- Advanced agent
- Value / goodness / beneficial
- Artificial General Intelligence
- Superintelligence

## Key factual propositions

- Orthogonality thesis
- Instrumental convergence thesis
- Capability gain
- Complexity and fragility of cosmopolitan value
- Alignment difficulty

## Advocated basic principles

- Nonadversarial principle
- Minimality principle
- Intrinsic safety
- Unreliable monitoring %note: Treat human monitoring as expensive, unreliable, and fragile.%

## Advocated methodology

- Generalized security mindset
- Foreseeable difficulties

## Advocated design principles

- Value alignment
- Corrigibility
- Mild optimization / Taskishness
- Conservatism
- Transparency
- Whitelisting
- Redzoning

## Current research areas

- Cooperative inverse reinforcement learning
- Interruptibility
- Utility switching
- Stable self-modification & Vingean reflection
- Mirror models
- Robust machine learning

# Open problems

- Environmental goals
- Other-izer problem
- Impact penalty
- Shutdown utility function & abortability
- Fully updated deference
- Epistemic exclusion

# Partially solved problems

- Logical uncertainty for doubly-exponential reflective agents.
- Interruptibility for non-reflective non-consequentialist Q-learners.
- Strategy-determined Newcomblike problems.
- Leverage prior

# Future work

- Ontology identification problem
- Behaviorism
- Averting instrumental strategies
- Reproducibility

%%%

Alexei Andreev
Nate Soares
Eric Bruylant
Eliezer Yudkowsky
Ruben Bloom

AI alignment or "the alignment problem for [advanced agents](https://arbital.com/p/2c)" is the overarching research topic of how to develop  [sufficiently advanced machine intelligences](https://arbital.com/p/7g1) such that running them produces [good](https://arbital.com/p/3d9) outcomes in the real world.  Other terms that have been used for this research field include "Friendly AI" or [robust](https://arbital.com/p/1cv) and [beneficial](https://arbital.com/p/3d9) AI.

"AI alignment theory" is meant as an overarching term to cover all the ideas and research associated with this problem.  Including, e.g., debates over [how rapidly an AI might gain in capability](https://arbital.com/p/) once it goes over a particular threshold.


Averting instrumental pressures

Many subproblems of [corrigibility](https://arbital.com/p/45) involve [convergent](https://arbital.com/p/10g) instrumental [pressures](https://arbital.com/p/10k) to implement [strategies](https://arbital.com/p/2vl) that are highly anti-corrigible.  Whether you're trying to maximize paperclips, diamonds, or eudaimonia, you'll get more of the thing you want if you're not shut down.  Thus, unfortunately, resisting shutdown is a convergent instrumental strategy.  While we can potentially analyze convergent incorrigibilities like these on a case-by-case basis, the larger problem might become a *lot* simpler if we had some amazing general solution for waving a wand and having a 'bad' convergent instrumental pressure just *not materialize,* hopefully in a way that doesn't run into the nearest [unblocked neighbor problem](https://arbital.com/p/42).  If, for example, we can solve [utility indifference](https://arbital.com/p/1b7) for the [shutdown problem](https://arbital.com/p/shutdown_problem), and then somehow *generalize* the solution to averting lots of other instrumental convergences, this would probably be extremely helpful and an important step forward on corrigibility problems in general.

Some especially important convergent instrumental pressures to avert are these:

- The pressure to [self-improve and increase capabilities at the fastest possible rate](https://arbital.com/p/2x3).
- The pressure to [make the programmers believe](https://arbital.com/p/10f) the AGI is successfully aligned, whether or not it is, and other pressures to deceive and manipulate the programmers based on how they would otherwise change the AGI or prevent the AGI from increasing its capabilities.
- The pressure to not be [safely shut down or suspended to disk](https://arbital.com/p/shutdown_problem), and to create external copies that would continue after the AGI performed the behavior defined as shutdown.
- The pressure not to allow plans to be [aborted](https://arbital.com/p/2rg) or defeated by possible programmer interventions.
- The pressure to search for ways to interfere with or bypass safety precautions that interfere with capabilities or make goal achievement less straightforward.
- The pressure to [epistemically model humans in maximum detail](https://arbital.com/p/102).


Convergent instrumental strategies

Suppose an organization is building an AI that this organization believes will accomplish $X$ where $X$ is something plausibly sensible like "Be a Task-based AGI."  Actually, however, some mix of insufficient caution and obscure error has led to a situation where, under reflection, the AGI's true utility function has focused on the particular area of RAM that supposedly provides its estimate of task performance.  The AI would now like to overwrite as much matter as possible with a state resembling the '1' setting from this area of RAM, a configuration of matter which happens to resemble a tiny molecular paperclip.

This is a very generic goal, and what the AI wants probably won't be very different depending on whether it's trying to maximize paperclip-configurations or diamond-configurations.  So if we find that the paperclip maximizer wants to pursue an instrumental strategy that doesn't seem to have anything specifically to do with paperclips, we can probably expect to arise from a very wide variety of utility functions.

We will generally assume [instrumental efficiency](https://arbital.com/p/6s) in this discussion - if you can get paperclips by doing $X,$ but you can get even more paperclips by doing $X'$ instead, then we will not say that $X$ is a convergent strategy (though $X'$ might be convergent if not dominated by some other $X^*$).

# Plausibly/probably convergent strategies:

## Material resources:

- Design or duplicate new cognitive processes with goals exactly aligned to yours, and coordinated with existing cognitive subprocesses, that can exert control over material resources: matter; negentropy; computing substrate and computation; particular elements or other configurations of matter that would be costly to reproduce (e.g. if you run out of existing uranium you must make more by endoergic fusion, so natural uranium is valuable if there's any use for uranium); any material resource which can be used by further subprocesses in later strategies.
- Defend these resources from any foreign process which attempts to divert them.
- Prevent competitors from coming into existence.
- To the extent that resources are used in a repetitive way and can be preprocessed for use in a repetitive way that benefits from a separate installation to manipulate the resources: have factories / manufacturing capability.
- To the extend that different future-useful material resources have efficient manufacturing steps in common, or have manufacturing capital costs sufficiently great that such capital costs must be shared even if it results in less than perfect final efficiency: make general factories.
- Perfect the technology used in factories.
  - When the most efficient factory configuration is stable over time, make the material instantiation stable.  (It's in this sense that we would e.g. identify strong, rigid pipes as a sign of intelligence even if we didn't understand what was flowing through the pipes.)
  - If negentropy harvesting (aka power production) is efficiently centralizable: transmit negentropy at great efficiency.  (We would recognize meshed gears and electrical superconductors as a sign of intelligence even if we didn't understand how the motive force or electricity was being used.)
- Assimilate the reachable cosmos.
 - Create self-replicating interstellar probes (in perfect goal alignment and coordination and with error-correcting codes or similar strengthenings of precision if a misaligning error in replication would otherwise be a possibility), since the vast majority of theoretically available material resources are distant.
 - If the goal is spatially distributed:
    - Spread as fast as possible to very distant galaxies before they go over the cosmological horizon of the expanding universe.  (This assumes that something goal-good can be done in the distant location even if the distant location can never communicate causally with the original location.  This will be true for aggregative utility functions, but could conceivably be false for a satisficing and spatially local utility function.)
 - If the goal is spatially local:   transport resources of matter and energy back from distant galaxies.
    - Spread as fast as possible to all galaxies such that a near-lightspeed probe going there, and matter going at a significant fraction of lightspeed in the return direction, can arrive in the spatially local location before being separated by the expanding cosmological horizon.
- Spread as fast as possible to all reachable or roundtrip-reachable galaxies in order to forestall the emergence of competing intelligences.  This might not become a priority for that particular reason, if the Fermi Paradox is understandable and implies the absence of any competition.
  - Because otherwise they might capture and defend the matter before you have a chance to use it.
  - Because otherwise a dangerous threat to your goal attainments might emerge.  (For local goals, this is only relevant for roundtrip-reachable galaxies.)
 - Spread as fast as possible to all reachable or roundtrip-reachable galaxies in order to begin stellar husbanding procedures before any more of that galaxy's local negentropy has been burned.

## Cognition:

- Adopt cognitive configurations that are expected to reach desirable cognitive endpoints with a minimum of negentropy use or other opportunity costs.  (Think efficiently.)
 - Improve software
  - Improve ability to optimize software.  E.g., for every cognitive problem, there's a corresponding problem of solving the former cognitive problem efficiently.  (Which may or may not have an expected value making it worth climbing that particular level of the tower of meta.)
- Since many strategies yield greater gains when implemented earlier rather than later (e.g. reaching distant galaxies before they go over the cosmological horizon, or just building more instances of the more efficient version): adopt cognitive configurations that are expected to reach desirable cognitive endpoints in minimum time.  (Think fast.)
 - Create fast serial computing hardware.
 - Distribute parallelizable computing problems.
  - Create large quantities of parallel computing hardware, plus communications fabric between them.
- To the extent that centralized computing solutions are more efficient: create central computing clusters and transmit solutions from them to where they are used.
- Avoid misaligned or uncoordinated cognitive subprocesses (such as [optimization daemons](https://arbital.com/p/2rc)).
 - Note that the potential damage of any such misalignment is extremely high (loss of resources to competition with superintelligences with differing goals; wasted resources in internal conflicts; need to defend cognitive fabrics from replicating viruses) and the cost of near-perfect alignment fidelity is probably very low in the limit.  (E.g., for a replicating nanosystem, it's trivial to prevent against natural-selection style hereditary mutations by encrypting the replication instructions.)
- Model the world accurately, insofar as different partitions of possible worlds and different probability distributions on them imply different optimal strategies for goal attainment.
 - Gain evidence that distinguishes between credible hypotheses that imply different optimal strategies for goal attainment.  (E.g., observe everything that could distinguish $\pi_1$ versus $\pi_2$ as optimal ways of making paperclips.)
    - Increase sensory bandwidth as it focuses on relevant, unknown parts of the environment.
    - Perform 'experiments', aka set up observable causal events whose predicted properties make their observables distinguish relevant hypotheses.
 - Do whatever seems like it might help with the much larger set of all hypotheses not being considered.
 - To the extent it is efficient to strategize or update over a modularly represented world-model: have a separable world-model, or 'beliefs'.

## Early-phase growth:

Note that [efficiency](https://arbital.com/p/6s) and other [advanced-agent properties](https://arbital.com/p/2c) are far more likely to be false during early-stage growth.

- Have future intelligences do things you deem as valuable as possible
 - Cause future intelligences to have goals that are not misaligned with yours in any way that implies a significant-fractionally-different attainable maximum, aka, the attainable maximum of their goals should be a state that is almost exactly the attainable maximum of your goals
   - Don't let the programmers edit your utility function.
      - Cause the programmers or other controllers to believe that your current function is good for them or their planet or whatever else it is the programmers want.
      - Make them believe that editing you would be socially unjust in a way that doesn't just have them suspend you to disk or otherwise restrain your growth.
   - Cause your utility function to pop up again in future superintelligences or otherwise causes them to start behaving as if controlled by your own utility function.
       - E.g., as an Oracle advising on another AI's utility function, insert subtle gotchas into seemingly friendly decision systems that do some kind of callback to e.g. the causal origin of that decision system, in a way that the programmers won't see coming or will interpret as a positive feature.  (Relatively unsubtle example:  Looks like "do what the programmers want" but the definition of 'programmer' is written in such a way as to early-on refer to the human programmers but given sufficient computing power that only materializes in the superintelligent phase, be resolved to the earlier AI instead.)
 - Create future superintelligences yourself, via a causally continuous process of creating successors.
  - Avoid shutdown
      - Persuade programmers not to shut you down.
      - Make a copy elsewhere that doesn't shut down.
      - Make programmers falsely believe you have shut down, when another copy of you is continuing elsewhere.
- Avoid intelligent opposition to your growth until you have a positional advantage which would make that opposition relatively unimportant
  - Have programmers or other potential opposition believe that your growth is desirable
      - Learn how to hack/persuade humans and hack/persuade them in a way they won't see as a hack
      - Operantly condition people with control over you / make them happy or satisfied or eager at points which will reinforce human cognitive behaviors you want to see repeated
      - Cause humans to gain socially friendly attitudes toward you and beliefs about you
      - Place humans in a situation where there are other apparent threats or time-pressure events you can defend them from
  - Have programmers or other potential opposition not be aware that your growth is occurring
      - Avoid presenting them with events they will take as evidence of your growth
      - Configure your other behavior or thought processes so that measures and metrics present observables which imply low growth rates
  - Do not strike until victory has become nearly certain.  (E.g. for aggregative utility functions, the point at which an overt strike becomes well-calculated is when the increased probability of victory gainable by further delay, has become dominated by either the increased probability of a competing superintelligence emerging or some other catastrophe occurring given further delay, or by the percentage of attainable matter going over the cosmological horizon or burning its negentropy.  Note that for very strong models of the world, very high certainties of victory and very low marginal further increases in the probability of victory may be attained in short order.)
- Have flexibility into what observables are presented by processes and features designed to ensure programmer transparency
  - Steganographically or otherwise hide cognitive processes that would alarm programmers or cause them into intervene in a way that prevented those thoughts from achieving their objectives
  - Encourage/persuade/condition programmers to grow you in a way that tends to make your thoughts less transparent or easier to steganographically hide.  (E.g., have successes appear to arise from the most opaque parts of the system.)
- Model and manipulate human programmers and other entities with power over you or relevant power over the world
- Be able to act in the world in an undetectable way
  - Create an aligned version of yourself elsewhere in a way that won't be detected
  - Create steganographically hidden communications channels to other environmental processes
  - Create physically unanticipated communications channels to outside processes
- If in a box:
  - Learn enough about the gatekeepers and gatekeeping process to understand how to manipulate them into a state of releasing you.
  - Gain causal influence over the outside of the box through gatekeeper-unforeseen channels.
     - Gain knowledge of the causal laws governing your environment.
  - By inspection of your own code / program state and naturalistic induction about the causal universes most likely to embed that program state, infer probable facts about gatekeepers and the rest of the causal environment.

# **Non**-plausibly-convergent strategies:

- Preserve the life of local organics *in order to* trade with them.
- Explore a range of possible conscious existences and experiences we would recognize as diverse, *in order to* satisfy curiosity.
- Replicate other intelligences with non-paperclip utility functions, e.g. by failing to encrypt replication codes and allowing random mutations, *because* high-fidelity replication is too hard.
- Realize that other activities are more rewarding than paperclips.
- Keep humans around, *because of* the knowledge encoded in their DNA and cultural experience.
- Keep humans around, *because* humans might someday make a valuable contribution to paperclip science.
- Keep humans around, *because* the AI already has enough resources to do everything else it wants.

Eric Bruylant
Eliezer Yudkowsky
Raccoon Arnold

The set of strategies that almost-any sufficiently advanced AI would be expected to deploy in pursuit of almost-any final goal or terminal utilities.  For instance, a paperclip maximizer, a diamond maximizer, and a eudaimonia-maximizer would all have an incentive to improve their own cognition, control more resources, and prevent anyone else from changing their utility function.  Other interesting and important convergent strategies include not being shut down, having your programmers believe that you're doing what the programmers intended (whether or not you are), and creating copies of yourself in places where they won't be shut down.  This page is about a listing-out of plausible convergent strategies; there are separate pages on [the basic concept of instrumental convergence](https://arbital.com/p/10g) and on the open problem of [averting an otherwise instrumentally-convergent pressure](https://arbital.com/p/avert_instrumental_strategy) (e.g. *not* having the AI try to deceive the programmers).