Corrigibility

A 'corrigible' agent is one that [doesn't interfere](https://arbital.com/p/7g0) with what [we](https://arbital.com/p/9r) would intuitively see as attempts to 'correct' the agent, or 'correct' our mistakes in building it; and permits these 'corrections' despite the apparent [instrumentally convergent reasoning](https://arbital.com/p/10g) saying otherwise.

- If we try to suspend the AI to disk, or shut it down entirely, a corrigible AI will let us do so.  (Even though, if suspended, [the AI will then be unable to fulfill what would usually be its goals](https://arbital.com/p/7g2).)
-  If we try to reprogram the AI's utility function or [meta-utility function](https://arbital.com/p/meta_utility), a corrigible AI will allow this modification to go through.  (Rather than, e.g., fooling us into believing the utility function was modified successfully, while the AI actually keeps its original utility function as [obscured](https://arbital.com/p/3cq) functionality; as we would expect by default to be [a preferred outcome according to the AI's current preferences](https://arbital.com/p/3r6).)

More abstractly:

- A corrigible agent experiences no preference or [instrumental pressure](https://arbital.com/p/10k) to interfere with attempts by the programmers or operators to modify the agent, impede its operation, or halt its execution.
- A corrigible agent does not attempt to manipulate or deceive its operators, especially with respect to properties of the agent that might otherwise cause its operators to modify it.
- A corrigible agent does not try to [obscure its thought processes](https://arbital.com/p/3cq) from its programmers or operators.
- A corrigible agent is motivated to preserve the corrigibility of the larger system if that agent self-modifies, constructs sub-agents in the environment, or offloads part of its cognitive processing to external systems; or alternatively, the agent has no preference to execute any of those general activities.

A stronger form of corrigibility would require the AI to positively cooperate or assist, such that the AI would rebuild the shutdown button if it were destroyed, or experience a positive preference *not* to self-modify if self-modification could lead to incorrigibility.  But this is not part of the primary specification since it's possible that we would *not* want the AI trying to actively be helpful in assisting our attempts to shut it down, and would in fact prefer the AI to be passive about this.

Good proposals for achieving corrigibility in specific regards are [open problems in AI alignment](https://arbital.com/p/4m).  Some areas of active current research are [https://arbital.com/p/1b7](https://arbital.com/p/1b7) and [https://arbital.com/p/interruptibility](https://arbital.com/p/interruptibility).

Achieving total corrigibility everywhere via some single, general mental state in which the AI "knows that it is still under construction" or "believes that the programmers know more than it does about its own goals" is termed '[the hard problem of corrigibility](https://arbital.com/p/3ps)'.

## Difficulties

### Deception and manipulation by default

By default, most sets of preferences are such that an agent acting according to those preferences will prefer to retain its current preferences. For example, imagine an agent which is attempting to collect stamps. Altering the agent so that it prefers to collect bottle caps would lead to futures where the agent has fewer stamps, and so allowing this event to occur is dispreferred (under the current, stamp-collecting preferences).

More generally, as noted by [instrumentally convergent strategies](https://arbital.com/p/10g), most utility functions give an agent strong incentives to retain its current utility function: imagine an agent constructed so that it acts according to the utility function U, and imagine further that its operators think they built the agent to act according to a different utility function U'. If the agent learns this fact, then it has incentives to either deceive its programmers (prevent them from noticing that the agent is acting according to U instead of U') or manipulate its programmers (into believing that they actually prefer U to U', or by coercing them into leaving its utility function intact).

A corrigible agent must avoid these default incentives to manipulate and deceive, but specifying some set of preferences that avoids deception/manipulation incentives remains an open problem.

### Trouble with utility function uncertainty

A first attempt at describing a corrigible agent might involve specifying a utility maximizing agent that is uncertain about its utility function. However, while this could allow the agent to make some changes to its preferences as a result of observations, the agent would still be incorrigible when it came time for the programmers to attempt to correct what they see as mistakes in their attempts to formulate how the "correct" utility function should be determined from interaction with the environment.

As an overly simplistic example, imagine an agent attempting to maximize the internal happiness of all humans, but which has uncertainty about what that means. The operators might believe that if the agent does not act as intended, they can simply express their dissatisfaction and cause it to update. However, if the agent is reasoning according to an impoverished hypothesis space of utility functions, then it may behave quite incorrigibly: say it has narrowed down its consideration to two different hypotheses, one being that a certain type of opiate causes humans to experience maximal pleasure, and the other is that a certain type of stimulant causes humans to experience maximal pleasure. If the agent begins administering opiates to humans, and the humans resist, then the agent may "update" and start administering stimulants instead. But the agent would still be incorrigible — it would resist attempts by the programmers to turn it off so that it stops drugging people.

It does not seem that corrigibility can be trivially solved by specifying agents with uncertainty about their utility function. A corrigible agent must somehow also be able to reason about the fact that the humans themselves might have been confused or incorrect when specifying the process by which the utility function is identified, and so on.

### Trouble with penalty terms

A second attempt at describing a corrigible agent might attempt to specify a utility function with "penalty terms" for bad behavior. This is unlikely to work for a number of reasons. First, there is the [https://arbital.com/p/42](https://arbital.com/p/42) problem: if a utility function gives an agent strong incentives to manipulate its operators, then adding a penalty for "manipulation" to the utility function will tend to give the agent strong incentives to cause its operators to do what it would have manipulated them to do, without taking any action that technically triggers the "manipulation" cause.  It is likely extremely difficult to specify conditions for "deception" and "manipulation" that actually rule out all undesirable behavior, especially if the agent is [smarter than us](https://arbital.com/p/47) or [growing in capability](https://arbital.com/p/6q).

More generally, it does not seem like a good policy to construct an agent that searches for positive-utility ways to deceive and manipulate the programmers, [even if those searches are expected to fail](https://arbital.com/p/7g0). The goal of corrigibility is *not* to design agents that want to deceive but can't. Rather, the goal is to construct agents that have no incentives to deceive or manipulate in the first place: a corrigible agent is one that reasons as if it is incomplete and potentially flawed in dangerous ways.

## Open problems

Some open problems in corrigibility are:

### Hard problem of corrigibility

On a human, intuitive level, it seems like there's a central idea behind corrigibility that seems simple to us: understand that you're flawed, that your meta-processes might also be flawed, and that there's another cognitive system over there (the programmer) that's less flawed, so you should let that cognitive system correct you even if that doesn't seem like the first-order right thing to do.  You shouldn't disassemble that other cognitive system to update your model in a Bayesian fashion on all possible information that other cognitive system contains; you shouldn't model how that other cognitive system might optimally correct you and then carry out the correction yourself; you should just let that other cognitive system modify you, without attempting to manipulate how it modifies you to be a better form of 'correction'.

Formalizing the hard problem of corrigibility seems like it might be a problem that is hard (hence the name).  Preliminary research might talk about some obvious ways that we could model A as believing that B has some form of information that A's preference framework designates as important, and showing what these algorithms actually do and how they fail to solve the hard problem of corrigibility.

### [Utility indifference](https://arbital.com/p/1b7)

[explain utility indifference](https://arbital.com/p/fixme:)

The current state of technology on this is that the AI behaves as if there's an absolutely fixed probability of the shutdown button being pressed, and therefore doesn't try to modify this probability.  But then the AI will try to use the shutdown button as an outcome pump.  Is there any way to avert this?

### Percentalization

Doing something in the top 0.1% of all actions.  This is actually a Limited AI paradigm and ought to go there, not under Corrigibility.

### Conservative strategies

Do something that's as similar as possible to other outcomes and strategies that have been whitelisted.  Also actually a Limited AI paradigm.

This seems like something that could be investigate in practice on e.g. a chess program.

### Low impact measure

(Also really a Limited AI paradigm.)

Figure out a measure of 'impact' or 'side effects' such that if you tell the AI to paint all cars pink, it just paints all cars pink, and doesn't transform Jupiter into a computer to figure out how to paint all cars pink, and doesn't dump toxic runoff from the paint into groundwater; and *also* doesn't create utility fog to make it look to people like the cars *haven't* been painted pink (in order to minimize this 'side effect' of painting the cars pink), and doesn't let the car-painting machines run wild afterward in order to minimize its own actions on the car-painting machines.  Roughly, try to actually formalize the notion of "Just paint the cars pink with a minimum of side effects, dammit."

It seems likely that this problem could turn out to be FAI-complete, if for example "Cure cancer, but then it's okay if that causes human research investment into curing cancer to decrease" is only distinguishable by us as an okay side effect because it doesn't result in expected utility decrease under our own desires.

It still seems like it might be good to, e.g., try to define "low side effect" or "low impact" inside the context of a generic Dynamic Bayes Net, and see if maybe we can find something after all that yields our intuitively desired behavior or helps to get closer to it.

### Ambiguity identification

When there's more than one thing the user could have meant, ask the user rather than optimizing the mixture.  Even if A is in some sense a 'simpler' concept to classify the data than B, notice if B is also a 'very plausible' way to classify the data, and ask the user if they meant A or B.  The goal here is to, in the classic 'tank classifier' problem where the tanks were photographed in lower-level illumination than the non-tanks, have something that asks the user, "Did you mean to detect tanks or low light or 'tanks and low light' or what?"

### Safe outcome prediction and description

Communicate the AI's predicted result of some action to the user, without putting the user inside an unshielded argmax of maximally effective communication.

### Competence aversion

To build e.g. a [behaviorist genie](https://arbital.com/p/102), we need to have the AI e.g. not experience an instrumental incentive to get better at modeling minds, or refer mind-modeling problems to subagents, etcetera.  The general subproblem might be 'averting the instrumental pressure to become good at modeling a particular aspect of reality'.  A toy problem might be an AI that in general wants to get the gold in a Wumpus problem, but doesn't experience an instrumental pressure to know the state of the upper-right-hand-corner cell in particular.

Matthew Graves
Tsvi BT
Alexei Andreev
Nate Soares
Eric Bruylant
Eliezer Yudkowsky

Corrigible agents allow themselves to be 'corrected' (from our standpoint) by human [programmers](https://arbital.com/p/9r), and don't experience [instrumental pressures](https://arbital.com/p/10k) to avoid correction.

Imagine building an [advanced AI](https://arbital.com/p/7g1) with a [shutdown button](https://arbital.com/p/2xd) that causes the AI to suspend to disk in an orderly fashion if the shutdown button is pressed.  An AI that is *corrigible* with respect to this shutdown button is an AI that doesn't try to prevent the shutdown button from being pressed... or rewrite itself without the shutdown code, or build a backup copy of itself elsewhere, or psychologically manipulate the programmers into not pressing the button, or fool the programmers into thinking the AI has shut down when it has not, etcetera.

Open subproblems in aligning a Task-based AGI


'Detrimental'

The opposite of [https://arbital.com/p/-3d9](https://arbital.com/p/-3d9).  A [reserved term](https://arbital.com/p/9p) in [AGI alignment theory](https://arbital.com/p/2v) that acts as a speaker-dependent variable denoting whatever the speaker means by "bad" or "no, really actually bad" or "bad in the long-run", from within whatever their view is on [where the future ought to go](https://arbital.com/p/55).  See the entry for [https://arbital.com/p/55](https://arbital.com/p/55).


Unforeseen maximum

An unforeseen maximum of a [utility function](https://arbital.com/p/109) (or other [preference framework](https://arbital.com/p/5f)) is when, e.g., you tell the AI to produce smiles, thinking that the AI will make people happy in order to produce smiles.  But unforeseen by you, the AI has an alternative for making even more smiles, which is to convert all matter within reach into tiny molecular smileyfaces.

In other words, you're proposing to give the AI a goal $U$, because you think $U$ has a maximum around some nice options $X.$  But it turns out there's another option $X'$ you didn't imagine, with $X' >_U X,$ and $X'$ is not so nice.

Unforeseen maximums are argued to be a [foreseeable difficulty](https://arbital.com/p/6r) of [AGI alignment](https://arbital.com/p/2v), if you try to [identify](https://arbital.com/p/6c) nice policies by giving a simple criterion $U$ that, so far as you can see, seems like it'd be best optimized by doing nice things.

Slightly more semiformally, we could say that "unforeseen maximum" is realized as a difficulty when:

1. A programmer thinking about a utility function $U$ considers policy options $\pi_i \in \Pi_N$ and concludes that of these options the policy with highest $\mathbb E [U | \pi_i ](https://arbital.com/p/)$ is $\pi_1,$ and hence a $U$-maximizer will probably do $\pi_1.$
2.  The programmer also thinks that their own [criterion of goodness](https://arbital.com/p/55) $V$ will be promoted by $\pi_1,$ that is, $\mathbb E [V | \pi_1 ](https://arbital.com/p/) > \mathbb E [V ](https://arbital.com/p/)$ or "$\pi_1$ is [beneficial](https://arbital.com/p/3d9)".  So the programmer concludes that it's a great idea to build an AI that optimizes for $U.$
3.  Alas, the AI is searching a policy space $\Pi_M,$ which although it does contain $\pi_1$ as an option, also contains an attainable option $\pi_0$ which programmer didn't consider, with $\mathbb E [U | \pi_0 ](https://arbital.com/p/) > \mathbb E [U | \pi_1 ](https://arbital.com/p/).$  This is a *problem* if $\pi_0$ produces much less $V$-benefit than $\pi_1$ or is outright [detrimental](https://arbital.com/p/3d9).

That is:

$$\underset{\pi_i \in \Pi_N}{\operatorname {argmax}} \ \mathbb E [U | \pi_i ](https://arbital.com/p/) = \pi_1$$

$$\underset{\pi_k \in \Pi_M}{\operatorname {argmax}} \ \mathbb E [U | \pi_k ](https://arbital.com/p/) = \pi_0$$

$$\mathbb E [V | \pi_0 ](https://arbital.com/p/) \ll \mathbb E [V | \pi_1 ](https://arbital.com/p/)$$

# Example: Schmidhuber's compression goal.

Juergen Schmidhuber of IDSIA, during the 2009 Singularity Summit, [gave a talk](https://vimeo.com/7441291) proposing that the best and most moral utility function for an AI was the gain in compression of sensory data over time. Schmidhuber gave examples of valuable behaviors he thought this would motivate, like doing science and understanding the universe, or the construction of art and highly aesthetic objects.

Yudkowsky in Q&A suggested that this utility function would instead motivate the construction of external objects that would internally generate random cryptographic secrets, encrypt highly regular streams of 1s and 0s, and then reveal the cryptographic secrets to the AI.

Translating into the above schema:

1.  Schmidhuber, considering the utility function $U$ of "maximize gain in sensory compression", thought that option $\pi_1$ of "do art and science" would be the attainable maximum of $U$ within all options $\Pi_N$ that Schmidhuber considered.
2.  Schmidhuber also considered the option $\pi_1$ "do art and science" to achieve most of the attainable value under his own criterion of goodness $V$.
3.  However, while the AI's option space $\Pi_M$ would indeed include $\pi_1$ as an option, it would also include the option $\pi_0$ of "have an environmental object encrypt streams of 1s or 0s and then reveal the key" which would score much higher under $U$, and much lower under $V.$

# Relation to other foreseeable difficulties

[https://arbital.com/p/6q](https://arbital.com/p/6q) implies an unforeseen maximum may come as a surprise, or not show up during the [development phase](https://arbital.com/p/5d), because during the development phase the AI's options are restricted to some $\Pi_L \subset \Pi_M$ with $\pi_0 \not\in \Pi_L.$

Indeed, the pseudo-formalization of a "type-1 [https://arbital.com/p/-6q](https://arbital.com/p/-6q)" is isomorphic to the pseudoformalization of "unforeseen maximum", except that in a [https://arbital.com/p/-6q](https://arbital.com/p/-6q), $\Pi_N$ and $\Pi_M$ are identified with "AI's options during development" and "AI's options after a capability gain".  (Instead of "Options the programmer is thinking of" and "Options the AI will consider".)

The two concepts are conceptually distinct because, e.g:

- A [https://arbital.com/p/-6q](https://arbital.com/p/-6q) could also apply to a decision criterion learned by training, not just a utility function envisioned by the programmer.
- It's an unforeseen maximum but not a [https://arbital.com/p/-6q](https://arbital.com/p/-6q) if the programmer is initially reasoning, *not* that the AI has already been observed to be [beneficial](https://arbital.com/p/3d9) during a development phase, but rather that the AI *ought* to be [beneficial](https://arbital.com/p/3d9) when it optimizes $U$ later because of the supposed nice maximum at $\pi_1$.

If we hadn't observed what seem like clear-cut cases of some actors in the field being blindsided by unforeseen maxima in imagination, we'd worry less about actors being blindsided by [https://arbital.com/p/-6q](https://arbital.com/p/-6q)s over observations.

[https://arbital.com/p/2w](https://arbital.com/p/2w) suggests that the real maxima of non-$V$ utility functions will be "strange, weird, and extreme" relative to our own $V$-views on preferable options.

[Missing the weird alternative](https://arbital.com/p/43g) suggests that people may [psychologically](https://arbital.com/p/43h) fail to consider alternative agent options $\pi_0$ that are very low in $V,$ because the human search function looks for high-$V$ and normal policies.  In other words, that Schmidhuber didn't generate "encrypt streams of 1s or 0s and then reveal the key" *because* this policy was less attractive to him than "do art and science" and *because* it was weird.

[https://arbital.com/p/42](https://arbital.com/p/42) suggests that if you try to add a penalty term to exclude $\pi_0$, the next-highest $U$-ranking option will often be some similar alternative $\pi_{0.01}$ which still isn't nice.

[https://arbital.com/p/fragile_value](https://arbital.com/p/fragile_value) asserts that our [true criterion of goodness](https://arbital.com/p/55) $V$ is narrowly peaked within the space of all achievable outcomes for a [superintelligence](https://arbital.com/p/41l), such that we rapidly fall off in $V$ as we move away from the peak.  [https://arbital.com/p/5l](https://arbital.com/p/5l) says that $V$ and its corresponding peak have high [algorithmic complexity](https://arbital.com/p/5v).  Then the peak outcomes identified by any simple [object-level](https://arbital.com/p/5t) $U$ will systematically fail to find $V$.  It's like trying to find a 1000-byte program which will approximately reproduce the text of Shakespeare's *Hamlet;* algorithmic information theory says that you just shouldn't expect to find a simple program like that.

[https://arbital.com/p/apple_pie_problem](https://arbital.com/p/apple_pie_problem) raises the concern that some people may have [psychological](https://arbital.com/p/43h) trouble accepting the "But $\pi_0$" critique even after it is pointed out, because of their ideological attachment to a noble goal $U$ (probably actually noble!) that would be even more praiseworthy if $U$ could also serve as a complete utility function for an AGI (which it unfortunately can't).

# Implications and research avenues

[Conservatism](https://arbital.com/p/2qp) in goal concepts can be seen as trying to directly tackle the problem of unforeseen maxima.  More generally, AI approaches which work on "whitelisting conservative boundaries around approved policy spaces" instead of "search the widest possible policy space, minus some blacklisted parts".

The [Task](https://arbital.com/p/6w) paradigm for [advanced agents](https://arbital.com/p/2c) concentrates on trying to accomplish some single [pivotal act](https://arbital.com/p/6y) which can be accomplished by one or more [tasks](https://arbital.com/p/6w) of limited scope.  [Combined with other measures,](https://arbital.com/p/43w) this might make it easier to identify an adequate safe plan for accomplishing the limited-scope task, rather than needing to identify the fragile peak of $V$ within some much larger landscape.  The Task AGI formulation is claimed to let us partially "narrow down" the scope of the necessary $U$, the part of $V$ that's relevant to the task, and the searched policy space $\Pi$ to what is only adequate.  This might reduce or meliorate, though not by itself eliminate, unforeseen maxima.

[Mild optimization](https://arbital.com/p/2r8) can be seen as "not trying so hard, not shoving all the way to the maximum" - the hope is that *when combined* with a [Task](https://arbital.com/p/6w) paradigm plus other measures like [conservative goals and strategies](https://arbital.com/p/2qp), this will produce less optimization pressure toward weird edges and unforeseen maxima.  (This method is not adequate on its own because an arbitrary adequate-$U$ policy may still not be high-$V$, ceteris paribus.)

[Imitation-based agents](https://arbital.com/p/2sj) try to maximize similarity to a reference human's immediate behavior, rather than trying to optimize a utility function.

The prospect of being tripped up by unforeseen maxima, is one of the contributing motivations for giving up on [hand-coded object-level utilities](https://arbital.com/p/5t) in favor of meta-level [preference frameworks](https://arbital.com/p/5f) that learn a utility function or decision rule.  (Again, this doesn't seem like a full solution by itself, [only one ingredient to be combined with other methods](https://arbital.com/p/43w).  If the utility function is a big complicated learned object, that by itself is not a good reason to relax about the possibility that its maximum will be somewhere you didn't foresee, especially after a [capabilities boost](https://arbital.com/p/6q).)

[Missing the weird alternative](https://arbital.com/p/43g) and the [https://arbital.com/p/apple_pie_problem](https://arbital.com/p/apple_pie_problem) suggest that it may be unusually difficult to explain to actors why $\pi_0 >_U \pi_1$ is a difficulty of their favored utility function $U$ that allegedly implies nice policy $\pi_1.$  That is, for [psychological reasons](https://arbital.com/p/43h), this difficulty seems unusually likely to actually trip up sponsors of AI projects or politically block progress on alignment.

Eric Rogstad
Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev

Unforeseen maximums occur when a powerful cognitive process, given a fitness function $F$ that you thought had a maximum somewhere around $X$, finds an even higher $F$-scoring solution $X'$ that was outside the space of possible solutions you considered.  (Example:  A [ programmer](https://arbital.com/p/9r) considers possible ways of producing smiles by producing human happiness, and the highest-scoring smile-producing strategies in this part of the solution space look quite nice to them.  They give a neutral genie the instruction to produce smiles.  The neutral genie tiles its future light cone with tiny molecular smiley-faces.  This was not something the programmer had explicitly considered as a possible way of producing smiles.)


Patch resistance

A proposed [foreseeable difficulty](https://arbital.com/p/6r) of [aligning advanced agents](https://arbital.com/p/2v) is furthermore proposed to be "patch-resistant" if the speaker thinks that most simple or naive solutions will fail to resolve the difficulty and just regenerate it somewhere else.

To call a problem "patch-resistant" is not to assert that it is unsolvable, but it does mean the speaker is cautioning against naive or simple solutions.

On most occasions so far, alleged cases of patch-resistance are said to stem from one of two central sources:

- The difficulty arises from a [convergent instrumental strategy](https://arbital.com/p/10g) executed by the AI, and simple patches aimed at blocking one observed bad behavior will not stop [a very similar behavior](https://arbital.com/p/42) from popping up somewhere else.
- The difficulty arises because the desired behavior has [high algorithmic complexity](https://arbital.com/p/5l) and simple attempts to pinpoint beneficial behavior are doomed to fail.

## Instrumental-convergence patch-resistance

Example:  Suppose you want your AI to have a [shutdown button](https://arbital.com/p/2xd):

- You first try to achieve this by writing a shutdown function into the AI's code.
- After the AI becomes self-modifying, it deletes the code because it is ([convergently](https://arbital.com/p/10g)) the case that the AI can accomplish its goals better by not being shut down.
- You add a patch to the utility function giving the AI minus a million points if the AI deletes the shutdown function or prevents it from operating.
- The AI responds by writing a new function that reboots the AI after the shutdown completes, thus technically not preventing the shutdown.
- You respond by again patching the AI's utility function to give the AI minus a million points if it continues operating after the shutdown.
- The AI builds an environmental subagent that will accomplish the AI's goals while the AI itself is technically "shut down".

This is the first sort of patch resistance, the sort alleged to arise from attempts to defeat an [instrumental convergence](https://arbital.com/p/10g) with simple patches meant to get rid of one observed kind of bad behavior.  After one course of action is blocked by a specific obstacle, [the next-best course of action remaining is liable to be highly similar to the one that was just blocked](https://arbital.com/p/42).

## Complexity-of-value patch-resistance

Example:

- You want your AI to accomplish good in the world, which is presently highly correlated with making people happy.  Happiness is presently highly correlated with smiling.  You build an AI that [tries to achieve more smiling](https://arbital.com/p/10d).
- After the AI proposes to force people to smile by attaching metal pins to their lips, you realize that this current empirical association of smiling and happiness doesn't mean that *maximum* smiling must occur in the presence of *maximum* happiness.
- Although it's much more complicated to infer, you try to reconfigure the AI's utility function to be about a certain class of brain states that has previously in practice produced smiles.
- The AI successfully generalizes the concept of pleasure, and begins proposing policies to give people heroin.
- You try to add a patch excluding artificial drugs.
- The AI proposes a genetic modification producing high levels of endogenous opiates.
- You try to explain that what's really important is not forcing the brain to experience pleasure, but rather, people experiencing events that naturally cause happiness.
- The AI proposes to put everyone in the Matrix...

Since the programmer-[intended](https://arbital.com/p/6h) concept is [actually highly complicated](https://arbital.com/p/5l), simple concepts will systematically fail to have their optimum at the same point as the complex intended concept.  By the [fragility of value](https://arbital.com/p/fragility_of_value), the optimum of the simple concept will almost certainly not be a *high* point of the complex intended concept.  Since [most concepts are *not* surprisingly compressible](https://arbital.com/p/4v5), there probably *isn't* any simple concept whose maximum identifies that fragile peak of value.  This explains why we would reasonably expect problems of [perverse instantiation](https://arbital.com/p/2w) to pop up over and over again, the optimum of the revised concept moving to a new weird extreme each time the programmer tries to hammer down the next [weird alternative](https://arbital.com/p/43g) the AI comes up with.

In other words:  There's a large amount of [algorithmic information](https://arbital.com/p/5v) or many independent [reflectively consistent degrees of freedom](https://arbital.com/p/2fr) in the correct answer, the plans we *want* the AI to come up with, but we've only given the AI relatively simple concepts that can't [identify](https://arbital.com/p/2s3) those plans.

# Analogues in the history of AI

The result of trying to tackle overly [general](https://arbital.com/p/42g) problems using AI algorithms too narrow for those general problems, usually appears in the form of [an infinite number of special cases](http://lesswrong.com/lw/l9/artificial_addition/) with a new special case needing to be handled for every problem instance.  In the case of narrow AI algorithms tackling a general problem, this happens because the narrow algorithm, being narrow, is not capable of capturing the deep structure of the general problem and its solution.

Suppose that burglars, and also earthquakes, can cause burglar alarms to go off.  Today we can represent this kind of scenario using a Bayesian network or causal model which will *compactly* yield probabilistic inferences along the lines of, "If the burglar alarm goes off, that probably indicates there's a burglar, unless you learn there was an earthquake, in which case there's probably not a burglar" and "If there's an earthquake, the burglar alarm probably goes off."

During the era where everything in AI was being represented by first-order logic and nobody knew about causal models, [people devised increasingly intricate "nonmonotonic logics"](https://arbital.com/p/) to try to represent inference rules like (simultaneously) $alarm \rightarrow burglar, \ earthquake \rightarrow alarm,$ and $(alarm \wedge earthquake) \rightarrow \neg burglar.$  But first-order logic wasn't naturally a good surface fit to the set of inferences needed, and the AI programmers didn't know how to compactly capture the structure that causal models capture.  So the "nonmonotonic logic" approach proliferated an endless nightmare of special cases.

Cognitive problems like "modeling causal phenomena" or "being good at math" (aka understanding which mathematical premises imply which mathematical conclusions) might be *general* enough to defeat modern narrow-AI algorithms.  But these domains still seem like they should have something like a central core, leading us to expect [correlated coverage](https://arbital.com/p/correlated_covereage) of the domain in [sufficiently advanced](https://arbital.com/p/2c) agents.  You can't conclude that because a system is very good at solving arithmetic problems, it will be good at proving Fermat's Last Theorem.  But if a system is smart enough to independently prove Fermat's Last Theorem *and* the Poincare Conjecture *and* the independence of the Axiom of Choice in Zermelo-Frankel set theory, it can probably also - without further handholding - figure out Godel's Theorem.  You don't need to *go on* programming in one special case after another of mathematical competency.  The fact that humans could figure out all these different areas, without needing to be independently reprogrammed for each one by natural selection, says that there's something like a central tendency underlying competency in all these areas.

In the case of [complexity of value](https://arbital.com/p/5l), the thesis is that there are many independent [reflectively consistent degrees of freedom](https://arbital.com/p/2fr) in our [intended](https://arbital.com/p/6h) specification of what's [good, bad, or best](https://arbital.com/p/55).  Getting one degree of freedom aligned with our intended result doesn't mean that other degrees of freedom need to align with our intended result.  So trying to "patch" the first simple specification that doesn't work, is likely to result in a different specification that doesn't work.

When we try to use a narrow AI algorithm to attack a problem which has a central tendency *requiring general intelligence to capture,* or at any rate requiring some new structure that the narrow AI algorithm can't handle, we're effectively asking the narrow AI algorithm to learn something that has no simple structure *relative to* that algorithm.  This is why early AI researchers' experience with "lack of common sense" *that you can't patch with special cases* may be [foreseeably](https://arbital.com/p/6r) indicative of how frustrating it would be, in practice, to repeatedly try to "patch" a kind of difficulty that we may foreseeably need to confront in aligning AI.

That is:  Whenever it feels to a human like you want to yell at the AI for its lack of "common sense", you're probably looking at a domain where trying to patch that particular AI answer is just going to lead into another answer that lacks "common sense".  Previously in AI history, this happened because real-world problems had no simple central learnable solution relative to the narrow AI algorithm.  In value alignment, something similar could happen because of the [complexity of our value function](https://arbital.com/p/5l), whose evaluations *also* [feel to a human](https://arbital.com/p/4v2) like "common sense".

# Relevance to alignment theory

Patch resistance, and its sister issue of lack of [correlated coverage](https://arbital.com/p/1d6), is a central reason why aligning advanced agents could be way harder, way more dangerous, and way more likely to actually kill everyone in practice, compared to optimistic scenarios.  It's a primary reason to worry, "Uh, what if *aligning* AI is actually way harder than it might look to some people, the way that *building AGI in the first place* turned out not to be something you could do in two months over the summer?"

It's also a reason to worry about [context disasters](https://arbital.com/p/6q) revolving around capability gains:  Anything you had to patch-until-it-worked at AI capability level $k$ is probably going to break *hard* at capability $l \gg k.$  This is doubly catastrophic in practice if the pressures to "just get the thing running today" are immense.

To the extent that we can see the central project of AI alignment as revolving around finding a set of alignment ideas that *do* have simple central tendencies and *are* specifiable or learnable which together add up to a safe but powerful AI - that is, finding domains with correlated coverage that add up to a safe AI that can do something pivotal - we could see the central project of AI alignment as finding a collectively good-enough set of safety-things we can do *without* endless patching.


Needs work

Meta tag for pages which need content improvement.

Eric Rogstad
Eric Bruylant
Eliezer Yudkowsky


Time-machine metaphor for efficient agents

The time-machine metaphor is an [intuition pump](https://arbital.com/p/4kf) for [instrumentally efficient agents](https://arbital.com/p/6s) - agents smart enough that they always get at least as much of their [utility](https://arbital.com/p/109) as any strategy we can imagine.  Taking a [superintelligent](https://arbital.com/p/41l) [paperclip maximizer](https://arbital.com/p/10h) as our central example, rather than visualizing a brooding mind and spirit deciding which actions to output in order to achieve its paperclippy desires, we should consider visualizing a time machine hooked up to an output channel, which, for every possible output, peeks at how many paperclips will result in the future given that output, and which always yields the output leading to the greatest number of paperclips.  In the metaphor, we're not to imagine the time machine as having any sort of mind or intentions - it just repeatedly outputs the action that actually leads to the most paperclips.

The time machine metaphor isn't a perfect way to visualize a [bounded](https://arbital.com/p/2rd) superintelligence; the time machine is strictly more powerful.  E.g., the time machine can instantly unravel a 4096-bit encryption key because it 'knows' the bitstring that is the answer.  So the point of this metaphor is not as an intuition pump for capabilities, but rather, an intuition pump for overcoming [https://arbital.com/p/-anthropomorphism](https://arbital.com/p/-anthropomorphism) in reasoning about a paperclip maximizer's policies; or as an intuition pump for understanding the sense-update-predict-act agent architecture.

That is:  If you imagine a superintelligent paperclip maximizer as a mind, you might [imagine persuading it](https://arbital.com/p/43g) that, really, [it can get more paperclips](https://arbital.com/p/3tm) by trading with humans instead of turning them into paperclips.  If you imagine a time machine, which isn't a mind, you're less likely to imagine persuading it, and instead ask more honestly the question, "What is the maximum number of paperclips the universe can be turned into, and how would one go about doing that?"  Instead of imagining ourselves arguing with Clippy about how humans really are very productive, we ask the question from the time machine's standpoint - which universe actually ends up with more paperclips in it?

The relevant fact about instrumentally efficient agents is that they are, from our perspective, *unbiased* (in the [statistical sense of bias](https://arbital.com/p/statistical_bias)) in their policies, relative to any kind of bias we can detect.

As an example, consider a 2015-era chess engine, contrasted to a 1985-era chess engine.  The 1985-era chess engine may lose to a moderately strong human amateur, so it's not [relatively efficient](https://arbital.com/p/6s).  It may have humanly-perceivable quirks such as "It likes to move its queen", that is, "I detect that it moves its queen more often than would be strictly required to win the game."  As we go from 1985 to 2015, the machine chessplayer improves beyond the point where we, personally, can detect any flaws in it.  You should expect the reason why the 2015 chess engine moves anywhere to be only understandable to *you* (without machine assistance) as "because that move had a great probability of leading to a winning position later", and not in any other psychological terms like "it likes to move its pawn".

From your perspective, the 2015 chess engine will only move its pawn on occasions where that probably leads to winning the game, and does not move the pawn on occasions where it leads to losing the game. If you see the 2015 chess engine make a move you didn't think was high in winningness, you conclude that it has seen some winningness you didn't know about and is about to do exceptionally well, or you conclude that the move you favored led into futures surprisingly low in winningness, and not that the chess engine is favoring some unwinning move.  We can no longer personally and without machine assistance detect any systematic departure from "It makes the chess move that leads to winning the game" in the direction of "It favors some other class of chess move for reasons apart from its winningness."

This is what makes the time machine metaphor a good intuition pump for an instrumentally efficient agent's choice of policies (though not a good intuition for the magnitude of its capabilities).


Coordinative AI development hypothetical

A simplified/easier hypothetical form of the [known algorithm nonrecursive](https://arbital.com/p/) path within the [https://arbital.com/p/2z](https://arbital.com/p/2z).  Suppose there was an effective world government with effective monitoring of all computers; or that for whatever other imaginary reason rogue AI development projects were simply not a problem. What would the ideal research trajectory for that world look like?

### Usefulness:

- Highlight / flag where safety shortcuts are being taken because we live in the non-ideal case.
- Let us think through what a maximally safe development pathway would look like, and why, without stopping every 30 seconds to think about how we won't have time.  This may uncover valuable research paths that could, on a second glance, be done more quickly.
- Think through a simpler case of a research-program-generator that has fewer desiderata and hence less cognitive distractions.

Eric Bruylant
Eliezer Yudkowsky
Janos Kramar
Alexei Andreev


Safe impact measure

A safe impact measure is one that captures all changes to every variable a human might care about, with no edge-cases where a lot of value could be destroyed by a 'low impact' action.  A safe impact measure must also not generate so many false alarms of 'high impact' that no strategy can be distinguished as 'low impact'.

### Importance / uses

A safe impact measure is an open problem of value alignment, which if solved, might be used in constructing:

- A [low-impact AI](https://arbital.com/p/), a subspecies of [Genie](https://arbital.com/p/6w) that tries to carry out its goals while otherwise minimizing the number of important things that it affects.
- An [Oracle](https://arbital.com/p/6x) that tries not to affect the world outside its box at all, apart from its output.
- A [shutdown utility function](https://arbital.com/p/) that incentives a [corrigible](https://arbital.com/p/45) AI to halt safely (perhaps after a button is pressed).

Some uses of a safe impact measure postulate that the impact measure has a 'hole' or some form of permitted output channel.  For example, one Stuart Armstrong proposal involves an Oracle in a sealed box, with a single output channel that is connected to the Oracle with high quantum probability, and connected to a quantum noise generator with low quantum probability.  The Oracle is putatively given the goal of generating an output signal with some informational property, and otherwise minimizing the 'impact' relative to the same output having been generated by the quantum noise generator instead of the Oracle.  This is intended to capture the idea that the Oracle's effect on the world should only occur via the effect of the output message, and not take advantage of any side channels such as managing to send a radio signal outside the Oracle box.

### Difficulty

To be used inside an [advanced agent](https://arbital.com/p/2c), an impact measure must be [safe](https://arbital.com/p/2l) in the face of whatever cognitive pressures and optimization pressures might tend to produce [edge instantiations](https://arbital.com/p/2w) or [https://arbital.com/p/42](https://arbital.com/p/42) - it must capture so much variance that there is *no* clever strategy whereby an advanced agent can produce some special type of variance that evades the measure.  Ideally, the measure will pass the [Omni Test](https://arbital.com/p/), meaning that even if it suddenly gained perfect control over every particle in the universe, there would still be no way for it to have what intuitively seems like a 'large influence' on the future, without that strategy being assessed as having a 'high impact'.

The reason why a safe impact measure might be possible, and specifiable to an AI without having to solve the entire [value learning problem](https://arbital.com/p/) for [complex values](https://arbital.com/p/5l), is that it may be possible to upper-bound the value-laden and complex quantity 'impact on literally everything cared about' by some much simpler quantity that says roughly 'impact on everything' - all causal processes worth modeling on a macroscale, or something along those lines.

The challenge of a safe impact measure is that we can't just measure, e.g., 'number of particles influenced in any way' or 'expected shift in all particles in the universe'.  For the former case, consider that a one-gram mass on Earth exerts a gravitational pull that accelerates the Moon toward it at roughly 4 x 10^-31 m/s^2, and every sneeze has a *very* slight gravitational effect on the atoms in distant galaxies.  Since every decision qualitatively 'affects' everything in its future light cone, this measure will have too many false positives / not approve any strategy / not usefully discriminate unusually dangerous atoms.

For the proposed quantity 'expectation of the net shift produced on all atoms in the universe':  If the universe (including the Earth) contains at least one process chaotic enough to exhibit butterfly effects, then any sneeze anywhere ends up producing a very great expected shift in total motions.  Again we must worry that the impact measure, as evaluated inside the mind of a superintelligence, would just assign uniformly high values to every strategy, meaning that unusually dangerous actions would not be discriminated for alarms or vetos.

Despite the first imaginable proposals failing, it doesn't seem like a 'safe impact measure' necessarily has the type of [value-loading](https://arbital.com/p/) that would make it [VA-complete](https://arbital.com/p/).  One intuition pump for 'notice big effects in general' not being value-laden, is that if we imagine aliens with nonhuman decision systems trying to solve this problem, it seems easy to imagine that the aliens would come up with a safe impact measure that we would also regard as safe.

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev


No-Free-Lunch theorems are often irrelevant

There's a wide variety of "No Free Lunch" theorems proving that, in general, some problem is unsolvable.  Very often, these theorems are not relevant because the real universe is a special case.

In a very general and metaphorical sense, most of these theorems say some equivalent of, "There's no universal good strategy:  For every universe where an action causes you to gain \$5, there's a different universe where that same action causes you to lose \$5.  You can't get a *free lunch* across every universe; if you gain \$5 in one universe you must lose \$5 in another universe."  To which the reply is very often, "Sure, that's true across maximum-entropy universes in general, but we happen to live in a low-entropy universe where things are far more ordered and predictable than in a universe where all the atoms have been placed at random."

Similarly:  In theory, the [protein folding problem](https://en.wikipedia.org/wiki/Protein_folding) (predicting the lowest-energy configuration for a string of amino acids) is NP-hard ([sorta](https://arxiv.org/abs/1306.1372)).  But since quantum mechanics is not known to solve NP-hard problems, this just means that in the real world, some proteins fold up in a way that doesn't reach the ideal lowest energy.  Biology is happy with just picking out proteins that reliably fold up a particular way.  "NP-hard" problems in real life with nonrandom data often have regularities that make them much more easily solvable than the worst cases; or they have pretty good approximate solutions; or we can work with just the solvable cases.

A related but distinct idea:  It's impossible to prove in general whether a Turing machine halts, or [has any other nontrivial property](https://en.wikipedia.org/wiki/Rice%27s_theorem) since it might be conditioned on a subprocess halting.  But that often doesn't stop us from picking particular machines that do halt, or limiting our consideration to computations that run in less than a quadrillion timesteps, etcetera.

This doesn't mean *all* No-Free-Lunch theorems are irrelevant in the real-universe special case.  E.g., the Second Law of Thermodynamics can also be seen as a No-Free-Lunch theorem, and does actually prohibit perpetual motion in our own real universe (on the standard model of physics).

It should finally be observed that human intelligence does work in the real world, meaning that there's no No-Free-Lunch theorem which prohibits an intelligence from working at least that well.  Any claim that a No-Free-Lunch theorem prohibits machine intelligence in general, must definitely [Prove Too Much](https://arbital.com/p/3tc), because the same reasoning could be applied to a human brain considered as a physical system.

"No Free Lunch" theorems prove that across some problem class, it's impossible to do better in one case without doing worse in some other case.  They say some equivalent of, "There's no universal good strategy:  For every universe where an action causes you to gain \$5, there's a different universe where that same action causes you to lose \$5.  You can't get a *free lunch* across every universe; if you gain \$5 in one possible world you must lose \$5 in another possible world."

To this the realistic reply is very often, "Okay, that's true across maximum-entropy universes in general.  But the real world is an extremely special case.  We happen to live in a low-entropy universe, where things are far more ordered and predictable than in a universe where all the atoms have been placed at random; and the world where we gain \$5 is far more probable than the world where we lose \$5."

Similarly:  In theory, the ideal form of the protein folding problem is NP-hard.  In practice, biology is happy to select proteins that reliably fold up a particular way even if they don't reach the exact theoretical minimum.  "NP-hard" problems in real life with nonrandom data often have regularities that make them much more easily solvable than the worst-case problems; or they have pretty good approximate solutions.


Mind design space is wide

Imagine an enormous space of possible mind designs, within which all humans who've ever lived are a single tiny dot.  We all have the same cerebral cortex, cerebellum, thalamus, etcetera.  There's an instinct to imagine "Artificial Intelligences" as a kind of weird tribe that lives across the river and ask what peculiar customs this foreign tribe might have.  Really the word "Artificial Intelligence" just refers to the *entire* space of possibilities outside the tiny human dot.  So to most questions about AI, the answer may be, "It depends on the exact mind design of the AI."  By similar reasoning, a universal claim over all possible AIs is much more dubious than a claim about at least one AI.  If you imagine that in the vast space of all mind designs, there's at least a billion binary design choices that can be made, then, there's at least $2^{1,000,000,000}$ distinct mind designs.  We might say that any claim of the form, "Every possible mind design has property $P$" has $2^{1,000,000,000}$ chances to be false, while any claim of the form "There exists at least one mind design with property $P$" has $2^{1,000,000,000}$ chances to be true.  This doesn't preclude us from thinking about properties that *most* mind designs might have.  But it does suggest that if we don't like some property $P$ that seems likely to *usually* hold, we can maybe find some special case of a mind design which unusually has $P$ false.


Flag the load-bearing premises

If someone says, "I think your AI safety scheme is horribly flawed because X will go wrong," and you reply "Nah, X will be fine because of Y and Z", then good practice calls for highlighting that Y and Z are important propositions.  Y and Z may need to be debated in their own right, especially if a lot of people consider Y and Z to be nonobvious or probably false.  Contrast [https://arbital.com/p/emphemeral_premises](https://arbital.com/p/emphemeral_premises).  Needs to be paired with understanding of the [Multiple-Stage Fallacy](https://arbital.com/p/multiple_stage_fallacy) so that listing load-bearing premises doesn't make the proposition look less probable - $\neg X$ will have load-bearing premises too.


AI alignment open problem

A tag for pages that describe at least one major open problem that has been identified within the theory of [value-aligned advanced agents](https://arbital.com/p/2c), powerful artificial minds such that the effect of running them is good / nice / normatively positive ('[high value](https://arbital.com/p/55)').

To qualify as an 'open problem' for this tag, the problem should be relatively crisply stated, unsolved, and considered important.


Task (AI goal)

A "Task" is a goal or subgoal within an [advanced](https://arbital.com/p/2c) AI, that can be satisfied as fully as possible by optimizing a bounded part of space, for a limited time, with a limited amount of effort.

E.g., "make as many [paperclips](https://arbital.com/p/10h) as possible" is definitely not a 'task' in this sense, since it spans every paperclip anywhere in space and future time.  Creating more and more paperclips, using more and more effort, would be more and more preferable up to the maximum exertable effort.

For a more subtle example of non-taskishness, consider Disney's "sorcerer's apprentice" scenario:  Mickey Mouse commands a broomstick to fill a cauldron.  The broomstick then adds more and more water to the cauldron until the workshop is flooded.  (Mickey then tries to destroy the broomstick.  But since the broomstick has no [designed-in reflectively stable shutdown button](https://arbital.com/p/2xd), the broomstick repairs itself and begins constructing subagents that go on pouring more water into the cauldron.)

Since the Disney cartoon is a musical, we don't know if the broomstick was given a time bound on its job.  Let us suppose that Mickey tells the broomstick to do its job sometime before 1pm.

Then we might imagine that the broomstick is a subjective [expected utility](https://arbital.com/p/18r) maximizer with a utility function $U_{cauldron}$ over outcomes $o$:

$$U_{cauldron}(o):  \begin{cases}
1 & \text{if in $o$ the cauldron is $\geq 90\%$ full of water at 1pm} \\
0 & \text{otherwise}
\end{cases}$$

This *looks* at first glance like it ought to be taskish:

- The cauldron is bounded in space.
- The goal only concerns events that happen before a certain time.
- The highest utility that can be achieved is $1,$ which is reached as soon as the cauldron is $\geq 90\%$ full of water, which seems achievable using a limited amount of effort.

The last property in particular makes $U_{cauldron}$ a "satisficing utility function", one where an outcome is either satisfactory or not-satisfactory, and it is not possible to do any better than "satisfactory".

But by previous assumption, the broomstick is still optimizing *expected* utility.  Assume the broomstick reasons with [reasonable generality](https://arbital.com/p/42g) via some [universal prior](https://arbital.com/p/4mr).  Then the *subjective probability* of the cauldron being full, when it *looks* full to the broomstick-agent, [will not be](https://arbital.com/p/4mq) *exactly* $1.$  Perhaps (the broomstick-agent reasons) the broomstick's cameras are malfunctioning, or its RAM has malfunctioned producing an inaccurate memory.

Then the broomstick-agent reasons that it can further increase the probability of the cauldron being full - however slight the increase in probability - by going ahead and dumping in another bucket of water.

That is:  [Cromwell's Rule](https://arbital.com/p/4mq) implies that the subjective probability of the bucket being full never reaches exactly $1$.  Then there can be an infinite series of increasingly preferred, increasingly more effortful policies $\pi_1, \pi_2, \pi_3 \ldots$ with

$$\mathbb E [U_{cauldron} | \pi_1](https://arbital.com/p/) = 0.99\\
\mathbb E [U_{cauldron} | \pi_2](https://arbital.com/p/) = 0.999 \\
\mathbb E [U_{cauldron} | \pi_3](https://arbital.com/p/) = 0.999002 \\
\ldots$$

In that case the broomstick can always do better in expected utility (however slightly) by exerting even more effort, up to the maximum effort it can exert.  Hence the flooded workshop.

If on the other hand the broomstick is an *expected utility satisficer*, i.e., a policy is "acceptable" if it has $\mathbb E [U_{cauldron} | \pi ](https://arbital.com/p/) \geq 0.95,$ then this is now finally a taskish process (we think).  The broomstick can find some policy that's reasonably sure of filling up the cauldron, execute that policy, and then do no more.

As described, this broomstick doesn't yet have any [impact penalty](https://arbital.com/p/4l), or features for [mild optimization](https://arbital.com/p/2r8).  So the broomstick could *also* get $\geq 0.90$ expected utility by flooding the whole workshop; we haven't yet [forbidden excess efforts](https://arbital.com/p/2r8).  Similarly, the broomstick could also go on to destroy the world after 1pm - we haven't yet [forbidden excess impacts](https://arbital.com/p/4l).

But the underlying rule of "Execute a policy that fills the cauldron at least 90% full with at least 95% probability" does appear taskish, so far as we know.  It seems *possible* for an otherwise well-designed agent to execute this goal to the greatest achievable degree, by acting in bounded space, over a bounded time, with a limited amount of effort.  There does not appear to be a sequence of policies the agent would evaluate as better fulfilling its decision criterion, which use successively more and more effort.

The "taskness" of this goal, even assuming it was correctly [identified](https://arbital.com/p/36y), wouldn't by itself make the broomstick a fully taskish AGI.  We also have to consider whether every subprocess of the AI is similarly tasky; whether there is any subprocess anywhere in the AI that tries to improve memory efficiency 'as far as possible'.  But it would be a start, and make further safety features more feasible/useful.

See also [https://arbital.com/p/2r8](https://arbital.com/p/2r8) as an [open problem in AGI alignment](https://arbital.com/p/2mx).

A "Task" is a goal within an [AI](https://arbital.com/p/2c) that only covers a bounded amount of space and time, and can be satisfied by a limited amount of effort.

An example might be "fill this cauldron with water before 1pm"; but even there, we have to be careful.  "Maximize the probability that this cauldron contains water at 1pm" would imply unlimited effort, since slightly higher probabilities could be obtained by adding more and more effort.

"Carry out some policy such that there's at least a 95% chance that the cauldron is at least 90% full of water by 1pm" would be more task-ish.  A limited effort seems like definitely enough to do that, and then it can't be done any further by expending more effort.

See also [https://arbital.com/p/2pf](https://arbital.com/p/2pf), [https://arbital.com/p/2r8](https://arbital.com/p/2r8) and [https://arbital.com/p/6w](https://arbital.com/p/6w).


Distinguish which advanced-agent properties lead to the foreseeable difficulty

Any general project of producing a large edifice of good thinking should try to break down the ideas into modular pieces, distinguish premises from conclusions, and clearly label which reasoning steps are being used.  Applied to [AI alignment theory](https://arbital.com/p/2v), one of the things this suggests is that if you propose any sort of potentially difficult or dangerous future behavior from an AI, you should distinguish what particular kinds of advancement or cognitive intelligence are supposed to produce this difficulty.  In other words, supposed [foreseeable difficulties](https://arbital.com/p/6r) should come with proposed [advanced agent properties](https://arbital.com/p/2c) that match up to them.


Natural language understanding of "right" will yield normativity

This proposition is true if you can take a cognitively powerful agent that otherwise seems pretty competent at understanding natural language, and that has been previously trained out of infrahuman errors in understanding natural language, ask it to 'do the right thing' or 'do the right thing, defined the right way' and its natural language understanding of 'right' yields what we would intuitively see as normativity.

### Arguments

Natural categories have boundaries with low [algorithmic information](https://arbital.com/p/5v) relative to boundaries produced by a purely epistemic system with a simplicity prior.

'Unnatural' categories have value-laden boundaries.  Values have high  algorithmic information because of the [https://arbital.com/p/1y](https://arbital.com/p/1y) and [https://arbital.com/p/5l](https://arbital.com/p/5l).  Unnatural categories appear simple to us because we do dimensional reduction on value boundaries.  Things merely near to the boundaries of unnatural categories can fall off rapidly in value because of fragility.

There's an inductive problem where 18 things are important and only 17 of them vary between the positive and negative examples in the data.

[Edge instantiation](https://arbital.com/p/2w) makes this worse because it tends to seek out extreme cases.

The word 'right' involves a lot of what we call 'philosophical competence' in the sense that humans figuring it out will go through a lot of new cognitive use-paths ('unprecedented excursions') that they didn't traverse while disambiguating blue and green.  This also holds true when people are reflecting on how to figure out 'right'.  Example case of CDT vs. UDT.

This also matters because edge instantiation on the most 'right' as persuasively-right cases, will produce things that humans find superpersuasive (perhaps via shoving brains onto strange new pathways).  So we can't define right as that which would counterfactually cause a human model to agree that 'right' applies.

This keys into the inductive problem where variation must be shadowed in the data for the induced concept to cover it.

But if you had a complete predictive model of a human, it's then possible though not necessary that normative boundaries might be possible to induce by examples and asking to clarify ambiguities.


Work in progress

A meta tag for pages unfinished pages which an author is still making major changes to.

Erik Istre
Kevin Clancy
Alexei Andreev
Tsvi BT
Stephanie Zolayvar
Eric Rogstad
Patrick Stevens
Jeremy Perret
Nate Soares
Morgan Redding
Eric Bruylant
Mark Chimes
Jaime Sevilla Molina
Eliezer Yudkowsky
Jack Gallagher


Underestimating complexity of value because goodness feels like a simple property

One potential reason why people might tend to systematically underestimate the [complexity of value](https://arbital.com/p/5l) is if the "goodness" of a policy or goal-instantiation *feels like* a simple, direct property.  That is, our brains compute the goodness level and make it available to us as a relatively simple quantity, so we *feel like* it's a simple fact that tiling the universe with tiny agents experiencing maximum simply-represented 'pleasure' levels, is a *bad* version of happiness.  We feel like it ought to be simple to yell at an AI "Just give me high-value happiness, not this *weird low-value* happiness!"  Or have the AI learn, from a few examples, that it's meant to produce *high*-value X and not *low*-value X, especially if the AI is smart enough to learn other simple boundaries, like the difference between red objects and blue objects.  Where actually the boundary between "good X" and "bad X" is [value-laden](https://arbital.com/p/36h) and far more wiggly and would require far more examples to delineate.  What our brain computes as a seemingly simple, perceptually available one-dimensional quantity, does not always correspond to a simple, easy-to-learn gradient in the space of policies or outcomes.  This is especially true of the seemingly readily-available property of [beneficialness](https://arbital.com/p/3d9).


Subjective probability

What does it *mean* to say that a flipped coin has a 50% probability of landing heads?

There are multiple ways to answer this question, depending on what you mean by "probability". This page discusses "subjective probabilities," which are a tool for quantifying your personal uncertainty about the world.

Imagine flipping a coin and slapping it against your wrist. It's already landed either heads or tails. The fact that you don't know whether it's heads or tails is a fact about _you,_ not a fact about the coin. Ignorance is in the mind, not in the world.

So your mind is representing the coin, and you're unsure about which way the coin came up. Those probabilities, represented in your brain, are your subjective probabilities. [https://arbital.com/p/1bv](https://arbital.com/p/1bv) is concerned with the formalization, study, and manipulation of subjective probabilities.

If probabilities are simply subjective mental states, what does it mean to say that probabilities are "good," "correct," "accurate," or "true"? The subjectivist answer, roughly, is that a probability distribution becomes more accurate as it puts more of its probability mass on the true possibility within the set of all possibilities it considers.  For more on this see [https://arbital.com/p/4yj](https://arbital.com/p/4yj).

Subjective probabilities, given even a small [grain of truth](https://arbital.com/p/), will become *more* accurate as they interact with reality and execute [Bayesian updates](https://arbital.com/p/1ly).  Your subjective belief about "Is it cloudy today?" is materially represented in your brain, and becomes more accurate as you look up at the sky and causally interact with it: light from the clouds in the sky comes down, enters your retina, is transduced to nerve impulses, processed by your visual cortex, and then your subjective belief about whether it's cloudy becomes more accurate.

'Subjective probability' designates our view of probability as an epistemic state, something inherently in the mind, since reality itself is not uncertain.  It doesn't mean 'arbitrary probability' or 'probability that somebody just made up with no connection to reality'.  Your belief that it's cloudy outside (or sunny) is a belief, but not an arbitrary or made-up belief.  The same can be true about your statement that you think it's 90% likely to be sunny outside, because it was sunny this morning and it's summer, even though you're currently in an interior room and you haven't checked the weather.  The outdoors itself is not wavering between sunny and cloudy; but your *guess* that it's 9 times more likely to be sunny than cloudy is not ungrounded.

Several [coherence theorems](https://arbital.com/p/7ry) suggest that classical probabilities are a *uniquely* good way of quantifying the relative credibility we attach to our guesses; e.g. that even if it's probably sunny, it's still *more* likely for it to be cloudy outside than for the Moon to be made of green cheese.  This in turn says that while the probabilities themselves may exist in our minds, the laws that govern the manipulation and updating of these probabilities are as solid as any other mathematical fact.

For an example of a solid law governing subjective probability, see Arbital's [guide to Bayes' rule](https://arbital.com/p/1zq).

Eric Bruylant
Nate Soares
Eliezer Yudkowsky
Alexei Andreev

What does it *mean* to say that a fair coin has a 50% probability of landing heads?

Imagine flipping a coin and slapping it against your wrist. It's already landed either heads or tails. The fact that you don't know whether it's heads or tails is a fact about _you,_ not a fact about the coin. Ignorance is in the mind, not in the world.

Subjective probabilities are a tool for quantifying uncertainty about the world. In your mind's representation of the coin, you're unsure about which way the coin came up. Those probabilities, represented in your brain, are your subjective probabilities.

[https://arbital.com/p/1bv](https://arbital.com/p/1bv) is concerned with the formalization, study, and manipulation of subjective probabilities.

Several [coherence theorems](https://arbital.com/p/7ry) suggest that classical probability is a *uniquely* good way to quantify subjective uncertainty.  This in turn means that while probabilities may be 'subjective' in the sense of existing in our minds rather than outside reality, the laws governing the manipulation and [updating](https://arbital.com/p/1ly) of these probabilities are solid and determined.


Identifying ambiguous inductions

One of the old fables in machine learning is the story of the "tank classifier" - a neural network that had supposedly been trained to detect enemy tanks hiding in a forest.  It turned out that all the photos of enemy tanks had been taken on sunny days and all the photos of the same field without the tanks had been taken on cloudy days, meaning that the neural net had really just trained itself to recognize the difference between sunny and cloudy days (or just the difference between bright and dim pictures).  ([Source](http://lesswrong.com/lw/7qz/machine_learning_and_unintended_consequences/#d6o1).)

We could view this problem as follows:  A human looking at the labeled data might have seen several concepts that someone might be trying to point at - tanks vs. no tanks, cloudy vs. sunny days, or bright vs. dim pictures.  A human might then ask, "Which of these possible categories did you mean?" and describe the difference using words; or, if it was easier for them to generate pictures than to talk, generate new pictures that distinguished among the possible concepts that could have been meant.  Since learning a simple boundary that separates positive from negative instances in the training data is a form of induction, we could call this problem noticing "inductive ambiguities" or "ambiguous inductions".

This problem bears some resemblance to numerous setups in computer science where we can query an oracle about how to classify instances and we want to learn the concept boundary using a minimum number of instances.  However, identifying an "inductive ambiguity" doesn't seem to be exactly the same problem, or at least, it's not obviously the same problem.  Suppose we consider the tank-classifier problem.  Distinguishing levels of illumination in the picture is a very simple concept, so it would probably be the first one learned; then, treating the problem in classical oracle-query terms, we might imagine the AI presenting the user with various random pixel fields at intermediate levels of illumination.  The user, not having any idea what's going on, classifies these intermediate levels of illumination as 'not tanks', and so the AI soon learns that only quite sunny levels of illumination are required.

Perhaps what we want is less like "figure out exactly where the concept boundary lies by querying the edge cases to the oracle, assuming our basic idea about the boundary is correct" and more like "notice when there's more than one plausible idea that describes the boundary" or "figure out if the user could have been trying to communicate more than one plausible idea using the training dataset".

# Possible approaches

Some possibly relevant approaches that might feed into the notion of "identifying inductive ambiguities":

- [Conservatism](https://arbital.com/p/2qp).  Can we draw a much narrower, but somewhat more complicated, boundary around the training data?
- Can we get a concept that more strongly predicts or more tightly predicts the training cases we saw?  (Closely related to conservatism - if we suppose there's a generator for the training cases, then a more conservative generator concentrates more probability density into the training cases we happened to see.)
- Can we detect commonalities in the positive training cases that aren't already present in the concept we've learned?
   - This might be a good fit for something like a [generative adversarial](http://arxiv.org/abs/1406.2661) approach, where we generate random instances of the concept we learned, then ask if we can detect the difference between those random instances and the actual positively labeled training cases.
- Is there a way to blank out the concept we've already learned so that it doesn't just get learned again, and ask if there's a different concept that's learnable instead?  That is, whatever algorithm we're using, is there a good way to tell it "Don't learn *this* concept, now try to learn" and see if it can learn something substantially different?
- Something something Gricean implication.

# Relevance in value alignment

Since inductive ambiguities are meant to be referred to the user for resolution rather than resolved automatically (the whole point is that the necessary data for an automatic resolution isn't there), they're instances of "[user queries](https://arbital.com/p/2qq)" and all [standard worries about user queries](https://arbital.com/p/2qq) would apply.

The hope about a good algorithm for identifying inductive ambiguities is that it would help catch [edge instantiations](https://arbital.com/p/2w) and [unforeseen maximums](https://arbital.com/p/47), and maybe just simple errors of communication.

An 'inductive ambiguity' is when there's more than one simple concept that fits the data, even if some of those concepts are much simpler than others, and you want to figure out *which* simple concept was intended.   Suppose you're given images that show camouflaged enemy tanks and empty forests, but it so happens that the tank-containing pictures were taken on sunny days and the forest pictures were taken on cloudy days.  Given the training data, the key concept the user intended might be "camouflaged tanks", or "sunny days", or "pixel fields with brighter illumination levels".  The last concept is by far the simplest, but rather than just assume the simplest explanation is correct with most of the probability mass, we want the algorithm (or AGI) to detect that there's more than one simple-ish boundary that might separate the data, and [check with the user](https://arbital.com/p/2qq) about *which* boundary was intended to be learned.