Edge instantiation

The edge instantiation problem is a hypothesized [patch-resistant problem](https://arbital.com/p/48) for [safe](https://arbital.com/p/2l) [value loading](https://arbital.com/p/) in [advanced agent scenarios](https://arbital.com/p/2c) where, for most utility functions we might try to formalize or teach, the maximum of the agent's utility function will end up lying at an edge of the solution space that is a 'weird extreme' from our perspective.

## Definition

On many classes of problems, the maximizing solution tends to lie at an extreme edge of the solution space.  This means that if we have an intuitive outcome X in mind and try to obtain it by giving an agent a solution fitness function F that sounds like it should assign X a high value, the maximum of F may be at an extreme edge of the solution space that looks to us like a very unnatural instance of X, or not an X at all.  The Edge Instantiation problem is a specialization of [unforeseen maximization](https://arbital.com/p/) which in turn specializes Bostrom's [perverse instantiation](https://arbital.com/p/) class of problems.

It is hypothesized (by e.g. [Yudkowsky](https://arbital.com/p/2)) that many classes of solution that have been proposed to [patch](https://arbital.com/p/48) Edge Instantiation would fail to resolve the entire problem and that further Edge Instantiation problems would remain.  For example, even if we consider a [satisficing](https://arbital.com/p/) utility function with only values 0 and 1 where 'typical' X has value 1 and no higher score is possible, an expected utility maximizer could still end up deploying an extreme strategy in order to maximize the *probability* that a satisfactory outcome is obtained.  Considering several proposed solutions like this and their failures suggests that Edge Instantiation is a [resistant](https://arbital.com/p/48) (not ultimately unsolvable, but with many attractive-seeming solutions failing to work) for the deep reason that many possible stages of an agent's cognition would potentially rank solutions and choose very-high-ranking solutions.

The proposition defined is true if Edge Instantiation does in fact surface as a pragmatically important problem for advanced agent scenarios, and would in fact resurface in the face of most 'naive' attempts to correct it.  The proposition is not that the Edge Instantiation Problem is unresolvable, but that it's real, important, doesn't have a *simple* answer, and resists most simple attempts to patch it.

### Example 1: Smiling faces

When Bill Hibbard was first beginning to consider the value alignment problem, he suggested giving AIs the goals of making humans smile, a goal that could be trained by recognizing pictures of smiling humans, and was intended to elicit human happiness.  Yudkowsky replied by suggesting that the true behavior elicited would be to tile the future light cone with tiny molecular smiley faces.  This is not because the agent was perverse, but because among the set of all objects that look like smiley faces, the solution with the most extreme value for achievable numerosity (that is, the strategy which creates the largest possible number of smiling faces) also sets the value for the size of individual smiling faces to an extremely small diameter.  The tiniest possible smiling faces are very unlike the archetypal examples of smiling faces that we had in mind when specifying the utility function; from a human perspective, the intuitively intended meaning has been replaced by a weird extreme.

Stuart Russell observes that maximizing some aspects of a solution tends to set all unconstrained aspects of the solution to extreme values.  The solution that maximizes the number of smiles minimizes the size of each individual smile.  The bad-seeming result is not just an accidental outcome of mere ambiguity in the instructions. The problem wasn't just that a wide range of possibilities corresponded to 'smiles' and a randomly selected possibility from this space surprised us by not being the central example we originally had in mind.  Rather, there's a systematic tendency for the highest-scoring solution to occupy an extreme edge of the solution space, which means that we are *systematically* likely to see 'extreme' or 'weird' solutions rather than the 'normal' examples we had in mind.

### Example 2: Sorcerer's Apprentice

In the hypothetical Sorcerer's Apprentice scenario, you instruct an artificial agent to add water to a cauldron, and it floods the entire workplace.  Hypothetically, you had in mind only adding enough water to fill the cauldron and then stopping, but some stage of the agent's solution-finding process optimized on a step where 'flooding the workplace' scored higher than 'add 4 buckets of water and then shut down safely', even though both of these qualify as 'filling the cauldron'.

This could be because (in the most naive case) the utility function you gave the agent was increasing in the amount of water in contiguous contact with the cauldron's interior - you gave it a utility function that implied 4 buckets of water were good and 4,000 buckets of water were better.

Suppose that, having foreseen in advance the above possible disaster, you try to [patch](https://arbital.com/p/48) the agent by instructing it not to move more than 50 kilograms of material total.  The agent promptly begins to build subagents (with the agent's own motions to build subagents moving only 50 kilograms of material) which build further agents and again flood the workplace.  You have run into a [Nearest Unblocked Neighbor](https://arbital.com/p/42) problem; when you excluded one extreme solution, the result was not the central-feeling 'normal' example you originally had in mind.  Instead, the new maximum lay on a new extreme edge of the solution space.

Another solution might be to define what you thought was a satisficing agent, with a utility function that assigned 1 in all cases where there were at least 4 buckets of water in the cauldron and 0 otherwise.  The agent then calculates that it could increase the *probability* of this condition obtaining from 99.9% to 99.99% by replicating subagents and repeatedly filling the cauldron, just in case one agent malfunctions or something else tries to remove water from the cauldron.   Since 0.9999 > 0.999, there is then a more extreme solution with greater *expected* utility, even though the utility function itself is binary and satisficing.

## Premises

### Assumes: Orthogonality thesis

As with most aspects of the value loading problem, [https://arbital.com/p/1y](https://arbital.com/p/1y) is an implicit premise of the Edge Instantiation problem; for Edge Instantiation to be a problem for advanced agents implies that 'what we really meant' or the outcomes of highest [normative value](https://arbital.com/p/) are not inherently picked out by every possible maximizing process; and that most possible utility functions do not care 'what we really meant' unless explicitly constructed to have a [do what I mean](https://arbital.com/p/) behavior.

### Assumes: [Complexity of values](https://arbital.com/p/5l)

If normative values were extremely simple (of very low algorithmic complexity), then they could be formally specified in full, and the most extreme strategy that scored highest on this formal measure simply *would* correspond with what we really wanted, with no downsides that hadn't been taken into account in the score.

## Arguments

### Interaction with nearest unblocked neighbor

The Edge Instantiation problem has the [https://arbital.com/p/42](https://arbital.com/p/42) pattern.  If you foresee one specific 'perverse' instantiation and try to prohibit it, the maximum over the remaining solution space is again likely to be at another extreme edge of the solution space that again seems 'perverse'.

### Interaction with [cognitive uncontainability](https://arbital.com/p/) of [advanced agents](https://arbital.com/p/2c)

Advanced agents search larger solution spaces than we do.  Therefore the project of trying to visualize all the strategies that might fit a utility function, to try to verify in our own minds that the maximum is somewhere safe, seems exceptionally untrustworthy (not [https://arbital.com/p/2l](https://arbital.com/p/2l)).

### Interaction with context change problem

Agents that acquire new strategic options or become able to search a wider range of the solution space may go from having only apparently 'normal' solutions to apparently 'extreme' solutions. This is known as the [context change problem](https://arbital.com/p/6q).  For example, an agent that inductively learns human smiles as a component of its utility function, might as a non-advanced agent have access only to strategies that make humans happy in an intuitive sense (thereby producing the apparent observation that everything is going fine and the agent is working as intended), and then after self-improvement, acquire as an advanced agent the strategic option of transforming the future light cone into tiny molecular smileyfaces.

### Strong pressures can arise at any stage of optimization

Suppose you tried to build an agent that was an *expected* utility satisficer - rather than having a 0-1 utility function and thus chasing probabilities of goal satisfaction ever closer to 1, the agent searches for strategies that have at least 0.999 *expected* utility.  Why doesn't this resolve the problem?

A bounded satisficer doesn't *rule out* the solution of filling the room with water, since this solution also has >0.999 expected utility.  It only requires the agent to carry out one cognitive algorithm which has at least one maximizing or highly optimizing stage, in order for 'fill the room with water' to be preferred to 'add 4 buckets and shut down safely' on that stage (while being equally acceptable at future satisficing stages).  E.g., maybe you build an expected utility satisficer and still end up with an extreme result because one of the cognitive algorithms suggesting solutions was trying to minimize its own disk space usage.

On a meta-level, we may run into problems of [https://arbital.com/p/71](https://arbital.com/p/71) for [reflective agents](https://arbital.com/p/).  Maybe one simple way of obtaining at least 0.999 expected utility is to create a subagent that *maximizes* expected utility?  It seems intuitively clear why bounded maximizers would build boundedly maximizing offspring, but a bounded satisficer doesn't need to build boundedly satisficing offspring - a bounded maximizer might also be 'good enough'.  (In the current theory of TilingAgents, we can prove that an expected utility satisficer can tile to an expected utility satisficer with some surprising caveats, but the problem is that it can tile to other things *besides* an expected utility satisficer.)

Since it seems very easy for at least one stage of a self-modifying agent to end up preferring solutions that have higher scores relative to some scoring rule, the [edge instantiation](https://arbital.com/p/EdgeInstantiation) problem can be expected to resist naive attempts to describe an agent that seems to have an overall behavior of 'not trying quite so hard'.  It's also not clear how to make the instruction 'don't try so hard' be ReflectivelyConsistent, or apply to every part of a considered subagent.  This is also why [limited optimization](https://arbital.com/p/) is an open problem.

Dispreferring solutions with 'extreme impacts' in general is the open problem of [low impact AI](https://arbital.com/p/).  Currently, no formalizable utility function is known that plausibly has the right intuitive meaning for this.  (We're working on it.)  Also note that not every extreme 'technically an X' that we think is 'not really an X' has an extreme causal impact in an intuitive sense, so not every case of the Edge Instantiation problem is blocked by dispreferring greater impacts.

## Implications

### One of [limited optimization](https://arbital.com/p/), [low Impact](https://arbital.com/p/), or [full coverage value loading](https://arbital.com/p/) seems critical for real-world agents

As Stuart Russell observes, solving an optimization problem where only some values are constrained or maximized, will tend to set unconstrained variables to extreme values.  The universe containing the maximum possible number of [paperclips](https://arbital.com/p/10h) contains no humans; optimizing for as much human safety as possible will drive human freedom to zero.

Then we must apparently do at least one of the following:

1.  Build [full coverage](https://arbital.com/p/) advanced agents whose utility functions lead them to terminally disprefer stomping on every aspect of value that we care about (or would care about under reflection).  In a full coverage agent there are no unconstrained variables *that we care about* to be set to extreme values that we would dislike; the AI's goal system knows and cares about *all* of these.  It will not set human freedom to an extremely low value in the course of following an instruction to optimize human safety, because it knows about human freedom and literally everything else.
2.  Build [powerful agents](https://arbital.com/p/2s) that are [limited optimizers](https://arbital.com/p/) which predictably invent only solutions we intuitively consider 'non-extreme', whose optimizations are such as to not drive to an extreme on any substage.  This leaves us with just ambiguity as a (severe) problem, but at least averts a systematic drive toward extremes that will systematically 'exploit' that ambiguity.
3.  Build [powerful agents](https://arbital.com/p/2s) that are [low impact](https://arbital.com/p/) and prefer to avoid solutions that produce greater impacts on *anything* we intuitively see as an important predicate, including both everything we value and a great many more things we don't particularly value.
4.  Find some other escape route from the [value achievement problem](https://arbital.com/p/2z).

### Insufficiently cautious attempts to build advanced agents are likely to be highly destructive

Edge Instantiation is one of the contributing reasons why value loading is hard and naive solutions end up doing the equivalent of tiling the future light cone with paperclips.

We've previously observed certain parties proposing utility functions for advanced agents that seem obviously subject to the Edge Instantiation problem.  Confronted with the obvious disaster forecast, they propose [patching](https://arbital.com/p/48) the utility function to eliminate that particular scenario (or rather, say that of course they would have written the utility function to exclude that scenario) or claim that the agent will not 'misinterpret' the instructions so egregiously (denying the [https://arbital.com/p/1y](https://arbital.com/p/1y) at least to the extent of proposing a universal preference for interpreting instructions 'as intended').  Mistakes of this type also belong to a class that potentially wouldn't show up during early stages of the AI, or would show up in an initially noncatastrophic way that seemed easily patched, so people advocating an [empirical first methodology](https://arbital.com/p/) would falsely believe that they had learned to handle them or eliminated all such tendencies already.

Thus the problem of Edge Instantiation (which is much less severe for nonadvanced agents than advanced agents, will not be solved in the advanced stage by patches that seem to fix weak early problems, and has empirically appeared in proposals by multiple speakers who rejected attempts to point out the Edge Instantiation problem) is a significant contributing factor to the overall expectation that the default outcome of developing advanced agents with current attitudes is disastrous.

### Relative to current attitudes, small increases in safety awareness do not produce significantly less destructive final outcomes

Simple patches to Edge Instantiation fail and the only currently known approaches would take a lot of work to solve problems like [limited optimization](https://arbital.com/p/) or [full coverage](https://arbital.com/p/) that are hard for deep reasons.  In other words, Edge Instantiation does not appear to be the sort of problem that an AI project can easily avoid just by being made aware of it.  (E.g. MIRI knows about it but hasn't yet come up with any solution, let alone one easily patched on to any cognitive architecture.)

This is one of the factors contributing to the general assessment that the curve of outcome goodness as a function of effort is flat for a significant distance around current levels of effort.

Eric Bruylant
Nate Soares
Eliezer Yudkowsky
Alexei Andreev


Omnipotence test for AI safety

Suppose your AI suddenly became omniscient and omnipotent - suddenly knew all facts and could directly ordain any outcome as a policy option.  Would the executing AI code lead to bad outcomes in that case?  If so, why did you write a program that in some sense 'wanted' to hurt you and was only held in check by lack of knowledge and capability?  Isn't that a bad way for you to configure computing power?  Why not write different code instead?

The Omni Test is that an advanced AI should be expected to remain aligned, or not lead to catastrophic outcomes, or fail safely, even if it suddenly knows all facts and can directly ordain any possible outcome as an immediate choice.  The policy proposal is that, among agents meant to act in the rich real world, any predicted behavior where the agent might act destructively if given unlimited power (rather than e.g. pausing for a [safe user query](https://arbital.com/p/2qq)) should be [treated as a bug](https://arbital.com/p/1cv).

# Safety mindset

The Omni Test highlights any reasoning step on which we've presumed, in a non-failsafe way, that the agent must not obtain definite knowledge of some fact or that it must not have access to some strategic option.  There are [epistemic obstacles](https://arbital.com/p/2j) to our becoming extremely confident of our ability to lower-bound the reaction times or upper-bound the power of an advanced agent.

The deeper idea behind the Omni Test is that any predictable failure in an Omni scenario, or lack of assured reliability, exposes some more general flaw.  Suppose NASA found that an alignment of four planets would cause their code to crash and a rocket's engines to explode.  They wouldn't say, "Oh, we're not expecting any alignment like that for the next hundred years, so we're still safe."  They'd say, "Wow, that sure was a major bug in the program."  Correctly designed programs just shouldn't explode the rocket, period.  If any specific scenario exposes a behavior like that, it shows that some general case is not being handled correctly.

The omni-safe mindset says that, rather than trying to guess what facts an advanced agent can't figure out or what strategic options it can't have, we just shouldn't make these guesses of ours *load-bearing* premises of an agent's safety.  Why design an agent that we expect will hurt us if it knows too much or can do too much?

For example, rather than design an AI that is meant to be monitored for unexpected power gains by programmers who can then press a pause button - which implicitly assumes that no capability gain can happen in fast enough that a programmer wouldn't have time to react - an omni-safe proposal would design the AI to detect unvetted capability gains and pause until the vetting had occurred.  Even if it [seemed improbable](https://arbital.com/p/1cv) that some amount of cognitive power could be gained faster than the programmers could react, especially when no such previous sharp power gain had occurred even in the course of a day, etcetera, the omni-safe [mindset](https://arbital.com/p/1cv) says to *just not build* an agent that is unsafe when such background variables have 'unreasonable' settings.  The correct general behavior is to, e.g., always pause when new capability has been acquired and a programmer has not yet indicated approval of its use.  It might not be possible for an AGI design to suddenly use unlimited power *optimally,* or even use it in any safe way at all, but that's still no excuse for building an omni-unsafe system; because it ought to be possible to detect that case, say "Something weird just happened!", and suspend to disk.

Similarly, consider the paradigm of [conservative planning](https://arbital.com/p/2qp).  Rather than thinking in terms of blacklisting features of bad plans, we think in terms of whitelisting allowed plans using [conservative generalizations](https://arbital.com/p/2qp).  So long as we're narrowly whitelisting rather than blacklisting, lots of new option space suddenly opening up shouldn't result in any of those strange new options being taken until the users can whitelist more things.

Similarly, the omni-safe mindset says that any time the AI is running a *cognitive search* for a way to do something harmful or bypass one of our security measures, we've *already* seen a bug in the design, even if we strongly expect that search to come up empty.  Why configure computing power in a way that will search for a way to hurt you?  This is clearly the wrong thing to do with computing power; we, the programmers, should have our code do something else instead.  This is part of the agenda motivating the study of [https://arbital.com/p/45](https://arbital.com/p/45).

Trying to guess an agent's actual capabilities is a reasonable project, especially if there are more 'adversarial' fallback precautions that revolve around our estimates of the agent's actual powers.  E.g., having designed an agent that doesn't *want* to gain unvetted communication channels with the outside world, we might then put the computing hardware into a Faraday cage and make sure there are no cellphone towers around - an 'adversarial' precaution meant to block communication even if the first line of defense fails and the AI starts wanting to communicate in an unvetted way.  But 'adversarial' precautions are *secondary* lines of defense; a reasonable expectation of omni-safety is the [first line of defense](https://arbital.com/p/2x4).  First we assume that all adversarial fallback measures will fail, and design the agent to remain nonharmful or fail safely no matter what new capability or knowledge is gained.  Then we assume the first line of defense has failed, and try, if it's at all possible or realistic, to put up fallback measures that will prevent total catastrophe so long as the agent has realistic amounts of power and *can't* violate what we think are 'the laws of physics' and so on.

Eric Bruylant
Olivia Schaefer
Eliezer Yudkowsky
Alexei Andreev

Suppose your AI suddenly became omniscient and omnipotent - suddenly knew all facts and could directly ordain any outcome as a policy option.  Would the executing AI code lead to bad outcomes in that case?  If so, why did you write a program that in some sense 'wanted' to hurt you and was only held in check by lack of knowledge and capability?  Isn't that a bad way for you to configure computing power?

The Omni Test suggests, e.g., that you should not rely on a human agent to monitor the AI's current growth rate and intervene if something goes visibly wrong.  Instead, growth should be measured internally, and cumulative growth should require external validation before proceeding.  The former case fails if the AI becomes suddenly omnipotent; the latter does not.  Or similarly, if weird new options open up to the AI, the AI should stay inside a [conservatively](https://arbital.com/p/2qp) whitelisted part of the option space until more user interactions have occurred.  Or similarly, we should never write an AI that we think will cognitively search for a way to defeat its own security measures, even if we *think* the search will probably fail.  See also [https://arbital.com/p/2x4](https://arbital.com/p/2x4).


Averting the convergent instrumental strategy of self-improvement

Rapid capability gains, or just large capability gains between a training paradigm and a test paradigm, are one of the primary expected reasons why AGI alignment might be hard.  We probably want the first AGI or AGIs ever built, tested, and used to *not* self-improve as quickly as possible.  Since there's a very strong [convergent incentive](https://arbital.com/p/10g) to self-improve and do things [neighboring](https://arbital.com/p/42) to self-improvement, by default you would expect an AGI to search for ways to defeat naive blocks on self-improvement, which violates the [nonadversarial principle](https://arbital.com/p/2x4). Thus, any proposals to limit an AGI's capabilities imply a very strong desideratum for us to figure out a way to [avert the instrumental incentive](https://arbital.com/p/2vk) to self-improvement in that AGI.  The alternative is failing the [Omni Test](https://arbital.com/p/2x), violating the nonadversarial principle, [having the AGI's code be actively inconsistent with what the AGI would approve of its own code being](https://arbital.com/p/2rb) (if the brake is a code-level measure), and setting up a safety measure that the AGI wants to defeat as the [only line of defense](https://arbital.com/p/2x4).


Niceness is the first line of defense

The first line of defense in constructing any [sufficiently advanced](https://arbital.com/p/2c) [Artificial General Intelligence](https://arbital.com/p/42g) is building an AI that does not *want* to hurt you.  Any other measures, like [AI-boxing](https://arbital.com/p/6z) or trying to [prevent the AI from accessing the Internet](https://arbital.com/p/airgap_ai), should be thought of only as backstops in case this first line of defense fails.  When designing the AGI we should first think as if all these [oppositional measures](https://arbital.com/p/7fx) don't exist, so that we aren't distracted while trying to envision an AGI that--[regardless](https://arbital.com/p/2x) of [how much power it has](https://arbital.com/p/capability_gain)--will not *want* to hurt us.

See also [the non-adversarial principle](https://arbital.com/p/7g0) and the distinction between [https://arbital.com/p/7fx](https://arbital.com/p/7fx).


Shutdown problem

The 'shutdown problem' is creating a [sufficiently advanced Artificial Intelligence](https://arbital.com/p/7g1) which will, on the press of a button, suspend itself safely to disk; such that the AI [does not object](https://arbital.com/p/7g0) to the existence of this button, nor [try to prevent it](https://arbital.com/p/45) from being pressed.

This is difficult because avoiding your own halt or suspension is a [convergent instrumental strategy](https://arbital.com/p/10g); even [a goal like "bring the user coffee" implies avoiding shutdown](https://arbital.com/p/7g2).

One possible research avenue is to:

 1. Describe a '[shutdown utility function](https://arbital.com/p/2rf)' whose [attainable optimum](https://arbital.com/p/7t9) is "Suspend safely to disk (and possibly [safely abort](https://arbital.com/p/2rg) plans in progress), without building a hidden environmental copy of yourself to carry out your previous goal and then assimilate all matter in the universe to make absolutely sure the original AI stays shut down".
 2.  Find [a sensible way to compose](https://arbital.com/p/1b7) the shutdown utility function with the agent's regular utility function, such that:
    - (2a) Which utility function the agent optimizes depends on whether a switch was pressed.
    - (2b) The AI experiences no incentive to cause the switch to be pressed or prevent the switch from being pressed.

Harder versions of this problem would add specifications such as:

- The AI [positively wants](https://arbital.com/p/nonadversarial_safety) the switch to exist and will restore the switch's existence if it is damaged.
- The AI will [safely abort](https://arbital.com/p/2rg) any plans in progress (and design plans such that they can be safely aborted).

See also [https://arbital.com/p/1b7](https://arbital.com/p/1b7), [https://arbital.com/p/2rf](https://arbital.com/p/2rf), [https://arbital.com/p/45](https://arbital.com/p/45), [https://arbital.com/p/interruptibility](https://arbital.com/p/interruptibility), [https://arbital.com/p/2pf](https://arbital.com/p/2pf), and [https://arbital.com/p/2rg](https://arbital.com/p/2rg).

The 'shutdown problem' is creating a [sufficiently advanced Artificial Intelligence](https://arbital.com/p/7g1) which will, on the press of a button, suspend itself safely to disk; such that the AI [does not object](https://arbital.com/p/7g0) to the existence of this button, nor [try to prevent it](https://arbital.com/p/45) from being pressed.

This is difficult because avoiding your own halt or suspension is a [convergent instrumental strategy](https://arbital.com/p/10g); even [a goal like "bring the user coffee" implies avoiding shutdown](https://arbital.com/p/7g2).

This problem is sometimes decomposed into (1) the [problem](https://arbital.com/p/2rf) of finding a utility function that [really actually means](https://arbital.com/p/47) "Suspend yourself safely to disk", and (2) the [problem](https://arbital.com/p/1b7) of building an agent that *wants* to switch to optimizing a different utility function if a button is pressed, but that *doesn't* want to press that button or prevent its being pressed.

See also [https://arbital.com/p/1b7](https://arbital.com/p/1b7), [https://arbital.com/p/2rf](https://arbital.com/p/2rf), [https://arbital.com/p/45](https://arbital.com/p/45), and [https://arbital.com/p/interruptibility](https://arbital.com/p/interruptibility).

Open subproblems in aligning a Task-based AGI


Relevant limited AI

It is an open problem to propose a [limited AI](https://arbital.com/p/5b3) that would be [relevant](https://arbital.com/p/2s) to the [value achievement dilemma](https://arbital.com/p/2z) - an agent cognitively constrained along some dimensions that render it much safer, but still able to perform some task useful enough to prevent catastrophe.

### Basic difficulty

Consider an [Oracle AI](https://arbital.com/p/6x) that is so constrained as to be allowed only to output proofs in HOL of input theorems; these proofs are then verified by a simple and secure-seeming verifier in a sandbox whose exact code is unknown to the Oracle, and this verifier outputs 1 if the proof is true and 0 otherwise, then discards the proof-data.  Suppose also that the Oracle is in a shielded box, etcetera.

It's possible that this Provability Oracle has been so constrained that it is [cognitively containable](https://arbital.com/p/2j) (it has no classes of options we don't know about).  If the verifier is unhackable, it gives us trustworthy knowledge that a theorem is provable.  But this limited system is not obviously useful in a way that enables humanity to extricate itself from its larger dilemma.  Nobody has yet stated a plan which could save the world *if only* we had a superhuman capacity to detect which theorems were provable in Zermelo-Fraenkel set theory.

Saying "The solution is for humanity to only build Provability Oracles!" does not resolve the [value achievement dilemma](https://arbital.com/p/2z) because humanity does not have the coordination ability to 'choose' to develop only one kind of AI over the indefinite future, and the Provability Oracle has no obvious use that prevents non-Oracle AIs from ever being developed.  Thus our larger value achievement dilemma would remain unsolved.  It's not obvious how the Provability Oracle would even constitute significant strategic progress.

### Open problem

Describe a cognitive task or real-world task for a AI to carry out, *that makes great progress upon the [value achievement dilemma](https://arbital.com/p/2z) if executed correctly*, and that can be done with a *limited* AI that:

1.  Has a real-world solution state that is exceptionally easy to pinpoint using a utility function, thereby avoiding some of [edge instantiation](https://arbital.com/p/2w), [unforeseen maximums](https://arbital.com/p/47), [context change](https://arbital.com/p/6q), [programmer maximization](https://arbital.com/p/), and the other pitfalls of [advanced safety](https://arbital.com/p/2l), if there is otherwise a trustworthy solution for [low-impact AI](https://arbital.com/p/); or
2.  Seems exceptionally implementable using a [known-algorithm non-self-improving agent](https://arbital.com/p/1fy), thereby averting problems of stable self-modification, if there is otherwise a trustworthy solution for a known-algorithm non-self-improving agent; or
3.  Constrains the agent's option space so drastically as to make the strategy space not be rich (and the agent hence containable), while still containing a trustworthy, otherwise unfindable solution to some challenge that resolves the larger dilemma.

Rob Bensinger
Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev


Value achievement dilemma

The value achievement dilemma is a way of framing the [AI alignment problem](https://arbital.com/p/2v) in a larger context.  This emphasizes that there might be possible solutions besides AI; and also emphasizes that such solutions must meet a high bar of potency or efficacy in order to *resolve* our basic dilemmas, the way that a sufficiently value-aligned and cognitively powerful AI could resolve our basic dilemmas.  Or at least [change the nature of the gameboard](https://arbital.com/p/6y), the way that a Task AGI could take actions to prevent destruction by later AGI projects, even if is only [narrowly](https://arbital.com/p/1vt) value-aligned and cannot solve the whole problem.

The point of considering posthuman scenarios in the long run, and not just an immediate [Task AGI](https://arbital.com/p/6w) as band-aid, can be seen in the suggestion by [https://arbital.com/p/2](https://arbital.com/p/2)  and [https://arbital.com/p/18k](https://arbital.com/p/18k)  that we can see Earth-originating intelligent life as having two possible [stable states](https://arbital.com/p/), [superintelligence](https://arbital.com/p/41l) and extinction.  If intelligent life goes extinct, especially if it drastically damages or destroys the ecosphere in the process, new intelligent life seems unlikely to arise on Earth.  If Earth-originating intelligent life becomes superintelligent, it will presumably expand through the universe and stay superintelligent for as long as physically possible.  Eventually, our civilization is bound to wander into one of these attractors or another.

Furthermore, by the [generic preference stability argument](https://arbital.com/p/3r6), any sufficiently advanced cognitive agent is very likely to be stable in its motivations or [meta-preference framework](https://arbital.com/p/5f). So if and when life wanders into the superintelligence attractor, it will either end up in a stable state of e.g. [fun-loving](https://arbital.com/p/10d) or [the reflective equilibrium of its creators' civilization](https://arbital.com/p/3c5) and hence achieving lots of [value](https://arbital.com/p/55), or a misaligned AI will go on [maximizing paperclips](https://arbital.com/p/10h) forever.

Among the dilemmas we face in getting into the high-value-achieving attractor, rather than the extinction attractor or the equivalence class of paperclip maximizers, are:

- The possibility of careless (or insufficiently cautious, or much less likely malicious) actors creating a non-value-aligned AI that undergoes an intelligence explosion.
- The possibility of engineered superviruses destroying enough of civilization that the remaining humans go extinct without ever reaching sufficiently advanced technology.
- Conflict between multipolar powers with nanotechnology resulting in a super-nuclear-exchange disaster that extinguishes all life.

Other positive events seem like they could potentially prompt entry into the high-value-achieving superintelligence attractor:

- Direct creation of a [fully](https://arbital.com/p/41k) normatively aligned [https://arbital.com/p/1g3](https://arbital.com/p/1g3) agent.
- Creation of a [Task AGI](https://arbital.com/p/6w) powerful enough to avert the creation of other [UnFriendly AI](https://arbital.com/p/).
- Intelligence-augmented humans (or 64-node clustered humans linked by brain-computer interface brain information exchange, etcetera) who are able and motivated to solve the AI alignment problem.

On the other hand, consider someone who proposes that "Rather than building AI, [we should](https://arbital.com/p/) build [Oracle AIs](https://arbital.com/p/) that just answer questions," and who then, after further exposure to the concept of the [AI-Box Experiment](https://arbital.com/p/) and [cognitive uncontainability](https://arbital.com/p/2j), further narrows their specification to say that [an Oracle running in three layers of sandboxed simulation must output only formal proofs of given theorems in Zermelo-Fraenkel set theory](https://arbital.com/p/70), and a heavily sandboxed and provably correct verifier will look over this output proof and signal 1 if it proves the target theorem and 0 otherwise, at some fixed time to avoid timing attacks.

This doesn't resolve the larger value achievement dilemma, because there's no obvious thing we can do with a ZF provability oracle that solves our larger problem.  There's no plan such that it would save the world *if only* we could take some suspected theorems of ZF and know that some of them had formal proofs.

The thrust of considering a larger 'value achievement dilemma' is that while imaginable alternatives to aligned AIs exist, they must pass a double test to be our best alternative:

- They must be genuinely easier or safer than the easiest (pivotal) form of the AI alignment problem.
- They must be game-changers for the overall situation in which we find ourselves, opening up a clear path to victory from the newly achieved scenario.

Any strategy that does not putatively open a clear path to victory if it succeeds, doesn't seem like a plausible policy alternative to trying to solve the AI alignment problem or to doing something else such that success leaves us a clear path to victory.  Trying to solve the AI alignment problem is something intended to leave us a clear path to achieving almost all of the achievable value for the future and its astronomical stakes.  Anything that doesn't open a clear path to getting there is not an alternative solution for getting there.

For more on this point, see the page on [pivotal events](https://arbital.com/p/6y).

# Subproblems of the larger value achievement dilemma

We can see the place of AI alignment in the larger scheme by considering its parent problem, its sibling problems, and examples of its child problems.

- The **value achievement dilemma**:  How does Earth-originating intelligent life achieve an acceptable proportion of its potential [value](https://arbital.com/p/55)?
   - The **AI alignment problem**:  How do we create AIs such that running them produces (global) outcomes of acceptably high value?
      -  The **value alignment problem**:  How do we create AIs that *want* or *prefer* to cause events that are of high value?  If we accept that we should solve the value alignment problem by creating AIs that prefer or want in particular ways, how do we do that?
          - The **[https://arbital.com/p/6c](https://arbital.com/p/6c)** or **value learning** problem:  How can we pinpoint, in the AI's decision-making, outcomes that have high 'value'?  (Despite all the [foreseeable difficulties](https://arbital.com/p/6r) such as [edge instantiation](https://arbital.com/p/2w) and [Goodhart's Curse](https://arbital.com/p/6g4).)
      - Other properties of aligned AIs such as e.g. **[https://arbital.com/p/-45](https://arbital.com/p/-45)**:  How can we create AIs such that, when we make an error in identifying value or specifying the decision system, the AI does not resist our attempts to correct what we regard as an error?
      - [Oppositional](https://arbital.com/p/7fx) features such as e.g. [boxing](https://arbital.com/p/6z) that are intended to mitigate harm if the AI's behavior has gone outside expected bounds.
   - The intelligence amplification problem.   How can we create smarter humans, preferably without driving them insane or otherwise ending up with evil ones?
   - The [value selection](https://arbital.com/p/) problem.  How can we figure out what to substitute in for the metasyntactic variable 'value'?  ([Answer](https://arbital.com/p/313).)

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev

The value achievement dilemma is the general, broad challenge faced by Earth-originating intelligent life in steering our [cosmic endowment](https://arbital.com/p/7cy) into a state of high [value](https://arbital.com/p/55) - successfully turning the stars into a happy civilization.

We face potential existential catastrophes (resulting in our extermination or the corruption of the cosmic endowment) such as sufficiently lethal engineered pandemics, non-value-aligned AIs, or insane smart uploads.  A strategy is [relevant](https://arbital.com/p/2s) to value achievement only if success is a [game-changer](https://arbital.com/p/6y) for the overall dilemma humanity faces.  E.g., [value-aligned](https://arbital.com/p/5s) [powerful AIs](https://arbital.com/p/2c) or [intelligence-enhanced humans](https://arbital.com/p/) both seem to qualify as strategically relevant; but an AI restricted to *only* [prove theorems in Zermelo-Frankel set theory](https://arbital.com/p/70) has no obvious game-changing use.


User manipulation

If there's anything an AGI wants whose achievement involves steps that interact with the AGI's programmers or users, then by default, the AGI will have an [instrumental incentive](https://arbital.com/p/10k) to optimize the programmers/users in the course of achieving its goal.  If the AGI wants to self-improve, then by default and unless specifically [averted](https://arbital.com/p/2vk), it also wants to have its programmers not interfere with self-improvement.  If a Task AGI has been aligned to the point of taking user instructions, then by default and unless otherwise averted, it will forecast greater success in the eventualities where it receives easier user instructions.


User maximization

A sub-principle of avoiding [user manipulation](https://arbital.com/p/309) - if you've formulated the AI in terms of an argmax over $X$ or an "optimize $X$" instruction, and $X$ includes a user interaction as a subpart, then you've just told the AI to optimize the user.  For example, let's say that the AI's criterion of action is "Choose a policy $p$ by maximizing/optimizing the probability $X$ that the user's instructions are carried out."  If the user's instructions are a variable inside the formula for $X$ rather than a constant outside it, you've just told the AI to try and get the user to give it easier instructions.


Extrapolated volition (normative moral theory)

(This page is about extrapolated volition as a normative moral theory - that is, the theory that extrapolated volition captures the concept of [value](https://arbital.com/p/55) or what outcomes we *should* want.  For the closely related proposal about what a sufficiently advanced [self-directed](https://arbital.com/p/1g3) AGI should be built to want/target/decide/do, see [coherent extrapolated volition](https://arbital.com/p/3c5).)

# Concept

Extrapolated volition is the notion that when we ask "What is right?", then insofar as we're asking something meaningful, we're asking about the result of running a certain logical function over possible states of the world, where this function is analytically identical to the result of *extrapolating* our current decision-making process in directions such as "What if I knew more?", "What if I had time to consider more arguments (so long as the arguments weren't hacking my brain)?", or "What if I understood myself better and had more self-control?"

A simple example of extrapolated volition might be to consider somebody who asks you to bring them orange juice from the refrigerator.  You open the refrigerator and see no orange juice, but there's lemonade.  You imagine that your friend would want you to bring them lemonade if they knew everything you knew about the refrigerator, so you bring them lemonade instead.  On an abstract level, we can say that you "extrapolated" your friend's "volition", in other words, you took your model of their mind and decision process, or your model of their "volition", and you imagined a counterfactual version of their mind that had better information about the contents of your refrigerator, thereby "extrapolating" this volition.

Having better information isn't the only way that a decision process can be extrapolated; we can also, for example, imagine that a mind has more time in which to consider moral arguments, or better knowledge of itself.  Maybe you currently want revenge on the Capulet family, but if somebody had a chance to sit down with you and have a long talk about how revenge affects civilizations in the long run, you could be talked out of that.  Maybe you're currently convinced that you advocate for green shoes to be outlawed out of the goodness of your heart, but if you could actually see a printout of all of your own emotions at work, you'd see there was a lot of bitterness directed at people who wear green shoes, and this would change your mind about your decision.

In Yudkowsky's version of extrapolated volition considered on an individual level, the three core directions of extrapolation are:

- Increased knowledge - having more veridical knowledge of declarative facts and expected outcomes.
- Increased consideration of arguments - being able to consider more possible arguments and assess their validity.
- Increased reflectivity - greater knowledge about the self, and to some degree, greater self-control (though this raises further questions about which parts of the self normatively get to control which other parts).

# Motivation

Different people react differently to the question "Where *should* we point an [autonomous](https://arbital.com/p/1g3) superintelligence, if we can point it exactly?" and approach it from different angles.  These angles include:

- All this talk of 'shouldness' is just a cover for the fact that whoever gets to build the superintelligence wins all the marbles; no matter what you do with your superintelligence, you'll be the one who does it.
- What if we tell the superintelligence what to do and it's the wrong thing?  What if we're basically confused about what's right?  Shouldn't we let the superintelligence figure that out on its own with its own superior intelligence?
- Imagine the Ancient Greeks telling a superintelligence what to do.  They'd have told it to optimize personal virtues, including, say, a glorious death in battle.  This seems like a bad thing and we need to figure out how not to do the analogous thing.  So telling an AGI to do what seems like a good idea to us will also end up seeming a very regrettable decision a million years later.
- Obviously we should just tell the AGI to optimize liberal democratic values.  Liberal democratic values are good.  The real threat is if bad people get their hands on AGI and build an AGI that doesn't optimize liberal democratic values.

Some corresponding initial replies might be:

- Okay, but suppose you're a programmer and you're trying *not to be a jerk.*  If you're like, "Well, whatever I do originates in myself and is therefore equally selfish, so I might as well declare myself God-Emperor of Earth," you're being a jerk.  Is there anything we can do which is less jerky, and indeed, minimally jerky?
- If you say you have no information at all about what's 'right', then what does the term even mean?  If I might as well have my AGI maximize paperclips and you have no ground on which to stand and say that's the wrong way to compute normativity, then what are we even talking about in the first place?  The word 'right' or 'should' must have some meaning that you know about, even if it doesn't automatically print out a list of everything you know is right.  Let's talk about hunting down that meaning.
- Okay, so what should the Ancient Greeks have done if they did have to program an AI?  How could they *not* have doomed future generations?  Suppose the Ancient Greeks are clever enough to have noticed that sometimes people change their minds about things and to realize that they might not be right about everything.  How can they use the cleverness of the AGI in a constructively specified, computable fashion that gets them out of this hole?  You can't just tell the AGI to compute what's 'right', you need to put an actual computable question in there, not a word.
- What if you would, after some further discussion, want to tweak your definition of "liberal democratic values" just a little?  What if it's *predictable* that you would do that?  Would you really want to be stuck with your off-the-cuff definition a million years later?

Arguendo by [CEV](https://arbital.com/p/3c5)'s advocates, these conversations eventually all end up converging on [Coherent Extrapolated Volition](https://arbital.com/p/3c5) as an alignment proposal by different roads.

"Extrapolated volition" is the corresponding normative theory that you arrive at by questioning the meaning of 'right' or trying to figure out what we 'should' really truly do.

# EV as [rescuing](https://arbital.com/p/3y6) the notion of betterness

We can see EV as trying to **[rescue](https://arbital.com/p/3y6)** the following pretheoretic intuitions (as they might be experienced by someone feeling confused, or just somebody who'd never questioned metaethics in the first place):

- (a)  It's possible to think that something is right, and be incorrect.
   - (a1)  It's possible for something to be wrong even if nobody knows that it's wrong.  E.g. an uneven division of an apple pie might be unfair even if all recipients don't realize this.
   - (a2)  We can learn more about what's right, and change our minds to be righter.
- (b)  Taking a pill that changes what you think is right, should not change what *is* right.  (If you're contemplating taking a pill that makes you think it's right to secretly murder 12-year-olds, you should not reason, "Well, if I take this pill I'll murder 12-year-olds... but also it *will* be all right to murder 12-year-olds, so this is a great pill to take.")
- (c)  We could be wrong, but it sure *seems* like the things on [Frankena's list](https://arbital.com/p/41r) are all reasonably good.  ("Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc...")
   - (c1)  The fact that we could be in some mysterious way "wrong" about what belongs on Frankena's list, doesn't seem to leave enough room for "[make as many paperclips as possible](https://arbital.com/p/10h)" to be the only thing on the list.  Even our state of confusion and possible ignorance doesn't seem to allow for that to be the answer.  We're at least pretty sure that *isn't* the total sum of goodness.
   - (c2)  Similarly, on the meta-level, it doesn't seem like the meta-level procedure "Pick whatever procedure for determining rightness, leads to the most paperclips existing after you adopt it" could be the correct answer.

We *cannot* [rescue](https://arbital.com/p/3y6) these properties by saying:

"There is an irreducible, non-natural 'rightness' XML tag attached to some objects and events.  Our brains perceive this XML tag, but imperfectly, giving us property (a) when we think the XML tag is there, even though it isn't.  The XML tags are there even if nobody sees them (a1).  Sometimes we stare harder and see the XML tag better (a2).  Obviously, doing anything to a brain isn't going to change the XML tag (b), just fool the brain or invalidate its map of the XML tag.  All of the things on Frankena's list have XML tags (c) or at least we think so.  For paperclips to be the total correct content of Frankena's list, we'd need to be wrong about paperclips not having XML tags *and* wrong about everything on Frankena's list that we think *does* have an XML tag (c1).  And on the meta-level, "Which sense of rightness leads to the most paperclips?" doesn't say anything *about* XML tags, and it doesn't lead *to* there being lots of XML tags, so there's no justification for it (c2)."

This doesn't work because:

- There are, [in fact](https://arbital.com/p/112), no tiny irreducible XML tags attached to objects.
- If there were little tags like that, there'd be no obvious normative justification for our caring about them.
- It doesn't seem like we should be able to make it good to murder 12-year-olds by swapping around the irreducible XML tags on the event.
- There's no way our brains could perceive these tiny XML tags even if they were there.
- There's no obvious causal story for how humans could have evolved such that we do in fact care about these tiny XML tags.  (A [descriptive rather than normative](https://arbital.com/p/3y9) problem with the theory as a whole; natural selection has no normative force or justificational power, but we *do* need our theory of how brains actually work to be compatible with it).

Onto what sort of entity can we then map our intuitions, if not onto tiny XML tags?

Consider the property of sixness possessed by six apples on a table.  The relation between the physical six apples on a table, and the logical number '6', is given by a logical function that takes physical descriptions as inputs: in particular, the function "count the number of apples on the table".

Could we rescue 'rightness' onto a logical function like this, only much more complicated?

Let's examine how the 6-ness property and the "counting apples" function behave:

- There are, in fact, no tiny tags saying '6' attached to the apples (and yet there are still six of them).
- It's possible to think there are 6 apples on the table, and be wrong.
- We can sometimes change our minds about how many apples there are on a table.
- There can be 6 apples on a table even if nobody is looking at it.
- Taking a pill that changes how many apples you think are on the table, doesn't change the number of apples on the table.
- You can't have a 6-tag-manipulator that changes the number of apples on a table without changing anything about the table or apples.
- There's a clear causal story for how we can see apples, and also for how our brains can count things, and there's an understandable historical fact about *why* humans count things.
- Changing the history of how humans count things could change *which* logical function our brains were computing on the table, so that our brains were no longer "counting apples", but it wouldn't change the number of apples on the table.  We'd be changing *which* logical function our brains were considering, not changing the logical facts themselves or making it so that identical premises would lead to different conclusions.
- Suppose somebody says, "Hey, you know, sometimes we're wrong about whether there's 6 of something or not, maybe we're just entirely confused about this counting thing; maybe the real number of apples on this table is this paperclip I'm holding."  Even if you often made mistakes in counting, didn't know how to axiomatize arithmetic, and were feeling confused about the nature of numbers, you would still know *enough* about what you were talking about to feel pretty sure that the number of apples on the table was not in fact a paperclip.
- If you could ask a superintelligence how many grains of sand your brain would think there were on a beach, in the limit of your brain representing everything the superintelligence knew and thinking very quickly, you would indeed gain veridical knowledge about the number of grains of sand on that beach.  Your brain doesn't determine the number of grains of sand on the beach, and you can't change the logical properties of first-order arithmetic by taking a pill that changes your brain.  But there's an *analytic* relation between the procedure your brain currently represents and tries to carry out in an error-prone way, and the logical function that counts how many grains of sand on the beach.

This suggests that 6-ness has the correct ontological nature for some much bigger and more complicated logical function than "Count the number of apples on the table" to be outputting rightness.  Or rather, if we want to [rescue our pretheoretic sense](https://arbital.com/p/3y6) of rightness in a way that [adds up to moral normality](https://arbital.com/p/3y6), we should rescue it onto a logical function.

This function, e.g., starts with the items on Frankena's list and everything we currently value; but also takes into account the set of arguments that might change our mind about what goes on the list; and also takes into account meta-level conditions that we would endorse as distinguishing "valid arguments" and "arguments that merely change our minds".  (This last point is pragmatically important if we're considering trying to get a [superintelligence](https://arbital.com/p/2c) to [extrapolate our volitions](https://arbital.com/p/3c5).  The list of everything that *does in fact change your mind* might include particular patterns of rotating spiral pixel patterns that effectively hack a human brain.)

The end result of all this work is that we go on guessing which acts are right and wrong as before, go on considering that some possible valid arguments might change our minds, go on weighing such arguments, and go on valuing the things on [Frankena's list](https://arbital.com/p/41r) in the meantime.  The theory as a whole is intended to [add up to the same moral normality as before](https://arbital.com/p/3y6), just with that normality embedded into the world of causality and logic in a non-confusing way.

One point we could have taken into our starting list of important properties, but deferred until later:

- It sure *feels* like there's a beautiful, mysterious floating 'rightness' property of things that are right, and that the things that have this property are terribly precious and important.

On the general program of "[rescuing the utility function](https://arbital.com/p/3y6)", we should not scorn this feeling, and should instead figure out how to map it onto what actually exists.

In this case, having preserved almost all the *structural* properties of moral normality, there's no reason why anything should change about how we experience the corresponding emotion in everyday life.  If our native emotions are having trouble with this new, weird, abstract, learned representation of 'a certain big complicated logical function', we should do our best to remember that the rightness is still there.  And this is not a retreat to second-best any more than "disordered kinetic energy" is some kind of sad consolation prize for [the universe's lack of ontologically basic warmth](https://arbital.com/p/3y6), etcetera.

## Unrescuability of moral internalism

In standard metaethical terms, we have managed to rescue 'moral cognitivism' (statements about rightness have truth-values) and 'moral realism' (there is a fact of the matter out there about how right something is).  We have *not* however managed to rescue the pretheoretic intuition underlying 'moral internalism':

- A moral argument, to be valid, ought to be able to persuade anyone.  If a moral argument is unpersuasive to someone who isn't making some kind of clear mistake in rejecting it, then that argument must rest on some appeal to a private or merely selfish consideration that should form no part of true morality that everyone can perceive.

This intuition cannot be preserved in any reasonable way, because [paperclip maximizers](https://arbital.com/p/10h) are in fact going to go on making paperclips (and not because they made some kind of cognitive error).  A paperclip maximizer isn't disagreeing with you about what's right (the output of the logical function), it's just following whatever plan leads to the most paperclips.

Since the paperclip maximizer's policy isn't influenced by any of our moral arguments, we can't preserve the internalist intuition without reducing the set of valid justifications and truly valuable things to the empty set - and even *that,* a paperclip maximizer wouldn't find motivationally persuasive!

Thus our options regarding the pretheoretic internalist intuition that a moral argument is not valid if not universally persuasive, seem to be limited to the following:

 1. Give up on the intuition in its intuitive form: a paperclip maximizer doesn't care if it's unjust to kill everyone; and you can't talk it into behaving differently; and this doesn't reflect a cognitive stumble on the paperclip maximizer's part; and this fact gives us no information about what is right or justified.
 2. Preserve, at the cost of all other pretheoretic intuitions about rightness, the intuition that only arguments that universally influence behavior are valid: that is, there are no valid moral arguments.
 3. Try to sweep the problem under the rug by claiming that *reasonable* minds must agree that paperclips are objectively pointless... even though Clippy is not suffering from any defect of epistemic or instrumental power, and there's no place in Clippy's code where we can point to some inherently persuasive argument being dropped by a defect or special case of that code.

It's not clear what the point of stance (2) would be, since even this is not an argument that would cause Clippy to alter its behavior, and hence the stance is self-defeating.  (3) seems like a mere word game, and potentially a *very dangerous* word game if it tricks AI developers into thinking that rightness is a default behavior of AIs, or even a function of low [algorithmic complexity](https://arbital.com/p/5v), or that [beneficial](https://arbital.com/p/3d9) behavior automatically correlates with 'reasonable' judgments about less [value-laden](https://arbital.com/p/36h) questions.  See "[https://arbital.com/p/1y](https://arbital.com/p/1y)" for the extreme practical importance of acknowledging that moral internalism is in practice false.

# Situating EV in contemporary metaethics

[Metaethics](https://arbital.com/p/41n) is the field of academic philosophy that deals with the question, not of "What is good?", but "What sort of property is goodness?"  As applied to issues in Artificial Intelligence, rather than arguing over which particular outcomes are better or worse, we are, from a standpoint of [executable philosophy](https://arbital.com/p/112), asking how to *compute* what is good; and why the output of any proposed computation ought to be identified with the notion of shouldness.

EV replies that for each person at a single moment in time, *right* or *should* is to be identified with a (subjectively uncertain) logical constant that is fixed for that person at that particular moment in time, where this logical constant is to be identified with the result of running the extrapolation process on that person.  We can't run the extrapolation process so we can't get perfect knowledge of this logical constant, and will be subjectively uncertain about what is right.

To eliminate one important ambiguity in how this might cash out, we regard this logical constant as being *analytically identified* with the extrapolation of our brains, but not *counterfactually dependent* on counterfactually varying forms of our brains.  If you imagine being administered a pill that makes you want to kill people, then you shouldn't compute in your imagination that different things are right for this new self.  Instead, this new self now wants to do something other than what is right.  We can meaningfully say, "Even if I (a counterfactual version of me) wanted to kill people, that wouldn't make it right" because the counterfactual alteration of the self doesn't change the logical object that you mean by saying 'right'.

However, there's still an *analytic* relation between this logical object and your *actual* mindstate, which is indeed is implied by the very meaning of discourse about shouldness, which means that you *can* get veridical information about this logical object by having a sufficiently intelligent AI run an approximation of the extrapolation process over a good model of your actual mind.  If a sufficiently intelligent and trustworthy AGI tells you that after thinking about it for a while you wouldn't want to eat cows, you have gained veridical information about whether it's right to eat cows.

Within the standard terminology of academic metaethics, "extrapolated volition" as a normative theory is:

- Cognitivist.  Normative propositions can be true or false.  You can believe that something is right and be mistaken.
- Naturalist.  Normative propositions are *not* irreducible or based on non-natural properties of the world.
- Externalist / not internalist.  It is not the case that all sufficiently powerful optimizers must act on what we consider to be moral propositions.  A paperclipper does what is clippy, not what is right, and the fact that it's trying to turn everything into paperclips does not indicate a disagreement with you about what is *right* any more than you disagree about what is clippy.
- Reductionist.  The whole point of this theory is that it's the sort of thing you could potentially compute.
- More synthetic reductionist than analytic reductionist.  We don't have *a priori* knowledge of our starting mindstate and don't have enough computing power to complete the extrapolation process over it.  Therefore, we can't figure out exactly what our extrapolated volition would say just by pondering the meaning of the word 'right'.

Closest antecedents in academic metaethics are Rawls and Goodman's [reflective equilibrium](https://en.wikipedia.org/wiki/Reflective_equilibrium), Harsanyi and Railton's [ideal advisor](https://intelligence.org/files/IdealAdvisorTheories.pdf) theories, and Frank Jackson's [moral functionalism](http://plato.stanford.edu/entries/naturalism-moral/#JacMorFun).

## Moore's Open Question

*Argument.*  If extrapolated volition is analytically equivalent to good, then the question "Is it true that extrapolated volition is good?" is meaningless or trivial.  However, this question is not meaningless or trivial, and seems to have an open quality about it.  Therefore, extrapolated volition is not analytically equivalent to goodness.

*Reply.*  Extrapolated volition is not supposed to be *transparently* identical to goodness.  The normative identity between extrapolated volition and goodness is allowed to be something that you would have to think for a while and consider many arguments to perceive.

Natively, human beings don't start out with any kind of explicit commitment to a particular metaethics; our brains just compute a feeling of rightness about certain acts, and then sometimes update and say that acts we previously thought were right are not-right.

When we go from that, to trying to draw a corresponding logical function that we can see our brains as approximating, and updating when we learn new things or consider new arguments, we are carrying out a project of "[rescuing the utility function](https://arbital.com/p/3y6)".  We are reasoning that we can best rescue our native state of confusion by seeing our reasoning about goodness as having its referent in certain logical facts, which lets us go on saying that it is better ceteris paribus for people to be happy than in severe pain, and that we can't reverse this ordering by taking a pill that alters our brain (we can only make our future self act on different logical questions), etcetera.  It's not surprising if this bit of philosophy takes longer than five minutes to reason through.

The notion that some act or policy "can in fact be wrong", even when you "think it is right", is more intuitive to some people than others; and it raises the question of what "rightness" is and [how to compute it](https://arbital.com/p/112).

On the extrapolated volition theory of metaethics, if you would change your mind about something after learning new facts or considering new arguments, then your updated state of mind is righter.  This can be true in advance of you knowing the facts.

E.g., maybe you currently want revenge on the Capulet family.  But if somebody had a chance to sit down with you and have a long talk about how revenge affects civilizations in the long run, you could be talked out of thinking that revenge, in general, is right.  So long as this *might* be true, it makes sense to say, "I want revenge on the Capulets, but maybe that's not really right."

Extrapolated volition is a normative moral theory.  It is a theory of how the concept of *shouldness* or *goodness* is or ought to be cashed out ([rescued](https://arbital.com/p/3y6)).  The corresponding proposal for [completely aligning](https://arbital.com/p/41k) a [fully self-directed superintelligence](https://arbital.com/p/1g3) is [coherent extrapolated volition](https://arbital.com/p/3c5).


Value-laden

An intuitive human category, or other humanly intuitive quantity or fact, is *value-laden* when it passes through human goals and desires, such that an agent couldn't reliably determine this intuitive category or quantity without knowing lots of complicated information about human goals and desires (and how to apply them to arrive at the intended concept).

In terms of [Hume's is-ought type distinction](https://arbital.com/p/), value-laden categories are those that humans compute using information from the ought side of the boundary, whether or not they notice they are doing so.

# Examples

## Impact vs. important impact

Suppose we want an AI to cure cancer, without this causing any *important side effects.*  What is or isn't an "important side effect" depends on what you consider "important".  If the cancer cure causes the level of thyroid-stimulating hormone to increase by 5%, this probably isn't very important.  If the cure increases the user's serotonin level by 5% and this significantly changes the user's emotional state, we'd probably consider that quite important.  But unless the AI already understands complicated human values, it doesn't necessarily have any way of knowing that one change in blood chemical levels is "not important" and the other is "important".

If you imagine the cancer cure as disturbing a set of variables $X_1, X_2, X_3...$ such that their values go from $x_1, x_2, x_3$ to $x_1^\prime, x_2^\prime, x_3^\prime$ then the question of which $X_i$ are *important* variables is value-laden.  If we temporarily mechanomorphize humans and suppose that we have a utility function, then we could say that variables are "important" when they're evaluated by our utility function, or when changes to those variables change our expected utility.

But by [orthogonality](https://arbital.com/p/1y) and [Humean freedom](https://arbital.com/p/2fs) of the utility function, there's an unlimited number of increasingly complicated utility functions that take into account different variables and functions of variables, so to know what we intuitively mean by "important", the AI would need information of high [algorithmic](https://arbital.com/p/5v) [complexity](https://arbital.com/p/5l) that the AI had no way to deduce *a priori*.  Which variables are "important" isn't a question of simple fact - it's on the "ought" side of the [Humean is-ought type distinction](https://arbital.com/p/) - so we can't assume that an AI which becomes increasingly good at answering "is"-type questions also knows which variables are "important".

Another way of looking at it is that if an AI merely builds a very good predictive model of the world, the set of "important variables" or "bad side effects" would be a squiggly category with a complicated boundary.  Even after the AI has already formed a rich natural is-language to describe concepts like "thyroid" and "serotonin" that are useful for modeling and predicting human biology, it might still require a long message in this language to exactly describe the wiggly boundary of "important impact" or the even more wiggly boundary of "bad impact".

This suggests that it might be simpler to try to tell the AI to cure cancer with [a minimum of *any* side effects](https://arbital.com/p/2pf), and [checking any remaining side effects](https://arbital.com/p/2qq) with the human operator.  If we have a set of "impacts" $X_k$ to be either minimized or checked which is broad enough to include, in passing, *everything* inside the squiggly boundary of the $X_h$ that humans care about, then this broader boundary of "any impact" might be smoother and less wiggly - that is, a short message in the AI's is-language, making it easier to learn.  For the same reason that a library containing every possible book has [less information](https://arbital.com/p/5v) than a library which only contains one book, a category boundary "impact" which includes everything a human cares about, *plus some other stuff,* can potentially be much simpler than an exact boundary drawn around "impacts we care about" which is value-laden because it involves caring.

From a human perspective, the complexity of our value system is already built into us and now appears as a deceptively simple-looking function call - relative to the complexity already built into us, "bad impact" sounds very obvious and very easy to describe.  This may lead people to underestimate the difficulty of training AIs to perceive the same boundary.  (Just list out all the impacts that potentially lower expected value, darn it!  Just the *important* stuff!)

## Faithful simulation vs. adequate simulation

Suppose we want to run an "adequate" or "good-enough" simulation of an uploaded human brain.  We can't say that an adequate simulation is one with identical input-output behavior to a biological brain, because the brain will almost certainly be a chaotic system, meaning that it's impossible for any simulation to get exactly the same result as the biological system would yield.  We nonetheless don't want the brain to have epilepsy, or to go psychopathic, etcetera.

The concept of an "adequate" simulation, in this case, is really standing in for "a simulation such that the *expected value* of using the simulated brain's information is within epsilon of using a biological brain".  In other words, our intuitive notion of what counts as a good-enough simulation is really a value-laden threshold because it involves an estimate of what's good enough.

So if we want an AI to have a notion of what kind of simulation is a faithful one, we might find it simpler to try to describe some superset of brain properties, such that if the simulated brain doesn't perturb the expectations of those properties, it doesn't perturb expected value either from our own intuitive standpoint (meaning the result of running the uploaded brain is equally valuable in our own expectation).  This set of faithfulness properties would need to automatically pick up on changes like psychosis, but could potentially pick up on a much wider range of other changes that we'd regard as unimportant, so long as all the important ones are in there.

## Personhood

Suppose that non-vegetarian programmers train an AGI on their intuitive category "person", such that:

- Rocks are not "people" and can be harmed if necessary.
- Shoes are not "people" and can be harmed if necessary.
- Cats are sometimes valuable to people, but are not themselves people.
- Alice, Bob, and Carol are "people" and should not be killed.
- Chimpanzees, dolphins, and the AGI itself: not sure, check with the users if the issue arises.

Now further suppose that the programmers haven't thought to cover, in the training data, any case of a cryonically suspended brain.  Is this a person?  Should it not be harmed?  On many 'natural' metrics, a cryonically suspended brain is more similar to a rock than to Alice.

From an intuitive perspective of avoiding harm to sapient life, a cryonically suspended brain has to be presumed a person until otherwise proven.  But the natural, or inductively simple category that covers the training cases is likely to label the brain a non-person, maybe with very high probability.  The fact that we want the AI to be careful not to hurt the cryonically suspended brain is the sort of thing you could only deduce by knowing which sort of things humans care about and why.  It's not a simple physical feature of the brain itself.

Since the category "person" is a value-laden one, when we extend it to a new region beyond the previous training cases, it's possible for an entirely new set of philosophical considerations to swoop in, activate, and control how we classify that case via considerations that didn't play a role in the previous training cases.

# Implications

Our intuitive evaluation of value-laden categories goes through our [Humean degrees of freedom](https://arbital.com/p/2fs).  This means that a value-laden category which a human sees as intuitively simple, can still have high [https://arbital.com/p/-5v](https://arbital.com/p/-5v), even relative to sophisticated models of the "is" side of the world.  This in turn means that even an AI understands the "is" side of the world very well, might not correctly and exactly learn a value-laden category from a small or incomplete set of training cases.

From the perspective of training an agent that hasn't yet been aligned along all the Humean degrees of freedom, value-laden categories are very wiggly and complicated relative to the agent's empirical language.  Value-laden categories are liable to contain exceptional regions that your training cases turned out not to cover, where from your perspective the obvious intuitive answer is a function of new value-considerations that the agent wouldn't be able to deduce from previous training data.

This is why much of the art in Friendly AI consists of trying to rephrase an alignment schema into terms that are simple relative to "is"-only concepts, where we want an AI with an impact-in-general metric, rather than AI which avoids only bad impacts.  "Impact" might have a simple, central core relative to a moderately sophisticated language for describing the universe-as-is.  "Bad impact" or "important impact" don't have a simple, central core and hence might be much harder to identify via training cases or communication.  Again, this is hard because humans do have all their subtle value-laden categories like 'important' built in as opaque function calls.  Hence people approaching value alignment for the first time often expect that various concepts are easy to identify, and tend to see the intuitive or intended values of all their concepts as "common sense" regardless of which side of the is-ought divide that common sense is on.

It's true, for example, that a modern chess-playing algorithm has "common sense" about when not to try to seize control of the gameboard's center; and similarly a sufficiently advanced agent would develop "common sense" about which substances would in empirical fact have which consequences on human biology, since this part is a strictly "is"-question that can be answered just by looking hard at the universe.  Not *wanting* to administer poisonous substances to a human requires a prior dispreference over the consequences of administering that poison, even if the consequences are correctly forecasted.  Similarly, the category "poison" could be said to really mean something like "a substance which, if administered to a human, produces low utility"; some people might classify vodka as poisonous, while others could disagree.  An AI doesn't necessary have common sense about the [intended](https://arbital.com/p/6h) evaluation of the "poisonous" category, even if it has fully developed common sense about which substances have which empirical biological consequences when ingested.  One of those forms of common sense can be developed by staring very intelligently at biological data, and one of them cannot.  But from a human intuitive standpoint, both of these can *feel* equally like the same notion of "common sense", which might lead to a dangerous expectation that an AI gaining in one type of common sense is bound to gain in the other.

# Further reading

- http://lesswrong.com/lw/td/magical_categories/

A category, estimate, or fact is *value-laden* when it passes through human goals and desires, so that an AI couldn't compute the [intended meaning](https://arbital.com/p/6h) without understanding [complicated human values](https://arbital.com/p/5l).

For instance, if you tell an AI to "cure cancer without bad side effects", the notion of what is a bad side effect is value-laden.  Even the notion of "important side effect" is value-laden, since by "important side effect" you really mean "a side effect I care about" or "a side effect that will perturb [value](https://arbital.com/p/55)".  It might be simpler to give the AI a category of "[all side effects](https://arbital.com/p/2pf)" to be minimized or checked with the user.

In terms of [Hume's is-ought type distinction](https://arbital.com/p/), value-laden categories are those that humans compute using information from the ought side of the boundary.  This means value-laden categories rely on our [Humean degrees of freedom](https://arbital.com/p/2fs) and therefore have high [https://arbital.com/p/-5v](https://arbital.com/p/-5v) and aren't automatically learned easily or induced as simple even by very advanced AIs.


Faithful simulation

The safe simulation problem is to start with some dynamical physical process $D$ which would, if run long enough in some specified environment, produce some trustworthy information of great value, and to compute some *adequate* simulation $S_D$ of $D$ faster than the physical process could have run.  In this context, the term "adequate" is [value-laden](https://arbital.com/p/36h) - it means that whatever we would use $D$ for, using $S_D$ instead produces within epsilon of the expected [value](https://arbital.com/p/55) we could have gotten from using the real $D.$  In more concrete terms, for example, we might want to tell a Task AGI "upload this human and run them as a simulation", and we don't want some tiny systematic skew in how the Task AGI models serotonin to turn the human into a psychopath, which is a *bad* (value-destroying) simulation fault.  Perfect simulation will be out of the question; the brain is almost certainly a chaotic system and hence we can't hope to produce *exactly* the same result as a biological brain.  The question, then, is what kind not-exactly-the-same-result the simulation is allowed to produce.

As with "[low impact](https://arbital.com/p/2pf)" hopefully being lower-[complexity](https://arbital.com/p/5v) than "[low bad impact](https://arbital.com/p/36h)", we might hope to get an *adequate* simulation via some notion of *faithful* simulation, which rules out bumps in serotonin that turn the upload into a psychopath, while possibly also ruling out any number of other changes we *wouldn't* see as important; with this notion of "faithfulness" still being permissive enough to allow the simulation to take place at a level above individual quarks.  On whatever computing power is available - possibly nanocomputers, if the brain was scanned via molecular nanotechnology - the upload must be runnable fast enough to [make the simulation task worthwhile](https://arbital.com/p/6y).

Since the main use for the notion of "faithful simulation" currently appears to be [identifying](https://arbital.com/p/2s3) a safe plan for uploading one or more humans as a [pivotal act](https://arbital.com/p/6y), we might also consider this problem in conjunction with the special case of wanting to avoid [mindcrime](https://arbital.com/p/6v).  In other words, we'd like a criterion of faithful simulation which the AGI can compute *without* it needing to observe millions of hypothetical simulated brains for ten seconds apiece, which could constitute creating millions of people and killing them ten seconds later.  We'd much prefer, e.g., a criterion of faithful simulation of individual neurons and synapses between them up to the level of, say, two interacting cortical columns, such that we could be confident that in aggregate the faithful simulation of the neurons would correspond to the faithful simulation of whole human brains.  This way the AGI would not need to think about or simulate whole brains in order to verify that an uploading procedure would produce a faithful simulation, and mindcrime could be avoided.

Note that the notion of a "functional property" of the brain - seeing the neurons as computing something important, and not wanting to disturb the computation - is still value-laden.  It involves regarding the brain as a means to a computational end, and what we see as the important computational end is value-laden, given that chaos guarantees the input-output relation won't be *exactly* the same.  The brain can equally be seen as implicitly computing, say, the parity of the number of synapse activations; it's just that we don't see this functional property as a valuable one that we want to preserve.

To the extent that some notion of function might be invoked in a notion of faithful, permitted speedups, we should hope that rather than needing the AGI to understand the high-level functional properties of the brain *and which details we thought were too important to simplify,* it might be enough to understand a 'functional' model of individual neurons and synapses, with the resulting transform of the uploaded brain still allowing for a pivotal speedup *and* knowably-faithful simulation of the larger brain.

At the same time, strictly local measures of faithfulness seem problematic if they can conceal *systematic* larger divergences.  We might think that any perturbation of a simulated neuron which has as little effect as adding one phonon is "within thermal uncertainty" and therefore unimportant, but if all of these perturbations are pointing in the same direction relative to some larger functional property, the difference might be very significant.  Similarly if all simulated synapses released slightly more serotonin, rather than releasing slightly more or less serotonin in no particular systematic pattern.


Identifying causal goal concepts from sensory data

Suppose we want an AI to carry out some goals involving strawberries, and as a result, we want to [identify](https://arbital.com/p/identify_goal_concept) to the AI the [concept](https://arbital.com/p/) of "strawberry".  One of the potential ways we could do this is by showing the AI objects that a teacher classifies as strawberries or non-strawberries.  However, in the course of doing this, what the AI actually sees will be e.g. a pattern of pixels on a webcam - the actual, physical strawberry is not directly accessible to the AI's intelligence.  When we show the AI a strawberry, what we're really trying to communicate is "A certain proximal [cause](https://arbital.com/p/) of this sensory data is a strawberry", not, "This arrangement of sensory pixels is a strawberry."  An AI that learns the latter concept might try to carry out its goal by putting a picture in front of its webcam; the former AI has a goal that actually involves something in its environment.

The open problem of "identifying causal goal concepts from sensory data" or "identifying environmental concepts from sensory data" is about getting an AI to form [causal](https://arbital.com/p/) goal concepts instead of [sensory](https://arbital.com/p/) goal concepts.  Since almost no [human-intended goal](https://arbital.com/p/6h) will ever be satisfiable solely in virtue of an advanced agent arranging to see a certain field of pixels, safe ways of identifying goals to sufficiently advanced goal-based agents will presumably involve some way of identifying goals among the *causes* of sense data.

A "toy" (and still pretty difficult) version of this open problem might be to exhibit a machine algorithm that (a) has a causal model of its environment, (b) can learn concepts over any level of its causal model including sense data, (c) can learn and pursue a goal concept, (d) has the potential ability to spoof its own senses or create fake versions of objects, and (e) is shown to learn a proximal causal goal rather than a goal about sensory data as shown by it pursuing only the causal version of that goal even if it would have the option to spoof itself.

For a more elaborated version of this open problem, see "[https://arbital.com/p/2s0](https://arbital.com/p/2s0)".

Look where I'm pointing, not at my finger


Goal-concept identification

The problem of trying to figure out how to communicate to an AGI an [intended](https://arbital.com/p/6h) goal [concept](https://arbital.com/p/) on the order of "give me a strawberry, and not a fake plastic strawberry either".

At this level of the problem, we're not concerned with e.g. larger problems of [safe plan identification](https://arbital.com/p/2s3) such as not mugging people for strawberries, or [minimizing side effects](https://arbital.com/p/2pf).  We're not (at this level of the problem) concerned with identifying [each and every one of the components of human value](https://arbital.com/p/6c), as they might be impacted by side effects more distant in the causal graph.  We're not concerned with [philosophical uncertainty](https://arbital.com/p/) about what we [should](https://arbital.com/p/normativity) mean by "strawberry".  We suppose that in an intuitive sense, we do have a pretty good idea of what we [intend](https://arbital.com/p/6h) by "strawberry", such that there are things that are definitely strawberries and we're pretty happy with our sense of that so long as nobody is deliberately trying to fool it.

We just want to communicate a *local* goal concept that distinguishes edible strawberries from plastic strawberries, or nontoxic strawberries from poisonous strawberries.  That is: we want to say "strawberry" in an understandable way that's suitable for fulfilling a [task](https://arbital.com/p/6w) of "just give Sally a strawberry", possibly in conjunction with other features like [conservatism](https://arbital.com/p/2qp) or [low impact](https://arbital.com/p/2pf) or [mild optimization](https://arbital.com/p/2r8).

For some open subproblems of the obvious approach that goes through showing actual strawberries to the AI's webcam, see "[https://arbital.com/p/36w](https://arbital.com/p/36w)" and "[https://arbital.com/p/2s0](https://arbital.com/p/2s0)".

The problem of communicating to an AI a very simple local concept on the order of "strawberries" or "give me a strawberry".  This level of the problem is meant to include subproblems of local categorization like "I meant a real strawberry like the ones I already showed you, not a fake strawberry that looks similar to your webcam".  It isn't meant to include larger problems like [verifying that a plan uses only known, whitelisted methods](https://arbital.com/p/2s3) or [identifying all possible harmful effects we could care about](https://arbital.com/p/6c).


Mechanical Turk (example)

In 1836, there was a sensation called the [Mechanical Turk](https://en.wikipedia.org/wiki/The_Turk), allegedly a chess-playing automaton. Edgar Allen Poe, who was also an amateur magician, wrote an essay arguing that the Turk must contain a human operator hidden in the apparatus (which it did).  Besides analyzing the Turk's outward appearance to locate the hidden compartment, Poe carefully argued as to why no arrangement of wheels and gears could ever play chess in the first place, explicitly comparing the Turk to "the calculating machine of Mr. Babbage":

> Arithmetical or algebraical calculations are, from their very nature, fixed and determinate. Certain data being given, certain results necessarily and inevitably follow [https://arbital.com/p/...](https://arbital.com/p/...)
> But the case is widely different with the Chess-Player. With him there is no determinate progression. No one move in chess necessarily follows upon any one other. From no particular disposition of the men at one period of a game can we predicate their disposition at a different period [https://arbital.com/p/...](https://arbital.com/p/...)
> Now even granting that the movements of the Automaton Chess-Player were in themselves determinate, they would be necessarily interrupted and disarranged by the indeterminate will of his antagonist. There is then no analogy whatever between the operations of the Chess-Player, and those of the calculating machine of Mr. Babbage [https://arbital.com/p/...](https://arbital.com/p/...)
> It is quite certain that the operations of the Automaton are regulated by *mind*, and by nothing else. Indeed this matter is susceptible of a mathematical demonstration, *a priori*.

(In other words:  In an algebraical problem, each step follows with the previous step of necessity, and therefore can be represented by the determinate motions of wheels and gears as in Charles Babbage's proposed computing engine.  In chess, the player's move and opponent's move don't follow with necessity from the board position, and therefore can't be represented by deterministic gears.)

This is an amazingly sophisticated remark, considering the era.  It even puts a finger on the part of chess that is computationally difficult, the combinatorial explosion of possible moves.  And it is still entirely wrong.

Even if you know an unbounded solution to chess, you might still be 47 years away from a bounded solution.  But if you can't state a program that solves the problem *in principle*, you are in some sense *confused* about the nature of the cognitive work needed to solve the problem.  If you can't even solve a problem given infinite computing power, you definitely can't solve it using bounded computing power.  (Imagine Poe trying to write a chess-playing program before he'd had the insight about search trees.)

For more on this point, see "[https://arbital.com/p/107](https://arbital.com/p/107)".

In 1836, there was a sensation called the [Mechanical Turk](https://en.wikipedia.org/wiki/The_Turk), allegedly a chess-playing automaton. Edgar Allen Poe, who was also an amateur magician, wrote an essay arguing that the Turk must contain a human operator hidden in the apparatus (which it did).

Besides analyzing the Turk's outward appearance to locate the hidden compartment, Poe carefully argued as to why no arrangement of wheels and gears could ever play chess in the first place, explicitly comparing the Turk to "the calculating machine of Mr. Babbage".  In an algebraical problem (Poe said) each step follows with the previous step of necessity, and therefore can be represented by the determinate motions of wheels and gears as in Charles Babbage's proposed computing engine.  In chess, the player's move and opponent's move don't follow with necessity from the board position, and therefore can't be represented by deterministic gears.

Thus, Poe concluded, "It is quite certain that the operations of the Automaton are regulated by *mind*, and by nothing else. Indeed this matter is susceptible of a mathematical demonstration, *a priori*."


Do we need to worry about AI?

Why worry about AI when nothing has gone wrong yet?