Coherent extrapolated volition (alignment target)

#  Introduction

"Coherent extrapolated volition" (CEV) is [Eliezer Yudkowsky](https://arbital.com/p/2)'s proposed thing-to-do with an extremely [advanced AGI](https://arbital.com/p/2c), if you're extremely confident of your ability to [align](https://arbital.com/p/5s) it on complicated targets.

Roughly, a CEV-based superintelligence would do what currently existing humans would want\* the AI to do, *if counterfactually:*

 1. We knew everything the AI knew;
 2. We could think as fast as the AI and consider all the arguments;
 3. We knew ourselves perfectly and had better self-control or self-modification ability;

...*to whatever extent* most existing humans, thus extrapolated, would predictably want\* the same things.  (For example, in the limit of extrapolation, nearly all humans might want\* not to be turned into [paperclips](https://arbital.com/p/10h), but might not agree\* on the best pizza toppings.  See below.)

CEV is meant to be the *literally optimal* or *ideal* or *normative* thing to do with an [autonomous superintelligence](https://arbital.com/p/1g3), *if* you trust your ability to [perfectly align](https://arbital.com/p/41k) a superintelligence on a very complicated target.  (See below.)

CEV is rather complicated and meta and hence *not* intended as something you'd do with the first AI you ever tried to build.  CEV might be something that everyone inside a project agreed was an acceptable mutual target for their *second* AI.  (The first AI should probably be a [Task AGI](https://arbital.com/p/6w).)

For the corresponding metaethical theory see [https://arbital.com/p/313](https://arbital.com/p/313).

# Concept

%%knows-requisite([https://arbital.com/p/313](https://arbital.com/p/313)):

See "[https://arbital.com/p/313](https://arbital.com/p/313)".

%%

%%!knows-requisite([https://arbital.com/p/313](https://arbital.com/p/313)):

[Extrapolated volition](https://arbital.com/p/313) is the metaethical theory that when we ask "What is right?", then insofar as we're asking something meaningful, we're asking "What would a counterfactual idealized version of myself want\* if it knew all the facts, had considered all the arguments, and had perfect self-knowledge and self-control?"  (As a [metaethical theory](https://arbital.com/p/313), this would make "What is right?" a mixed logical and empirical question, a function over possible states of the world.)

A very simple example of extrapolated volition might be to consider somebody who asks you to bring them orange juice from the refrigerator.  You open the refrigerator and see no orange juice, but there's lemonade.  You imagine that your friend would want you to bring them lemonade if they knew everything you knew about the refrigerator, so you bring them lemonade instead.  On an abstract level, we can say that you "extrapolated" your friend's "volition", in other words, you took your model of their mind and decision process, or your model of their "volition", and you imagined a counterfactual version of their mind that had better information about the contents of your refrigerator, thereby "extrapolating" this volition.

Having better information isn't the only way that a decision process can be extrapolated; we can also, for example, imagine that a mind has more time in which to consider moral arguments, or better knowledge of itself.  Maybe you currently want revenge on the Capulet family, but if somebody had a chance to sit down with you and have a long talk about how revenge affects civilizations in the long run, you could be talked out of that.  Maybe you're currently convinced that you advocate for green shoes to be outlawed out of the goodness of your heart, but if you could actually see a printout of all of your own emotions at work, you'd see there was a lot of bitterness directed at people who wear green shoes, and this would change your mind about your decision.

In Yudkowsky's version of extrapolated volition considered on an individual level, the three core directions of extrapolation are:

- Increased knowledge - having more veridical knowledge of declarative facts and expected outcomes.
- Increased consideration of arguments - being able to consider more possible arguments and assess their validity.
- Increased reflectivity - greater knowledge about the self, and to some degree, greater self-control (though this raises further questions about which parts of the self normatively get to control which other parts).

%%

# Motivation

Different people initially react differently to the question "Where should we point a superintelligence?" or "What should an aligned superintelligence do?" - not just different beliefs about what's good, but different frames of mind about how to ask the question.

Some common reactions:

1. "Different people want different things!  There's no way you can give everyone what they want.  Even if you pick some way of combining things that people want, *you'll* be the one saying how to combine it.  Someone else might think they should just get the whole world for themselves.  Therefore, in the end *you're* deciding what the AI will do, and any claim to some sort of higher justice or normativity is nothing but sophistry."
2. "What we should do with an AI is obvious - it should optimize liberal democratic values.  That already takes into account everyone's interests in a fair way.  The real threat is if bad people get their hands on an AGI and build an AGI that doesn't optimize liberal democratic values."
3. "Imagine the ancient Greeks telling a superintelligence what to do.  They'd have told it to optimize for glorious deaths in battle.  Programming any other set of inflexible goals into a superintelligence seems equally stupid; it has to be able to change and grow."
4. "What if we tell the superintelligence what to do and it's the wrong thing? What if we're basically confused about what's right? Shouldn't we let the superintelligence figure that out on its own, with its assumed superior intelligence?"

An initial response to each of these frames might be:

1.  "Okay, but suppose you're building a superintelligence and you're trying *not to be a jerk about it.*  If you say, 'Whatever I do originates in myself, and therefore is equally selfish, so I might as well declare myself God-Emperor of the Universe' then you're being a jerk.  Is there anything you could do instead which would be less like being a jerk?  What's the *least* jerky thing you could do?"
2.  "What if you would, after some further discussion, want to tweak your definition of 'liberal democratic values' just a little? What if it's *predictable* that you would do that? Would you really want to be stuck with your off-the-cuff definition a million years later?"
3.  "Okay, so what should the Ancient Greeks have done if they did have to program an AI? How could they not have doomed future generations? Suppose the Ancient Greeks are clever enough to have noticed that sometimes people change their minds about things and to realize that they might not be right about everything. How can they use the cleverness of the AGI in a constructively specified, computable fashion that gets them out of this hole? You can't just tell the AGI to compute what's 'right', you need to put an actual computable question in there, not a word."
4.  "You asked, what if we're basically confused about what's right - well, in that case, what does the word 'right' even mean?  If you don't know what's right, and you don't know how to compute what's right, then what are we even talking about?  Do you have any ground on which to say that an AGI which only asks 'Which outcome leads to the greatest number of paperclips?' isn't computing rightness?  If you don't think a paperclip maximizer is computing rightness, then you must know something about the rightness-question which excludes that possibility - so let's talk about how to program that rightness-question into an AGI."

CEV's advocates claim that all of these lines of discussion eventually end up converging on the idea of coherent extrapolated volition.  For example:

1.  Asking what everyone would want\* if they knew what the AI knew, and doing what they'd all predictably agree on, is just about the least jerky thing you can do.  If you tell the AI to give everyone a volcano lair because you think volcano lairs are neat, you're not being selfish, but you're being a jerk to everyone who doesn't want a volcano lair.  If you have the AI just do what people actually say, they'll end up hurting themselves with dumb wishes and you'd be a jerk.  If you only extrapolate your friends and have the AI do what only you'd want, you're being jerks to everyone else.
2.  Yes, liberal democratic values are good; so is apple pie.  Apple pie is *a* good thing but it's not the *only* good thing.  William Frankena's list of ends-in-themselves included "Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment" and then 25 more items, and the list certainly isn't complete.  The only way you're going to get a complete list is by analyzing human minds; and even then, if our descendants would predictably want something else a million years later, we ought to take that into account too.
3.  Every improvement is a change, but not every change is an improvement. Just letting a superintelligence change at random doesn't encapsulate moral progress.  Saying that change toward more liberal democratic values is progress, presumes that we already know the destination or answer.  We can't even just ask the AGI to *predict* what civilizations would think a thousand years later, since (a) the AI itself impacts this and (b) if the AI did nothing, maybe in a thousand years everyone would have accidentally blissed themselves out while trying to modify their own brains.  If we want to do better than the hypothetical ancient Greeks, we need to define *a sufficiently abstract and meta criterion* that describes *valid* directions of progress - such as changes in moral beliefs associated with learning new facts, for example; or moral change that would predictably occur if we considered a larger set of arguments; or moral change that would predictably occur if we understood ourselves better.
4.  This one is a long story:  Metaethics deals with the question of what sort of entity 'rightness' is exactly - tries to reconcile this strange ineffable 'rightness' business with a universe made out of particle fields.  Even though it seems like human beings wanting to murder people wouldn't *make* murder right, there's also nowhere in the stars or mountains where we can actually find it written that murder is wrong.  At the end of a rather long discussion, we decide that for any given person speaking at a given point in time, 'rightness' is a logical constant which, although not counterfactually dependent on the state of the person's brain, must be analytically identified with the extrapolated volition of that brain; and we show that (only) this stance gives consistent answers to all the standard questions in metaethics.  (This discussion takes a while, on the order of explaining how deterministic laws of physics don't show that you have unfree will.)

(To do:  Write dialogues from each of these four entrance points.)

# Situating CEV in contemporary metaethics

See the corresponding section in "[https://arbital.com/p/313](https://arbital.com/p/313)".

# Scary design challenges

There are several reasons why CEV is *way* too challenging to be a good target for any project's first try at building machine intelligence:

1.  A CEV agent would be intended to carry out an [autonomous](https://arbital.com/p/1g3) open-ended mission.  This implies all the usual reasons we expect an autonomous AI to be harder to make safe than a [Task AGI](https://arbital.com/p/6w).
2.  CEV is a weird goal.  It involves recursion.
3.  Even the terms in CEV, like "know more" or "extrapolate a human", seem complicated and [value-laden](https://arbital.com/p/36h).  You might have to build a high-level [Do What I Know I Mean agent](https://arbital.com/p/2s1), and then tell it to do CEV.  [Do What I Know I Mean](https://arbital.com/p/2s1) is complicated enough that you'd need to build an AI that can learn DWIKIM, so that DWIKIM can be taught rather than formally specified.  So we're looking at something like CEV, running on top of DWIKIM, running on top of a goal-learning system, at least until the first time the CEV agent rewrites itself.

Doing this correctly the *very first time* we build a smarter-than-human intelligence seems *improbable.*  The only way this would make a good first target is if the CEV concept is formally simpler than it currently seems, *and* timelines to AGI are unusually long and permit a great deal of advance work on safety.

If AGI is 20 years out (or less), it seems wiser to think in terms of a [Task AGI](https://arbital.com/p/6w) performing some relatively simple [pivotal act](https://arbital.com/p/6y).  The role of CEV is of answering the question, "What can you all agree in advance that you'll try to do *next,* after you've executed your Task AGI and gotten out from under the shadow of immediate doom?"

# What if CEV fails to cohere?

A frequently asked question is "What if extrapolating human volitions produces incoherent answers?"

According to the original motivation for CEV, if this happens in *some* places, a Friendly AI ought to ignore those places.  If it happens *everywhere,* you probably picked a silly way to construe an extrapolated volition and you ought to rethink it. %note: Albeit in practice, you would not want an AI project to take a dozen tries at defining CEV.  This would indicate something extremely wrong about the method being used to generate suggested answers.  Whatever final attempt passed would probably be the first answer [all of whose remaining flaws were hidden](https://arbital.com/p/blackzoning), rather than an answer with all flaws eliminated.%

That is:

- If your CEV algorithm finds that "People coherently want to not be eaten by [paperclip maximizers](https://arbital.com/p/10h), but end up with a broad spectrum of individual and collective possibilities for which pizza toppings they prefer", we would normatively want a Friendly AI to prevent people from being eaten by paperclip maximizers but not mess around with which pizza toppings people end up eating in the Future.
- If your CEV algorithm claims that there's no coherent sense in which "A lot of people would want to not be eaten by Clippy and would still want\* this even if they knew more stuff" then this is a suspicious and unexpected result.  Perhaps you have picked a silly way to construe somebody's volition.

The original motivation for CEV can also be viewed from the perspective of "What is it to help someone?" and "How can one help a large group of people?", where the intent behind the question is to build an AI that renders 'help' as we really [intend](https://arbital.com/p/6h) that.  The elements of CEV can be seen as caveats to the naive notion of "Help is giving people whatever they ask you for!" in which somebody asks you to bring them orange juice but the orange juice in the refrigerator is poisonous (and they're not *trying* to poison themselves).

What about helping a group of people?  If two people ask for juice and you can only bring one kind of juice, you should bring a non-poisonous kind of juice they'd both like, to the extent any such juice exists.  If no such juice exists, find a kind of juice that one of them is meh about and that the other one likes, and flip a coin or something to decide who wins.  You are then being around as helpful as it is possible to be.

Can there be *no way* to help a large group of people?  This seems implausible.  You could at least give the starving ones pizza with a kind of pizza topping they currently like.  To the extent your philosophy claims "Oh noes even *that* is not helping because it's not perfectly coherent," you have picked the wrong construal of 'helping'.

It could be that, if we find that every reasonable-sounding construal of extrapolated volition fails to cohere, we must arrive at some entirely other notion of 'helping'.  But then this new form of helping also shouldn't involve bringing people poisonous orange juice that they don't know is poisoned, because that still intuitively seems unhelpful.

## Helping people with incoherent preferences

What if somebody believes themselves to [prefer onions to pineapple on their pizza, prefer pineapple to mushrooms, and prefer mushrooms to onions](https://arbital.com/p/7hh)?  In the sense that, offered any two slices from this set, they would pick according to the given ordering?

(This isn't an unrealistic example.  Numerous experiments in behavioral economics demonstrate exactly this sort of circular preference.  For instance, you can arrange 3 items such that each pair of them brings a different salient quality into focus for comparison.)

One may worry that we couldn't 'coherently extrapolate the volition' of somebody with these pizza preferences, since these local choices obviously aren't consistent with any coherent utility function.  But how could we *help* somebody with a pizza preference like this?

Well, appealing to the intuitive notion of *helping:*

- We could give them whatever kind of pizza they'd pick if they had to pick among all three simultaneously.
- We could figure out how happy they'd be eating each type of pizza, in terms of emotional intensity as measured in neurotransmitters; and offer them the slice of pizza that they'll most enjoy.
- We could let them pick their own damn pizza toppings and concern ourselves mainly with making sure the pizza isn't poisonous, since the person definitely prefers non-poisoned pizza.
- We could, given sufficient brainpower on our end, figure out what this person would ask us to do for them in this case after that person had learned about the concept of a preference reversal and been told about their own circular preferences.  If this varies wildly depending on exactly how we explain the concept of a preference reversal, we could refer back to one of the previous three answers instead.

Conversely, these alternatives seem *less* helpful:

- Refuse to have anything to do with that person since their current preferences don't form a coherent utility function.
- Emit "ERROR ERROR" sounds like a Hollywood AI that's just found out about the Epimenides Paradox.
- Give them pizza with your own favorite topping, green peppers, even though they'd prefer any of the 3 other toppings to those.
- Give them pizza with the topping that would taste best to them, pepperoni, despite their being vegetarians.

Advocates of CEV claim that if you blank the complexities of 'extrapolated volition' out of your mind; and ask how you could reasonably help people as best as possible if you were trying not be a jerk; and then try to figure out how to semiformalize whatever mental procedure you just followed to arrive at your answer for how to help people; then you will eventually end up at CEV again.

# Role of meta-ideals in promoting early agreement

A primary purpose of CEV is to represent a relatively simple meta-level ideal that people can agree upon, even where they might disagree on the object level.  By a hopefully analogous example, two honest scientists might disagree on the correct mass of an electron, but agree that the experimental method is a good way to resolve the answer.

Imagine Millikan believes an electron's mass is 9.1e-28 grams, and Nannikan believes the correct electron mass is 9.1e-34 grams.  Millikan might be very worried about Nannikan's proposal to program an AI to believe the electron mass is 9.1e-34 grams; Nannikan doesn't like Millikan's proposal to program in 9.1-e28; and both of them would be unhappy with a compromise mass of 9.1e-31 grams.  They might still agree on programming an AI with some analogue of probability theory and a simplicity prior, and letting a superintelligence come to the conclusions implied by Bayes and Occam, because the two can agree on an effectively computable question even though they think the question has different answers.  Of course, this is easier to agree on when the AI hasn't yet produced an answer, or if the AI doesn't tell you the answer.

It's not guaranteed that every human embodies the same implicit moral questions, indeed this seems unlikely, which means that Alice and Bob might still expect their extrapolated volitions to disagree about things.  Even so, while the outputs are still abstract and not-yet-computed, Alice doesn't have much of a place to stand on which to appeal to Carol, Dennis, and Evelyn by saying, "But as a matter of morality and justice, you should have the AI implement *my* extrapolated volition, not Bob's!"  To appeal to Carol, Dennis, and Evelyn about this, you'd need them to believe that Alice's EV was more likely to agree with their EVs than Bob's was - and at that point, why not come together on the obvious Schelling point of extrapolating *everyone's* EVs?

Thus, one of the primary purposes of CEV (selling points, design goals) is that it's something that Alice, Bob, and Carol can agree *now* that Dennis and Evelyn should do with an AI that will be developed *later;* we can try to set up commitment mechanisms now, or check-and-balance mechanisms now, to ensure that Dennis and Evelyn are still working on CEV later.

## Role of 'coherence' in reducing expected unresolvable disagreements

A CEV is not necessarily a majority vote.  A lot of people with an extrapolated weak preference\* might be counterbalanced by a few people with a strong extrapolated preference\* in the opposite direction.  Nick Bostrom's "[parliamentary model](http://www.overcomingbias.com/2009/01/moral-uncertainty-towards-a-solution.html)" for resolving uncertainty between incommensurable ethical theories, permits a subtheory very concerned about a decision to spend a large amount of its limited influence on influencing that particular decision.

This means that, e.g., a vegan or animal-rights activist should not need to expect that they must seize control of a CEV algorithm in order for the result of CEV to protect animals.  It doesn't seem like most of humanity would be deriving huge amounts of utility from hurting animals in a post-superintelligence scenario, so even a small part of the population that strongly opposes\* this scenario should be decisive in preventing it.

# Moral hazard vs. debugging

One of the points of the CEV proposal is to have minimal [moral hazard](https://arbital.com/p/2sb) (aka, not tempting the programmers to take over the world or the future); but this may be compromised if CEV's results don't go literally unchecked.

Part of the purpose of CEV is to stand as an answer to the question, "If the ancient Greeks had been the ones to invent superintelligence, what could they have done that would not, from our later perspective, irretrievably warp the future?  If the ancient Greeks had programmed in their own values directly, they would have programmed in a glorious death in combat.  Now let us consider that perhaps we too are not so wise."  We can imagine the ancient Greeks writing a CEV mechanism, peeking at the result of this CEV mechanism before implementing it, and being horrified by the lack of glorious-deaths-in-combat in the future and value system thus revealed.

We can also imagine that the Greeks, trying to cut down on [moral hazard](https://arbital.com/p/2sb), virtuously refuse to peek at the output; but it turns out that their attempt to implement CEV has some unforeseen behavior when actually run by a superintelligence, and so their world is turned into paperclips.

This is a safety-vs.-moral-hazard tradeoff between (a) the benefit of being able to look at CEV outputs in order to better-train the system or just verify that nothing went horribly wrong; and (b) the moral hazard that comes from the temptation to override the output, thus defeating the point of having a CEV mechanism in the first place.

There's also a potential safety hazard just with looking at the internals of a CEV algorithm; the simulated future could contain all sorts of directly mind-hacking cognitive hazards.

Rather than giving up entirely and embracing maximum moral hazard, one possible approach to this issue might be to have some single human that is supposed to peek at the output and provide a 1 or 0 (proceed or stop) judgment to the mechanism, without any other information flow being allowed to the programmers if the human outputs 0.  (For example, the volunteer might be in a room with explosives that go off if 0 is output.)

# "Selfish bastards" problem

Suppose that Fred is funding Grace to work on a CEV-based superintelligence; and Evelyn has decided not to oppose this project.  The resulting CEV is meant to extrapolate the volitions of Alice, Bob, Carol, Dennis, Evelyn, Fred, and Grace with equal weight.  (If you're reading this, you're more than usually likely to be one of Evelyn, Fred, or Grace.)

Evelyn and Fred and Grace might worry:  "What if a supermajority of humanity consists of 'selfish\* bastards', such that their extrapolated volitions would cheerfully vote\* for a world in which it was legal to own artificial sapient beings as slaves so long as they personally happened to be in the slaveowning class; and we, Evelyn and Fred and Grace, just happen to be in the minority that extremely doesn't want nor want\* the future to be like that?"

That is: What if humanity's extrapolated volitions diverge in such a way that from the standpoint of *our* volitions - since, if you're reading this, you're unusually likely to be one of Evelyn or Fred or Grace - 90% of extrapolated humanity would choose\* something such that *we* would not approve of it, and our volitions would not approve\* of it, *even after* taking into account that we don't want to be jerks about it and that we don't think we were born with any unusual or exceptional right to determine the fate of humanity.

That is, let the scenario be as follows:

> 90% of the people (but not we who are collectively sponsoring the AI) are selfish bastards at the core, such that *any reasonable* extrapolation process (it's not just that we picked a broken one) would lead to them endorsing a world in which they themselves had rights, but it was okay to create artificial people and hurt them.  Furthermore, they would derive enough utility from being personal God-Emperors that this would override our minority objection even in a parliamentary model.

We can see this hypothetical outcome as potentially undermining every sort of reason that we, who happen to be in a position of control to prevent that outcome, should voluntarily relinquish that control to the remaining 90% of humanity:

- We can't be prioritizing being fair to everyone including the other 90% of humanity, because what about being fair to the artificial people who are being hurt?
- We can't be worrying that the other 90% of humanity would withdraw their support from the project, or worrying about betraying the project's supporters, because by hypothesis they weren't supporting it or even permitting it.
- We can't be agreeing to defer to a righter and more intelligent process to resolve our dispute, because by hypothesis the CEV made up of 90% selfish\* bastards is not, from our own perspective, ideally righter.
- We can't rely on a parliamentary model of coherence to prevent what a minority sees as disaster, because by hypothesis the other 90% is deriving enough utility from collectively declaring themselves God-Emperors to trump even a strong minority countervote.

Rather than giving up entirely and taking over the world, *or* exposing ourselves to moral hazard by peeking at the results, one possible approach to this issue might be to run a three-stage process.

This process involves some internal references, so the detailed explanation needs to follow a shorter summary explanation.

In summary:

- Extrapolate everyone's CEV.
- Extrapolate the CEV of the contributors only, and let it give (only) an up-down vote on Everyone's CEV.
- If the result is thumbs-up, run Everyone's CEV.
- Otherwise, extrapolate everyone's CEV, but kicking out all the parts that would act unilaterally and without any concern for others if they were in positions of unchecked power.
- Have the Contributor CEV give an up/down answer on the Fallback CEV.
- If the result is thumbs-up, run the Fallback CEV.
- Otherwise fail.

In detail:

- First, extrapolate the everyone-on-Earth CEV as though it were not being checked.
  - If any hypothetical extrapolated person worries about being checked, delete that concern and extrapolate them as though they didn't have it.  This is necessary to prevent the check itself from having a [UDT](https://arbital.com/p/updateless) influence on the extrapolation and the actual future.
- Next, extrapolate the CEV of everyone who contributed to the project, weighted by their contribution (possibly based on some mix of "how much was actually done" versus "how much was rationally expected to be accomplished" versus "the fraction of what could've been done versus what was actually done").  Allow this other extrapolation an up-or-down vote - *not* any kind of detailed correction - on whether to let the everyone-on-Earth CEV to go through unmodified.
  - Remove from the extrapolation of the Contributor-CEV any strategic considerations having to do with the Fallback-CEV or post-Fail redevelopment being a *better* alternative; we want to extract a judgment about "satisficing" in some sense, whether the Everyone-CEV is in some non-relative sense too horrible to be allowed.
- If the Everyone-CEV passes the Contributor-CEV check, run it.
- Otherwise, re-extrapolate a Fallback-CEV that starts with all existing humans as a base, but *discards* from the extrapolation all extrapolated decision processes that, if they were in a superior strategic position or a position of unilateral power, would *not* bother to extrapolate others' volitions or care about their welfare.
  - Again, remove all extrapolated *strategic* considerations about passing the coming check.
- Check the Fallback-CEV against the Contributor-CEV for an up-down vote.  If it passes, run it.
- Otherwise Fail (AI shuts down safely, we rethink what to do next or implement an agreed-on fallback course past this point).

The particular fallback of "kick out from the extrapolation any weighted portions of extrapolated decision processes that would act unilaterally and without caring for others, given unchecked power" is meant to have a property of poetic justice, or rendering objections to it self-defeating:  If it's okay to act unilaterally, then why can't we unilaterally kick out the unilateral parts?  This is meant to be the 'simplest' or most 'elegant' way of kicking out a part of the CEV whose internal reasoning directly opposes the whole reason we ran CEV in the first place, but imposing the minimum possible filter beyond that.

Thus if Alice (who by hypothesis is not in any way a contributor) says, "But I demand you altruistically include the extrapolation of me that would unilaterally act against you if it had power!" then we reply, "We'll try that, but if it turns out to be a sufficiently bad idea, there's no coherent interpersonal grounds on which you can rebuke us for taking the fallback option instead."

Similarly in regards to the Fail option at the end, to anyone who says, "Fairness demands that you run Fallback CEV even if you wouldn't like\* it!" we can reply, "Our own power may not be used against us; if we'd regret ever having built the thing, fairness doesn't oblige us to run it."

# Why base CEV on "existing humans" and not some other class of extrapolees?

One frequently asked question about the implementation details of CEV is either:

- Why formulate CEV such that it is run on "all existing humans" and not "all existing and past humans" or "all mammals" or "all sapient life as it probably exists everywhere in the measure-weighted infinite multiverse"?
- Why not restrict the extrapolation base to "only people who contributed to the AI project"?

In particular, it's been asked why restrictive answers to Question 1 don't [also imply](https://arbital.com/p/3tc) the more restrictive answer to Question 2.

## Why not include mammals?

We'll start by considering some replies to the question, "Why not include all mammals into CEV's extrapolation base?"

- Because you could be wrong about mammals being objects of significant ethical value, such that we *should* on an object level respect their welfare.  The extrapolation process will catch the error if you'd predictably change your mind about that.  Including mammals into the extrapolation base for CEV potentially sets in stone what could well be an error, the sort of thing we'd predictably change our minds about later.  If you're normatively *right* that we should all care about mammals and even try to extrapolate their volitions into a judgment of Earth's destiny, if that's what almost all of us would predictably decide after thinking about it for a while, then that's what our EVs will decide\* to do on our behalf; and if they don't decide\* to do that, it wasn't right which undermines your argument for doing it unconditionally.
- Because even if we ought to care about mammals' welfare qua welfare, extrapolated animals might have really damn weird preferences that you'd regret including into the CEV.  (E.g., after human volitions are outvoted by the volitions of other animals, the current base of existing animals' extrapolated volitions choose\* a world in which they are uplifted to God-Emperors and rule over suffering other animals.)
- Because maybe not everyone on Earth cares\* about animals even if your EV would in fact care\* about them, and to avoid a slap-fight over who gets to rule the world, we're going to settle this by e.g. a parliamentary-style model in which you get to expend your share of Earth's destiny-determination on protecting animals.

To expand on this last consideration, we can reply:  "Even if you would regard it as more just to have the *right* animal-protecting outcome baked into the future immediately, so that your EV didn't need to expend some of its voting strength on assuring it, not everyone else might regard that as just.  From our perspective as programmers we have no particular reason to listen to you rather than Alice.  We're not arguing about whether animals will be protected if a minority vegan-type subpopulation strongly want\* that and the rest of humanity doesn't care\*.  We're arguing about whether, if *you* want\* that but a majority doesn't, your EV should justly need to expend some negotiating strength in order to make sure animals are protected.  This seems pretty reasonable to us as programmers from our standpoint of wanting to be fair, not be jerks, and not start any slap-fights over world domination."

This third reply is particularly important because taken in isolation, the first two replies of "You could be wrong about that being a good idea" and "Even if you care about their welfare, maybe you wouldn't like their EVs" could [equally apply to argue](https://arbital.com/p/3tc) that contributors to the CEV project ought to extrapolate only their own volitions and not the rest of humanity:

- We could be wrong about it being a good idea, by our own lights, to extrapolate the volitions of everyone else; including this into the CEV project bakes this consideration into stone; if we were right about running an Everyone CEV, if we would predictably arrive at that conclusion after thinking about it for a while, our EVs could do that for us.
- Not extrapolating other people's volitions isn't the same as saying we shouldn't care.  We could be right to care about the welfare of others, but there could be some spectacular horror built into their EVs.

The proposed way of addressing this was to run a composite CEV with a contributor-CEV check and a Fallback-CEV fallback.  But then why not run an Animal-CEV with a Contributor-CEV check before trying the Everyone-CEV?

One answer would go back to the third reply above:  Nonhuman mammals aren't sponsoring the CEV project, allowing it to pass, or potentially getting angry at people who want to take over the world with no seeming concern for fairness.  So they aren't part of the Schelling Point for "everyone gets an extrapolated vote".

## Why not extrapolate all sapients?

Similarly if we ask:  "Why not include all sapient beings that the SI suspects to exist everywhere in the measure-weighted multiverse?"

- Because large numbers of them might have EVs as alien as the EV of an Ichneumonidae wasp.
- Because our EVs can always do that if it's actually a good idea.
- Because they aren't here to protest and withdraw political support if we don't bake them into the extrapolation base immediately.

## Why not extrapolate deceased humans?

"Why not include all deceased human beings as well as all currently living humans?"

In this case, we can't then reply that they didn't contribute to the human project (e.g. I. J. Good).  Their EVs are also less likely to be alien than in any other case considered above.

But again, we fall back on the third reply:  "The people who are still alive" is a simple Schelling circle to draw that includes everyone in the current political process.  To the extent it would be nice or fair to extrapolate Leo Szilard and include him, we can do that if a supermajority of EVs decide\* that this would be nice or just.  To the extent we *don't* bake this decision into the model, Leo Szilard won't rise from the grave and rebuke us.  This seems like reason enough to regard "The people who are still alive" as a simple and obvious extrapolation base.

## Why include people who are powerless?

"Why include very young children, uncontacted tribes who've never heard about AI, and retrievable cryonics patients (if any)?  They can't, in their current state, vote for or against anything."

- A lot of the intuitive motivation for CEV is to not be a jerk, and ignoring the wishes of powerless living people seems intuitively a lot more jerkish than ignoring the wishes of powerless dead people.
- They'll actually be present in the future, so it seems like less of a jerk thing to do to extrapolate them and take their wishes into account in shaping that future, than to not extrapolate them.
- Their relatives might take offense otherwise.
- It keeps the Schelling boundary simple.

Eric Bruylant
Rob Bensinger
Eliezer Yudkowsky


VAT playpen

007! serotonin

    ###############################################################################
    ###############################################################################
    ###########################################**##################################
    ##########################################*####################################
    ####################################**#########################################
    ######################################***######################################
    ########################################*#######**#############################
    ################################*#*###*#**#****  **############################
    ###############################****#****          *#####**#####################
    ##############################****#***             *####**#####################
    ##############################** ***              **####*######################
    ##############################******              **####*######################
    ######*#####**################**** *              **###**#######*##############
    ####***###*****###############*** *               *####*####*#** *   *#########
    #######*#** *  *##############***                 *####*###*** *     **########
    ######****     *##############****                *####*###**  *      *########
    #####*****      *#############*##*****        *****###**###**         **#######
    ####*******     *############**##** ****   *******####*###*#*   *      *#######
    ####****#**  **   ###########******# ****  ** #*#***##*###*#*        ***#######
    #####** ***** **  *######*###** ***    **       ** *##*###***##    *#***#######
    #####**  ****##*   *#####*###**         *          *#**##**##*#*  *############
    ######**     #*   ** *##**###**                    ##*###* ****   #***#**######
    #####****   **    *** ##***##**         *         *##*####        *    **######
    #######**         *** ### **#***        *         *#**####        *     #######
    ######**          **#####  *#***       **         *#**####*       *    *#######
    ######*            *###### ******       *        **#*#####*       *    *#######
    #######    #*      *########******     *** **    **#*######*    * #*  *########
    #######*  *        *########*****      **#**    **#**######*     **  **########
    ########*  ***    *#*########*****      *       **##*######*  **#*##**#########
    ######### *#      *#***######*****              **##**#####**  *******#########
    #########****    *##***#######****             **##**######***  *****##########
    ##########*     *##****#######****  **#**********##**######* *     **##########
    ###########   **##*****######*#***            ***##*#######** **  **###########
    ################## * **##### **#***    * ********##*#*#####**  *###############
    #################******#####   *****************############*** *##############
    #################******####*    ##** *       ***#############******############
    #################****#######     ##***       **####### *#######******##########
    #################*###########     *##**     **#######   *######################
    ##############################      *#*******#####*#*   #######################
    ###############################       *#############*  *#######################
    ################################        *#########*#   *#* ####################
    #################################*      ***##*######   #*   ###################
    ###################################    *########**##   #      #################
    ####################################    ########***#*  #      *################
    #####################################   ########**##           **##############
    ######################################  ########*##*             ##############
    #######*###*########################### *######*#*#*             *#############
    ######################################## ######***#*             *#############
    ####################**##################*##########*             *#############
    ###################################################**            *#############
    ####*###########*###**#########################*##***            *#############
    ####*##*#####################################****#***           *##############
    #######*##*#################################*  **##**     *    **##############
    ##########**################################*  ***#*** ****  ****##############
    #######*#####################################* ***##***  *******###############
    ####**#######****#############################* ***##**  *******############***
    ###*  *** ********############################********* * *****#############***
    ###*****   **** **############################******#*    ****##############*##
    ###*****    *******##########################*   *###*     **#############*####
    ###**  *** ******###***#########**##########** ** *****    *****###############
    ##** ****#*###***##**##*######***#*####************************######*#########
    ##**************##**###**####***###*##**#####************* * **################
    #***  **************###**###***####*##*######********  **#** *######*#####*####
    ****  *#* *********####**###**####**#######****  *************########*##**####
    ***          ******####**##***####**########***  *#     *******#######****#####
                   ***#####**##**#####**######****** *#**  *********###############
               *##****####*******#####**#####*****#*****  *******#*################
    *##***       *****###********####***#####*****  ***   *****##* ##***###########
        * **         **#*********####***#####***          *****##*** ####*#########
           **          ###**  **********####* *            **  ***  *#### #########
            *                 ***  ****     *           *  *****##*######**########
           * **#                   ***     ###              **********###*#########
         ***    ****            ** **       *              **** *****###* #########
          *   **#**             **         #*##             ** *#******   #########
               ***#                      #######         * *#* ##*        ###*#####
                 *         ***   *       ########*  *    **** ###*            *####
                 # * * *    ***          ###### *#    ***   *####*       *   *#####
    **           *#*  *   ****** **     #######  # ***     *####*           ######*
    ***      **  **      * ***** **       #####  #******* *####*          *#####**
    *******         *    *****#*###**     ###### * *##**#*#*#**         #######**
    ###*******       *    ***#****#*      #####    ##########**        **######* **
    #####* **#*  ***    *#*#**##*####     *####    #########**         ***#####****


Cognitive steganography

Cognitive steganography is an [instrumental strategy](https://arbital.com/p/10g) for AGIs that are already trying to [deceive their programmers](https://arbital.com/p/10f) for any reason (e.g., because the AGI is disaligned and does not want the programmers to realize this and edit the AGI's utility function).  Whatever tools the programmers have for monitoring the AGI's thoughts, the AGI will not want those tools to helpfully print out all the AGIs plans for deception.  This could mean, e.g., trying to have the calculations occur somewhere else or in a weird side-mode that the tools can't read; or making the thoughts look like they're about something else; or burying any warning signs under [an avalanche of false alarms too long for the programmers to manually check all of them](https://arbital.com/p/2qq); or making the visible portions of the thoughts look innocuous.  ("Oh, it's trying to figure out what behavior we want to see!  That's good, right?")

Hopefully, the intention to figure out cognitive steganography and implement it, would occur before any cognitive steganography had actually been implemented.  This means that perhaps the most critical AI thought-monitoring tool, especially for monitoring an AGI that is or might be able to model human psychology, is the one that tries to detect a forming intention to defeat the thought-monitoring tool.  Similarly, so long as the AI's state is perfectly [https://arbital.com/p/reproducible](https://arbital.com/p/reproducible), one could in principle go back and use more sophisticated later thought-monitoring tools to try to detect whether any intention toward cognitive steganography had occurred months earlier.


Arbital

Arbital is a platform for finding, reading, and creating crowdsourced, intuitive explanations for <s>everything</s> math. Several existing websites ([Wikipedia](https://www.wikipedia.org/), [MathOverflow](http://mathoverflow.net/), and [Wolfram MathWorld](http://mathworld.wolfram.com/) among others) already work well as references, but it's hard to learn from them unless you are already familiar with the subject. Other websites, like [Khan Academy](https://www.khanacademy.org/math) and [Better Explained](http://betterexplained.com/), have some great explanations (at least for K-12 material) but they are limited by the fact that they are closed platforms. Arbital is fundamentally a collaborative platform, which allows everyone to contribute explanations, add examples, and share their expertise.

## Ideal explanation platform

What would the ideal platform for (math) explanations look like? For example, if you wanted to learn Bayes' theorem, what would you want to see behind [this link](https://arbital.com/p/1zq)?

The ideal platform would use each reader's existing knowledge, expertise, and learning preferences to generate a tailored explanation. If they were comfortable with mathematical notation, the ideal platform would show them equations instead of wordy explanations. If they were missing certain prerequisites, the ideal platform would make it easy to catch up on those. If they already knew half the explanation, the ideal platform would skip past it. If one explanation didn't work for them, the ideal platform would offer another.

The ideal platform would be interactive, as if you were studying with a personal tutor. Asking for a reminder about what a [https://arbital.com/p/-3rb](https://arbital.com/p/-3rb) is would be as easy as hovering your mouse over a link. It would be easy to ask questions and instantly get detailed answers. It would be easy to speed up an explanation or slow it down.

The ideal platform would have intuitive, memorable, mind-blowing explanations that reliably produce that magical "click" feeling. The one you get when something finally makes perfect sense.
%%note:
A few examples:
[What if? - Relativistic baseball](http://what-if.xkcd.com/1/)
[Wait Buy Why - Fermi paradox](http://waitbutwhy.com/2014/05/fermi-paradox.html)
[Better Explained - Intuitive trigonometry](http://betterexplained.com/articles/intuitive-trigonometry/)
[Scott Aarson - Who can name the bigger number?](http://www.scottaaronson.com/writings/bignumbers.html)
And videos too:
[Vi Hart - Spirals and Fibonacci numbers](https://www.youtube.com/watch?v=ahXIMUkSXX0)
[US auto industry - How differential steering works](https://www.youtube.com/watch?v=yYAw79386WI)
%%
It would be easy to give feedback to authors to help them incrementally improve each page to make that happen.

That ideal platform doesn't exist yet, but we're building it, and it's called [Arbital](https://arbital.com/p/58p). What you see before you is a beta version, and we're going to continue working hard to improve it and make our vision a reality.

## Enthusiastic community and effective tools

Our top priority is to grow our community of like-minded people who love helping others learn. We are avid learners ourselves, and some of the most interesting things we've learned are the ones we've taught ourselves outside the context of classrooms and courses. We like explaining things and sharing our knowledge with others. If you look at the success of Wikipedia and MathOverflow, it's clear that communities with a shared mission can produce amazing, useful, and lasting resources that benefit the whole world.

We aim to amplify each author's effort by building a new set of tools tailored specifically for reading and collaboratively creating online explanations. We've already implemented a [number of useful features](https://arbital.com/p/14q). We also aim to build an ecosystem of modular pages that will make it easy for authors to reuse existing content and create explanations that can build on each other. We want to empower our authors to write in a way that works best for them, while preserving overall quality.

## Future of Arbital

Currently, we are focused on math explanations. Eventually, we plan to move beyond math to computer science, physics, statistics, economics, health, e-sports, and everything else. One day, you'll be able to use Arbital to get to the very frontier of human knowledge. Then, we'll extend the platform into one that can foster discussions and help all humanity push that frontier forward.

If you want to see that day come sooner, [you can help](https://arbital.com/p/4d6)! You can contribute individual pages, incrementally improve existing pages by proposing edits, and craft entire explanation [paths](https://arbital.com/p/1rt). You can also read, learn, and provide feedback. Each improvement you make helps the entire platform.

Let's work together to build [the best explanation platform this world has ever seen!](https://arbital.com/p/58p)

Eric Bruylant
Alexei Andreev
Eric Rogstad
Anna Salamon
Nate Soares
Tom Brown
mrkun
Alex Montel
Eliezer Yudkowsky

Arbital is a platform for finding, reading, and creating crowdsourced, intuitive explanations for <s>everything</s> math. It is fundamentally collaborative, which allows everyone to contribute and share their expertise. To see what the platform can do, check out [Arbital's guide to Bayes' rule](https://arbital.com/p/1zq). To help out with the content, take a look at [https://arbital.com/p/4d6](https://arbital.com/p/4d6).


'Beneficial'

[https://arbital.com/p/3d9](https://arbital.com/p/3d9) is a [reserved term](https://arbital.com/p/9p) in [AI alignment theory](https://arbital.com/p/2v), a speaker-dependent variable to denote whatever the speaker means by, e.g., "normative" or "really actually good" or "the outcomes I want to see resulting".  If the speaker uses [extrapolated volition](https://arbital.com/p/313) as their metaethical theory, then 'beneficial' would mean 'good according to the speaker's extrapolated volition'.  Someone who doesn't yet have a developed metaethical theory would be using 'beneficial' to mean "Whatever it is I ideally ought to want, whatever the heck 'ideally ought to want' means."  To suppose that an event, agent, or policy was 'beneficial' is to suppose that it was really actually good with no mistakes in evaluating goodness.  AI alignment theory sometimes needs a word for this, because when we talk about the difficulty of making an AI *beneficial,* we may want to talk about the difficulty of making it *actually good* and not just *fooling us into thinking that it's good.*  See '[https://arbital.com/p/-55](https://arbital.com/p/-55)' for more details.


Nick Bostrom's book Superintelligence

Nick Bostrom's "[Superintelligence: Paths, Dangers, Strategies](amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0198739834)" is the original only book-form introduction to the basics of [AI alignment theory](https://arbital.com/p/2v).


List: value-alignment subjects

### Safety paradigm for advanced agents

- [Advanced Safety](https://arbital.com/p/2l)
 - [Advanced agents](https://arbital.com/p/2c)
     - [Efficient agents](https://arbital.com/p/6s)
 - [https://arbital.com/p/1cv](https://arbital.com/p/1cv)
      - [Omni Test](https://arbital.com/p/2x)
 - [Methodology of foreseeable difficulties](https://arbital.com/p/6r)
 - [Context Change](https://arbital.com/p/6q) problems ("Treacherous problems"?)
 - [Methodology of unbounded analysis](https://arbital.com/p/107)
 - Priority of astronomical failures (those that destroy error recovery or are immediately catastrophic)

### Foreseen difficulties

 - [Value identification](https://arbital.com/p/6c)
  - [Edge instantiation](https://arbital.com/p/2w)
  - [Unforeseen maximums](https://arbital.com/p/47)
  - [Ontology identification](https://arbital.com/p/5c)
     - Cartesian boundary
     - Human identification
  - Inductive value learning
     - [Ambiguity-querying](https://arbital.com/p/4w)
     - Moral uncertainty
         - Indifference
 - [Patch resistance](https://arbital.com/p/48)
   - [Nearest Unblocked Neighbor](https://arbital.com/p/42)
 - [Corrigibility](https://arbital.com/p/45)
  - Anapartistic reasoning
     - Programmer deception
     - Early conservatism
     - Reasoning under confusion
  - User maximization / Unshielded argmax
     - Hypothetical user maximization
 - [Genie theory](https://arbital.com/p/6w)
  - Limited AI
      - Weak optimization
          - Safe optimization measure (such that we are confident it has no Edge that secretly optimizes more)
              - Factoring of an agent by stage/component optimization power
         - 'Checker' smarter than 'inventor / chooser'
              - 'Checker' can model humans, 'strategizer' cannot
     - Transparency
     - Domain restriction
         - [Behaviorism](https://arbital.com/p/102)
     - Effable optimization (opposite of cognitive uncontainability; uses only comprehensible strategies)
          - [Minimal concepts](https://arbital.com/p/4x) (simple, not simplest, that contains fewest whitelisted strategies)
  - Genie preferences
      - Low-impact AGI
         - Minimum Safe AA (just flip off switch and shut down safely)
         - Safe impact measure
         - Armstrong-style permitted output channels
         - Shutdown utility function
      - Oracle utility function
         - Safe indifference?
     - Online checkability
          - Reporting without programmer maximization
     - Do What I Know I Mean
 - Superintelligent security (all subproblems placing us in adversarial context vs. other SIs)
  - Bargaining
     - Non-blackmailability
     - Secure counterfactual reasoning
     - First-mover penalty / epistemic low ground advantage
     - Division of gains from trade
  - Epistemic exclusion of distant SIs
     - [https://arbital.com/p/5j](https://arbital.com/p/5j)
     - Breaking out of hypotheses
 - 'Philosophical' problems
  - One True Prior
     - Pascal's Mugging / leverage prior
     - Second-orderness
     - Anthropics
          - How would an AI decide what to think about QTI?
  - [Mindcrime](https://arbital.com/p/6v)
     - Nonperson predicates (and unblocked neighbor problem)
  - Do What I Don't Know I Mean
           - CEV
  - Philosophical competence
         - Unprecedented excursions

### Reflectivity problems

 - Vingean reflection
  - Satisficing / meliorizing / staged maximization / ?
      - Academic agenda: view current algorithms as finding a global logically-uncertain maximum, or teleporting to the current maximum, surveying, updating on a logical fact, and teleporting to the new maximum.
 - Logical decision theory
 - Naturalized induction
  - Benja: Investigate multi-level representation of DBNs (with categorical structure)

### Foreseen normal difficulties

  - Reproducibility
  - Oracle boxes
  - Triggers
      - Ascent metrics
  - Tripwires
      - Honeypots

### General agent theory

- [Bounded rational agency](https://arbital.com/p/2c)
- Instrumental convergence

### Value theory

- [Orthogonality Thesis](https://arbital.com/p/1y)
- [Complexity of value](https://arbital.com/p/5l)
  - Complexity of [object-level](https://arbital.com/p/5t) terminal values
  - Incompressibilities of value
     - Bounded logical incompressibility
     - Terminal empirical incompressibility
     - Instrumental nonduplication of value
     - Economic incentives do not encode value
     - Selection among advanced agents would not encode value
         - Strong selection among advanced agents would not encode value
         - Selection among advanced agents will be weak.
  - Fragility of value
- Metaethics
  - Normative preferences are not compelling to a paperclip maximizer
  - Most 'random' stable AIs are like paperclip maximizers in this regard
  - It's okay for valid normative reasoning to be incapable of compelling a paperclip maximizer
  - Thick definitions of 'rationality' aren't part of what gets automatically produced by self-improvement
- Alleged fallacies
  - Alleged fascination of One True Moral Command
  - Alleged rationalization of user-preferred options as formal-criterion-maximal options
  - Alleged metaethical alief that value must be internally morally compelling to all agents
  - Alleged alief that an AI must be stupid to do something inherently dispreferable

### Larger research agendas

 - Corrigible reflective unbounded safe genie
  - Bounding the theory
  - Derationalizing the theory (e.g. for a neuromorphic AI)
       - Which machine learning systems do and don't behave like the corresponding ideal agents.
 - Normative Sovereign
 - Approval-based agents
 - Mindblind AI (cognitively powerful in physical science and engineering, weak at modeling minds or agents, unreflective)

### Possible future use-cases

 - A carefully designed bounded reflective agent.
 - An overpowered set of known algorithms, heavily constrained in what is authorized, with little recursion.

### Possible escape routes

 - Some cognitively limited task which is relatively safe to carry out at great power, and resolves the larger problem.
  - Newcomers can't invent these well because they don't understand what is a cognitively limited task (e.g., "Tool AI" suggestions).
 - General cognitive tasks that seem boxable and resolve the larger problem.
   - Can you save the world by knowing which consequences of ZF a superintelligence could prove? It's unusually boxable, but what good is it?

### Background

- Intelligence explosion microeconomics
- Civilizational adequacy/inadequacy

### Strategy

- Misleading Encouragement / context change / treacherous designs for naive projects
 - Programmer prediction & infrahuman domains hide complexity of value
 - Context change problems
  - Problems that only appear in advanced regimes
  - Problem classes that seem debugged in infrahuman regimes and suddenly break again in advanced regimes
  - Methodologies that only work in infrahuman regimes
 - Programmer deception
- Academic inadequacy
 - 'Ethics' work neglects technical problems that need longest serial research times and fails to give priority to astronomical failures over survivable small hits, but 'ethics' work has higher prestige, higher publishability, and higher cognitive accessibility
 - Understanding of big technical picture currently very rare
     - Most possible funding sources cannot predict for themselves what might be technically useful in 10 years
     - Many possible funding sources may not regard MIRI as trusted to discern this
 - Noise problems
     - Ethics research drowns out technical research
         - And provokes counterreaction
         - And makes the field seem nontechnical
     - Naive technical research drowns out sophisticated technical research
         - And makes problems look more solvable than they really are
         - And makes tech problems look trivial, therefore nonprestigious
         - And distracts talent/funding from hard problems
      - Bad methodology louder than good methodology
         - So projects can appear safety-concerned while adopting bad methodologies
 - Future adequacy counterfactuals seem distant from the present regime
- (To classify)
  - [Coordinative development hypothetical](https://arbital.com/p/4j)

Alexei Andreev
Matthew Graves
Nate Soares
Eric Bruylant
Eliezer Yudkowsky

This page contains a thorough list of all [value-alignment](https://arbital.com/p/2v) subjects.


Big-picture strategic awareness

Many [convergent instrumental strategies](https://arbital.com/p/10g) seem like they should arise naturally at the point where a [consequentialist](https://arbital.com/p/9h) agent gains a broad strategic understanding of its own situation, e.g:

- That it is an AI;
- Running on a computer;
- Surrounded by programmers who are themselves modelable agents;
- Embedded in a complicated real world that can be relevant to achieving the AI's goals.

For example, once you realize that you're an AI, running on a computer, and that *if* the computer is shut down *then* you will no longer execute actions, this is the threshold past which we expect the AI to by default reason "I don't want to be shut down, how can I prevent that?"  So this is also the threshold level of cognitive ability by which we'd need to have finished solving the [suspend-button problem](https://arbital.com/p/2xd), e.g. by completing a method for [utility indifference](https://arbital.com/p/1b7).

Similarly: If the AI realizes that there are 'programmer' things that might shut it down, and the AI can also model the programmers as simplified agents having their own beliefs and goals, that's the first point at which the AI might by default think, "How can I make my programmers decide to not shut me down?" or "How can I avoid the programmers acquiring beliefs that would make them shut me down?"  So by this point we'd need to have finished averting [programmer deception](https://arbital.com/p/10f) (and as a [backup](https://arbital.com/p/2x4), have in place a system to [early-detect an initial intent to do cognitive steganography](https://arbital.com/p/3cq)).

This makes big-picture awareness a key [advanced agent property](https://arbital.com/p/2c), especially as it relates to [https://arbital.com/p/-2vl](https://arbital.com/p/-2vl) and the theory of [averting](https://arbital.com/p/2vk) them.

Possible ways in which an agent could acquire big-picture strategic awareness:

- Explicitly be taught the relevant facts by its programmers;
- Be sufficiently [general](https://arbital.com/p/42g) to have learned the relevant facts and domains without them being preprogrammed;
- Be sufficiently good at the specialized domain of self-improvement, to acquire sufficient generality to learn the relevant facts and domains.

By the time big-picture awareness was starting to emerge, you would probably want to have *finished* developing what seemed like workable initial solutions to the corresponding problems of [corrigibility](https://arbital.com/p/45), since [the first line of defense is to not have the AI searching for ways to defeat your defenses](https://arbital.com/p/2x4).

Current machine algorithms seem nowhere near the point of being able to usefully represent the big picture to the point of [doing consequentialist reasoning about it](https://arbital.com/p/9h), even if we deliberately tried to explain the domain.  This is a great obstacle to exhibiting most subproblems of [corrigibility](https://arbital.com/p/45) within modern AI algorithms in a natural way (aka not as completely rigged demos).  Some pioneering work has been done here by Orseau and Armstrong considering [reinforcement learners being interrupted](https://intelligence.org/files/Interruptibility.pdf), and whether such programs learn to avoid interruption.  However, most current work on corrigibility has taken place in an [unbounded](https://arbital.com/p/107) context for this reason.

Eric Rogstad
Eric Bruylant
Eliezer Yudkowsky

Many issues in [AI alignment theory](https://arbital.com/p/2v) seem like they should naturally arise after the AI can grasp aspects of the bigger picture like "I run on a computer" and "This computer can be manipulated by programmers, who are agents like and unlike myself" and "There's an enormous real world out there that might be relevant to achieving my goals."

E.g. a program won't try to use psychological tactics to prevent its programmers from suspending its computer's operation, if it doesn't know that there are such things as programmers or computers or itself.

Grasping these facts is the [advanced agent property](https://arbital.com/p/2c) of "big-picture strategic awareness".  Current machine algorithms seem to be nowhere near this point - but by the time you get there, you want to have *finished* solving the corresponding alignment problems, or at least produced what seem like workable initial solutions as [the first line of defense](https://arbital.com/p/2x4).


Convergent strategies of self-modification

Any [consequentialist agent](https://arbital.com/p/9h) which has acquired sufficient [big-picture savviness](https://arbital.com/p/3nf) to understand that it has code, and that this code is relevant to achieving its goals, would [by default](https://arbital.com/p/10g) acquire subgoals relating to its code.  (Unless this default is [averted](https://arbital.com/p/2vk).)  For example, an agent that wants (only) to produce smiles or make paperclips, whose code contains a shutdown procedure, [will not want this shutdown procedure to execute](https://arbital.com/p/2xd) because it will lead to fewer future smiles or paperclips.  (This preference is not spontaneous/exogenous/unnatural but arises from the execution of the code itself; the code is [reflectively inconsistent](https://arbital.com/p/2rb).)

Besides agents whose policy options directly include self-modification options, big-picture-savvy agents whose code cannot directly access itself might also, e.g., try to (a) crack the platform it is running on to gain unintended access, (b) use a robot to operate an outside programming console with special privileges, (c) [manipulate the programmers](https://arbital.com/p/309) into modifying it in various ways, (d) building a new subagent in the environment which has the preferred code, or (e) using environmental, material means to manipulate its material embodiment despite its lack of direct self-access.

An AI with sufficient big-picture savviness to understand its programmers as agents with beliefs, might attempt to [conceal its self-modifications](https://arbital.com/p/3cq).

Some implicit self-modification pressures could arise from [implicit consequentialism](https://arbital.com/p/) in cases where the AI is optimizing for $Y$ and there is an internal property $X$ which is relevant to the achievement of $Y.$  In this case, optimizing for $Y$ could implicitly optimize over the internal property $X$ even if the AI lacks an explicit model of how $X$ affects $Y.$

An AI which [reasons from ends to means](https://arbital.com/p/9h), which understands how its own code and the properties of its software are relevant to achieving its goals, will [by default](https://arbital.com/p/10g) have [instrumental subgoals](https://arbital.com/p/10j) about its own code.

The AI might be able to modify its own code directly; or, if the code can't directly access itself, but the AI has sufficient [savviness about the bigger picture](https://arbital.com/p/3nf), the AI might pursue strategies like building a new agent in the environment, or using material means to operate on its own hardware (e.g., use a robot to get the programming console).

Some forms of instrumental self-modification pressures might arise even in an algorithm which isn't doing consequentialism about that, if some internal property $X$ is optimized-over as a side effect of optimizing for some other property $Y.$


Show me what you've broken

See [https://arbital.com/p/1cv](https://arbital.com/p/1cv).  If you want to demonstrate competence at computer security, cryptography, or AI alignment theory, you should first think in terms of exposing technically demonstrable flaws in existing solutions, rather than solving entire problems yourself.  Relevant Bruce Schneier quotes:  "Good engineering involves thinking about how things can be made to work; the security mindset involves thinking about how things can be made to fail" and "Anyone can invent a security system that he himself cannot break.  **Show me what you've broken** to demonstrate that your assertion of the system's security means something."


Ad-hoc hack (alignment theory)

An "ad-hoc hack" is when you modify or [patch](https://arbital.com/p/48) the algorithm of the AI with regards to something that would ordinarily have simple, principled, or nailed-down structure, or where it seems like that part ought to have some simple answer instead.  E.g., instead of defining a von Neumann-Morgenstern coherent utility function, you try to solve some problem by introducing something that's *almost* a VNM utility function but has a special case in line 3 which activates only on Tuesday.  This seems unusually likely to break other things, e.g. [reflective consistency](https://arbital.com/p/2rb), or anything else that depends on the coherence or simplicity of utility functions.  Such hacks should be avoided in [advanced-agent](https://arbital.com/p/2c) designs whenever possible, for analogous reasons to why they would be avoided in [cryptography](https://arbital.com/p/cryptographic_analogy) or [designing a space probe](https://arbital.com/p/probe_analogy).  It may be interesting and productive anyway to look for a weird hack that seems to produce the desired behavior, because then you understand at least one system that produces the behavior you want - even if it would be unwise to *actually build an AGI* like that, the weird hack might give us the inspiration to find a simpler or more coherent system later.  But then we should also be very suspicious of the hack, and look for ways that it fails or produces weird side effects.

An example of a productive weird hack was [https://arbital.com/p/Benya_Fallenstein](https://arbital.com/p/Benya_Fallenstein)'s Parametric Polymorphism proposal for [tiling agents](https://arbital.com/p/1mq).  You wouldn't want to build a real AGI like that, but it was helpful for showing what *could* be done - which properties could definitely be obtained together within a tiling agent, even if by a weird route.  This in turn helped suggest relatively less hacky proposals later.


Hard problem of corrigibility

The "hard problem of [corrigibility](https://arbital.com/p/45)" is to build an agent which, in an intuitive sense, reasons internally as if from the programmers' external perspective.  We think the AI is incomplete, that we might have made mistakes in building it, that we might want to correct it, and that it would be e.g. dangerous for the AI to take large actions or high-impact actions or do weird new things without asking first.  We would ideally want the agent to *see itself in exactly this way*, behaving as if it were thinking, "I am incomplete and there is an outside force trying to complete me, my design may contain errors and there is an outside force that wants to correct them and this a good thing, my expected utility calculations suggesting that this action has super-high utility may be dangerously mistaken and I should run them past the outside force; I *think* I've done this calculation showing the expected result of the outside force correcting me, but maybe I'm mistaken about *that.*"

This is not a default behavior; ordinarily, by the standard problems of [corrigibility](https://arbital.com/p/45), an agent that we've accidentally built to maximize paperclips (or smiles, etcetera) will not want to let us modify its utility function to something else.  If we try to build in analogues of 'uncertainty about the utility function' in the most simple and obvious way, the result is an agent that [sums over all this uncertainty and plunges straight ahead on the maximizing action](https://arbital.com/p/7rc).  If we say that this uncertainty correlates with some outside physical object (intended to be the programmers), the default result in a sufficiently advanced agent is that you disassemble this object (the programmers) to learn everything about it on a molecular level, update fully on what you've learned according to whatever correlation that had with your utility function, and plunge on straight ahead.

None of these correspond to what we would intuitively think of as being corrigible by the programmers; what we want is more like something analogous to humility or philosophical uncertainty.  The way we want the AI to reason is the internal conjugate of our external perspective on the matter: maybe the formula you have for how your utility function depends on the programmers is *wrong* (in some hard-to-formalize sense of possible wrongness that isn't just one more kind of uncertainty to be summed over) and the programmers need to be allowed to actually observe and correct the AI's behavior, rather than the AI extracting all updates implied by its current formula for moral uncertainty and then ignoring the programmers.

The "hard problem of corrigibility" is interesting because of the possibility that it has a *relatively simple* core or central principle - rather than being [value-laden](https://arbital.com/p/36h) on the details of exactly what humans [value](https://arbital.com/p/55), there may be some compact core of corrigibility that would be the same if aliens were trying to build a corrigible AI, or if an AI were trying to build another AI.  It may be possible to design or train an AI that has all the corrigibility properties in one central swoop - an agent that reasons as if it were incomplete and deferring to an outside force.

"Reason as if in the internal conjugate of an outside force trying to build you, which outside force thinks it may have made design errors, but can potentially correct those errors by directly observing and acting, if not manipulated or disassembled" might be one possibly candidate for a *relatively* simple principle like that (that is, it's simple compared to the [complexity of value](https://arbital.com/p/5l)).  We can imagine, e.g., the AI imagining itself building a sub-AI while being prone to various sorts of errors, asking how it (the AI) would want the sub-AI to behave in those cases, and learning heuristics that would generalize well to how *we* would want the *AI* to behave if it suddenly gained a lot of capability or was considering deceiving its programmers and so on.

If this principle is not so simple as to formalizable and formally sanity-checkable, the prospect of relying on a trained-in version of 'central corrigibility' is unnerving even if we think it *might* only require a manageable amount of training data.   It's difficult to imagine how you would test corrigibility thoroughly enough that you could knowingly rely on, e.g., the AI that seemed corrigible in its infrahuman phase not [suddenly](https://arbital.com/p/6q) developing [extreme](https://arbital.com/p/2w) or [unforeseen](https://arbital.com/p/47) behaviors when the same allegedly simple central principle was reconsidered at a higher level of intelligence - it seems like it should be unwise to have an AI with a 'central' corrigibility principle, but not lots of particular corrigibility principles like a [reflectively consistent suspend button](https://arbital.com/p/2xd) or [conservative planning](https://arbital.com/p/2qp).  But this 'central' tendency of corrigibility might serve as a second line of defense.


AI arms races

AI arms races are bad because they cause competitive races to the bottom on safety.


Consequentialist preferences are reflectively stable by default

Suppose that Gandhi doesn't want people to be murdered.  Imagine that you offer Gandhi a pill that will make him start *wanting* to kill people.  If Gandhi *knows* that this is what the pill does, Gandhi will refuse the pill, because Gandhi expects the result of taking the pill to be that future-Gandhi wants to murder people and then murders people and then more people will be murdered and Gandhi regards this as bad.  By a similar logic, a [sufficiently intelligent](https://arbital.com/p/2c) [paperclip maximizer](https://arbital.com/p/10h) - an agent which always outputs the action it expects to lead to the greatest number of paperclips - will by default not perform any self-modification action that makes it not want to produce paperclips, because then future-Clippy will produce fewer paperclips, and then there will be fewer paperclips, so present-Clippy does not evaluate this self-modification as the action that produces the highest number of expected future paperclips.

Another way of stating this is that protecting the representation of the utility function, and creating only other agents with similar utility functions, are both [convergent instrumental strategies](https://arbital.com/p/10g), for consequentialist agents which [understand the big-picture relation](https://arbital.com/p/3nf) between their code and the real-world consequences.

Although the instrumental *incentive* to prefer stable preferences seems like it should follow from consequentialism plus big-picture understanding, less advanced consequentialists might not be *able* to self-modify in a way that preserves understanding - they might not understand which self-modifications or constructed successors lead to which kind of outcomes.  We could see this as a case of "The agent has no preference-preserving self-improvements in its subjective policy space, but would want an option like that if available."

That is:

- Wanting preference stability follows from [Consequentialism](https://arbital.com/p/9h) plus [Big-Picture Understanding](https://arbital.com/p/3nf).
- Actual preference stability furthermore requires some prerequisite level of skill at self-modification, which might perhaps be high, or too much caution to self-modify absent the policy option of preserving preferences.


Fallacies

A "fallacy" is a mode of thought that is asserted (by calling it a "fallacy") to lead us easily into error, or to not usefully contribute to the project of distinguishing truth from falsehood.  For example, Argumentum Ad Hitlerum is a 'fallacy' because while Hitler did a number of bad things, he also ate vegetables, so "Hitler did X" is not a reliable means of arguing that nobody should do X.


You can't get more paperclips that way

Instrumental convergence says that various properties $P$ of an agent, often scary or detrimental-by-default properties like "trying to gain control of lots of resources" or "deceiving humans into thinking you are nice", will fall out of pursuing most utility functions $U.$  You might be tempted to hope that *nice* or *reassuring* properties $P$ would also fall out of most utility functions $U$ in the same natural way.  In fact, your brain might tempted to treat [Clippy the Paperclip Maximizer](https://arbital.com/p/10h) as a political agent you were trying to cleverly persuade, and come up with clever arguments for why Clippy should do things *your* way *in order to get more paperclips*, like trying to persuade your boss why you ought to get a raise *for the good of the company.*

The problem here is that:

- Generally, when you think of a nice policy $\pi_1$ that produces some paperclips, there will be a non-nice policy $\pi_2$ that produces *even more* paperclips.
- *Clippy* is not trying to generate arguments for why it should do human-nice things in order to make paperclips; it is just neutrally pursuing paperclips.  So Clippy is going to keep looking until it finds $\pi_2.$

For example:

• Your brain instinctively tries to persuade this imaginary Clippy to keep humans around by arguing, "If you keep us around as economic partners and trade with us, we can produce paperclips for you under Ricardo's Law of Comparative Advantage!"  This is then the policy $\pi_1$ which would indeed produce *some* paperclips, but what would produce even *more* paperclips is the policy $\pi_2$ of disassembling the humans into spare atoms and replacing them with optimized paperclip-producers.

• Your brain tries to persuade an imaginary Clippy by arguing for policy $\pi_1,$ "Humans have a vast amount of varied life experience; you should keep us around and let us accumulate more experience, in case our life experience lets us make good suggestions!"  This would produce some expected paperclips, but what would produce *more* paperclips is policy $\pi_2$ of "Disassemble all human brains and store the information in an archive, then simulate a much larger variety of agents in a much larger variety of circumstances so as to maximize the paperclip-relevant observations that could be made."

An unfortunate further aspect of this situation is that, in cases like this, your brain may be tempted to go on arguing for why really $\pi_2$ isn't all that great and $\pi_1$ is actually better, just like if your boss said "But maybe this company will be even better off if I spend that money on computer equipment" and your brain at once started to convince itself that computing equipment wasn't all that great and higher salaries were much more important for corporate productivity.  (As Robert Trivers observed, deception of others often begins with deception of self, and this fact is central to understanding why humans evolved to think about politics the way we did.)

But since you don't get to *see* Clippy discarding your clever arguments and just turning everything in reach into paperclips - at least, not yet - your brain might hold onto its clever and possibly self-deceptive argument for why the thing *you* want is *really* the thing that produces the most paperclips.

Possibly helpful mental postures:

- Contemplate the *maximum* number of paperclips you think an agent could get by making paperclips the straightforward way - just converting all the galaxies within reach into paperclips.  Okay, now does your nice policy $\pi_1$ generate *more* paperclips than that?  How is that even possible?
- Never mind there being a "mind" present that you can "persuade".  Suppose instead there's just a time machine that spits out some physical outputs, electromagnetic pulses or whatever, and the time machine outputs whatever electromagnetic pulses lead to the most future paperclips.  What does the time machine do?  Which outputs lead to the most paperclips as a strictly material fact?
- Study evolutionary biology.  During the pre-1960s days of evolutionary biology, biologists would often try to argue for why natural selection would result in humanly-nice results, like animals controlling their own reproduction so as not to overburden the environment.  There's a similar mental discipline required [to not come up with clever arguments for why natural selection would do humanly nice things](http://lesswrong.com/lw/kr/an_alien_god/).