Pivotal act

The term 'pivotal act' in the context of [AI alignment theory](https://arbital.com/p/2v) is a [guarded term](https://arbital.com/p/10l) to refer to actions that will make a large positive difference a billion years later. Synonyms include 'pivotal achievement' and 'astronomical achievement'.

We can contrast this with *existential catastrophes* (or 'x-catastrophes'), events that will make a large *negative* difference a billion years later. Collectively, this page will refer to pivotal acts and existential catastrophes as *astronomically significant events* (or 'a-events').

'Pivotal *event*' is a deprecated term for referring to astronomically significant events, and 'pivotal catastrophe' is a deprecated term for existential catastrophes. 'Pivotal' was originally used to refer to the superset (a-events), but AI alignment researchers kept running into the problem of lacking a crisp way to talk about 'winning' actions in particular, and their distinctive features.

Usage has therefore shifted such that (as of late 2021) researchers use 'pivotal' and 'pivotal act' to refer to *good* events that upset the current gameboard - events that decisively settle a [win](https://arbital.com/p/55), or drastically increase the probability of a win.

### Reason for guardedness

[Guarded definitions](https://arbital.com/p/10l) are deployed where there is reason to suspect that a concept will otherwise be over-extended.  The case for having a guarded definition of 'pivotal act' (and another for 'existential catastrophe') is that, after it's been shown that event X is maybe not as important as originally thought, one side of that debate may be strongly tempted to go on arguing that, wait, really it could be "relevant" (by some [strained](https://arbital.com/p/10m) line of possibility).

**Example 1**:  In the [Zermelo-Fraenkel provability oracle](https://arbital.com/p/70) dialogue, Alice and Bob consider a series of possible ways that an untrusted [oracle](https://arbital.com/p/6x) could break an attempt to [box](https://arbital.com/p/6z) it. We end with an extremely boxed oracle that can only output machine-checkable proofs of predefined theorems in Zermelo-Fraenkel set theory, with the proofs themselves being thrown away once machine-verified.

The dialogue then observes that there isn't currently any obvious way to save the world by finding out that particular pre-chosen theorems are provable.

It may then be tempting to argue that this device could greatly advance the field of mathematics, and that math is relevant to the AI alignment problem.  However, at least given that particular proposal for using the ZF oracle, the basic rules of the AI-development playing field would remain the same, the AI alignment problem would not be *finished* nor would it have moved on to a new phase, the world would still be in danger (neither safe nor destroyed), etcetera.

(This doesn't rule out that tomorrow some reader will think of some spectacularly clever use for a ZF oracle that *does* upset the chessboard and get us on a direct path to winning where we know what we need to do from there - and in this case MIRI would reclassify the ZF oracle as a high-priority research avenue!)

**Example 2**:  Suppose a funder, worried about the prospect of advanced AIs wiping out humanity, starting offering grants for "AI safety".  It may then be tempting to try to write papers that you know you can finish, like a paper on robotic cars [causing unemployment](https://arbital.com/p/3b) in the trucking industry, or a paper on who holds legal liability when a factory machine crushes a worker. These have the advantage of being much less difficult problems than those involved in making something actually smarter than you be safe.

But while it's true that crushed factory workers and unemployed truckers are both, ceteris paribus, bad, they are not *existential catastrophes that transform all galaxies inside our future light cone into paperclips*, and the latter category seems worth distinguishing.

This definition needs to be guarded because there may then be a temptation for the grantseeker to argue, "Well, if AI causes unemployment, that could slow world economic growth, which will make countries more hostile to each other, which would make it harder to prevent an AI arms race."  The possibility of something ending up having a *non-zero impact* on astronomical stakes is not the same concept as events that have a *game-changing impact* on astronomical stakes.

The question is what are the largest lowest-hanging fruit in astronomical stakes, not whether something can be argued as defensible by pointing to a non-zero astronomical impact.

**Example 3**:  Suppose a [behaviorist genie](https://arbital.com/p/102) is restricted from modeling human minds in any great detail, but is still able to build and deploy molecular nanotechnology.  Moreover, the AI is able to understand the instruction, "Build a device for scanning human brains and running them at high speed with minimum simulation error", and is able to work out a way to do this without simulating whole human brains as test cases.  The genie is then used to upload a set of, say, fifty human researchers, and run them at 10,000-to-1 speeds.

This accomplishment would not of itself save the world or destroy it - the researchers inside the simulation would still need to solve the alignment problem, and might not succeed in doing so.

But it would (positively) *upset the gameboard* and change the major determinants of winning, compared to the default scenario where the fifty researchers are in an equal-speed arms race with the rest of the world, and don't have practically-unlimited time to check their work.  The event where the genie was used to upload the researchers and run them at high speeds would be a critical event, a hinge where the optimum strategy was drastically different before versus after that pivotal act.

**Example 4**:  Suppose a paperclip maximizer is built, self-improves, and converts everything in its future light cone into paperclips.  The fate of the universe is then settled in the negative direction, so building the paperclip maximizer was an existential catastrophe.

**Example 5**:  A mass simultaneous malfunction of robotic cars causes them to deliberately run over pedestrians in many cases.  Humanity buries its dead, picks itself up, and moves on.  This was not an existential catastrophe, even though it may have nonzero influence on future AI development.

**Discussion**: Many [strained arguments](https://arbital.com/p/10m) for X being a pivotal act have a step where X is an input into a large pool of goodness that also has many other inputs.  A ZF provability oracle would advance mathematics, and mathematics can be useful for alignment research, but there's nothing obviously game-changing about a ZF oracle that's specialized for advancing alignment work, and it's unlikely that the effect on win probabilities would be large relative to the many other inputs into total mathematical progress.

Similarly, handling trucker disemployment would only be one factor among many in world economic growth.

By contrast, a genie that uploaded human researchers putatively would *not* be producing merely one upload among many; it would be producing the only uploads where the default was otherwise no uploads.  In turn, these uploads could do decades or centuries of unrushed serial research on the AI alignment problem, where the alternative was rushed research over much shorter timespans; and this can plausibly make the difference by itself between an AI that achieves ~100% of value versus an AI that achieves ~0% of value.  At the end of the extrapolation where we ask what difference everything is supposed to make, we find a series of direct impacts producing events qualitatively different from the default, ending in a huge percentage difference in how much of all possible value gets achieved.

By having narrow and guarded definitions of 'pivotal acts' and 'existential catastrophes', we can avoid bait-and-switch arguments for the importance of research proposals, where the 'bait' is raising the apparent importance of 'AI safety' by discussing things with large direct impacts on astronomical stakes (like a paperclip maximizer or Friendly sovereign) and the 'switch' is to working on problems of dubious astronomical impact that are inputs into large pools with many other inputs.

### 'Dealing a deck of cards' metaphor

There's a line of reasoning that goes, "But most consumers don't want general AIs, they want voice-operated assistants.  So companies will develop voice-operated assistants, not general AIs."  But voice-operated assistants are themselves not astronomically significant events; developing them doesn't prevent general AIs from being developed later.  So even though this non-astronomically-significant event precedes a more significant event, it doesn't mean we should focus on the earlier event instead.

No matter how many non-game-changing 'AIs' are developed, whether playing great chess or operating in the stock market or whatever, the underlying research process will keep churning and keep turning out other and more powerful AIs.

Imagine a deck of cards which has some aces (superintelligences) and many more non-aces.  We keep dealing through the deck until we get a black ace, a red ace, or some other card that *stops the deck from dealing any further*.

A non-ace Joker card that permanently prevents any aces from being drawn would be 'astronomically significant' (not necessarily good, but definitely astronomically significant).

A card that shifts the further distribution of aces in the deck from 10% red to 90% red would be pivotal; we could see this as a metaphor for the hoped-for result of Example 3 (uploading the researchers), even though the game is not then stopped and assigned a score.

A card that causes the deck to be dealt 1% slower or 1% faster, that eliminates a non-ace card, that adds a non-ace card, that changes the proportion of red non-ace cards, etcetera, would not be astronomically significant.  A card that raises the probability of a red ace from 50% to 51% would be highly desirable, but not pivotal - it would not qualitatively change the nature of the game.

Giving examples of non-astronomically-significant events that could precede or be easier to accomplish than astronomically significant ones doesn't change the nature of the game where we keep dealing until we get a black ace or red ace.

### Examples of possible events

Existential catastrophes:

- non-value-aligned AI is built, takes over universe
- a complete and detailed synaptic-vesicle-level scan of a human brain results in cracking the cortical and cerebellar algorithms, which rapidly leads to non-value-aligned neuromorphic AI

Potential pivotal acts:

- human intelligence enhancement powerful enough that the best enhanced humans are qualitatively and significantly smarter than the smartest non-enhanced humans
- a limited [Task AGI](https://arbital.com/p/6w) that can:
 - upload humans and run them at speeds more comparable to those of an AI
 - prevent the origin of all hostile superintelligences (in the nice case, only temporarily and via strategies that cause only acceptable amounts of collateral damage)
 - design or deploy nanotechnology such that there exists a direct route to the operators being able to do one of the other items on this list (human intelligence enhancement, prevent emergence of hostile SIs, etc.)

Non-astronomically-significant events:

- curing cancer (good for you, but it didn't resolve the alignment problem)
- proving the Riemann Hypothesis (ditto)
- an extremely expensive way to augment human intelligence by the equivalent of 5 IQ points that doesn't work reliably on people who are already very smart
- making a billion dollars on the stock market
- robotic cars devalue the human capital of professional drivers, and mismanagement of aggregate demand by central banks plus burdensome labor market regulations is an obstacle to their re-employment

Borderline-astronomically-significant cases:

- unified world government with powerful monitoring regime for 'dangerous' technologies
- widely used gene therapy that brought anyone up to a minimum equivalent IQ of 120

### Centrality to limited AI proposals

We can view the general problem of Limited AI as having the central question: **What is a pivotal act, such that an AI which does that thing and not some other things is therefore a whole lot safer to build?**

This is not a trivial question because it turns out that most interesting things require general cognitive capabilities, and most interesting goals can require arbitrarily complicated value identification problems to pursue safely.

It's trivial to create an "AI" which is absolutely safe and can't be used for any pivotal acts.  E.g. Google Maps, or a rock with "2 + 2 = 4" painted on it.

(For arguments that Google Maps could potentially help researchers drive to work faster or that a rock could potentially be used to bash in the chassis of a hostile superintelligence, see the pages on [guarded definitions](https://arbital.com/p/10l) and [strained arguments](https://arbital.com/p/10m).)

### Centrality to concept of 'advanced agent'

We can view the notion of an advanced agent as "agent with enough cognitive capacity to cause an astronomically significant event, positive or negative"; the [advanced agent properties](https://arbital.com/p/2c) are either those properties that might lead up to participation in an astronomically significant event, or properties that might play a critical role in determining the AI's trajectory and hence how the event turns out.

### Policy of focusing effort on enacting pivotal acts or preventing existential catastrophes

Obvious utilitarian argument: doing something with a big positive impact is better than doing something with a small positive impact.

In the larger context of [effective altruism](https://arbital.com/p/) and [adequacy theory](https://arbital.com/p/), the issue is a bit more complicated.  Reasoning from [adequacy theory](https://arbital.com/p/) says that there will often be barriers (conceptual or otherwise) to the highest-return investments.  When we find that hugely important things seem *relatively neglected* and hence promising of high marginal returns if solved, this is often because there's some conceptual barrier to running ahead and doing them.

For example: tackling the hardest problems is often much scarier (you're not sure if you can make any progress on describing a self-modifying agent that provably has a stable goal system) than 'bouncing off' to some easier, more comprehensible problem (like writing a paper about the impact of robotic cars on unemployment, where you're very sure you can in fact write a paper like that at the time you write the grant proposal).

The obvious counterargument is that perhaps you can't make progress on your problem of self-modifying agents, perhaps it's too hard.  But from this it doesn't follow that the robotic-cars paper is what we should be doing instead - the robotic cars paper only makes sense if there are *no* neglected tractable investments that have bigger relative marginal inputs into astronomically significant events.

If there are in fact some neglected tractable investments in gameboard-flipping acts, then we can expect a search for gameboard-flipping acts to turn up superior places to invest effort.  But a failure mode of this search is if we fail to cognitively guard the concept of 'pivotal act'.

In particular, if we're allowed to have indirect arguments for 'relevance' that go through big common pools of goodness like 'friendliness of nations toward each other', then the pool of interventions inside that concept is so large that it will start to include things that are optimized to be appealing under more usual metrics, such as papers that don't seem unnerving and that somebody knows they can write.

So if there's no guarded concept of research on 'pivotal' things, we will end up with very standard research being done, the sort that would otherwise be done by academia anyway, and our investment will end up having a low expected marginal impact on the final outcome.

This sort of qualitative reasoning about what is or isn't 'pivotal' wouldn't be necessary if we could put solid numbers on the impact of each intervention on the probable achievement of astronomical goods.  But that is an unlikely 'if'.  Thus, there's some cause to reason qualitatively about what is or isn't 'pivotal', as opposed to just calculating out the numbers, when we're trying to pursue [astronomical altruism](https://arbital.com/p/).

Alexei Andreev
Rob Bensinger
Nate Soares
Eric Bruylant
Eliezer Yudkowsky


Boxed AI

AI-boxing is the theory that deals in machine intelligences that are allegedly safer due to allegedly having extremely restricted manipulable channels of causal interaction with the outside universe.

AI-boxing theory includes:

- The straightforward problem of building elaborate sandboxes (computers and simulation environments designed not to have any manipulable channels of causal interaction with the outside universe).
- [Foreseeable difficulties](https://arbital.com/p/6r) whereby the remaining, limited channels of interaction may be exploited to manipulate the outside universe, especially the human operators.
- The attempt to design [preference frameworks](https://arbital.com/p/5f) that are not incentivized to go outside the Box, not incentivized to manipulate the outside universe or human operators, and incentivized to answer questions accurately or perform whatever other activity is allegedly to be performed inside the box.

The central difficulty of AI boxing is to describe a channel which cannot be used to manipulate the human operators, but which provides information relevant enough to be [pivotal or game-changing](https://arbital.com/p/6y) relative to larger events.  For example, it seems not unthinkable that [we could safely extract, from a boxed AI setup, reliable information that prespecified theorems had been proved within Zermelo-Fraenkel set theory](https://arbital.com/p/70), but there is no known way to save the world if only we could sometimes know that prespecified theorems had been reliably proven in Zermelo-Fraenkel set theory.

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev


Zermelo-Fraenkel provability oracle

The Zermelo-Fraenkel provability [oracle](https://arbital.com/p/6x) is the endpoint of the following conversation:

Alice:  Let's build [an AI that only answers questions](https://arbital.com/p/6x).  That sounds pretty safe to me. In fact, I don't understand why all you lot keep talking about building AIs that *don't* just answer questions, when the question-answering systems sound so much safer to me.

Bob:  Suppose you screw up the goal system and the AI starts wanting to do something besides answer questions?  This system is only as safe as your ability to make an AI with a [stable goal system](https://arbital.com/p/71) that actually implements your [intended goal](https://arbital.com/p/6h) of "not doing anything except answering questions", which is [difficult to formalize](https://arbital.com/p/oracle_utility_function).

Alice:  But surely it's *safer*? Surely it's *easier* to develop a goal system that just safely answers questions?

Bob:  How much safer?  How much easier?  If we hypothesize restricting the abilities of the AI in this way, we're reducing the potential payoff if we actually succeed.  An AI that can act in the world and build its own nanotechnology can potentially protect us from other AIs becoming superintelligent, at least as a temporary fix to the immediate dilemma and the immediate danger of death, if not as an eternal solution.  An AI that can act in the real world using nanotechnology would be, in our parlance, a [pivotal achievement](https://arbital.com/p/6y).  Getting from a question-answering AI to a safer world seems a lot harder, unless the question you ask the AI is "How to do I build an intelligent, directed nanotechnological system?" in which case you might as not well bother pretending it's an Oracle.  So we're losing something by supposing that the system only answers questions - we're losing some of the pivotal-ness of the hypothetical success.  If you'll pardon the extreme example, if the Oracle system is only 0.01% safer or 99.9% as difficult to build as a Genie, then it definitely wouldn't be worth the decreased probability of being able to win the larger dilemma using a successful Oracle instead of a successful Genie.  So the question is, *how* useful is the Oracle compared to the Genie, and given that you're going to use it that way, *how* much safer is it, or easier to build, compared to the Genie?  An "Oracle" that immediately writes the code for a Genie is equally useful, but it's definitely not any easier to build safely.

Alice:  But there *are* substantial gains in safety, or in ease of building the system to adequate safety, from adopting a question-answering design.  Like, imagine that the computer is in a sealed chamber on the Moon, without any attached robots or robot arms, and just a single radio downlink to send the answers to the questions we ask.  Then we'll be safe even if the goal design fails and it turns against us, because, you know, it'll be stuck on the Moon without any hands.  To me this obviously seems a *lot* safer than, say, a Genie running around in the world with its own nanotechnology.  Less useful assuming it works, yes, but also a lot less dangerous assuming it fails.

Bob:  Are you sure you're properly [respecting the intelligence of something vastly smarter than you](respect_intelligence-1)?  We're not smart enough to [consider all the strategies it might imagine](cognitive_uncontainability-1).  What if there's some creative thing you can do even if you're in a sealed chamber on the Moon?  There are radio waves leaving the chamber that depend on the flows of electrons in transistors -

Alice:  Okay, a sealed chamber *and* we're running it in three layers of sandboxed simulation *and* we haven't told the AI about the hardware it's running on *or* about any of the emulation layers.  I think it'd be pretty hard for the AI to manipulate the radio waves in that condition.  It just wouldn't be able to figure out a relation between anything it could control, and the radio waves going out, and even if it could modulate the radio waves some tiny amount, do you really think that could somehow end the world?

**[https://arbital.com/p/4v](https://arbital.com/p/4v)**

bob:  it can manipulate the human

alice:  we won't tell it about humans

bob:  it may be able to deduce an awful lot by looking at its own code.

alice: okay it gives a bare answer, like just a few bits.

bob:  timing problem

alice:  time it, obviously.

bob:  what the heck do you think you can do with just a few bits that's interesting?

alice:  we'll ask it which of two policies is the better policy for getting the world out of the AI hole.

bob:  but how can you trust it?

alice:  it can't coordinate with future AIs, but humans are trustworthy because we're honorable.

bob:  cognitive uncontainability zaps you!  it can totally coordinate.  we have proof thereof.  so any question it answers could just be steering the future toward a hostile AI.  you can't trust it if you screwed up the utility function.

alice: okay fine, it outputs proofs of predetermined theorems in ZF that go into a verifier which outputs 0 or 1 and is then destroyed by thermite.

bob:  I agree that this complete system might actually work.

alice:  yay!

bob:  but unfortunately it's now useless.

alice:  useless?  we could ask it if the Riemann Hypothesis is provable in ZF -

bob:  and how is that going to save the world, exactly?

alice:  it could generally advance math progress.

bob:  and now we're back at the "contributing one more thingy to a big pool" that people do when they're trying to promote their non-pivotal exciting fun thing to pivotalness.

alice:  that's not fair.  it could advance math a lot.  beyond what humans could do for two hundred years.

bob:  but the kind of math we do at MIRI frankly doesn't depend on whether RH is true.  it's very rare that we know exactly what theorem we might want to prove, and we don't really know whether it's a consequence of ZF, and if we do know, we can move on.  that's basically never happened in our whole history, except for maybe 7 days at a time.  if we'd had your magic box the whole time, there are certain fields of math that would be decades ahead, but our field would be maybe a week or a month ahead.  most of our work is in thinking up the theorems, not proving them.

alice:  so what if it could suggest theorems to prove?  like show you the form of a tiling agent?

bob:  if it's showing us the outline of a framework for AI and proving something about it, I'm now very worried that it's going to manage to smuggle in some deadly trick of decision theory that lets it take over somehow.  we've found all sorts of gotchas that could potentially do that, like Pascal's Mugging, [probable environment hacking](https://arbital.com/p/5j), blackmail... by unrestricting it to be able to suggest which theorems it can prove, you've given it a rich output channel and a wide range of options for influencing future AI design.

points:

- supports methodology of foreseeable difficulties and its rule of explicitization and critique: when something looks easy or safe to someone, it often isn't.
- intuitive notions of what sort of restricted or limited AI should be "much easier to build" are often wrong, and this screws up the first intuition of how much safety we're gaining in exchange for putatively relinquishing some capabilities
  - by the time you correct this bias, the amount of capability you actually have to relinquish in or order to get a bunch of safety and development ease, often makes the putative AI not have any obvious use
  - it is wise to be explicit about how you intend to use the restricted AI to save the world, because trying to think of concrete details here will often reveal that anything you think of either (1) isn't actually very safe or (2) isn't actually very useful.

Rob Bensinger
Alexei Andreev
Eric Bruylant
Eliezer Yudkowsky
Gurkenglas

The Zermelo-Fraenkel provability [oracle](https://arbital.com/p/6x) is a [thought experiment](https://arbital.com/p/) illustrating the safety-capability tradeoff: a narrowly [Boxed](https://arbital.com/p/6z) advanced agent, wrapped in multiple layers of software simulation, that can only output purported proofs of prespecified theorems in Zermelo-Fraenkel set theory, which proofs are then routed to a Boxed [https://arbital.com/p/proof-verifier](https://arbital.com/p/proof-verifier) which checks the proof to make sure all the derivations follow syntactically.  Then the verifier outputs a 0 or 1 saying whether the prespecified theorem was proven from the Zermelo-Fraenkel axioms.  While this *might* be restrictive enough to actually be safe, and the result trustworthy, it's not clear what [pivotal achievement](https://arbital.com/p/6y) could be carried out thereby.


Stub

Stub-Class pages are tiny, often a single paragraph, or a couple of sentences.  Stubs may have been created to provide a [https://arbital.com/p/-5xs](https://arbital.com/p/-5xs) for parenting, tagging, or reference purposes in a section under construction.  An editor expanding a Stub page might start over from scratch.

Contrast:

- [https://arbital.com/p/3rk](https://arbital.com/p/3rk), which is a larger page, but still visibly incomplete or disorganized.
- [https://arbital.com/p/4gs](https://arbital.com/p/4gs), for brief pages which only contain a formal definition and don't try to explain the topic.

**Also** use the tag [https://arbital.com/p/4v](https://arbital.com/p/4v) if the page is being actively edited.

**[Quality scale](https://arbital.com/p/4yg)**

* [https://arbital.com/p/4ym](https://arbital.com/p/4ym)
* [https://arbital.com/p/4gs](https://arbital.com/p/4gs)
* [https://arbital.com/p/5xq](https://arbital.com/p/5xq)
* [https://arbital.com/p/72](https://arbital.com/p/72)
* [https://arbital.com/p/3rk](https://arbital.com/p/3rk)
* [https://arbital.com/p/4y7](https://arbital.com/p/4y7)
* [https://arbital.com/p/4yd](https://arbital.com/p/4yd)
* [https://arbital.com/p/4yf](https://arbital.com/p/4yf)
* [https://arbital.com/p/4yl](https://arbital.com/p/4yl)

Kevin Clancy
Joe Zeng
Alexei Andreev
Eric Rogstad
Tsvi BT
Jessica Taylor
Stephanie Zolayvar
Patrick Stevens
Jeremy Perret
Nate Soares
Silas Barta
Eric Bruylant
Jaime Sevilla Molina
Eliezer Yudkowsky
Jack Gallagher


Real-world domain

A 'domain', in Artificial Intelligence or machine learning, is a problem class on which an AI program can be targeted.  Face recognition is a 'domain', as is playing chess, or driving a car.  Each of these domains presents characteristic options and modes of action for accomplishing the task: a Go game can be won by outputting good Go moves, a car can be moved safely to its definition by turning, accelerating, and braking.  The *real-world domain* is the superdomain that contains all of reality.  In reality, you could try to win a chess game by making funny faces to distract the opponent, and a driving problem has solutions like 'work for a salary and use the money to pay for an Uber'.  If you can reason like this then you are acting in the 'real-world domain'.

Being able to act in the real-world domain is distinct from *realistic* domains.  Driving a robotic car is a *realistic* domain: solutions must be computed in real-time, everything is continuous rather than turn-based, a deer could wander across the road at any time, sensors and knowledge are imperfect, and sometimes a tire blows out.  Even so, a standard robotic driver that faces this class of events will act through the medium of steering and acceleration.  If instead the robotic driver switched to running a hedge fund and hired a human to drive the car using the money it earned, it would have taken advantage of such a breadth of options that the only domain wide enough to contain those options is 'the real world', reality itself where anything goes so long as it works.

Acting in the real world is an [advanced agent property](https://arbital.com/p/2c) and would likely follow from the advanced agent property of [Artificial General Intelligence](https://arbital.com/p/42g): the ability to learn new domains would lead to the ability to act through modalities covering many different kinds of real-world options.


Paperclip

A 'paperclip', in the context of [AI alignment](https://arbital.com/p/2v), is any configuration of matter which would seem boring and [valueless](https://arbital.com/p/55) even from a very [cosmopolitan](https://arbital.com/p/7cl) perspective.

If some bizarre physics catastrophe, spreading out at the speed of light, permanently transformed all matter it touched into paperclips, this would be morally equivalent to a physics catastrophe that destroys the reachable universe outright.  There is no deep missing moral insight we could have, no broadening of perspective and understanding, that would make us realize that little bent pieces of metal without any thought or internal experiences are the best possible use of our [cosmic endowment](https://arbital.com/p/7cy).  It's true that we don't know what epiphanies may lie in the future for us, but that *particular* paperclip-epiphany seems *improbable.*  If you are tempted to argue with this statement as it applies to actual non-metaphorical paperclips, you are probably being overly contrary.  This is why we consider actual non-metaphorical paperclips as the case in point of 'paperclips'.

From our perspective, any entity that did in fact go around transforming almost all reachable matter into literal actual nonmetaphorical paperclips, would be doing something incredibly pointless; there would almost certainly be no hidden wisdom in the act that we could perceive on deeper examination or further growth of our own intellectual capacities.  By the definition of the concept, this would be equally true of anything more generally termed a [paperclip maximizer](https://arbital.com/p/10h).  Anything claimed about 'paperclips' or a ['paperclip' maximizer](https://arbital.com/p/10h) (such as the claim that [such an entity can exist without having any special intellectual defects](https://arbital.com/p/1y)) must go through without any change for actual paperclips.  Actual paperclips are meant to be a central example of 'paperclips'.

The only distinction between paperclips and 'paperclips' is that the category 'paperclips' is far *wider* than the category 'actual non-metaphorical paperclips' and includes many more specific configurations of matter.  Pencil erasers, tiny molecular smileyfaces, and [enormous diamond masses](https://arbital.com/p/5g) are all 'paperclips'.  Even under the [Orthogonality Thesis](https://arbital.com/p/1y), an AI maximizing actual non-metaphorical paperclips would be an improbable *actual* outcome of screwing up on [value alignment](https://arbital.com/p/5s); but only because there are so many other possibilities.  A 'red actual-paperclip maximizer' would be even more improbable than an actual-paperclip maximizer to find in real life; but this is not because redness is antithetical to the nature of intelligent goals.  The 'redness' clause is one more added piece of complexity in the specification that drives down the probability of that exact outcome.

The popular press has sometimes distorted the notion of a paperclip maximizer into a story about an AI running a paperclip factory that takes over the universe.  (Needless to say, the kind of AI used in a paperclip-manufacturing facility is unlikely to be a frontier research AI.)  The concept of a 'paperclip' is not that it's an explicit goal somebody foolishly gave an AI, or even a goal comprehensible in human terms at all.  To imagine a central example of a supposed paperclip maximizer, imagine a research-level AI that did not stably preserve what its makers thought was supposed to be its utility function, or an AI with a poorly specified value learning rule, etcetera; such that the configuration of matter that [actually happened to max out the AI's utility function](https://arbital.com/p/47) looks like a [tiny](https://arbital.com/p/2w) string of atoms in the shape of a paperclip.

A 'paperclip', in the context of [AI alignment](https://arbital.com/p/2v), is any configuration of matter which would seem very boring and pointless even from a very [cosmopolitan](https://arbital.com/p/7cl) perspective.  If some bizarre physics catastrophe, spreading out at the speed of light, permanently transformed all matter it touched into paperclips, this would be morally equivalent to a physics catastrophe that destroys the reachable universe outright.  There is no deep missing moral insight we could have, no broadening of perspective and understanding, that would make us realize that little bent pieces of metal are a wonderful use of a cosmic endowment that could otherwise be transformed into intelligent life.  This is why we consider actual non-metaphorical paperclips as the case in point of 'paperclips'.


Cosmopolitan value

'Cosmopolitan', lit. "of the city of the cosmos", intuitively implies a very broad, embracing standpoint that is tolerant of other people (entities) and ways that may at first seem strange to us; trying to step out of our small, parochial, local standpoint and adopt a broader one.

From the perspective of [volitional metaethics](https://arbital.com/p/313), this would [normatively](https://arbital.com/p/3y9) cover a case where what we humans *currently* value doesn't cover as much as what we would predictably come to value\* in the limit of better knowledge, greater comprehension, longer thinking, higher intelligence, or better understanding our own natures and changing ourselves in directions we thought were right.  An alien civilization might at first seem completely bizarre to us, and hence scarce in events that we intuitively understood how to value; but if we really understood what was going on, and tried to take additional steps toward widening our circle of concern, we'd see it was a galaxy no less to be valued than our own.

From outside the perspective of any particular metaethics, the notion of 'cosmopolitan' may be viewed as more like a historical generalization about moral progress: many times in human history, we get a first look at people different from us, find their ways repugnant or just confusing, and then later on we bring these people into our circle of concern and learn that they had their own nice things even if we didn't understand those nice things.  Afterwards, in these cases, we look back and say 'moral progress has occurred'.  Anyone pointing at people and claiming they are *not* to be valued as our fellow sapients, or asserting that their ways are objectively inferior to our own, is refusing to learn this lesson of history and unable to appreciate what we would see if we could really adopt their perspective.  To be 'cosmopolitan' is to learn from this generalization, and accept in advance that other beings may have valuable lives and ways even if we don't find them immediately easy to understand.

People who've adopted this viewpoint often start out with a strong prior that anyone talking about *not* just letting AIs do their own thing, figure out their own path, and create whatever kind of intergalactic civilization they want, must have failed to learn the cosmopolitan lesson.  To which at least some AI alignment theorists reply: "No!  You don't understand!  You're completely failing to pass our [Ideological Turing Test](https://arbital.com/p/7cm)!  We *are* cosmopolitans!  We *also* grew up reading science fiction about aliens that turned out to have their own perspectives, and AIs willing to extend a hand in friendship but being mistreated by carbon chaunivists!  We'd be *fine* with a weird and wonderful intergalactic civilization full of non-organic beings appreciating their own daily life in ways we wouldn't understand.  But [paperclip maximizers](https://arbital.com/p/10h) *don't do that!*  We predict that if you got to see the use a paperclip maximizer would make of the cosmic endowment, if you really understood what was going on inside that universe, you'd be as horrified as we are.  You and I have a difference of empirical predictions about the consequences of running a paperclip maximizer, not a values difference about how far to widen the circle of concern."

"Fragility of Cosmopolitan Value" could denote the form of the [Fragility of Value](https://arbital.com/p/) / [https://arbital.com/p/5l](https://arbital.com/p/5l) thesis that is relevant to intuitive cosmopolitans:  Agents with [random utility functions](https://arbital.com/p/7cp) wouldn't use the cosmic endowment in ways that achieve a tiny fraction of the achievable [value](https://arbital.com/p/55), even in the limit of our understanding exactly what was going on and trying to take a very embracing perspective.

'Cosmopolitan', lit. "of the city of the cosmos", intuitively refers to a broad, widely embracing standpoint that tolerates and appreciates other people (entities) whose ways may at first seem very strange to us; trying to step out of our small, parochial, local instincts.  An alien civilization might at first seem completely bizarre to us, but if we really understood what was going on and tried to open our minds and hearts, we'd see it was a galaxy no less to be valued than our own.

People who feel strongly about this 'citizen of the cosmos' perspective often start out with a strong [prior](https://arbital.com/p/1rm) that anyone talking about *not* just letting AIs do their own thing must be taking a parochial, humans-first, carbon-chauvinist viewpoint.  To which at least some AI alignment theorists reply: "No!  You don't understand!  We *are* cosmopolitans!  We *also* grew up reading science fiction about aliens that turned out to have their own perspectives, and AIs willing to extend a hand in friendship being mistreated by carbon chauvinists!  But [paperclip maximizers](https://arbital.com/p/10h) are really genuinely different from that!  We predict that if you got to see the use a paperclip maximizer would make of the cosmic endowment, you'd be as horrified as we are; we have a difference of empirical predictions about what happens when you run a paperclip maximizer, not a values difference about how far to widen the circle of concern."


Random utility function

A 'random utility function' is a utility function selected according to some simple probability measure over a logical space of formal, compact specifications of utility functions.

For example: suppose utility functions are specified by computer programs (e.g. a program that maps an output description to a rational number).  We then draw a random computer program from the standard [universal](https://arbital.com/p/4mr) [prior](https://arbital.com/p/27p) on computer programs:  $2^{-\operatorname K(U)}$ where $\operatorname K(U)$ is the algorithmic complexity ([Kolmogorov complexity](https://arbital.com/p/5v)) of the utility-specifying program $U.$

This obvious measure could be amended further to e.g. take into account non-halting programs; to not put almost all of the probability mass on extremely simple programs; to put a satisficing criterion on whether it's computationally tractable and physically possible to optimize for $U$ (as assumed in the [Orthogonality Thesis](https://arbital.com/p/1y)); etcetera.

[https://arbital.com/p/5l](https://arbital.com/p/5l) is the thesis that the attainable optimum of a random utility function has near-null [goodness](https://arbital.com/p/55) with very high probability.  That is: the attainable optimum configurations of matter for a random utility function are, with very high probability, the moral equivalent of [paperclips](https://arbital.com/p/7ch).  This in turn implies that a [superintelligence](https://arbital.com/p/41l) with a random utility function is with very high probability the moral equivalent of a [paperclip maximizer](https://arbital.com/p/10h).

A 'random utility function' is *not:*

-  A utility function randomly selected from whatever distribution of utility functions may actually exist among agents within the generalized universe.  That is, a random utility function is not the utility function of a random actually-existing agent.
- A utility function with maxentropy content.  That is, a random utility function is not one that independently assigns a uniform random value between 0 and 1 to every distinguishable outcome.  (This utility function would not be tractable to optimize for--we couldn't optimize it ourselves even if somebody paid us--so it's not covered by e.g. the [Orthogonality Thesis](https://arbital.com/p/1y).)

A 'random utility function' is a utility function that's been drawn according to some simple probability measure over a logical space of formal, compact specifications for utility functions.  For example, we might say that a random utility function is a utility function specified by a random program drawn from the [algorithmic complexity](https://arbital.com/p/5v) [prior](https://arbital.com/p/27p) on programs.


Cosmic endowment

The 'cosmic endowment' comprises the $\approx 4 \times 10^{20}$ estimated stars that could potentially be reached by probes originating from modern Earth, before the expanding universe carries those stars over the [cosmological horizon](https://en.wikipedia.org/wiki/Hubble_volume).  It has been [estimated](https://www.gwern.net/docs/ai/1999-bradbury-matrioshkabrains.pdf) that simply surrounding a star with solar panels would yield on the order of enough energy to perform $\approx 10^{42}$ computer operations per second.  Using 100 billion neurons x 1000 synapses per neuron x 100 operations per synapse per second as a standard handwavy estimate of the computing power required to support a human-equivalent existence--although it is doubtful that humans are using this computational power very efficiently, or that 100 ops/second could faithfully simulate one synapse--this suggests that a star could support, at minimum, $\approx 10^{25}$ human-equivalent sapient lives.  Glossing a star's lifetime as 1 billion years, and assuming the quality of posthuman existence to count for no more than 1 Quality-Adjusted Life Year (QALY), this suggests that the cosmic endowment could be used (extreme handwaving) to realize *at least* $\approx 10^{54}$ QALYs.

Eric Bruylant
Nate Soares
Rob Bensinger
Eliezer Yudkowsky


Directing, vs. limiting, vs. opposing

'Directing' versus 'limiting' versus 'opposing' is a proposed conceptual distinction between 3 ways of getting [good](https://arbital.com/p/3d9) outcomes and avoiding [bad](https://arbital.com/p/450) outcomes, when running a [sufficiently advanced Artificial Intelligence](https://arbital.com/p/7g1):

- **Direction** means the AGI wants to do the right thing in a domain;
- **Limitation** is the AGI not thinking or acting in places where it's not aligned;
- **Opposition**  is when we try to prevent the AGI from successfully doing the wrong thing, *assuming* that it would act wrongly given the power to do so.

For example:

- A successfully **directed** AI, given full Internet access, will do [beneficial](https://arbital.com/p/3d9) things rather than [detrimental](https://arbital.com/p/450) things using Internet access, because it wants to do good and understands sufficiently well which actions have good or bad outcomes;
- A **limited AI**, suddenly given an Internet feed, will not do *anything* with that Internet access, because its programmers haven't [whitelisted](https://arbital.com/p/whitelisting) this new domain for being okay to think about;
- **Opposition** is [airgapping](https://en.wikipedia.org/wiki/Air_gap_) (networking)) the AI from the Internet and then putting the AI's processors inside a [Faraday cage](https://en.wikipedia.org/wiki/Faraday_cage), in the hope that even if the AI *wants* to get to the Internet, the AI won't be able to [produce GSM cellphone signals by modulating its memory accesses](https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-guri-update.pdf).

A fourth category not reducible to the other three might be **stabilizing,** e.g. numerical stability of floating-point algorithms, not having memory leaks in the code, etcetera.  These are issues that a sufficiently advanced AI would fix in itself automatically, but an [insufficiently advanced](https://arbital.com/p/1fy) AI might not, which causes problems either if early errors introduce changes that are reflectively stable later, or if we are intending to run the AI in insufficiently-advanced mode.

# Not running adversarial searches

Under the [https://arbital.com/p/agents_as_searches](https://arbital.com/p/agents_as_searches) view, we can see an agent as embodying computations that search out actions within a domain.  The [non-adversarial principle](https://arbital.com/p/7g0) then says that we should not be running a search which is looking for ways to hurt us.

We can achieve this desideratum in one of two ways:

- The AI is sufficiently nice / [value-aligned](https://arbital.com/p/5s) that it's okay for the AI to think about this domain / run searches here; whatever the AI is searching for, we don't mind if that search succeeds.
- The AI is *not* aligned in this domain and therefore should not be *running searches* here--we didn't finish the alignment here, and therefore, we shouldn't be running this computation.

If both of these measures fail, then as a [fallback second line of defense](https://arbital.com/p/strictly), we could try to make a bad search return a null result (the AI doesn't find any way to do the harmful thing).

For example:  The [non-adversarial principle](https://arbital.com/p/7g0) implies that if you build a [shutdown button](https://arbital.com/p/2xd) for the AI that suspends the AI to disk when pressed, you must *also* ensure:

- That the AI *wants* there to be a shutdown button;
- That the AI *wants* the state of this shutdown switch to reflect the dynamic decisions of the human operators;
- That the AI does not *want* to influence the operators to press the switch or not press it;
- That the AI does *not* want to cause anything besides an orderly suspend-to-disk to happen after this switch is pressed.

*Or:*

- That the AI doesn't *want* to think about the shutdown button at all (e.g. because it's not inside a [whitelisted cognitive domain](https://arbital.com/p/whitelisting)).

The first set of measures would be 'direction'--if those measures work, it's okay for the AI to think here because we think those thoughts are pointed in the right direction.  The second approach would be [limitation](https://arbital.com/p/7fx)--we don't think the AI's computations are aligned inside this domain yet, so until then, we shouldn't be running computations there.

As a [final, strictly fallback](https://arbital.com/p/2x4) line of defense, you could put a big red lever in the computing center that shut off all the electricity even if the AI was failing to suspend to disk.  (But you're not relying on this working; it's just there as a maybe-it-might-work measure that might buy you a little more probable safety even if the primary lines of defense failed.)

# Relation to other non-adversarial ideas

The direction/limitation/opposition distinction can help state other ideas from the [https://arbital.com/p/1cv](https://arbital.com/p/1cv).  For example:

The principle [niceness is the first line of defense](https://arbital.com/p/2x4) can be rephrased as follows:  When designing an AGI, we should imagine that all 'oppositional' measures are absent or failed, and think only about 'direction' and 'limitation'.  Any oppositional measures are then added on top of that, just in case.

Similarly, the [https://arbital.com/p/2x](https://arbital.com/p/2x) says that when thinking through our primary design for alignment, we should think as if the AGI just *will* get Internet access on some random Tuesday.  This says that we should design an AGI that is limited by [not wanting to act in newly opened domains without some programmer action](https://arbital.com/p/whitelisting), rather than relying on the AI to be *unable* to reach the Internet until we've finished aligning it.

Eric Rogstad
Eric Bruylant
Eliezer Yudkowsky

With respect to the [theory](https://arbital.com/p/2v) of constructing [sufficiently advanced AIs](https://arbital.com/p/7g1) in ways that yield [good outcomes](https://arbital.com/p/3d9):

- **Direction** is when it's okay for the AI to compute plans, because the AI will end up choosing rightly;
- **Limitation** is shaping an insufficiently-aligned AI so that it doesn't run a computation, if we expect that computation to produce bad results;
- **Opposition**  is when we try to prevent the AI from successfully doing something we don't like, *assuming* the AI would act wrongly given the power to do so.

For example:

- A successfully **directed** AI, given full Internet access, will do [beneficial](https://arbital.com/p/3d9) things given that Internet access;
- A **limited AI**, suddenly given an Internet feed, will not do *anything* with that Internet access, because its programmers haven't [whitelisted](https://arbital.com/p/whitelisting) this new domain as okay to think about;
- **Opposition** is [airgapping](https://en.wikipedia.org/wiki/Air_gap_) (networking)) the AI from the Internet and then putting the AI's processors inside a [Faraday cage](https://en.wikipedia.org/wiki/Faraday_cage), in the hope that even if the AI *wants* to get to the Internet, the AI won't be able to, say, [produce GSM cellphone signals by modulating its memory accesses](https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-guri-update.pdf).


Non-adversarial principle

The 'Non-Adversarial Principle' is a proposed design rule for [sufficiently advanced Artificial Intelligence](https://arbital.com/p/7g1) stating that:

*By design, the human operators and the AGI should never come into conflict.*

Special cases of this principle include [https://arbital.com/p/2x4](https://arbital.com/p/2x4) and [The AI wants your safety measures](https://arbital.com/p/ai_wants_security).

According to this principle, if the AI has an off-switch, our first thought should not be, "How do we have guards with guns defending this off-switch so the AI can't destroy it?" but "How do we make sure the AI *wants* this off-switch to exist?"

If we think the AI is not ready to act on the Internet, our first thought should not be "How do we [airgap](https://arbital.com/p/airgapping) the AI's computers from the Internet?" but "How do we construct an AI that wouldn't *try* to do anything on the Internet even if it got access?"  Afterwards we may go ahead and still not connect the AI to the Internet, but only as a fallback measure.  Like the containment shell of a nuclear power plant, the *plan* shouldn't call for the fallback measure to ever become necessary.  E.g., nuclear power plants have containment shells in case the core melts down.  But this is not because we're planning to have the core melt down on Tuesday and have that be okay because there's a containment shell.

# Why run code that does the wrong thing?

Ultimately, every event inside an AI--every RAM access and CPU instruction--is an event set in motion by our own design.  Even if the AI is modifying its own code, the modified code is a causal outcome of the original code (or the code that code wrote etcetera).  Everything that happens inside the computer is, in some sense, our fault and our choice.  Given that responsibility, we should not be constructing a computation that is *trying* to hurt us.  At the point that computation is running, we've already done something foolish--willfully shot ourselves in the foot.  Even if the AI doesn't find any way to do the bad thing, we are, at the very least, wasting computing power.

No aspect of the AI's design should ever put us in an adversarial position vis-a-vis the AI, or pit the AI's wits against our wits.  If a computation starts *looking* for a way to outwit us, then the design and methodology has *already* failed.  We just shouldn't be putting an AI in a box and then having the AI search for ways to get out of the box.  If you're building a toaster, you don't build one element that heats the toast and then add a tiny refrigerator that cools down the toast.

# Not running searches for harmful strategies

Under the [https://arbital.com/p/agents_as_searches](https://arbital.com/p/agents_as_searches) viewpoint, we can see an AI as embodying computations that search out strategies within a domain.  E.g., an AI carrying out the "[put one damn strawberry on a damn plate](https://arbital.com/p/one_damn_tomato)" task is searching the space of imaginable strategies for a strategy such that the consequence of executing that strategy is one strawberry being on the plate.

The non-adversarial principle then says that we should at no point be running a computation which is *searching* for ways to hurt us.

Suppose you don't want the AI to go on the Internet.  If your first thought is [airgapping](https://arbital.com/p/airgapping) the AI's hardware from the Internet or otherwise [boxing the AI](https://arbital.com/p/6z), then what you're trying to do is ensure that *a computation searching for strategies to reach the Internet returns no result.*

If the primary plan is "the AI searches, but doesn't find anything", then we need to worry that the AI's search [considers some class of strategies we didn't imagine](https://arbital.com/p/9f).  We are pitting our own wits against a potentially [weird](https://arbital.com/p/9f) or [superhuman](https://arbital.com/p/41l) search.  Even if we did successfully rule out all possibilities for escape, it would be hard for us to ever be confident that we had done so.  It would be very hard to rule out the *conceptual* possibility of some entirely unexpected side-channel attack like the AI [producing GSM cellphone signals by cleverly modulating its memory accesses](https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-guri-update.pdf), or the AI [persuading its human operators to let it out of the box](https://arbital.com/p/aibox_experiment).  (Except that neither of those two particular possibilities would be *unexpected;* any specific strategy we write down will be the kind of possibility that humans consider.)

Even if the AI's search does come up empty, you were tempting fate and wasting computing power by instantiating that search in the first place.

# Oppositional measures are fallbacks

*After* we ensure that the escape-search computation is not running in the first place, we can think about fallback [oppositional](https://arbital.com/p/7fx) lines of defense; e.g. putting the AI's processors inside a [Faraday cage](https://en.wikipedia.org/wiki/Faraday_cage).

But the *plan* is not that the AI tries to escape and we successfully prevent the escape.  If we imagine the AI trying to escape, we ought to [feel very surprised in our imagination](https://arbital.com/p/rule_of_surprise).  If the AI actually does try to escape, we should be crying 'What?' and tracking down which strong-seeming assumption has failed.

No matter what clever design we use or how strong it seems, we'd probably *still* be wise to [airgap](https://arbital.com/p/airgapping) an AI under development, not say "Oh it's supposed to be safe!" and connect it to the Internet and saunter on.  But that doesn't mean we *rely* on the airgap.  It doesn't mean the plan calls for the airgap to be necessary.

Nuclear power plants have concrete containment units in case the core melts down; but the *design* is not that they melt down.  The master plan doesn't say "And then on Tuesday the core melts down, but that's fine because of the containment unit."  By design, that enormous concrete shell isn't supposed to actually ever become necessary.  And then we build it anyway, because the best-laid plans etcetera.

Similarly, when designing an AI, we should pretend that the airgap doesn't exist or that the AI will [suddenly get Internet access anyway](https://arbital.com/p/2x) on Tuesday; our *primary* thought should be to design AI that doesn't need an airgap to be safe.  And *then* we add the airgap, making sure that we're not thinking the equivalent of "Oh, it doesn't *really* matter if the core melts down, because we've got a containment structure there anyway."

# Challenges in implementing non-adversarialism

The main difficulties foreseen so far for implementing the non-adversarial principle, tend to center around [https://arbital.com/p/10g](https://arbital.com/p/10g) plus [https://arbital.com/p/42](https://arbital.com/p/42) behavior.

For example, if you build a [shutdown button](https://arbital.com/p/2xd) for a [Task AGI](https://arbital.com/p/6w) that suspends the AI to disk when pressed, the nonadversarial principle implies you must also ensure:

- That the AI *wants* there to be a shutdown button;
- That the AI *wants* to be suspended to disk after this button is pressed;
- That the AI *wants* the state of this shutdown button to reflect the dynamic decisions of the human operators;
- That the AI does not *want* to influence the operators to decide to not press the switch, or to press it;
- That the AI does *not* want anything *besides* an orderly suspend-to-disk to happen, or not happen, after this button is pressed.

*Or:*

- The AI does not think about or make plans involving the shutdown button, e.g. because that domain was not [whitelisted](https://arbital.com/p/whitelisting) for cognition.
- None of the AI's other models end up reflecting the existence of the shutdown button or none of its other plans end up taking into account that part of the model.

The difficulties here center around ["You can't fetch the coffee if you're dead"](https://arbital.com/p/7g2).  This reasoning is very general, so [even if we try to make it not apply at one point, it tends to pop up somewhere else](https://arbital.com/p/48):

- If you naively try to add in a special-case clause to the utility function for wanting a shutdown button to exist, the AI wants the shutdown button to not be pressed.
- If you successfully add a special case saying that the AI wants the button to be pressed if the humans want that button to be pressed, the AI wants the humans to not want to press the button.
- If you naively try to add in a special clause for the AI wanting to shut down after the button is pressed, the AI wants to create a [subagent](https://arbital.com/p/environmental_subagent) to make sure the coffee gets pressed anyway.
- If you try to make an AI that [doesn't think about the shutdown button](https://arbital.com/p/1g4) or model it at all, this seems potentially difficult because in reality the best hypothesis to explain the world *does* contain a shutdown button.  A general search for good hypotheses may tend to create cognitive tokens that represent the shutdown button, and it's not clear (yet) how this could in general be prevented by trying to divide the world into domains.

More generally: by default a lot of *high-level* searches we do want to run, [have *subsearches* we'd prefer *not* to run](https://arbital.com/p/10g).  If we run an agent that searches *in general* for ways to fetch the coffee, that search would, by default and if smart enough, also search for ways to prevent itself from being shut down.

How exactly to implement the non-adversarial principle is thus a major open problem.  We may need to be more clever about shaping which computations give rise to which other computations than the default "Search for any action in any domain which achieves X."

# See also

- [https://arbital.com/p/2x4](https://arbital.com/p/2x4)
- [The omnipotence/omniscience test](https://arbital.com/p/2x)
- [The AI should not want to defeat your safety measures](https://arbital.com/p/nonadversarial_safety)
- [https://arbital.com/p/7fx](https://arbital.com/p/7fx)

Ananya Aloke
Eric Bruylant
Eric Rogstad
Eliezer Yudkowsky

The 'non-adversarial principle' states:  *By design, the human operators and the AGI should never come into conflict.*

Since every event inside an AI is ultimately the causal result of choices by the human programmers, we should not choose so as to run computations that are searching for a way to hurt us.  At the point the AI is even *trying* to outwit us, we've already screwed up the design; we've made a foolish use of computing power.

E.g., according to this principle, if the AI's server center has [a switch that shuts off the electricity](https://arbital.com/p/2xd), our first thought should not be, "How do we have guards with guns defending this off-switch so the AI can't destroy it?"  Our first thought should be, "How do we make sure the AI *wants* this off-switch to exist?"

Open subproblems in aligning a Task-based AGI


Sufficiently advanced Artificial Intelligence

A 'sufficiently advanced Artificial Intelligence' is a cognitive [agent](https://arbital.com/p/6t) or cognitive algorithm with capabilities great enough that we need to think about it in a qualitatively different way from robotic cars; a machine intelligence smart enough that we need to start doing [AI alignment theory](https://arbital.com/p/2v) to it.

For example, we probably don't need to worry about an AI that tries to prevent us from pressing its off-switch, until the AI knows that (a) it has an off-switch and (b) [pressing the off-switch will prevent the AI from achieving its other goals](https://arbital.com/p/7g2).

In turn, this knowledge is a special case of [https://arbital.com/p/3nf](https://arbital.com/p/3nf), which might follow from learning many facts about many domains via [https://arbital.com/p/42g](https://arbital.com/p/42g).

The page on [advanced agent properties](https://arbital.com/p/2c) starts to list out some of the different ways that an AI could be 'smart enough' in this sense, along with the particular problems that might be encountered with an AI that smart.

A 'sufficiently advanced' Artificial Intelligence is one smart enough that we need to start thinking about some potential difficulty being discussed.

For example:  We probably don't need to worry about an AI that tries to prevent us from pressing its off-switch, until the AI knows that (a) it has an off-switch and (b) [pressing the off-switch will prevent the AI from achieving its other goals](https://arbital.com/p/7g2).

In turn, this knowledge is a special case of [https://arbital.com/p/3nf](https://arbital.com/p/3nf), which might follow from learning many facts about many domains via [https://arbital.com/p/42g](https://arbital.com/p/42g).

See [https://arbital.com/p/2c](https://arbital.com/p/2c) for a list of some different ways a cognitive agent or algorithm could be smart enough that we would need to start doing [AI alignment theory](https://arbital.com/p/2v) to it.


You can't get the coffee if you're dead

"You can't get the coffee if you're dead" is [Stuart Russell's](https://arbital.com/p/18m) capsule description of why almost any agent goal implies an instrumental strategy of surviving / not being shut down--by default and barring other measures to prevent this from happening; and assuming the agent has sufficient [big-picture awareness](https://arbital.com/p/3nf) to understand the relation between the agent's non-operation and the coffee not being brought.  See [https://arbital.com/p/2xd](https://arbital.com/p/2xd) and [https://arbital.com/p/10g](https://arbital.com/p/10g).


Infrahuman, par-human, superhuman, efficient, optimal

Some thresholds in '[sufficiently advanced](https://arbital.com/p/2c)' machine intelligence are not absolute ability levels within a domain, but abilities relative to the human programmers or operators of the AI.  When this is true, it's useful to think about *relative* ability levels within a domain; and one generic set of distinguished thresholds in relative ability is:

- **Strictly infrahuman:**  The AI cannot do anything its human operators / programmers cannot do.  Computer chess in 1966 relative to a human master.
- **Infrahuman:**  The AI is definitely weaker than its operators but can deploy some surprising moves.  Computer chess in 1986 relative to a human master.
- **Par-human** (or more confusingly "**human-level**"):  If competing in that domain, the AI would sometimes win, sometimes lose; it's better than human at some things and worse in others; it just barely wins or loses.  Computer chess in 1991 on a home computer, relative to a strong amateur human player.
- **High-human**:  The AI performs as well as exceptionally competent humans.  Computer chess just before [1996](https://arbital.com/p/1bx).
- **Superhuman:**  The AI always wins.  Computer chess in 2006.
- **[Efficient](https://arbital.com/p/6s):**  Human advice contributes no marginal improvement to the AI's competence.  Computer chess was somewhere around this level in 2016, with "advanced" / "freestyle" / "hybrid" / "centaur" chess starting to lose out against purely machine players. %note: Citation solicited. Googling gives the impression that nothing has been heard from 'advanced chess' in the last few years.%
- **Strongly superhuman:**
  - The ceiling of possible performance in the domain is far above the human level; the AI can perform orders of magnitudes better.  E.g., consider a human and computer competing at *how fast* they can do arithmetic.  In principle the domain is simple, but competing with respect to speed leaves room overhead for the computer to do literally billions of times better.
  - [The domain is rich enough](https://arbital.com/p/9j) that humans don't understand key generalizations, leaving them shocked at *how* the AI wins.  Computer Go relative to human masters in 2017 was just starting to exhibit the first signs of this ("We thought we were one or two stones below God, but after playing AlphaGo, we think it is more like three or four").  Similarly, consider a human grandmaster playing Go against a human novice.
- **Optimal:**  The AI's performance is perfect for the domain; God could do no better.  Computer play in checkers as of 2007.

The *ordering* of these thresholds isn't always as above.  For example, in the extremely simple domain of [logical Tic-Tac-Toe](https://arbital.com/p/9s), humans can play optimally after a small amount of training.  Optimal play in Tic-Tac-Toe is therefore not superhuman.  Similarly, if an AI is playing in a rich domain but still has strange weak spots, the AI might be strongly superhuman (its play is *much* better and shocks human masters) but not [efficient](https://arbital.com/p/6s) (the AI still sometimes plays wrong moves that human masters can see are wrong).

The term "human-equivalent" is deprecated because it confusingly implies a roughly human-style balance of capabilities, e.g., an AI that is roughly as good at conversation as a human and also roughly as good at arithmetic as a human.  This seems pragmatically unlikely.

The [other Wiki](https://en.wikipedia.org/wiki/Progress_in_artificial_intelligence) lists the categories "optimal, super-human, high-human, par-human, sub-human".

# Relevant thresholds for AI alignment problems

Considering these categories as [thresholds of advancement](https://arbital.com/p/2c) relevant to the point at which AI alignment problems first materialize:

- "Strictly infrahuman" means we don't expect to be surprised by any tactic the AI uses to achieve its goals (within a domain).
- "Infrahuman" means we might be surprised by a tactic, but not surprised by overall performance levels.
- "Par-human" means we need to start worrying that humans will lose in any event determined by a competition (although this seems to imply the [non-adversarial principle](https://arbital.com/p/7g0) has already been violated); we can't rely on humans winning some event determined by a contest of relevant ability.  Or this may suppose that the AI gains access to resources or capabilities that we have strong reason to believe are protected by a lock of roughly human ability levels, even if that lock is approached in a different way than usual.
- "High-human" means the AI will *probably* see strategies that a human sees in a domain; it might be possible for an AI of par-human competence to miss them, but this is much less likely for a high-human AI.  It thus behaves like a slightly weaker version of postulating [efficiency](https://arbital.com/p/6s) for purposes of expecting the AI to see some particular strategy or point.
- "Superhuman" implies at least [weak cognitive uncontainability](https://arbital.com/p/9f) by [Vinge's Law](https://arbital.com/p/1bt).  Also, if something is known to be difficult or impossible for humans, but seems possibly doable in principle, we may need to consider it becoming possible given some superhuman capability level.
- "Efficiency" is a fully sufficient condition for the AI seeing any opportunity that a human sees; e.g., it is a fully sufficient condition for many instrumentally convergent strategies.  Similarly, it can be postulated as a fully sufficient condition to refute a claim that an AI will take a path such that some other path would get more of its utility function.
- "Strongly superhuman" means we need to expect that an AI's strategies may deploy faster than human reaction times, or overcome great starting disadvantages.  Even if the AI starts off in a much worse position it may still win.
- "Optimality" doesn't obviously correspond to any particular threshold of results, but is still an important concept in the hierarchy, because only by knowing the absolute limits on optimal performance can we rule out strongly superhuman performance as being possible.  See also the claim [https://arbital.com/p/9t](https://arbital.com/p/9t).

# 'Human-level AI' confused with 'general intelligence'

The term "human-level AI" is sometimes used in the literature to denote [https://arbital.com/p/42g](https://arbital.com/p/42g).  This should probably be avoided, because:

- Narrow AIs have achieved par-human or superhuman ability in many specific domains without [general intelligence](https://arbital.com/p/7vh).
- If we consider [general intelligence](https://arbital.com/p/7vh) as a capability, a kind of superdomain, it seems possible to imagine infrahuman levels of general intelligence (or superhuman levels).  The apparently large jump from humans to chimpanzees mean that we mainly see human levels of general intelligence with no biological organisms exhibiting the same ability at a lower level; but, at least so far as we currently know, AI could possibly take a different developmental path.  So alignment thresholds that could plausibly follow from general intelligence, like [big-picture awareness](https://arbital.com/p/3nf), aren't necessarily locked to par-human performance overall.

Arguably, the term 'human-level' should just be avoided entirely, because it's been pragmatically observed to function as a [gotcha button](https://arbital.com/p/7mz) that derails the conversation some fraction of the time; with the interrupt being "Gotcha!  AIs won't have a humanlike balance of abilities!"

- **Strictly infrahuman**: The AI can't do better than a human in any regard (in that domain).
- **Infrahuman**:  The AI almost always loses to the human (in that domain).
- **Par-human**:  The AI sometimes wins and sometimes loses; it's weaker in some places and stronger in others (in that domain).
- **High-human**:  The AI performs around as well as exceptionally competent humans.
- **Superhuman**:  The AI almost always wins.
- **[Efficient](https://arbital.com/p/6s)**:  Human advice contributes no marginal improvement to the AI's competence.
- **Strongly superhuman**:  The AI is *much* better than human; [the domain is rich enough](https://arbital.com/p/9j) for humans to be [surprised](https://arbital.com/p/9f) at the AI's tactics.
- **Optimal**:  Perfect performance for the domain.

These thresholds aren't always ordered as above.  For example, [logical Tic-Tac-Toe](https://arbital.com/p/9s) is simple enough that humans and AIs can both play optimally; so, in the Tic-Tac-Toe domain, optimal play isn't superhuman.