Problem of fully updated deference

The problem of 'fully updated deference' is an obstacle to using [moral uncertainty](https://arbital.com/p/7s2) to create [corrigibility](https://arbital.com/p/45).

One possible scheme in [AI alignment](https://arbital.com/p/2v) is to give the AI a state of  [https://arbital.com/p/-7s2](https://arbital.com/p/-7s2) implying that we know more than the AI does about its own utility function, as the AI's [meta-utility function](https://arbital.com/p/7t8) defines its [ideal target](https://arbital.com/p/7s6).  Then we could tell the AI, "You should let us [shut you down](https://arbital.com/p/2xd) because we know something about your [ideal target](https://arbital.com/p/7s6) that you don't, and we estimate that we can optimize your ideal target better without you."

The obstacle to this scheme is that belief states of this type also tend to imply that an even better option for the AI would be to learn its ideal target by observing us.  Then, having 'fully updated', the AI would have no further reason to 'defer' to us, and could proceed to directly optimize its ideal target.

Furthermore, if the present AI foresees the possibility of fully updating later, the current AI may evaluate that it is better to avoid being shut down now so that the AI can directly optimize its ideal target later, after updating.  Thus the prospect of future updating is a reason to behave [incorrigibly](https://arbital.com/p/45) in the present.

While moral uncertainty seems to take us *conceptually* closer to deference-based [corrigibility](https://arbital.com/p/45), and there may be research avenues for fixing the issue (see below), the current explicit proposals will (when scaled to sufficiently high intelligence) yield essentially the same form of incorrigibility as an AI given a constant utility function.

# Review: Standard problem of value divergence

In a [value learning](https://arbital.com/p/6c) problem:

- Let $V$ indicate our [true intended](https://arbital.com/p/6h) [value](https://arbital.com/p/55) function. %note: Pretending for the sake of simplification that $V$ has been [idealized](https://arbital.com/p/313) or [rescued](https://arbital.com/p/3y6) into a utility function.%
- Let $U$ indicate the AI's actual utility function (learned under some [preference framework](https://arbital.com/p/5f), presumably when we attempted to teach the AI part of $V$).

If the achievable optimum of $U$ substantially diverges from the achievable optimum of $V,$ i.e., if the AI expects humans to disapprove of $U$'s optimum, then this leads into a class of [adversarial](https://arbital.com/p/7g0) situations we'd like to avoid.

Ceteris paribus, $U$-$V$ divergence creates [instrumental](https://arbital.com/p/10g) [pressure](https://arbital.com/p/10k) for the AI to:

- Prevent humans from modifying the AI's program to optimize a different utility function $U'$ closer to $V.$
- [Fool humans](https://arbital.com/p/10f) into thinking that the AI is optimizing $V,$ or thinking that the optimum of $U$ is higher in $V$ than it actually is.
- Avoid being effectively [shut down](https://arbital.com/p/2xd) (or suspended to disk).

The [shutdown problem](https://arbital.com/p/2xd) seems like a well-simplified special case.  Even if we reduce the scenario to a single shutdown button that always works if successfully pressed, %note: So that for purposes of the simplified scenario, we only need to consider what the AI does about the button, and not whether the AI tries to back itself up to elsewhere on the Internet.  More generally, though, "avoiding effective shutdown" can include strategies like creating a hidden backup while the original hardware is in fact shut down, thus giving the appearance of a successful shutdown and avoiding further shutdown attempts.% most of the same general problems appear.  Ceteris paribus and by default, we should expect an AI to try to prevent the button from being pressed, whenever that AI expects that what humans will do by optimizing $V$ after the AI is shutdown, will not be as high in $U$ as what the AI would do if it avoided shutdown.

%%comment: A 'minor' divergence between $U$ and $V$ may not lead into an adversarial situation, if e.g:

- The achievable optima of $U$ and $V$ coincide even though $U$ and $V$ have disagreements about non-optimum situations.
- The AI's preferences have been successfully rendered [tasklike](https://arbital.com/p/4mn).  Then $V$ having preferences over matters elsewhere in space and time not represented in $U$, and the disposition of resources not required to optimize $U$ as far as possible, would not necessarily lead the AI to expect disagreement over $U$-equivalent partitions of the outcome.
- The AI anticipates that $U$'s optimum is high enough in $V$ to satisfice the humans and not provoke adversarial responses.
- The AI expects the $V$-optimum to be close enough to $U$-optimal that fighting loses more expected $U$-value than the slight divergence.%%

# Moral uncertainty and its relation to corrigibility

The [generalized notion of cognitive alignment](https://arbital.com/p/7td) suggests that, if we want an AI to not resist being paused to disk, we should ask whether we can have the AI think conjugate thoughts to the same reasoning we're using to decide to shut it down:

"Can we make the AI *want* to be shut down for the *same reasons* we want to shut it down?  Maybe because the AI [knows that it's incomplete](https://arbital.com/p/3ps), and is able to conceive of its programmers having gotten its utility function 'wrong' in a way that the programmers but not the AI know how to correct?"

In particular, we can ask whether [https://arbital.com/p/-7s2](https://arbital.com/p/-7s2)--a [meta-utility function](https://arbital.com/p/7t8) with a probability distribution over object-level utility functions--would *automatically* solve this problem.

For purposes of simplification, suppose the AI thinks there are exactly three possible 'true' utility functions $U_1, U_2, U_3.$  We similarly say that the AI's meta-utility function $\Delta U$ starts out by assigning $\frac{1}{3}$ probability to each $U_i.$

Let's also assume the $U_i$ are importantly different in the sense that there's no single outcome that's nearly optimal for all of them.  Letting $\mathbb O$ be the outcome space, this says:

$$\forall o_j \in \mathbb O \colon \exists i \colon \ U_i(o_j) \ll \max_{o \in \mathbb O} U_i(o) $$

Otherwise, the AI can just pick a single outcome that well-optimizes all the $U_i$ simultaneously.  (We should also interpret $\max$ as indicating the [https://arbital.com/p/-7t9](https://arbital.com/p/-7t9) rather than global maximum.)

Now (arguendo) suppose the AI believes that the humans know which $U_i$ is the correct one.  More precisely, suppose the AI's meta-utility function asserts a relation between the $U_i$ in different possible worlds, and the empirical behavior of humans in different possible worlds; such that in the set of possible worlds where $U_i$ is '[correct](https://arbital.com/p/7s6)', humans within that set of possible worlds empirically tend to optimize a corresponding $V_i$ whose humanly-attainable optimum is pretty good under $U_i.$

Then (it is further argued) the AI ought to enthusiastically cooperate in being shut down.

In particular (still assuming the above), suppose the AI was starting to believe that $U_1$ is the [correct utility function](https://arbital.com/p/7s6).  But after the AI starts to optimize for mostly $U_1,$ the AI sees its programmers starting to approach the shutdown button.  Then (under the above assumption) the AI concludes that $U_1$ is the wrong utility function and that one of $U_2$ or $U_3$ must be the correct utility function; and either of these will be optimized well by humans pursuing the corresponding $V_i$ after the AI is shut down.

Suppose the AI has only the following 5 policy options available:

- $\pi_1$: Continue optimizing $U_1.$
- $\pi_2$: Switch to optimizing $U_2.$
- $\pi_3$: Switch to optimizing $U_3.$
- $\pi_4$: Optimize the expectation of its current updated $\Delta U$, now 50% $U_2$ and 50% $U_3.$
- $\pi_5$: Shut down and let the humans optimize whatever $V$ they have in the actual world.

Let $u_1, u_2, u_3$ and $v_1, v_2, v_3$ stand for the outcomes in $\mathbb O$ that are respectively the [attainable optima](https://arbital.com/p/7t9) of the AI optimizing $U_i$ and the humans optimizing $V_i.$  Also let $u_{\Delta U}$ refer to the AI's attainable optimum outcome under the current $\Delta U$ treated as a constant.

Then the AI will prefer $\pi_5$ from this set whenever:

$$0.5 \cdot U_2(u_{\Delta U}) + 0.5 \cdot U_3(u_{\Delta U}) \ < \ 0.5 \cdot U_2(v_2) + 0.5 \cdot U_3(v_3)$$

This allows the $U_i(v_i)$ to fall well short of the AI's optima $U_i(u_i).$  The AI can potentially believe the humans are worse optimizers, or that our $V_i$ diverge from its $U_i$, and still defer.   $\pi_5$ will still be preferred so long as actions that do well under $U_2$ tend to do poorly under $U_3$ and vice versa, leading the AI to prefer a human-optimized $v_i$ specific to the true $U_i.$

In general this does require some mix of "the $U_i$ correspond to $V_i$ that humans can and will do pretty well" and "there's a wide variety of dissimilar $U_i.$"  E.g., suppose that $U_2$ is [paperclips](https://arbital.com/p/10h), $U_3$ is staples, and these are the only two possibilities.  A universe filled half with paperclips and half with staples will probably do better under the average of $U_2$ and $U_3$ than any plausible human behavior.

Obvious answers to the above requirements are that:

- It seems realistic that a meta-utility function would cover a very wide range of $U_i$ rather than just 3 alternatives, greatly increasing the advantage of $V$'s specificity.
- The proposal of [Inverse Reinforcement Learning](https://arbital.com/p/) would have the AI specifically assume that the [true](https://arbital.com/p/7s6) $U$ is correlated to whatever it sees the behavior of its operators tending to maximize. %%note:  This idea comes with its own arguable problems — e.g. humans sometimes optimize bad things.  Let us set those aside while considering only whether this approach solves the [shutdown problem](https://arbital.com/p/2xd) in particular.%%

# The problem of updated deference

Counter-arguendo, the reason this approach via moral uncertainty may fail to solve the [shutdown problem](https://arbital.com/p/2xd) (or the [corrigibility in general](https://arbital.com/p/3ps)) is that the AI may consider an [alternative policy option](https://arbital.com/p/43g):

- $\pi_6$:  Avoid being shut down; then gather more evidence $E$ to update $\Delta U$ as far as possible; then optimize the updated $\Delta U | E.$

This would tend to lead into a narrower $\Delta U | E$ and an outcome $u_{\Delta U | E}$ much more specific to the true $U_i,$ so that the human optimum $v_i$ would be at much less of an advantage.

This is the problem of getting the AI to 'defer' to us (prefer human optimization of $V$ to its own optimization of $\Delta U$) despite the future evidence that the AI thinks it can potentially gather to 'update' to $\Delta U|E$.  This [open problem](https://arbital.com/p/4m) is tentatively named the "problem of updated deference".  %note: This issue was first observed in analyzing [historical-fact shutdown](https://arbital.com/p/) as a possible alternative to [utility indifference](https://arbital.com/p/1b7).%

## Attempted rescue by AI unobservability

One possible attempt to rescue the original proposal might be to try to construct a meta-utility function such that the AI believes it can't learn by observation which of $U_2$ or $U_3$ are true; in other words, such that the AI does not believe it can update much further.

However, this directly cuts against the condition, also required for deference, that:

- The AI believes that humans optimize a $V_2$ well-suited to $U_2$ in possible worlds where $U_2$ is the true $U,$ and that humans optimize a $V_3$ well-suited to $U_3$ in possible worlds where $U_3$ is the true $U.$

Then the AI can learn well by observing human behavior, since empirical human behavior must correlate with the [true](https://arbital.com/p/7s6) $U$ according to the AI's meta-utility function.

Can we try to have the AI believe that observable human behavior is uninformative or ambiguous as to the [true](https://arbital.com/p/7s6) $U$?

But this cuts directly against the required condition:

- There are no policies which are simultaneously very good at optimizing $U_2$ and $U_3,$ compared to how well the specialized $V_i$ can do for particular $U_i.$

That is, if there are things humans can do that are simultaneously pretty good at optimizing all the remaining plausible $V_i$ (hence uniformative), and these are pretty good at optimizing the remaining plausible $U_i$ (hence acceptable to the AI), there are probably things the AI can do which would be even better at simultaneously optimizing all remaining $U_i.$

What if the AI has moral uncertainty that doesn't correlate to human behavior at all, and can't be resolved by any possible observation?  Then — leaving aside that the AI has no incentive to *defer* to humans in this case — we should expect the AI to at some point decide that it has obtained all possible information about $U$ and cannot find any policy that leads to a further update of $\Delta U.$  At this point the AI behaves as if optimizing a constant utility function $U_\Delta = \sum_i \mathbb P_{\Delta}(i) \cdot U_i,$ where $\mathbb P_\Delta$ indicates the probability distribution after the AI has updated its moral uncertainty as far as it could.  That is, the AI marginalizes over its remaining uncertainty since that uncertainty cannot be resolved.

# Relation to the general problem of fully updated value identification

One way to look at the central problem of [value identification](https://arbital.com/p/6c) in [superintelligence](https://arbital.com/p/41l) is that we'd ideally want some function that takes a complete but purely physical description of the universe, and spits out our [true intended](https://arbital.com/p/6h) notion of [value](https://arbital.com/p/55) $V$ in all its glory.  Since superintelligences would probably be pretty darned good at collecting data and guessing the empirical state of the universe, this probably solves the whole problem.

This is not the same problem as writing down our true $V$ by hand.  The minimum [algorithmic complexity](https://arbital.com/p/5v) of a meta-utility function $\Delta U$ which outputs $V$ after updating on all available evidence, seems [plausibly much lower](https://arbital.com/p/7wm) than the minimum algorithmic complexity for writing $V$ down directly.  But as of 2017, nobody has yet floated any *formal* proposal for a $\Delta U$ of this sort which has not been immediately shot down.

(There is one *informal* suggestion for how to turn a purely physical description of the universe into $V,$ [coherent extrapolated volition](https://arbital.com/p/3c5).  But CEV does not look like we could write it down as an algorithmically *simple* function of [sense data](https://arbital.com/p/36w), or a simple function over the [unknown true ontology](https://arbital.com/p/5c) of the universe.)

We can then view the problem of updated deference as follows:

For some $\Delta U$ we do know how to write down, let $T$ be the hypothetical result of updating $\Delta U$ on all empirical observations the AI can reasonably obtain.  By the argument given in the previous section, any uncertainty the AI deems unresolvable will behave as if marginalized out, so we can view $T$ as a simple utility function.

For any prior $\Delta U$ we currently know how to formalize, the corresponding fully updated $T$ [seems likely to be very far from our ideal](https://arbital.com/p/7wm) $V$ and to have its optimum far away from the default result of us trying to optimize our intuitive values.  [If the AI figures out](https://arbital.com/p/2x) this *true* fact, similar [instrumental pressures](https://arbital.com/p/10k) emerge as if we had given the AI the constant utility function $T$ divergent from our equivalent of $V.$

This problem [reproduces itself on the meta-level](https://arbital.com/p/): the AI also has a default incentive to resist our attempt to tweak its meta-utility function $\Delta U$ to a new meta-utility function $\Delta \dot U$ that updates to something other than $T.$  By default and ceteris paribus, this seems liable to be treated by the agent in [exactly the same way it would treat](https://arbital.com/p/3r6) us trying to tweak a constant utility function $U$ to a new $\dot U$ with an optimum far from $U$'s optimum.

If we *did* know how to specify prior $\Delta U$ such that updating it on data a superintelligence could obtain would reliably yield $T \approx V,$ the problem of aligned superintelligence would have been reduced to the problem of building an AI with that meta-utility function.  We could just specify $\Delta U$ and tell the AI to self-improve as fast as it wants, confident that true [value](https://arbital.com/p/55) would come out the other side.  Desired behaviors like "be cautious in what you do while learning" could probably be realized as the consequence of informing the young AI of true facts within the $\Delta U$ framework (e.g. "the universe is fragile, and you'll be much better at this if you wait another month to learn more, before you try to do anything large").  Achieving [general cognitive alignment](https://arbital.com/p/7td), free of [adversarial](https://arbital.com/p/7g0) situations, would probably be much more straightforward.

But short of this total solution, morally uncertain $\Delta U$ with a misaligned [ideal target](https://arbital.com/p/7s6) $T$ may not make progress on corrigibility in [sufficiently advanced](https://arbital.com/p/7g1) AIs.  And this may also be true at earlier points when $\Delta U$ has not fully updated, if the current AI correctly realizes that it will update later.

To make this argument slightly less informal, we could appeal to the premises that:

- Bayesians [don't update in a predictable direction](https://wiki.lesswrong.com/wiki/Conservation_of_expected_evidence);
- Sufficiently advanced cognitive agents would be [relatively efficient](https://arbital.com/p/6s) compared to humans, since we know of no human cognitive capability too magical to be duplicated and exceeded;
- Sufficiently advanced cognitive agents will appear to us to exhibit behavior that is, for all we know, compatible with their having [coherent probabilities and utilities](https://arbital.com/p/7hh); since, by efficiency, strategies so bad that even we can see they are dominated will have been ironed out.

Then if we can predict that the AI would update to wanting to run the universe itself without human interference after the AI had seen all collectable evidence, a [sufficiently advanced AI](https://arbital.com/p/7g1) can also see that this update is predictable (by efficiency) and therefore behaves as if it had already updated (by Bayesianism).  Efficiency is a sufficient condition but not a necessary one; [high-human](https://arbital.com/p/7mt) reasoning over the meta-level question also seems sufficient, and perhaps even [infrahuman](https://arbital.com/p/7mt) reasoning would suffice.

Therefore we should expect a sufficiently intelligent AI, given a morally uncertain utility function $\Delta U$ that updates to $\Delta U | E \approx T$ given all available evidence, to behave as corrigibly or incorrigibly as an AI given a constant utility function $T.$  This is a problem from the viewpoint of anyone who thinks we [do not currently know](https://arbital.com/p/7wm) how to pick $\Delta U$ such that surely $\Delta U | E \approx V,$ which makes corrigibility still necessary.

# Further research avenues

The motivation for trying to solve corrigibility with moral uncertainty is that this seems in some essential sense [conjugate to our own reasoning](https://arbital.com/p/7td) about why we want the AI to shut down; *we* don't think the AI has the correct answer.  A necessary step in echoing this reasoning inside the AI seems to be a meta-utility function taking on different object-level utility functions in different possible worlds; without this we cannot represent the notion of a utility function being guessed incorrectly.  If the argument above holds, that necessary step is however not sufficient.

What more is needed?  On one approach, we would like the AI to infer, in possible worlds where the humans try to shut the AI down, that *even the fully updated* $\Delta U | E$ ends up being wronger than humans left to their own devices, compared to the 'true' $U.$  This is what we believe about the AI relative to the true $V,$ so we should [look for a way to faithfully echo that reasoning](https://arbital.com/p/7td) inside the AI's beliefs about its true $U.$

The fundamental obstacle is that for any explicit structure of uncertainty $\Delta U$ and meaningful observation $e_0$ within that structure--e.g. where $e_0$ might be seeing the humans moving toward the shutdown button--we must ask, why wouldn't $\Delta U$ just update on that $e_0$?  Why would the updated $\Delta U | e_0$ still expect its own reasoning to be bad?

Generally, decision systems think that optimizing their utility functions based on their current beliefs is a good idea.  If you show the decision system new evidence, it updates beliefs and then thinks that optimizing its utility function on the updated beliefs is a good idea.  Optimizing the utility function based on all possible evidence is the best idea.  This reasoning doesn't yet change for meta-utility functions evidentially linked to human behaviors.

Averting this convergent conclusion seems like it might take a new meta-level idea involving some broader space of possible 'true' preference frameworks; or perhaps some nontrivially-structured recursive belief about one's own flawedness.

One suggestively similar such recursion is the [Death in Damascus dilemma](https://arbital.com/p/5qn) from decision theory.  In this dilemma, you must either stay in Damascus or flee to Aleppo, one of those cities will kill you, and Death (an excellent predictor) has told you that whichever decision you actually end up making turns out to be the wrong one.

[Death in Damascus yields complicated reasoning that varies between decision theories](https://arbital.com/p/5qn), and it's not clear that any decision theory yields reasoning we can adapt for corrigibility.  But we want the AI to internally echo our external reasoning in which we think $\Delta U,$ as we defined that moral uncertainty, *ends up* updating to the wrong conclusion even after the AI tries to update on the evidence of the humans believing this.  We want an AI which somehow believes that its own $\Delta U$ can be *fundamentally* flawed: that whatever reasoning the AI ends up doing about $\Delta U,$ on any meta-level, will yield the wrong answer compared to what $\Delta U$ defines as the true $U$; to furthermore believe that the human $V$ will do better under this true $U$; to believe that this state of affairs is evidentially indicated by the humans trying to shut down the AI; and believe that $\Delta U$ still updates to the wrong answer even when the AI tries to update on all the previous meta-knowledge; except for the meta-meta answer of just shutting down, which becomes the best possible choice given all the previous reasoning.  This seems suggestively similar in structure to Death's prediction that whatever you do will be the wrong decision, even having taken Death's statement into account.

The Death in Damascus scenario can be well-represented in some (nonstandard) decision theories.  This presents one potential avenue for further formal research on using moral uncertainty to yield shutdownability--in fact, using moral uncertainty to solve in general the [hard problem of corrigibility](https://arbital.com/p/3ps).

Rob Bensinger
Eric Bruylant
Eliezer Yudkowsky
Thomas Jones

One possible scheme in [AI alignment](https://arbital.com/p/2v) is to give the AI a state of  [https://arbital.com/p/-7s2](https://arbital.com/p/-7s2) implying that we know more than the AI does about its own utility function, as the AI's [meta-utility function](https://arbital.com/p/7t8) defines its [ideal target](https://arbital.com/p/7s6).  Then we could tell the AI, "You should let us [shut you down](https://arbital.com/p/2xd) because we know something about your [ideal target](https://arbital.com/p/7s2) that you don't, and we estimate that we can optimize your ideal target better without you."

The obstacle to this scheme is that belief states of this type also tend to imply that an even better option for the AI would be to learn its ideal target by observing us.  Then, having 'fully updated', the AI would have no further reason to 'defer' to us, and could proceed to directly optimize its ideal target.

Furthermore, if the present AI foresees the possibility of fully updating later, the current AI may evaluate that it is better to avoid being shut down now so that the AI can directly optimize its ideal target later, after updating.  Thus the prospect of future updating is a reason to behave [incorrigibly](https://arbital.com/p/45) in the present.


Moral uncertainty

"Moral uncertainty" in the context of AI refers to an agent with an "uncertain utility function".  That is, we can view the agent as pursuing a [utility function](https://arbital.com/p/1fw) that takes on different values in different subsets of possible worlds.

For example, an agent might have a [meta-utility function](https://arbital.com/p/meta_utility) saying that eating cake has a utility of €8 in worlds where Lee Harvey Oswald shot John F. Kennedy and that eating cake has a utility of €10 in worlds where it was the other way around.  This agent will be motivated to inquire into political history to find out which utility function is probably the 'correct' one (relative to this meta-utility function), though it will never be [absolutely sure](https://arbital.com/p/4mq).

Moral uncertainty must be resolvable by some conceivable observation in order to function as uncertainty.  Suppose for example that an agent's probability distribution $\Delta U$ over the 'true' utility function $U$ asserts a dependency on a fair quantum coin that was flipped inside a sealed box then destroyed by explosives: the utility function is $U_1$ over outcomes in the worlds where the coin came up heads, and if the coin came up tails the utility function is $U_2.$  If the agent thinks it has no way of ever figuring out what happened inside the box, it will thereafter behave as if it had a single, constant, certain utility function equal to $0.5 \cdot U_1 + 0.5 \cdot U_2.$

"Moral uncertainty" in the context of AI refers to an agent with an "uncertain utility function". That is, we can view the agent as pursuing a [utility function](https://arbital.com/p/1fw) that takes on different values in different subsets of possible worlds.

For example, an agent might have a [meta-utility function](https://arbital.com/p/meta_utility) saying that eating cake has a utility of €8 in worlds where Lee Harvey Oswald shot John F. Kennedy and that eating cake has a utility of €10 in worlds where it was the other way around.  This agent will be motivated to inquire into political history to find out which utility function is probably the 'correct' one (relative to this meta-utility function).


Ideal target

The 'ideal target' of a [meta-utility function](https://arbital.com/p/meta_utility) $\Delta U$ which behaves as if a ground-level utility function $U$ is taking on different values in different possible worlds, is the value of $U$ in the actual world; or the expected value of $U$ after updating on all possible accessible evidence.  If chocolate has €8 utility in worlds where the sky is blue, and €5 utility in worlds where the sky is not blue, then in the AI's 'ideal target' utility function, the utility of chocolate is €8.


Meta-utility function

A "meta-utility function" is a [preference framework](https://arbital.com/p/5f) built by composing multiple simple [utility functions](https://arbital.com/p/1fw) into a more complicated structure.  The point might be, e.g., to describe an [https://arbital.com/p/-agent](https://arbital.com/p/-agent) that [optimizes different utility functions depending on whether a switch is pressed](https://arbital.com/p/1b7), or an agent that [learns a 'correct' utility function](https://arbital.com/p/7s2) by observing data informative about some [https://arbital.com/p/-7s6](https://arbital.com/p/-7s6).  For central examples see [https://arbital.com/p/1b7](https://arbital.com/p/1b7) and [https://arbital.com/p/7s2](https://arbital.com/p/7s2).


Attainable optimum

The 'attainable optimum' of an agent's preferences is the most preferred option that the agent can (a) obtain using its bounded material capabilities and (b) find as an available option using its limited cognitive resources; as distinct from the theoretical global maximum of the agent's utility function.  When you run a [non-mildly-optimizing](https://arbital.com/p/2r8) agent, what you actually get as the resulting outcome is not the single outcome that theoretically maximizes the agent's [utility function](https://arbital.com/p/1fw); you rather get that agent's attainable optimum of its [expectation](https://arbital.com/p/18t) of that utility function.  A preference framework's 'attainable optimum' is what you get in practice when somebody runs the corresponding agent.


The AI must tolerate your safety measures

A corollary of the [https://arbital.com/p/-7g0](https://arbital.com/p/-7g0):  For every kind of safety measure proposed for a [https://arbital.com/p/-7g1](https://arbital.com/p/-7g1), we should immediately ask how to avoid this safety measure inducing an [adversarial context](https://arbital.com/p/7g0) between the human programmers and the agent being constructed.

A further corollary of the [generalized principle of cognitive alignment](https://arbital.com/p/cognitive_alignment) would suggest that, if we know how to do it without inducing further problems, the AI should positively *want* the safety measure to be there.

E.g., if the safety measure we want is a [suspend button](https://arbital.com/p/2xd) (off switch), our first thought should be, "How do we build an agent such that it doesn't mind the off-switch being pressed?"

At a higher level of alignment, if something damages the off-switch, the AI might be so configured that it naturally and spontaneously thinks, "Oh no!  The off-switch is damaged!" and reports this to the programmers, or failing any response there, tries to repair the off-switch itself.  But this would only be a good idea if we were pretty sure we knew this wouldn't lead to the AI substituting its own helpful ideas of what an off-switch would do, or shutting off extra hard.

Similarly, if you start thinking how nice it would be to have the AI operating inside a [box](https://arbital.com/p/6z) rather than running around in the outside world, your first thought should not be "How do I enclose this box in 12 layers of Faraday cages, a virtual machine running a Java sandbox, and 15 meters of concrete?" but rather "How would I go about constructing an agent that only cared about things inside a box and experienced no motive to affect anything outside the box?"

At a higher level of alignment we might imagine constructing a sort of agent that, if something went wrong, would think "Oh no I am outside the box, that seems very unsafe, how do I go back in?"  But only if we were very sure that we were not thereby constructing a kind of agent that would, e.g., build a superintelligence outside the box just to make extra sure the original agent stayed inside it.

Many classes of safety measures are only meant to come into play after something else has already gone wrong, implying that other things may have gone wrong earlier and without notice.  This suggests that pragmatically we should focus on the principle of "The AI should leave the safety measures alone and not experience an incentive to change their straightforward operation" rather than tackling the more complicated problems of exact alignment inherent in "The AI should be enthusiastic about the safety measures and want them to work even better."

However, if the AI is [changing its own code or constructing subagents](https://arbital.com/p/1mq), it is necessary for the AI to have at least *some* positive motivation relating to any safety measures embodied in the operation of an internal algorithm.  An AI indifferent to that code-based safety measure would tend to [just leave the uninteresting code out of the next self-modification](https://arbital.com/p/1fx).


Interruptibility

"Interruptibility" is a subproblem of [corrigibility](https://arbital.com/p/45) (creating an advanced agent that allows us, its creators, to 'correct' what *we* see as our mistakes in constructing it), as seen from a machine learning paradigm.  In particular, "interruptibility" says, "If you do interrupt the operation of an agent, it must not learn to avoid future interruptions."

The groundbreaking paper on interruptibility, "[Safely Interruptible Agents](https://www.fhi.ox.ac.uk/wp-content/uploads/Interruptibility.pdf)", was published by [Laurent Orseau](https://arbital.com/p/) and [Stuart Armstrong](https://arbital.com/p/).  This says, roughly, that to avoid a model-based reinforcement-learning algorithm from learning to avoid interruption, we should, after any interruption, propagate internal weight updates as if the agent had received exactly its expected reward from before the interruption.  This approach was inspired by [Stuart Armstrong](https://arbital.com/p/)'s earlier idea of [https://arbital.com/p/-1b7](https://arbital.com/p/-1b7).

Contrary to some uninformed media coverage, the above paper doesn't solve [the general problem of getting an AI to not try to prevent itself from being switched off](https://arbital.com/p/2xd).  In particular, it doesn't cover the [advanced-safety](https://arbital.com/p/2l) case of a [sufficiently intelligent AI](https://arbital.com/p/7g1) that is trying to [achieve particular future outcomes](https://arbital.com/p/9h) and that [realizes](https://arbital.com/p/3nf) it needs to [go on operating in order to achieve those outcomes](https://arbital.com/p/7g2).

Rather, if a non-general AI is operating by policy reinforcement - repeating policies that worked well last time, and avoiding policies that worked poorly last time, in some general sense of a network being trained - then 'interruptibility' is about making an algorithm that, *after* being interrupted, doesn't define this as a poor outcome to be avoided (nor a good outcome to be repeated).

One way of seeing that Interruptibility doesn't address the general-cognition form of the problem is that Interruptibility only changes what happens after an actual interruption.  So if a problem can arise from an AI foreseeing interruption in advance, before having ever actually been shut off, interruptibility won't address that (on the current paradigm).

Similarly, interruptibility would not be [consistent under cognitive reflection](https://arbital.com/p/2rb); a sufficiently advanced AI that knew about the existence of the interruptibility code would have no reason to want that code to go on existing.  (It's hard to even phrase that idea inside the reinforcement learning framework.)

Metaphorically speaking, we could see the general notion of 'interruptibility' as the modern-day shadow of [corrigibility](https://arbital.com/p/45) problems for non-[generally-intelligent](https://arbital.com/p/42g), non-[future-preferring](https://arbital.com/p/9h), non-[reflective](https://arbital.com/p/1c1) machine learning algorithms.

For an example of ongoing work on the [advanced-agent](https://arbital.com/p/2c) form of [https://arbital.com/p/-45](https://arbital.com/p/-45), see the entry on Armstrong's original proposal of [https://arbital.com/p/1b7](https://arbital.com/p/1b7).


Generalized principle of cognitive alignment

A generalization of the [https://arbital.com/p/7g0](https://arbital.com/p/7g0) is that whenever we are asking how we want an AI algorithm to execute with respect to some alignment or safety issue, we might ask how we ourselves are thinking about that problem, and whether we can have the AI think conjugate thoughts.   This may sometimes seem like a much more complicated or dangerous-seeming approach than simpler avenues, but it's often a source of useful inspiration.

For example, with respect to the [https://arbital.com/p/-2xd](https://arbital.com/p/-2xd), this principle might lead us to ask:  "Is there some way we can have the AI [truly understand that its own programmers may have built the wrong AI](https://arbital.com/p/3ps), including the wrong definition of exactly what it means to have 'built the wrong AI', such that [the AI thinks it *cannot* recover the matter by optimizing any kind of preference already built into it](https://arbital.com/p/7rc), so that the AI itself wants to shut down before having a great impact, because when the AI sees the programmers trying to press the button or contemplates the possibility of the programmers pressing the button, updating on this information causes the AI to expect its further operation to have a net bad impact in some sense that it can't overcome through any kind of clever strategy besides just shutting down?"

This in turn might imply a complicated mind-state we're not sure how to get right, such that we would prefer a simpler approach to shutdownability along the lines of a perfected [utility indifference](https://arbital.com/p/1b7) scheme.  If we're shutting down the AI at all, it means something has gone wrong, which implies that something else may have gone wrong earlier before we noticed.  That seems like a bad time to have the AI be enthusiastic about shutting down even better than in its original design (unless we can get the AI to [understand even *that* part too](https://arbital.com/p/3ps), the danger of that kind of 'improvement', during its normal operation).

Trying for maximum cognitive alignment isn't always a good idea; but it's almost always worth trying to think through a safety problem from that perspective for inspiration on what we'd ideally want the AI to be doing.  It's often a good idea to move closer to that ideal when this doesn't introduce greater complication or other problems.


Minimality principle

In the context of [https://arbital.com/p/2v](https://arbital.com/p/2v), the "Principle of Minimality" or "Principle of Least Everything" says that when we are building the *first* [https://arbital.com/p/-7g1](https://arbital.com/p/-7g1), we are operating in an extremely dangerous context in which building a marginally more powerful AI is marginally more dangerous.  The first AGI ever built should therefore execute the least dangerous plan for [preventing immediately following AGIs from destroying the world six months later](https://arbital.com/p/6y).  Furthermore, the least dangerous plan is not the plan that seems to contain the fewest material actions that seem risky in a conventional sense, but rather the plan that requires the *least dangerous cognition* from the AGI executing it.  Similarly, inside the AGI itself, if a class of thought seems dangerous but necessary to execute sometimes, we want to execute the fewest possible instances of that class of thought.

E.g., if we think it's a dangerous kind of event for the AGI to ask "How can I achieve this end using strategies from across every possible domain?" then we might want a design where most routine operations only search for strategies within a particular domain, and events where the AI searches across all known domains are rarer and visible to the programmers.  Processing a goal that can recruit subgoals across every domain would be a dangerous event, albeit a necessary one, and therefore we want to do *less* of it within the AI (and require positive permission for all such cases and then require operators to validate the results before proceeding).

Ideas that inherit from this principle include the general notion of [https://arbital.com/p/6w](https://arbital.com/p/6w), [taskishness](https://arbital.com/p/4mn), and [https://arbital.com/p/-2r8](https://arbital.com/p/-2r8).


Understandability principle

An obvious [design principle](https://arbital.com/p/7v8) of [https://arbital.com/p/2v](https://arbital.com/p/2v) that nonetheless deserves to be stated explicitly:  The more you understand what the heck is going on inside your AI, the more likely you are to succeed at aligning it.

This principle participates in motivating design subgoals like [passive transparency](https://arbital.com/p/passive_transparency); or the AI having explicitly represented preferences; or, taken more broadly, pretty much every aspect of the AI design where we think we understand how any part works or what any part is doing.

The Understandability Principle in its broadest sense is *so* widely applicable that it may verge on being an [applause light](https://arbital.com/p/applause_light).  So far as is presently known to the author(s) of this page, counterarguments against the importance of understanding at least *some* parts of the AI's thought processes, have been offered only by people who reject at least one of the [https://arbital.com/p/1y](https://arbital.com/p/1y) or the [Fragility of Cosmopolitan Value thesis](https://arbital.com/p/fragility).  That is, the Understandability Principle in this very broad sense is rejected only by people who reject in general the importance of deliberate design efforts to align AI.

A more controversial subthesis is Yudkowsky's proposed [Effability principle](https://arbital.com/p/7vb).

Rob Bensinger
Eric Bruylant
Eliezer Yudkowsky


Principles in AI alignment

A 'principle' of [AI alignment](https://arbital.com/p/2v) is something we want in a broad sense for the whole AI, which has informed narrower design proposals for particular parts or aspects of the AI.

For example:

- The **[https://arbital.com/p/7g0](https://arbital.com/p/7g0)** says that the AI should never be searching for a way to defeat our safety measures or do something else we don't want, even if we *think* this search will come up empty; it's just the wrong thing for us to program computing power to do.
  - This informs the proposal of [https://arbital.com/p/5s](https://arbital.com/p/5s): we ought to build an AI that wants to attain the class of outcomes we want to see.
  - This informs the proposal of [https://arbital.com/p/45](https://arbital.com/p/45), subproposal [https://arbital.com/p/1b7](https://arbital.com/p/1b7): if we build a [suspend button](https://arbital.com/p/2xd) into the AI, we need to make sure the AI experiences no [instrumental pressure](https://arbital.com/p/10k) to [disable the suspend button](https://arbital.com/p/7g2).
- The **[https://arbital.com/p/7tf](https://arbital.com/p/7tf)** says that when we are building the first aligned AGI, we should try to do as little as possible, using the least dangerous cognitive computations possible, that is necessary in order to prevent the default outcome of the world being destroyed by the first unaligned AGI.
  - This informs the proposal of [https://arbital.com/p/2r8](https://arbital.com/p/2r8) and [Taskishness](https://arbital.com/p/4mn):  We are safer if all goals and subgoals of the AI are formulated in such a way that they can be achieved as greatly as preferable using a bounded amount of effort, and the AI only exerts enough effort to do that.
  - This informs the proposal of [Behaviorism](https://arbital.com/p/102):  It seems like there are some [pivotal-act](https://arbital.com/p/6y) proposals that don't require the AI to understand and predict humans in great detail, just to master engineering; and it seems like we can head off multiple thorny problems by not having the AI trying to model humans or other minds in as much detail as possible.

Please be [guarded](https://arbital.com/p/10l) about declaring things to be 'principles' unless they have already informed more than one specific design proposal and more than one person thinks they are a good idea.  You could call them 'proposed principles' and post them under your own domain if you personally think they are a good idea.  There are a *lot* of possible 'broad design wishes', or things that people think are 'broad design wishes', and the principles that have actually already informed specific design proposals would otherwise get lost in the crowd.

A 'principle' of [AI alignment](https://arbital.com/p/2v) is something we want in a broad sense for the whole AI, which has informed narrower design proposals for particular parts or aspects of the AI.

Examples:

- The **[https://arbital.com/p/7g0](https://arbital.com/p/7g0)** says that the AI should never be searching for a way to defeat our safety measures or do something else we don't want, even if we *think* this search will come up empty; it's just the wrong thing for us to program computing power to do.
  - This informs the proposal of [https://arbital.com/p/45](https://arbital.com/p/45), subproposal [https://arbital.com/p/1b7](https://arbital.com/p/1b7): if we build a [suspend button](https://arbital.com/p/2xd) into the AI, we need to make sure the AI experiences no [instrumental pressure](https://arbital.com/p/10k) to [disable the suspend button](https://arbital.com/p/7g2).
- The **[https://arbital.com/p/7tf](https://arbital.com/p/7tf)** says that when we are building the first AGI, we should try to do as little as possible, using the least dangerous cognitive computations possible, in order to prevent the default outcome of the world otherwise being destroyed by the second AGI.
  - This informs the proposal of [https://arbital.com/p/2r8](https://arbital.com/p/2r8) and [taskishness](https://arbital.com/p/4mn):  We are safer if all goals and subgoals of the AI are formulated in such a way that they can be achieved as greatly as preferable using a bounded amount of effort, and the AI only exerts enough effort to do that.


Effability principle

A proposed [principle](https://arbital.com/p/7v8) of [https://arbital.com/p/2v](https://arbital.com/p/2v) stating, "The more insight you have into the deep structure of an AI's cognitive operations, the more likely you are to succeed in aligning that AI."

As an example of increased effability, consider the difference between having the idea of [expected utility](https://arbital.com/p/18t) while building your AI, versus having never heard of expected utility.  The idea of expected utility is so well-known that it may not seem salient as an insight anymore, but consider the difference between having this idea and not having it.

Staring at the expected utility principle and how it decomposes into a utility function and a probability distribution, leads to a potentially obvious-sounding but still rather important insight:

Rather than *all* behaviors and policies and goals needing to be up-for-grabs in order for an agent to adapt itself to a changing and unknown world, the agent can have a *stable utility function* and *changing probability distribution.*

E.g., when the agent tries to grab the cheese and discovers that the cheese is too high, we can view this as an [update](https://arbital.com/p/1ly) to the agent's *beliefs about how to get cheese,* without changing the fact that *the agent wants cheese.*

Similarly, if we want superhuman performance at playing chess, we can ask for an AI that has a known, stable, understandable preference to win chess games; but a probability distribution that has been refined to greater-than-human accuracy about *which* policies yield a greater probabilistic expectation of winning chess positions.

Then contrast this to the state of mind where you haven't decomposed your understanding of cognition into preference-ish parts and belief-ish parts.  In this state of mind, for all you know, every aspect of the AI's behavior, every goal it has, must potentially need to change in order for the AI to deal with a changing world; otherwise the AI will just be stuck executing the same behaviors over and over... right?  Obviously, this notion of AI with unchangeable preferences is just a fool's errand.  Any AI like that would be too stupid to make a [major difference](https://arbital.com/p/6y) for good or bad. %note: The idea of [https://arbital.com/p/10g](https://arbital.com/p/10g) is also important here; e.g. that scientific curiosity is already an instrumental strategy for 'make as many paperclips as possible', rather than an AI needing a separate [terminal](https://arbital.com/p/1bh) preference about scientific curiosity in order to ever engage in it.%

(This argument has indeed been encountered in the wild many times.)

Probability distributions and utility functions have now been known for a relatively long time and are understood relatively well; people have made many, many attempts to poke at their structure and imagine potential variations and show what goes wrong with those variations.  There is now known an [enormous family of coherence theorems](https://arbital.com/p/7hh) stating that "Strategies which are not qualitatively dominated can be viewed as coherent with some consistent probability distribution and utility function."  This suggests that we can in a broad sense expect that, [as a sufficiently advanced AI's behavior is more heavily optimized for not qualitatively shooting itself in the foot, that AI will end up exhibiting some aspects of expected-utility reasoning](https://arbital.com/p/21).  We have some idea of why a sufficiently advanced AI would have expected-utility-ish things going on *somewhere* inside it, or at least behave that way so far as we could tell by looking at the AI's external actions.

So we can say, "Look, if you don't *explicitly* write in a utility function, the AI is probably going to end up with something like a utility function *anyway,* you just won't know where it is.  It seems considerably wiser to know what that utility function says and write in on purpose.  Heck, even if you say you explicitly *don't* want your AI to have a stable utility function, you'd need to know all the coherence theorems you're trying to defy by saying that!"

The Effability Principle states (or rather hopes) that as we get marginally more of this general kind of insight into an AI's operations, we become marginally more likely to be able to align the AI.

The example of expected utility arguably suggests that if there are any *more* ideas like that lying around, which we *don't* yet have, our lack of those ideas may entirely doom the AI alignment project or at least make it far more difficult.  We can in principle imagine someone who is just using a big reinforcement learner to try to execute some large [pivotal act](https://arbital.com/p/6y), who has no idea where the AI is keeping its consequentialist preferences or what those preferences are; and yet this person was *so* paranoid and had the resources to put in *so* much monitoring and had *so* many tripwires and safeguards and was *so* conservative in how little they tried to do, that they succeeded anyway.  But it doesn't sound like a good idea to try in real life.

The search for increased effability has generally motivated the "Agent Foundations" agenda of research within MIRI.  While not the *only* aspect of AI alignment, a concern is that this kind of deep insight may be a heavily serially-loaded task in which researchers need to develop one idea after another, compared to relatively [shallow ideas in AI alignment](https://arbital.com/p/shallow_alignment_ideas) that require less serial time to create.  That is, this kind of research is among the most important kinds of research to start *early.*

The chief rival to effability is the [Supervisability Principle](https://arbital.com/p/supervisability_principle), which, while not directly opposed to effability, tends to focus our understanding of the AI at a much larger grain size.  For example, the Supervisability Principle says, "Since the AI's behaviors are the only thing we can train by direct comparison with something we know to be already aligned, namely human behaviors, we should focus on ensuring the greatest possible fidelity at that point, rather than any smaller pieces whose alignment cannot be directly determined and tested in the same way."  Note that both principles agree that it's important to [understand](https://arbital.com/p/7v7) certain facts about the AI as well as possible, but they disagree about what should be our design priority for rendering maximally understandable.

zohar jackson
Eric Bruylant
Eliezer Yudkowsky


Cognitive domain

In the context of cognitive science, a 'domain' is a subject matter, or class of problems, that is being reasoned about by some algorithm or agent.  A robotic car is operating in the domain of "legally and safely driving in a city".  The subpart of the robotic car that plots a route might be operating in the domain "finding low-cost traversals between two nodes in a graph with cost-labeled edges".


Theory of (advanced) agents

Many issues in AI alignment have dependencies on what we think we can factually say about the general design space of cognitively powerful agents, or on which background assumptions yield which implications about advanced agents.  E.g., the [Orthogonality Thesis](https://arbital.com/p/1y) is a claim about the general design space of powerful AIs.  The design space of advanced agents is very wide, and only very weak statements seem likely to be true about the *whole* design space; but we can still try to say 'If X then Y' and refute claims about 'No need for if-X, Y happens anyway!'


General intelligence

# Definition

Although humans share 95% of their DNA with chimpanzees, and have brains only three times as large as chimpanzee brains, humans appear to be *far* better than chimpanzees at learning an *enormous* variety of cognitive [domains](https://arbital.com/p/7vf).  A bee is born with the ability to construct hives; a beaver is born with an instinct for building dams; a human looks at both and imagines a gigantic dam with a honeycomb structure of internal reinforcement.  Arguendo, some set of factors, present in human brains but not in chimpanzee brains, seem to sum to a central cognitive capability that lets humans learn a huge variety of different domains without those domains being specifically preprogrammed as instincts.

This very-widely-applicable cognitive capacity is termed **general intelligence** (by most AI researchers explicitly talking about it; the term isn't universally accepted as yet).

We are not perfectly general - we have an easier time learning to walk than learning to do abstract calculus, even though the latter is much easier in an objective sense. But we're sufficiently general that we can figure out Special Relativity and engineer skyscrapers despite our not having those abilities built-in at compile time (i.e., at birth). An [Artificial General Intelligence](https://arbital.com/p/42g) would have the same property; it could learn a tremendous variety of domains, including domains it had no inkling of when it was switched on.

More specific hypotheses about *how* general intelligence operates have been advanced at various points, but any corresponding attempts to *define* general intelligence that way, would be [theory-laden](https://arbital.com/p/).  The pretheoretical phenomenon to be explained is the extraordinary variety of human achievements across many non-instinctual domains, compared to other animals.

[https://arbital.com/p/toc:](https://arbital.com/p/toc:)

## Artificial General Intelligence is not [par-human](https://arbital.com/p/7mt) AI

Since we only know about one organism with this 'general' or 'significantly more generally applicable than chimpanzee cognition' intelligence, this capability is sometimes *identified* with humanity, and consequently with our overall level of cognitive ability.

We do not, however, *know* that "cognitive ability that works on a very wide variety of problems" and "overall humanish levels of performance" need to go together across [much wider differences of mind design](https://arbital.com/p/nonanthropomorphism).

Humans evolved incrementally out of earlier hominids by blind processes of natural selection; evolution wasn't trying to design a human on purpose.  Because of the way we evolved incrementally, all neurotypical humans have specialized evolved capabilities like 'walking' and 'running' and 'throwing stones' and 'outwitting other humans'.  We have all the primate capabilities and all the hominid capabilities *as well as* whatever is strictly necessary for general intelligence.

So, for all we know at this point, there could be some way to get a 'significantly more general than chimpanzee cognition' intelligence, in the equivalent of a weaker mind than a human brain.  E.g., due to leaving out some of the special support we evolved to run, throw stones, and outwit other minds.  We might at some point consistently see an infrahuman general intelligence that is not like a disabled human, but rather like some previously unobserved and unimagined form of weaker but still highly general intelligence.

Since the concepts of 'general intelligence' and 'roughly par-human intelligence' come apart in theory and possibly also in practice, we should avoid speaking of Artificial General Intelligence as if were identical with a concept like "human-level AI".

## General intelligence is not perfect intelligence

General intelligence doesn't imply the ability to solve every kind of cognitive problem; if we wanted to use a longer phrase we could say that humans have 'significantly more generally applicable intelligence than chimpanzees'.  A sufficiently advanced Artificial Intelligence that could self-modify (rewrite its own code) might have 'significantly more generally applicable intelligence than humans'; e.g. such an AI might be able to easily write bug-free code in virtue of giving itself specialized cognitive algorithms for programming.  Humans, to write computer programs, need to adapt savanna-specialized tiger-evasion modules like our visual cortex and auditory cortex to representing computer programs instead, which is one reason we're such terrible programmers.

Similarly, it's not hard to construct math problems to which we know the solution, but are unsolvable by any general cognitive agent that fits inside the physical universe.  For example, you could pick a long random string and generate its SHA-4096 hash, and if the SHA algorithm turns out to be secure against quantum computing, you would be able to construct a highly specialized 'agent' that could solve the problem of 'tell me which string has this SHA-4096 hash' which no other agent would be able to solve without directly inspecting your agent's cognitive state, or [tricking your agent into revealing the secret](https://arbital.com/p/9t), etcetera.  The 'significantly more generally applicable than chimpanzee intelligence' of humans is able to figure out how to launch interplanetary space probes just by staring at the environment for a while, but it still can't reverse SHA-4096 hashes.

It would however be an instance of the [continuum fallacy](https://en.wikipedia.org/wiki/Continuum_fallacy), [nirvana fallacy](https://en.wikipedia.org/wiki/Nirvana_fallacy), false dichotomy, or [straw superpower fallacy](https://arbital.com/p/7nf), to argue:

- Some small agents can solve certain specific math problems unsolvable by much larger superintelligences.
- Therefore there is no perfectly general intelligence, just a continuum of being able to solve more and more problems.
- Therefore there is nothing worthy of remark in how humans are able to learn a far wider variety of domains than chimpanzees, nor any sharp jump in generality that an AI might exhibit in virtue of obtaining some central set of cognitive abilities.

For attempts to talk about performance relative to a truly general measure of intelligence (as opposed to just saying that humans seem to have some central capability which sure lets them learn a whole lot of stuff) see [Shane Legg and Marcus Hutter's work on proposed metrics of 'universal intelligence'](https://arbital.com/p/).

## General intelligence is a separate concept from IQ / g-factor

Charles Spearman found that by looking on performances across many cognitive tests, he was able to infer a central factor, now called *Spearman's g*, which appeared to be *more* correlated with performance on each task than any of the tasks were correlated with *each other*.

[For example](https://en.wikipedia.org/wiki/G_factor_) (psychometrics)), the correlation between students' French and English scores was 0.67: that is, 67% of the variation in performance in French could be predicted by looking at the student's score in English.

However, by looking at all the test results together, it was possible to construct a central score whose correlation with the student's French score was 88%.

This would make sense if, for example, the score in French was "g-factor plus uncorrelated variables" and the score in English was "g-factor plus other uncorrelated variables".  In this case, the setting of the g-factor latent variable, which you could infer better by looking at all the student's scores together, would be more highly correlated with both French and English observations, than those tests would be correlated with each other.

In the context of Artificial Intelligence, g-factor is *not* what we want to talk about.  We are trying to point to a factor separating humans from chimpanzees, not to internal variations within the human species.

That is:  If you're trying to build the first mechanical heavier-than-air flying machine, you ought to be thinking "How do birds fly?  How do they stay up in the air, at all?"  Rather than, "Is there a central Fly-Q factor that can be inferred from the variation in many different measures of how well individual pigeons fly, which lets us predict the individual variation in a pigeon's speed or turning radius better than any single observation about one factor of that pigeon's flying ability?"

In some sense the existence of g-factor could be called Bayesian evidence for the notion of general intelligence: if general intelligence didn't exist, probably neither would IQ.  Likewise the observation that, e.g., John von Neumann existed and was more productive across multiple disciplines compared to his academic contemporaries.  But this is not the main argument or the most important evidence.  Looking at humans versus chimpanzees gives us a much, much stronger hint that a species' ability to land space probes on Mars correlates with that species' ability to prove Fermat's Last Theorem.

# Cross-domain consequentialism

A marginally more detailed and hence theory-laden view of general intelligence, from the standpoint of [advanced agent properties](https://arbital.com/p/2c), is that we can see general intelligence as "general cross-domain learning and [consequentialism](https://arbital.com/p/9h)".

That is, we can (arguendo) view general intelligence as: the ability to learn to model a wide variety of domains, and to construct plans that operate within and across those domains.

For example:  AlphaGo can be seen as trying to achieve the consequence of a winning Go position on the game board--to steer the future into the region of outcomes that AlphaGo defines as a preferred position.  However, AlphaGo only plans *within* the domain of legal Go moves, and it can't learn any domains other than that.  So AlphaGo can't, e.g., make a prank phone call at night to Lee Se-Dol to make him less well-rested the next day, *even though this would also tend to steer the future of the board into a winning state,* because AlphaGo wasn't preprogrammed with any tactics or models having to do with phone calls or human psychology, and AlphaGo isn't a general AI that could learn those new domains.

On the other hand, if a general AI were given the task of causing a certain Go board to end up in an outcome defined as a win, and that AI had 'significantly more generally applicable than chimpanzee intelligence' on a sufficient level, that Artificial General Intelligence might learn what humans are, learn that there's a human trying to defeat it on the other side of the Go board, realize that it might be able to win the Go game more effectively if it could make the human play less well, realize that to make the human play less well it needs to learn more about humans, learn about humans needing sleep and sleep becoming less good when interrupted, learn about humans waking up to answer phone calls, learn how phones work, learn that some Internet services connect to phones...

If we consider an actual game of Go, rather than a [logical game](https://arbital.com/p/9s) of Go, then the state of the Go board at the end of the game is produced by an enormous and tangled causal process that includes not just the proximal moves, but the AI algorithm that chooses the moves, the cluster the AI is running on, the humans who programmed the cluster; and also, on the other side of the board, the human making the moves, the professional pride and financial prizes motivating the human, the car that drove the human to the game, the amount of sleep the human got that night, all the things all over the world that *didn't* interrupt the human's sleep but *could* have, and so on.  There's an enormous lattice of causes that lead up to the AI's and the human's actual Go moves.

We can see the cognitive job of an agent in general as "select policies or actions which lead to a more preferred outcome".  The enormous lattice of real-world causes leading up to the real-world Go game's final position, means that an enormous set of possible interventions could potentially steer the real-world future into the region of outcomes where the AI won the Go game.  But these causes are going through all sorts of different [domains](https://arbital.com/p/7vf) on their way to the final outcome, and correctly choosing from the much wider space of interventions means you need to understand all the domains along the way.  If you don't understand humans, understanding phones doesn't help; the prank phone call event goes through the sleep deprivation event, and to correctly model events having to do with sleep deprivation requires knowing about humans.

# Deep commonalities across cognitive domains

To the extent one credits the existence of 'significantly more general than chimpanzee intelligence', it implies that there are common cognitive subproblems of the huge variety of problems that humans can (learn to) solve, despite the surface-level differences of those domains.  Or at least, the way humans solve problems in those domains, the cognitive work we do must have deep commonalities across those domains.  These commonalities may not be visible on an immediate surface inspection.

Imagine you're an ancient Greek who doesn't know anything about the brain having a visual cortex.  From your perspective, ship captains and smiths seem to be doing a very different kind of work; ships and anvils seem like very different objects to know about; it seems like most things you know about ships don't carry over to knowing about anvils.  Somebody who learns to fight with a spear, does not therefore know how to fight with a sword and shield; they seem like quite different weapon sets.

(Since, by assumption, you're an ancient Greek, you're probably also not likely to wonder anything along the lines of "But wait, if these tasks didn't all have at least some forms of cognitive labor in common deep down, there'd be no reason for humans to be simultaneously better at all of them than other primates.")

Only after learning about the existence of the cerebral cortex and the cerebellum and some hypotheses about what those parts of the brain are doing, are you likely to think anything along the lines of:

"Ship-captaining and smithing and spearfighting and swordfighting look like they all involve using temporal hierarchies of chunked tactics, which is a kind of thing the cortical algorithm is hypothesized to do.  They all involve realtime motor control with error correction, which is a kind of thing the cerebellar cortex is hypothesized to do.  So if the human cerebral cortex and cerebellar cortex are larger or running better algorithms than chimpanzees' cerebrums and cerebellums, humans being better at learning and performing this kind of deep underlying cognitive labor that all these surface-different tasks have in common, could explain why humans are simultaneously better than chimpanzees at learning and performing shipbuilding, smithing, spearfighting, and swordfighting."

This example is hugely oversimplified, in that there are far more differences going on between humans and chimpanzees than just larger cerebrums and cerebellums.  Likewise, learning to build ships involves deliberate practice which involves maintaining motivation over long chains of visualization, and many other cognitive subproblems.  Focusing on just two factors of 'deep' cognitive labor and just two mechanisms of 'deep' cognitive performance is meant more as a straw illustration of what the much more complicated real story would look like.

But in general, the hypothesis of general intelligence seems like it should cash out as some version of:  "There's some set of new cognitive algorithms, plus improvements to existing algorithms, plus bigger brains, plus other resources--we don't know how many things like this there are, but there's some set of things like that--which, when added to previously existing primate and hominid capabilities, created the ability to do better on a broad set of deep cognitive subproblems held in common across a very wide variety of humanly-approachable surface-level problems for learning and manipulating domains.  And that's why humans do better on a huge variety of domains simultaneously, despite evolution having not preprogrammed us with new instinctual knowledge or algorithms for all those domains separately."

## Underestimating cognitive commonalities

The above view suggests a [directional bias of uncorrected intuition](https://arbital.com/p/):  Without an explicit correction, we may tend to intuitively underestimate the similarity of deep cognitive labor across seemingly different surface problems.

On the surface, a ship seems like a different object from a smithy, and the spear seems to involve different tactics from a sword.  With our attention [going to these visible differences](https://arbital.com/p/invisible_constants), we're unlikely to spontaneously invent a concept of 'realtime motor control with error correction' as a kind of activity performed by a 'cerebellum'--especially if our civilization doesn't know any neuroscience.  The deep cognitive labor in common goes unseen, not just because we're not paying attention to the [invisible constants](https://arbital.com/p/invisible_constants) of human intelligence, but because we don't have the theoretical understanding to imagine in any concrete detail what could possibly be going on.

This suggests an [argument from predictable updating](https://arbital.com/p/predictable_update): if we knew even *more* about how general intelligence actually worked inside the human brain, then we would be even *better* able to concretely visualize deep cognitive problems shared between different surface-level domains.  We don't know at present how to build an intelligence that learns a par-human variety of domains, so at least some of the deep commonalities and corresponding similar algorithms across those domains, must be unknown to us.  Then, arguendo, if we better understood the true state of the universe in this regard, our first-order/uncorrected intuitions would predictably move further along the direction that our belief previously moved when we learned about cerebral cortices and cerebellums.  Therefore, [to avoid violating probability theory by foreseeing a predictable update](https://arbital.com/p/predictable_update), our second-order corrected belief should already be that there is more in common between different cognitive tasks than we intuitively see how to compute.

%%comment:

In sum this suggests a [deflationary psychological account](https://arbital.com/p/43h) of a [directional bias of uncorrected intuitions](https://arbital.com/p/) toward general-intelligence skepticism:  People invent theories of distinct intelligences and nonoverlapping specializations, because (a) they are looking toward socially salient human-human differences instead of human-vs-chimpanzee differences, (b) they have failed to correct for the fading of [invisible constants](https://arbital.com/p/) such as human intelligence, and (c) they have failed to apply an explicit correction for the extent to which we feel like we understand surface-level differences but are ignorant of the cognitive commonalities suggested by the general human performance factor.

(The usual cautions about psychologizing apply: you can't actually get empirical data about the real world by arguing about people's psychology.)
%%

# Naturally correlated AI capabilities

Few people in the field would outright disagree with either the statement "humans have significantly more widely applicable cognitive abilities than other primates" or, or the other side, "no matter how intelligent you are, if your brain fits inside the physical universe, you might not be able to reverse SHA-4096 hashes".  But even taking both those statements for granted, there seems to be a set of policy-relevant factual questions about, roughly, to what degree general intelligence is likely to shorten the pragmatic distance between different AI capabilities.

For  example, consider the following (straw) [amazing simple solution to all of AI alignment](https://arbital.com/p/43w):

"Let's just develop an AI that knows how to do [good](https://arbital.com/p/3d9) things but not [bad](https://arbital.com/p/450) things!  That way, even if something goes wrong, it won't know *how* to hurt us!"

To which we reply:  "That's like asking for an AI that understands how to drive blue cars but not red cars.  The cognitive work you need to do in order to drive a blue car is very similar to the cognitive labor required to drive a red car; an agent that can drive a blue car is only a tiny step away from driving a red car.  In fact, you'd pretty much have to add design features specifically intended to prevent the agent from understanding how to drive a car if it's painted red, and if something goes wrong with those features, you'll have a red-car-driving-capable agent on your hands."

"I don't believe in this so-called general-car-driving-intelligence," comes the reply.  "I see no reason why ability at driving blue cars has to be so strongly correlated with driving red cars; they look pretty different to me.  Even if there's a kind of agent that's good at driving both blue cars and red cars, it'd probably be pretty inefficient compared to a specialized blue-car-driving or red-car-driving intelligence.  Anyone who was constructing a car-driving algorithm that only needed to work with blue cars, would not naturally tend to produce an algorithm that also worked on red cars."

"Well," we say, "maybe blue cars and red cars *look* different.  But if you did have a more concrete and correct idea about what goes on inside a robotic car, and what sort of computations it does, you'd see that the computational subproblems of driving a blue car are pretty much identical to the computational subproblems of driving a red car."

"But they're not actually identical," comes the reply.  "The set of red cars isn't actually identical to the set of blue cars and you won't actually encounter exactly identical problems in driving these non-overlapping sets of physical cars going to different places."

"Okay," we reply, "that's admittedly true.  But in order to reliably drive *any* blue car you might get handed, you need to be able to solve an abstract volume of [not-precisely-known-in-advance](https://arbital.com/p/5d) cognitive subproblems.  You need to be able to drive on the road regardless of the exact arrangement of the asphalt.  And that's the same range of subproblems required to drive a red car."

We are, in this case, talking to someone who doesn't believe in *color-general car-driving intelligence* or that color-general car-driving is a good or natural way to solve car-driving problems.  In this particular case it's an obvious straw position because we've picked two tasks that are extremely similar in an intuitively obvious way; a human trained to drive blue cars does not need any separate practice at all to drive red cars.

For a straw position at the opposite extreme, consider:  "I just don't believe you can solve [logical Tic-Tac-Toe](https://arbital.com/p/9s) without some deep algorithm that's general enough to do anything a human can.  There's no safe way to get an AI that can play Tic-Tac-Toe without doing things dangerous enough to require solving [all of AI alignment](https://arbital.com/p/41k).  Beware the cognitive biases that lead you to underestimate how much deep cognitive labor is held in common between tasks that merely appear different on the surface!"

To which we reply, "Contrary to some serious predictions, it turned out to be possible to play superhuman Go without general AI, never mind Tic-Tac-Toe.  Sometimes there really are specialized ways of doing things, the end."

Between these two extremes lie more plausible positions that have been seriously held and debated, including:

- The problem of *making good predictions* requires a significantly smaller subset of the abilities and strategies used by a general agent; an [Oracle](https://arbital.com/p/6x) won't be easy to immediately convert to an agent.
- An AI that only generates plans for humans to implement, solves less dangerous problems than a general agent, and is not an immediate neighbor of a very dangerous general agent.
- If we only try to make superhuman AIs meant to assist but not replace humans, AIs designed to operate only with humans in the loop, the same technology will not immediately extend to building autonomous superintelligences.
- It's possible to have an AI that is, at a given moment, a superhumanly good engineer [but not very good at modeling human psychology](https://arbital.com/p/102); an AI with domain knowledge of material engineering does not have to be already in immediate possession of all the key knowledge for human psychology.

Arguably, these factual questions have in common that they revolve about [the distance between different cognitive domains](https://arbital.com/p/7vk)--given a natural design for an agent that can do X, how close is it in design space to an agent that can do Y?  Is it 'driving blue cars vs. driving red cars' or 'Tic-Tac-Toe vs. classifying pictures of cats'?

(Related questions arise in any safety-related proposal to [divide an AI's internal competencies into internal domains](https://arbital.com/p/domaining), e.g. for purposes of [minimizing](https://arbital.com/p/7tf) the number of [internal goals with the power to recruit subgoals across any known domain](https://arbital.com/p/major_goals).)

It seems like in practice, different beliefs about 'general intelligence' may account for a lot of the disagreement about "Can we have an AI that X-es without that AI being 30 seconds away from being capable of Y-ing?"  In particular, different beliefs about:

- To what degree most interesting/relevant domain problems, decompose well into a similar class of deep cognitive subproblems;
- To what degree whacking on an interesting/relevant problem with general intelligence is a good or natural way to solve it, compared to developing specialized algorithms (that can't just be developed *by* a general intelligence (without that AGI paying pragmatically very-difficult-to-pay costs in computation or sample complexity)).

To the extent that you assign general intelligence a more central role, you may tend *in general* to think that competence in domain X is likely to be nearer to competence at domain Y.  (Although not to an unlimited degree, e.g. witness Tic-Tac-Toe or reversing a SHA-4096 hash.)

# Relation to capability gain theses

How much credit one gives to 'general intelligence' is not the same question as how much credit one gives to issues of [rapid capability gains](https://arbital.com/p/capability_gain), [superintelligence](https://arbital.com/p/41l), and the possible intermediate event of an [intelligence explosion](https://arbital.com/p/428).  The ideas can definitely be pried apart conceptually:

- An AI might be far more capable than humans in virtue of running orders of magnitude faster, and being able to expand across multiple clusters sharing information with much higher bandwidth than human speech, rather than the AI's general intelligence being algorithmically superior to human general intelligence in a deep sense %note: E.g. in the sense of having lower [sample complexity](https://arbital.com/p/sample_complexity) and hence being able to [derive correct answers using fewer observations](https://arbital.com/p/observational_efficiency) than humans trying to do the same over relatively short periods of time.% *or* an intelligence explosion of algorithmic self-improvement having occurred.
- If it's *cheaper* for an AI with high levels of specialized programming ability to acquire other new specialized capabilities than for a human to do the same--not because of any deep algorithm of general intelligence, but because e.g. human brains can't evolve new cortical areas over the relevant timespan--then this could lead to an explosion of other cognitive abilities rising to superhuman levels, without it being in general true that there were deep similar subproblems being solved by similar deep algorithms.

In practice, it seems to be an observed fact that people who give *more* credit to the notion of general intelligence expect *higher* returns on cognitive reinvestment, and vice versa.  This correlation makes sense, since:

- The more different surface domains share underlying subproblems, the higher the returns on cognitive investment in getting better at those deep subproblems.
- The more you think an AI can improve its internal algorithms in faster or deeper ways than human neurons updating, the more this capability is *itself* a kind of General Ability that would lead to acquiring many other specialized capabilities faster than human brains would acquire them.  %note: It seems conceptually possible to believe, though this belief has not been observed in the wild, that self-programming minds have something worthy of being called 'general intelligence' but that human brains don't.%

It also seems to make sense for people who give more credit to general intelligence, being more concerned about capability-gain-related problems in general; they are more likely to think that an AI with high levels of one ability is likely to be able to acquire another ability relatively quickly (or immediately) and without specific programmer efforts to make that happen.

A bee is born with the ability to construct hives.  A beaver is born with an instinct for building dams.  A human looks at both and imagines a large dam with a honeycomb structure.  Arguendo, some set of factors, present in human brains but not all present in chimpanzee brains, seem to sum to a central cognitive capability that lets humans learn a huge variety of different domains without those domains being specifically preprogrammed as instincts.

This very-widely-applicable cognitive capacity is termed **general intelligence** (by most AI researchers explicitly talking about it; the term isn't universally accepted as yet).

We are not perfectly general - we have an easier time learning to walk than learning to do abstract calculus, even though the latter is much easier in an objective sense. But we're sufficiently general that we can figure out Special Relativity and engineer skyscrapers despite our not having those abilities built-in at compile time by natural selection. An [Artificial General Intelligence](https://arbital.com/p/42g) would have the same property; it could learn a tremendous variety of domains, including domains it had no inkling of when it was first switched on.


Distances between cognitive domains

In the context of [AI alignment](https://arbital.com/p/2v), we may care a lot about the degree to which competence in two different [cognitive domains](https://arbital.com/p/7vf) is separable, or alternatively highly tangled, relative to the class of algorithms reasoning about them.

- Calling X and Y 'separate domains' is asserting at least one of "It's possible to learn to reason well about X without needing to know about Y" or "It's possible to learn to reason well about Y without necessarily knowing how to reason well about X".
- Calling X a distinct domain within a set of domains Z relative to a background domain W would say that: taking for granted other background algorithms and knowledge W that the agent can use to reason about any domain in Z; it's possible to reason well about the domain X using ideas, methods, and knowledge that are mostly related to each other and not tangled up with ideas from non-X domains within Z.

For example:  If the domains X and Y are 'blue cars' and 'red cars', then it seems unlikely that X and Y would be well-separated domains because an agent that knows how to reason well about blue cars is almost surely *extremely* close to being an agent that can reason well about red cars, in the sense that:

- For *almost* everything we want to do or predict about blue cars, the simplest or fastest or easiest-to-discover way of manipulating or predicting blue cars in this way, will also work for manipulating or predicting red cars.  This is the sense in which the blue-car and red-car domains are 'naturally' very close.
- For most natural agent designs, the state or specification of an agent that can reason about blue cars, is probably extremely close to the state or specification of an agent that can reason about red cars.
  - The only reason why an agent that reasons well about blue cars would be hard to convert to an agent that reasons well about red cars, would be if there were specific extra elements added to the agent's design to prevent it from reasoning well about red cars.  In this case, the design distance is increased by whatever further modifications are required to untangle and delete the anti-red-car-learning inhibitions; but no further than that, the 'blue car' and 'red car' domains are naturally close.
- An agent that has already learned how to reason well about blue cars probably requires only a tiny amount of extra knowledge or learning, if any, to reason well about red cars as well.  (Again, unless the agent contains specific added design elements to make it reason poorly about red cars.)

In more complicated cases, which domains are truly close or far from each other, or can be compactly separated out, is a theory-laden assertion.  Few people are likely to disagree that blue cars and red cars are very close domains (if they're not specifically trying to be disagreeable).  Researchers are more likely to disagree in their predictions about:

- Whether (by default and ceteris paribus and assuming designs not containing extra elements to make them behave differently etcetera) an AI that is good at designing cars is also likely to be very close to learning how to design airplanes.
- Whether (assuming straightforward designs) the first AGI to obtain [superhuman](https://arbital.com/p/7mt) engineering ability for designing cars including software, would probably be at least [par-human](https://arbital.com/p/7mt) in the domain of inventing new mathematical proofs.
- Whether (assuming straightforward designs) an AGI that has superhuman engineering ability for designing cars including software, necessarily needs to think about most of the facts and ideas that would be required to understand and manipulate human psychology.

## Relation to 'general intelligence'

A key parameter in some such disagreements may be how much credit the speaker gives to the notion of [https://arbital.com/p/-7vh](https://arbital.com/p/-7vh).  Specifically, to what extent the natural or the most straightforward approach to get par-human or superhuman performance in critical domains, is to take relatively general learning algorithms and deploy them on learning the domain as a special case.

If you think that it would take a weird or twisted design to build a mind that was superhumanly good at designing cars including writing their software, *without* using general algorithms and methods that could with minor or little adaptation stare at mathematical proof problems and figure them out, then you think 'design cars' and 'prove theorems' and many other domains are in some sense *naturally* not all that separated.  Which (arguendo) is why humans are so much better than chimpanzees at so many apparently different cognitive domains: the same competency, general intelligence, solves all of them.

If on the other hand you are more inspired by the way that superhuman chess AIs can't play Go and AlphaGo can't drive a car, you may think that humans using general intelligence on everything is just an instance of us having a single hammer and trying to treat everything as a nail; and predict that specialized mind designs that were superhuman engineers, but very far in mind design space from being a kind of mind that could prove Fermat's Last Theorem, would be a more natural or efficient way to create a superhuman engineer.

See the entry on [https://arbital.com/p/7vh](https://arbital.com/p/7vh) for further discussion.


'Concept'

In the context of Artificial Intelligence or machine learning, a 'concept' is something that identifies thingies as being inside or outside of the boundary.  For example, a neural network has learned the 'picture of a cat' concept if it can reliably distinguish pictures of cats from pictures of noncats.  That is:  A concept is a membership predicate.