# How easily can we separate a friendly AI in design space from one which would bring about a hyperexistential catastrophe?

(I've written about this in my [Shortform](https://www.lesswrong.com/posts/LDPcAEhn35uoC2Hz3/shortform & may regurgitate some stuff) and may regurgitate some stuff from there.)

  

Eliezer proposes that we [separate an AI in design space from one that would constitute a fate worse than death](https://arbital.com/p/hyperexistential_separation/) if e.g. the reward model's sign (+/-) were flipped or the direction of updates to the reward model were reversed. This seems absolutely crucial, although I'm not yet aware of any robust way of doing this. Eliezer proposes assigning the AI a utility function of:

> _U_ = _V_ \+ _W_

Where _V_ refers to human values & _W_ takes a very negative value for some arbitrary variable (e.g. diamond paperclips of length 5cm). So if the AI instead maximises -_U_, it'd realise that it can gain more utility by just tiling the universe with garbage.

But it seems entirely plausible that the error could occur with _V_ instead of _U_, resulting in the AI maximising _U_ = _W_ \- _V_, which would result in torture.

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

Another proposition I found briefly described in a [Facebook discussion that was linked to by somewhere](https://www.facebook.com/yudkowsky/posts/10155975880959228 ). Stuart Armstrong proposes the following:

> Let B1 and B2 be excellent, bestest outcomes. Define _U_(B1) = 1, _U_(B2) = -1, and _U_ = 0 otherwise. Then, under certain assumptions about what probabilistic combinations of worlds it is possible to create, maximising or minimising _U_ leads to good outcomes.

> Or, more usefully, let _X_ be some trivial feature that the agent can easily set to -1 or 1, and let _U_ be a utility function with values in \[0, 1\]. Have the AI maximise or minimise _XU_. Then the AI will always aim for the same best world, just with a different _X_ value.

Later, he suggests that _X_ should be a historical fact (i.e. the value of _X_ would be set in stone 10 seconds after the system is turned on.) As _XU_ can only take positive values (because _U_ has values in \[0, 1\]), the greatest value -_XU_ could take would be 0 (which suggests merely killing everyone.)

But this could still be problematic if e.g. the bug occurred in the reward function/model such that it gave positive values for bad things and negative values for bad things. Although I'm not sure how frequently errors effectively multiply everything in the reward by -1. I'm also unsure how this would interact with an error that reverses the direction of updates to a reward _model_.

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

A few possible (?) causes for this type of error include: (list obviously not exhaustive)

*   Bug caused by AI's developers. For example, [GPT-2 experienced an interesting bug](https://openai.com/blog/fine-tuning-gpt-2/):

> **Bugs can optimise for bad behavior**

> One of our code refactors introduced a bug which flipped the sign of the reward. Flipping the reward would usually produce incoherent text, but the same bug also flipped the sign of the KL penalty. The result was a model which optimized for negative sentiment while preserving natural language. Since our instructions told humans to give very low ratings to continuations with sexually explicit text, the model quickly learned to output only content of this form.

The responses to [this](https://www.lesswrong.com/posts/WMhiJf3xx9ZopC2tP/likelihood-of-hyperexistential-catastrophe-from-a-bug) thread suggest that this type of thing would be noticed and redressed immediately, although this view [doesn't appear to be held unanimously](https://www.lesswrong.com/posts/9nRfQBCGDdfMTmxgR/open-and-welcome-thread-august-2020-1?commentId=4SreftCqGKWyQYmnG). See also Gwern's [follow-up comment](https://www.lesswrong.com/posts/9nRfQBCGDdfMTmxgR/open-and-welcome-thread-august-2020-1?commentId=TENcarNEgJzgXLgoB).

*   Errors in self-modification/improvement
*   [Programmers making a mistake in a relevant database](https://www.lesswrong.com/posts/9nRfQBCGDdfMTmxgR/open-and-welcome-thread-august-2020-1?commentId=9GRTobpRt9XS63JgQ#4SreftCqGKWyQYmnG )
*   Some sort of weird vulnerability getting exploited by [daemons](https://arbital.com/p/daemons/) (unsure on this one)

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

So, yeah. Are there any mechanisms to prevent this sort of thing from happening other than the two that I listed; and if not, would you expect the examples provided to robustly prevent this type of error from happening _regardless of the cause_?