# An onion strategy for AGI discussion

Cross-posted from [my blog](http://lukemuehlhauser.com/an-onion-strategy-for-agi-discussion/).

"[The stabilization of environments](http://lukemuehlhauser.com/wp-content/uploads/Hammond-et-al-The-stabilization-of-environments.pdf)" is a paper about AIs that reshape their environments to make it easier to achieve their goals. This is typically called _enforcement_, but they prefer the term _stabilization_ because it "sounds less hostile."

"I'll open the pod bay doors, Dave, but then I'm going to stabilize the ship..."

[Sparrow (2013)](http://commonsenseatheism.com/wp-content/uploads/2013/08/Sparrow-In-vitro-eugenics.pdf) takes the opposite approach to plain vs. dramatic language. Rather than using a modest term like _[iterated embryo selection](http://theuncertainfuture.com/faq.html#7)_, Sparrow prefers the phrase _in vitro eugenics_. Jeepers.

I suppose that's more likely to provoke public discussion, but... will much good will come of that public discussion? The public had a needless freak-out about in vitro fertilization back in the 60s and 70s and then, as soon as the first IVF baby was born in 1978, [decided they were in favor of it](http://intelligence.org/files/EmbryoSelection.pdf).

Someone recently suggested I use an "**onion strategy**" for the discussion of novel technological risks. The outermost layer of the communication onion would be aimed at the general public, and focus on benefits rather than risks, so as not to provoke an unproductive panic. A second layer for a specialist audience could include a more detailed elaboration of the risks. The most complete discussion of risks and mitigation options would be reserved for technical publications that are read only by professionals.

Eric Drexler [seems](http://lukemuehlhauser.com/wp-content/uploads/Giles-Nanotech-takes-small-steps-toward-burying-grey-goo.pdf) to wish he had more successfully used an onion strategy when writing about nanotechnology. [_Engines of Creation_](http://smile.amazon.com/Engines-Creation-Nanotechnology-Library-Science/dp/0385199732/) included frank discussions of both the benefits and risks of nanotechnology, including the "[grey goo](http://en.wikipedia.org/wiki/Grey_goo)" scenario that was discussed widely in the media and used as the premise for the bestselling novel _[Prey](http://en.wikipedia.org/wiki/Prey_(novel))._

Ray Kurzweil may be using an onion strategy, or at least keeping his writing in the outermost layer. If you look carefully, chapter 8 of _[The Singularity is Near](http://smile.amazon.com/Singularity-Near-Humans-Transcend-Biology-ebook/dp/B000QCSA7C/)_ takes technological risks pretty seriously, and yet it's written in such a way that most people who read the book seem to come away with an overwhelmingly optimistic perspective on technological change.

George Church may be following an onion strategy. _[Regenesis](http://smile.amazon.com/Regenesis-Synthetic-Biology-Reinvent-Ourselves-ebook/dp/B00HTQ326W/)_ also contains a chapter on the risks of advanced bioengineering, but it's presented as an "epilogue" that many readers will skip.

Perhaps those of us writing about AGI for the general public should try to discuss:

*   _astronomical stakes_ rather than _existential risk_
*   _Friendly AI_ rather than _AGI risk_ or the _superintelligence control problem_
*   the orthogonality _thesis_ and _convergent instrumental values_ and _complexity of values_ rather than "doom by default"
*   etc.

MIRI doesn't have any official recommendations on the matter, but these days I find myself leaning toward an onion strategy.