# AI Risk and Opportunity: A Strategic Analysis

![](http://lukeprog.com/images/AI%20risk%20and%20opportunity.png)

Suppose you buy [the argument](http://facingthesingularity.com/) that humanity faces both the _risk_ of AI-caused extinction and the _opportunity_ to shape an AI-built utopia. What should we do about that? As Wei Dai [asks](/lw/6mi/some_thoughts_on_singularity_strategies/), "In what direction should we nudge the future, to maximize the chances and impact of a positive intelligence explosion?"

This post serves as a table of contents and an introduction for an ongoing strategic analysis of AI risk and opportunity.

**Contents:**

1.  Introduction (this post)
2.  [Humanity's Efforts So Far](/r/discussion/lw/b0v/ai_risk_and_opportunity_humanitys_efforts_so_far/)
3.  [A Timeline of Early Ideas and Arguments](/r/discussion/lw/bd6/ai_risk_opportunity_a_timeline_of_early_ideas_and/)
4.  [Questions We Want Answered](/r/discussion/lw/bdt/ai_risk_opportunity_questions_we_want_answered/)
5.  [Strategic Analysis Via Probability Tree](/r/discussion/lw/bjl/ai_risk_opportunity_strategic_analysis_via/)
6.  [Intelligence Amplification and Friendly AI](/r/discussion/lw/iqi/intelligence_amplification_and_friendly_ai/)
7.  ...

###   

### Why discuss AI safety strategy?

The main reason to discuss AI safety strategy is, of course, to draw on a wide spectrum of human expertise and processing power to clarify our understanding of the factors at play and the expected value of particular interventions we could invest in: raising awareness of safety concerns, forming a Friendly AI team, differential technological development, investigating AGI confinement methods, and others.

Discussing AI safety strategy is also a challenging exercise in _applied rationality_. The relevant issues are complex and uncertain, but we need to take advantage of the fact that [rationality is faster than science](/lw/qi/faster_than_science/): we can't "try" a bunch of intelligence explosions and see which one works best. We'll have to predict in advance how the future will develop and what we can do about it.

###   

### Core readings

Before engaging with this series, I recommend you read _at least_ the following articles:

*   Muehlhauser & Salamon, [Intelligence Explosion: Evidence and Import](http://intelligence.org/files/IE-EI.pdf) (2013)
*   Yudkowsky, [AI as a Positive and Negative Factor in Global Risk](http://intelligence.org/files/AIPosNegFactor.pdf) (2008)
*   Chalmers, [The Singularity: A Philosophical Analysis](http://consc.net/papers/singularityjcs.pdf) (2010)

###   

### Example questions

Which strategic questions would we like to answer? [Muehlhauser (2011)](http://lukeprog.com/SaveTheWorld.html) elaborates on the following questions:

*   What methods can we use to predict technological development?
*   Which kinds of differential technological development should we encourage, and how?
*   Which open problems are safe to discuss, and which are potentially dangerous?
*   What can we do to reduce the risk of an AI arms race?
*   What can we do to raise the "sanity waterline," and how much will this help?
*   What can we do to attract more funding, support, and research to x-risk reduction and to specific sub-problems of successful Singularity navigation?
*   Which interventions should we prioritize?
*   How should x-risk reducers and AI safety researchers interact with governments and corporations?
*   How can optimal philanthropists get the most x-risk reduction for their philanthropic buck?
*   How does AI risk compare to other existential risks?
*   Which problems do we need to solve, and which ones can we have an AI solve?
*   How can we develop microeconomic models of WBEs and self-improving systems?
*   How can we be sure a Friendly AI development team will be altruistic?

[Salamon & Muehlhauser (2013)](http://intelligence.org/files/SS11Workshop.pdf) list several other questions gathered from the participants of a workshop following Singularity Summit 2011, including:

*   How hard is it to create Friendly AI?
*   What is the strength of feedback from neuroscience to AI rather than brain emulation?
*   Is there a safe way to do uploads, where they don't turn into neuromorphic AI?
*   How possible is it to do FAI research on a seastead?
*   How much must we spend on security when developing a Friendly AI team?
*   What's the best way to recruit talent toward working on AI risks?
*   How difficult is stabilizing the world so we can work on Friendly AI slowly?
*   How hard will a takeoff be?
*   What is the value of strategy vs. object-level progress toward a positive Singularity?
*   How feasible is Oracle AI?
*   Can we convert environmentalists into people concerned with existential risk?
*   Is there no such thing as bad publicity \[for AI risk reduction\] purposes?

These are the kinds of questions we will be tackling in this series of posts for [Less Wrong Discussion](/r/discussion/new/), in order to improve our predictions about which direction we can nudge the future to maximize the chances of a positive intelligence explosion.