# Intelligence Amplification and Friendly AI

Part of the series [AI Risk and Opportunity: A Strategic Analaysis](/lw/ajm/ai_risk_and_opportunity_a_strategic_analysis/). Previous articles on this topic: [Some Thoughts on Singularity Strategies](/lw/6mi/some_thoughts_on_singularity_strategies/), [Intelligence enhancement as existential risk mitigation](/lw/10l/intelligence_enhancement_as_existential_risk/), [Outline of possible Singularity scenarios that are not completely disastrous](/lw/6j9/outline_of_possible_singularity_scenarios_that/).

Below are my quickly-sketched thoughts on intelligence amplification and FAI, without much effort put into organization or clarity, and without many references.[\[1\]](#fn:1 "see footnote") But first, I briefly review some strategies for increasing the odds of FAI, one of which is to work on intelligence amplification (IA).

###   

### Some possible “best current options” for increasing the odds of FAI

Suppose you find yourself in a pre-AGI world,[\[2\]](#fn:2 "see footnote") and you’ve been convinced that the status quo world is unstable, and within the next couple centuries we’ll likely[\[3\]](#fn:3 "see footnote") settle into one of [four stable outcomes](http://intelligenceexplosion.com/2012/ai-the-problem-with-solutions/): FAI, uFAI, non-AI extinction, or a sufficiently powerful global government which can prevent AGI development[\[4\]](#fn:4 "see footnote"). And you _totally_ prefer the FAI option. What should you do to get there?

*   Obvious direct approach: start solving the technical problems that must be solved to get FAI: goal stability under self-modification, decision algorithms that handle counterfactuals and logical uncertainty properly, indirect normativity, and so on. ([MIRI’s work](http://intelligence.org/research/), some FHI work.)
*   Do strategy research, to potentially identify superior alternatives to the other items on this list, or superior versions of the things on this list already. ([FHI’s work](http://www.fhi.ox.ac.uk/), some MIRI work, etc.)
*   Accelerate IA technologies, so that smarter humans can tackle FAI. (E.g. [cognitive genomics](http://intelligence.org/2013/08/31/stephen-hsu-on-cognitive-genomics/).)
*   Try to make sure we get high-fidelity [WBEs](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf) before AGI, without WBE work first enabling dangerous neuromorphic AGI. ([Dalyrmple’s work?](http://nemaload.davidad.org/))
*   Improve political and scientific institutions so that the world is [more likely to handle AGI wisely](http://intelligence.org/2013/09/12/how-well-will-policy-makers-handle-agi-initial-findings/) when it comes. ([Prediction markets?](http://en.wikipedia.org/wiki/Prediction_market) [Vannevar Group?](http://www.vannevargroup.org/))
*   Capacity-building. Grow the rationality community, the x-risk reduction community, the effective altruism movement, etc.
*   Other stuff. (More in later posts).

###   

### The IA route

Below are some key considerations about the IA route. I’ve numbered them so they’re easy to refer to later. My discussion assumes [MIRI’s basic assumptions](http://intelligence.org/2013/05/05/five-theses-two-lemmas-and-a-couple-of-strategic-implications/), including timelines similar to [my own AGI timelines](http://intelligence.org/2013/05/15/when-will-ai-be-created/).

1.  Maybe FAI is so hard that we can _only_ get FAI with a large team of IQ 200+ humans, whereas uFAI can be built by a field of IQ 130–170 humans with a few more decades and lots of computing power and trial and error. So to have any chance of FAI at all, we’ve got to do WBE or IA first.
2.  You could accelerate FAI relative to AGI if you somehow kept IA technology secret, for use only by FAI researchers (and maybe their supporters).
3.  Powerful IA technologies would likely get wide adoption, and accelerate economic growth and scientific progress in general. If you think [Earths with slower economic growth have a better chance at FAI](/lw/hoz/do_earths_with_slower_economic_growth_have_a/), that could be bad for our FAI chances. If you think the opposite, then broad acceleration from IA could be good for FAI.
4.  Maybe IA increases one’s “rationality” and “philosophical ability” (in scare quotes because we mostly don’t know how to measure them yet), and thus IA increases the frequency with which people will realize the risks of AGI and do sane things about it.
5.  Maybe IA increases the role of intelligence and designer understanding, relative to hardware and accumulated knowledge, in AI development.[\[5\]](#fn:5 "see footnote")

Below are my thoughts about all this. These are only _my current views_: other MIRI personnel (including Eliezer) disagree with some of the points below, and I wouldn’t be surprised to change my mind about some of these things after extended discussion (hopefully in public, on Less Wrong).

I doubt (1) is true. I think IQ 130–170 humans could figure out FAI in 50–150 years if they were trying to solve the right problems, and if FAI development wasn’t in a death race with the strictly easier problem of uFAI. If normal smart humans _aren’t_ capable of building FAI in that timeframe, that’s probably for lack of rationality and philosophical skill, not for lack of IQ. And I’m not confident that rationality and philosophical skill predictably improve with IQ after about IQ 140. It’s a good sign that [atheism increases with IQ after IQ 140](http://www.karolek.w.tkb.pl/nature98.pdf), but on the other hand I know too many high-IQ people who think that (e.g.) an AI that maximizes K-complexity is a win, and also there’s [Stanovich’s research](http://keithstanovich.com/Site/Research_on_Reasoning_files/Stanovich_CDPS_2013.pdf) on how IQ and rationality come apart. For these reasons, I’m also not convinced (4) would be a large positive effect on our FAI chances.

Can we train people in rationality and philosophical skill beyond that of say, the 95th percentile Less Wronger? [CFAR](http://rationality.org/) has plans to find out, but they need to grow a lot first to execute such an ambitious research program.

(2) looks awfully hard, unless we can find a powerful IA technique that also, say, gives you a 10% chance of cancer. Then some EAs devoted to building FAI might just use the technique, and maybe the AI community in general doesn’t.

(5) seems right, though I doubt it’ll be a big enough effect to make a difference for the final outcome.

I think (3) is the dominant consideration here, along with the worry about lacking the philosophical skill (but not IQ) to build FAI at all. At the moment, I (sadly) lean toward the view that slower Earths have a better chance at FAI. (Much of my brain doesn’t know this, though: I remember reading the [Summers news](http://www.themoneyillusion.com/?p=23653) with glee, and then remembering that on my current model this was actually bad news for FAI.)

I could say more, but I’ll stop for now and see what comes up in discussion.

* * *

1.  My thanks to Justin Shovelain for sending me his [old notes](/lw/6mi/some_thoughts_on_singularity_strategies/4ikh) on the “IA first” case, and to Wei Dai, Carl Shulman, and Eliezer Yudkowsky for their feedback on this post. [ ↩](#fnref:1 "return to article")
    
2.  Not counting civilizations that might be simulating our world. This matters, but I won’t analyze that here. [ ↩](#fnref:2 "return to article")
    
3.  There are other possibilities. For example, there could be a global nuclear war that kills all but about 100,000 people, which could set back social, economic, and technological progress by centuries, thus delaying the crucial point in Earth’s history in which it settles into one of the four stable outcomes. [ ↩](#fnref:3 "return to article")
    
4.  And perhaps also advanced nanotechnology, intelligence amplification technologies, and whole brain emulation. [ ↩](#fnref:4 "return to article")
    
5.  Thanks to Carl Shulman for making this point. [ ↩](#fnref:5 "return to article")