# New article from Oren Etzioni

(Cross-posted from [EA Forum](https://forum.effectivealtruism.org/posts/7x2BokkkemjnXD9B6/new-article-from-oren-etzioni).)

This just appeared in this week’s MIT Technology Review: Oren Etzioni, “[How to know if AI is about to destroy civilization](https://www.technologyreview.com/s/615264/artificial-intelligence-destroy-civilization-canaries-robot-overlords-take-over-world-ai/).” Etzioni is a noted skeptic of AI risk. Here are some things I jotted down:

Etzioni’s key points / arguments:

*   Warning signs that AGI is coming soon (like canaries in a coal mine, where if they start dying we should get worried)

*   Automatic formulation of learning problems
*   Fully self-driving cars
*   AI doctors
*   Limited versions of the Turing test (like Winograd Schemas)

*   If we get to the Turing test itself then it'll be too late

*   \[Note: I think if we get to practically deployed fully self-driving cars and AI doctors, then we will have already had to solve more limited versions of AI safety. It’s a separate debate whether those solutions would scale up to AGI safety though. We might also get the capabilities without actually being able to deploy them due to safety concerns.\]

*   We are decades away from the versatile abilities of a 5 year old
*   Preparing anyway even if it's very low probability because of extreme consequences is Pascal's Wager

*   \[Note: This is a decision theory question, and I don’t think that's his area of expertise. I’ve researched PW extensively, and it’s not at all clear to me where to draw the line between low probability - high consequence scenarios that we should be factoring into our decisions, vs. very low probability – very high consequence that we should not factor into our decisions. I’m not sure there is any principled way of drawing a line between those, which might be a problem if it turns out that AI risk is a borderline case.\]

*   If and when a canary "collapses" we will have ample time to design off switches and identify red lines we don't want AI to cross
*   "AI eschatology without empirical canaries is a distraction from addressing existing issues like how to regulate AI’s impact on employment or ensure that its use in criminal sentencing or credit scoring doesn’t discriminate against certain groups."
*   Agrees with Andrew Ng that it's too far off to worry about now

But he seems to agree with the following:

*   If we don’t end up doing anything about it then yes, superintelligence would be incredibly dangerous
*   If we get to human level AI then superintelligence will be very soon afterwards so it'll be too late at that point
*   If it were a lot sooner (as other experts expect) then it sounds like he would agree with the alarmists
*   Even if it was more than a tiny probability then again it sounds like he'd agree because he wouldn't consider it Pascal's Wager
*   If there's not ample time between "canaries collapsing" and AGI (as I think other experts expect) then we should be worried a lot sooner
*   If it wouldn't distract from other issues like regulating AI's impact on employment, it sounds like he might agree that it's reasonable to put some effort into it (although this point is a little less clear)

See also Eliezer Yudkowsky, “[There's no fire alarm for Artificial General Intelligence](https://www.lesswrong.com/There's%20no%20fire%20alarm%20for%20Artificial%20General%20Intelligence)”