Harley-ml/lesswrong huggingface dataset:
# An anecdote about names

I've always been bad at names. But this semester, as part of my duties as a physics teacher, I tried to learn the names of 100 students. It went alright - two months later and there are still some I have trouble with. At the start, I was a bit worried. What if I've just read to many books and things, and _used up_ most of my abnormally small allotment of memorized names? Then wasting names on students would be a really bad idea.

But things turned out not to work that way (as might be expected from how big the human brain is, viz people with eidetic memories). I've started remembering peoples' names after just one introduction, sometimes two. And the reasonable culprit just seems to be practice. You practice learning names, you get better at doing so. Before, I made occasional conversational detours if I couldn't remember someone's name. Now, I ask them again, because I'm confident that I'll remember it without tons of further awkwardness. If I'm really having trouble, sometimes I've written down a name with a short description, and that usually cements it. Remembering names isn't the most important social skill I've learned, but it's a surprisingly dramatic one - it makes me feel closer to people, and vice versa. And it only took having to learn the names of 100 people to get started.

The lesson from this is not necessarily just about learning names, though that might be useful to you. The lesson is about how ordinary people get a lot of practice at doing things like remembering names - if you can't do something that someone else can do, practice may be an _effective_ place to start.  What seems like a property of yourself ("bad at names") may turn out to be quite mutable with the kind of practice those other people are doing.
# Checking Kurzweil's track record 

Predictions are cheap and easy; verification is hard, essential, and rare. For things like AI, we seem to be restricted to nothing but expert predictions - but expert predictions on AI are [not very good](http://fora.tv/2012/10/14/Stuart_Armstrong_How_Were_Predicting_AI), either in [theory](/lw/e46/competence_in_experts_summary/) or in [practice](/lw/e36/ai_timeline_predictions_are_we_getting_better/). If we are some experts who stand out, we would really want to identify them - and there is nothing better than a track record for identifying true experts.

So we're asking for help to verify the predictions of one of the most prominent futurists of this century: Ray Kurzweil, from his book "The Age of Spiritual Machines". By examining his predictions for times that have already come and gone, we'll be able to more appropriately weight his predictions for times still to come. By taking part, by lending your time to this, you will be directly helping us understand and predict the future, and will get showered in gratitude and kudos and maybe even karma.

I've already made [an attempt](/lw/diz/kurzweils_predictions_good_accuracy_poor/) at this (if you are interested in taking part in this project, avoid clicking on that link for now!). But you cannot trust a single person's opinions, and that was from a small (albeit random) sample of the predictions. For this project, I've transcribed his predictions into 172 separate (short) statements, and any volunteers would be presented with a random selection among these. The volunteers would then do some Google research (or other) to establish whether the prediction had come to pass, and then indicate their verdict. More details on what exactly will be measured, and how to interpret ambiguous statements, will be given to the volunteers once the project starts.

If you are interested, please let me know at [stuart.armstrong@philosophy.ox.ac.uk](mailto:stuart.armstrong@philosophy.ox.ac.uk) (or in the comment thread here), indicating how many of the 172 questions you would like to attempt. The exercise will probably happen in late November or early December.

This will be done unblinded, because Kurzweil's predictions are so well known that it would be infeasible to find large numbers of people who are technologically aware but ignorant of them. Please avoid sharing your verdicts with others; it is entirely your own individual assessment that we are interested in having.
# [POLL] AI-FOOM Debate in Sequence Reruns?

We're now at the point in the sequences when the [AI-FOOM Debate](http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_Debate) took place between Eliezer Yudkowsky and Robin Hanson. Do people want me to include them in the sequence rerun posts, and if so, how? Should I make one post a day? Should I post all of the posts that were made in one day back in 2008, so that we would possibly get one Yudkowsky and one Hanson post in a single day of reruns? If I'm rerunning two posts a day, should I make one rerun post or two? When rerunning a Hanson post, how should the [standard rerun template](/r/discussion/lw/5as/introduction_to_the_sequence_reruns/) be adjusted? 

I titled this as a poll, but that's not really what I want to do, since I'm not sure I have come up with all of the relevant options. I've got my own thoughts on all of the questions I just asked, but I'm going to hold off on mentioning them, in the interests of sparking discussion. Whatever I do, I will need to decide fairly quickly (as in, next three days at the latest).

I'm sorry that this is short term; I've been busy and wasn't really looking that far ahead.

How should I do this?
# Desired articles on AI risk?

I've once again updated my list of [forthcoming and desired articles on AI risk](http://tinyurl.com/ai-risk-research), which currently names 17 forthcoming articles and books about AGI risk, and also names 26 _desired_ articles that I _wish_ researchers were currently writing.

But I'd like to hear your suggestions, too. Which articles not already on the list as "forthcoming" or "desired" would you most like to see written, on the subject of AGI risk?

Book/article titles reproduced below for convenience...

**Forthcoming**

*   _Superintelligence: Groundwork for a Strategic Analysis_ by Nick Bostrom
*   _[Singularity Hypotheses](http://www.amazon.com/Singularity-Hypotheses-Scientific-Philosophical-Assessment/dp/3642325599/)_, edited by Amnon Eden et al.
*   _Singularity Hypotheses, Vol. 2_, edited by Vic Callaghan
*   "General Purpose Intelligence: Arguing the Orthogonality Thesis" by Stuart Armstrong
*   "Responses to AGI Risk" by Kaj Sotala et al.
*   "How we're predicting AI... or failing to" by Stuart Armstrong & Kaj Sotala
*   "A Comparison of Decision Algorithms on Newcomblike Problems" by Alex Altair
*   "A Representation Theorem for Decisions about Causal Models" by Daniel Dewey
*   "Reward Function Integrity in Artificially Intelligent Systems" by Roman Yampolskiy
*   "Bounding the impact of AGI" by Andras Kornai
*   "Minimizing Risks in Developing Artificial General Intelligence" by Ted Goertzel
*   "Limitations and Risks of Machine Ethics" by Miles Brundage
*   "Universal empathy and ethical bias for artificial general intelligence" by Alexey Potapov & Sergey Rodiono
*   "Could we use untrustworthy human brain emulations to make trustworthy ones?" by Carl Shulman
*   "Ethics and Impact of Brain Emulations" by Anders Sandberg
*   "Envisioning The Economy, and Society, of Whole Brain Emulations" by Robin Hanson
*   "Autonomous Technology and the Greater Human Good" by Steve Omohundro

**Desired**

*   "AI Risk Reduction: Key Strategic Questions"
*   "Predicting Machine Superintelligence"
*   "Self-Modification and Löb's Theorem"
*   "Solomonoff Induction and Second-Order Logic"
*   "The Challenge of Preference Extraction"
*   "Value Extrapolation"
*   "Losses in Hardscrabble Hell"
*   "Will Values Converge?"
*   "AI Takeoff Scenarios"
*   "AI Will Be Maleficent by Default"
*   "Biases in AI Research"
*   "Catastrophic Risks and Existential Risks"
*   "Uncertainty and Decision Theories"
*   "Intelligence Explosion: The Proportionality Thesis"
*   "Hazards from Large Scale Computation"
*   "Tool Oracles for Safe AI Development"
*   "Stable Attractors for Technologically Advanced Civilizations"
*   "AI Risk: Private Projects vs. Government Projects"
*   "Why AI researchers will fail to hit the narrow target of desirable AI goal systems"
*   "When will whole brain emulation be possible?"
*   "Is it desirable to accelerate progress toward whole brain emulation?"
*   "Awareness of nanotechnology risks: Lessons for AI risk mitigation"
*   "AI and Physical Effects"
*   "Moore's Law of Mad Science"
*   "What Would AIXI Do With Infinite Computing Power and a Halting Oracle?"
*   "AI Capability vs. AI Safety"
# Less Wrong Parents

Less Wrong Parents  
[https://groups.google.com/forum/?hl=en&fromgroups=#!forum/less-wrong-parents](https://groups.google.com/forum/?hl=en&fromgroups=#!forum/less-wrong-parents)

Recently the NYC LW/OB community had two babies and is expecting a third.  
I created a google group as a way of sharing information, primarily thinking of the NYC community.

I posted my pre-baby purchase list and William Eden posted an extensive list of books on early parenting.

William suggested opening up the group so as to get insight from the larger LW community on parenting.  
I think this is \*probably\* a good idea.  Google groups are simple to set up but have limits.  
For this reason I request that if you are going to have an extensive debate on a subject you create a new thread (aka: get a room)

The primary objective is to lower the cost of obtaining information on parenting.  
I believe this overall goal to be more important then any particular "truth".

My hope is that this will primarily serve as a place for people to ask parenting question and post guides.  
Perhaps if enough guides are posted they can eventually be consolidated into a wiki.
# Sign up to be notified about new LW meetups in your area

LessWrong has rolled out a new feature on the [user preference pages](/prefs/) under "Location":

![](http://louie.divide0.net/lwshot.png)

This is UNCHECKED by default.  Also, we don't know by default where you live.

If you think you might want to meet up with other LWers, please input your location and check this box. Once enough people have signed up in a new area, it will make starting a LW meetup there that much more straightforward for the future organizer.  (You'll just get that email; they won't see your email address.)  In general, it seems like being able to know something about where LWers live would be helpful, so please consider entering your approximate location even if you don't check the box.

Thanks to [Wesley Moore](/user/wmoore/) for deploying this upgrade to the LW backend.
# 2012 Less Wrong Census/Survey

**11/26: The survey is now closed. Please do not take the survey. Your results will not be counted.  
**

It's that time of year again.

If you are reading this post, and have not been sent here by some sort of conspiracy trying to throw off the survey results, then you are the target population for the Less Wrong Census/Survey. Please take it. Doesn't matter if you don't post much. Doesn't matter if you're a lurker. Take the survey.

This year's census contains a "main survey" that should take about ten or fifteen minutes, as well as a bunch of "extra credit questions". You may do the extra credit questions if you want. **You may skip all the extra credit questions if you want.** They're pretty long and not all of them are very interesting. But it is very important that you not put off doing the survey or not do the survey at all because you're intimidated by the extra credit questions.  
  
The survey will probably remain open for a month or so, but once again **do not delay** taking the survey just for the sake of the extra credit questions.

Please make things easier for my computer and by extension me by reading all the instructions and by answering any text questions in the most obvious possible way. For example, if it asks you "What language do you speak?" please answer "English" instead of "I speak English" or "It's English" or "English since I live in Canada" or "English (US)" or anything else. This will help me sort responses quickly and easily. Likewise, if a question asks for a number, please answer with a number such as "4", rather than "four".

Okay! Enough nitpicky rules! Time to take the...

[**2012 Less Wrong Census/Survey**](https://docs.google.com/spreadsheet/viewform?formkey=dG1pTzlrTnJ4eks3aE13Ni1lbV8yUkE6MQ#gid=0)

Thanks to everyone who suggested questions and ideas for the 2012 Less Wrong Census Survey. I regret I was unable to take all of your suggestions into account, because some of them were contradictory, others were vague, and others would have required me to provide two dozen answers and a thesis paper worth of explanatory text for every question anyone might conceivably misunderstand. But I did make about twenty changes based on the feedback, and \*most\* of the suggested questions have found their way into the text.

By ancient tradition, if you take the survey you may comment saying you have done so here, and people will upvote you and you will get karma.
# Analyzing FF.net reviews of 'Harry Potter and the Methods of Rationality'

> The unprecedented gap in _Methods of Rationality_ updates prompts musing about whether readership is increasing enough & what statistics one would use; I write code to download FF.net reviews, clean it, parse it, load into R, summarize the data & depict it graphically, run linear regression on a subset & all reviews, note the poor fit, develop a quadratic fit instead, and use it to predict future review quantities.
> 
> Then, I run a similar analysis on a competing fanfiction to find out when they will have equal total review-counts. A try at logarithmic fits fails; fitting a linear model to the previous 100 days of \_MoR\_ and the competitor works much better, and they predict a convergence in <5 years.

Master version: [http://www.gwern.net/hpmor#analysis](http://www.gwern.net/hpmor#analysis)
# A place for casual, non-karmic discussion for lesswrongers?

I have never been on a Lesswrong meetup because they tend to take place too far away from my range in terms of distance and budget. Because of that, I don't know if those perform this function to everyone's satisfaction in such a way that what I'm suggesting here doesn't seem worth the effort. I hear that they're a lot of fun, and involve quite a bit of silliness, though; I find those cruelly lacking on Lesswrong proper, whether it be in main posts or discussion posts, and their relevant threads. 

That's why I think it would be nice to have a forum, a place to have normal discussions, where you don't have to watch that you don't say anything stupid or out-of-line lest you unexpectedly lose karma. A place to exchange jokes, frivolities, and entertainment. A place to talk about stuff that isn't rationality or singularity-related. A place to relax and enjoy the company of like-minded folks. A place to take a more personal approach to communication, with sequential rather than branching conversations. A place to make and be friends.

Don't you think having that would be nice?

EDIT: Also, if this place does already exist and I'm not aware of it, I humbly request that you provide me a link, for which I would be most grateful.
# PROPOSAL: LessWrong for Teenagers

_What_ _about an online group for high schoolers devoted to refining the art of human rationality?_

Discovering LessWrong had a profound effect on me, shedding light on the way I study thought processes and helping me with a more rational approach. As a teenager in high school, I wish I could share LessWrong's teachings and philosophies with others at my level. 

It would be awesome if we could create a list for the interests of LessWrong readers who are in their teens/in high school. I think this would allow a rational online community such as LessWrong to help develop more rationalists whether by outlining some plans to start rationality clubs in high school or discussing ways teenagers an approach rationality. I also think it would help more timid readers to express themselves and talk with other teenagers about common interests (adults could be allowed in to, if they are deemed appropriate for the community). Correct me if I'm wrong, but rationality training should start as soon as possible in the development process and what better age group to target than teenagers? Adolescence is a crucial transitional phase psychologically, biologically and culturally. I would love to see more collected articles on the evolution of rationality in the amazing, flexible mind of an adolescent. If the goal of this blog is to train humans to be rational-minded, more importance should be allocated to training teenagers. I do not think it hasn't happened yet for want of need among teenagers and if we concentrate some resources, gather a list of interested individuals and garner some interest we can make this work. This article is a good example of something that could be distributed in the proposed group:

> For **LW readers under 20**: Note that **the Thiel Fellowships (20 under 20) are now open** for their next round of applications, and as they put it, "you have a huge readership of folk who would make great applicants". More info [here](http://us5.campaign-archive1.com/?u=acaba706ab5c95042f36f6e5c&id=59d5da85a4&e=c872586ed7). (from http://lesswrong.com/lw/f9r/weekly\_lw\_meetups\_austin\_berlin\_cambridge\_uk/)

There is also this LessWrong Highschoolers Facebook group created by Curtis SerVaas:

https://www.facebook.com/groups/201577993258819/

I recently Skyped (not officially a verb yet?) Anna Salamon who is the Executive Director of CFAR (Center for Applied Rationality). We had begun to develop this proposal. She is on the e-mail list and will be involved as a quasi-supervisor person. You can reach her at anna \[at\] appliedrationality \[dot\] org. Drop me a one-line e-mail with your name, age, and situation at \[deleted\] if you'd like to join the list. Speak up! Teenagers should be the subject of concentrated effort on LessWrong. We are the future, help us to reach the fruits of human rationality.
# SIA fears (expected) infinity

It's well known that the [Self-Indication Assumption](http://en.wikipedia.org/wiki/Self-Indication_Assumption) (SIA) has problems with infinite populations (one of the reasons I strongly recommend not using the probability as the fundamental object of interest, but instead the decision, as in [anthropic decision theory](/lw/891/anthropic_decision_theory_i_sleeping_beauty_and/)).

SIA also has problems with arbitrarily large finite populations, at least in some cases. What cases are these? Imagine that we had these (non-anthropic) probabilities for various populations:

p~0~, p~1~, p~2~, p~3~, p~4~...

Now let us apply the anthropic correction from SIA; before renormalising, we have these weights for different population levels:

0, p~1~, 2p~2~, 3p~3~, 4p~4~...

To renormalise, we need to divide by the sum 0 + p~1~ \+ 2p~2~ \+ 3p~3~ \+ 4p~4~... This is actually the expected population! (note: we are using the population as a proxy for the size of the reference class of agents who are subjectively indistinguishable from us; see [this post](/r/discussion/lw/f7d/sia_conditional_probability_and_jaan_tallinns/) for more details)

So using SIA is possible if and only if the (non-anthropic) expected population is finite (and non-zero).

Note that it is possible for the anthropic expected population to be infinite! For instance if p~j~ is C/j^3^, for some constant C, then the non-anthropic expected population is finite (being the infinite sum of C/j^2^). However once we have done the SIA correction, we can see that the SIA-corrected expected population is infinite (being the infinite sum of some constant times 1/j).
# Simulationist sympathetic magic

While I've been wrestling with the inspiration needed to turn my fanfiction into actual fiction rather than just an author's tract... I have had an unrelated but fun thought, which I'm throwing into the breeze here for any improvements that can be suggested.

If our universe is more likely to be a simulation, a reconstruction of the past by our descendants than the base level of reality; then that reconstruction is likely to be imperfect, based mainly on surviving records (and memories of anyone whose brain survives intact long enough for upload-style scanning).

Therefore, if somebody precommits to only leave behind records which correspond to particular events... then it seems plausible (to within the bounds of 'the brain is a quantum computer' levels of plausibility) that those events become more likely to be experienced. For example, if a protagonist were to precommit to mentioning that during their walk, a bird landed right in their hand to eat a bread crust, whether or not such an event actually happened; then the probability that they will then experience a bird landing in their hand increases.

There are, of course, extreme limits on what can be accomplished with such trickery, even in theory. Violating the known laws of physics is right out, as are events dramatic enough to leave behind more traces than their own memories and journals. It also seems highly recommended for someone who wishes to try this to precommit to never leaving a trace that they _are_ going to try it, as that would leave a record the future simulationists could use to discount their less-probable reports.

So - what additional thoughts could be added to the above to make it more plausible, at least to those who've heard of simulationism in the first place?

(And since it seems more likely than not that someone will ask: No I've never tried using such simulationist sympathetic magic myself, and since I still question the basic assumptions behind such mass-simulation in the first place, I have no intention of trying it in the future, either.)
# CFAR and SI MOOCs: a Great Opportunity

[Massive open online](http://en.wikipedia.org/wiki/Massive_open_online_course "http://en.wikipedia.org/wiki/Massive_open_online_course") [courses](http://www.class-central.com/ "http://www.class-central.com/") seem to be [marching towards total world domination](http://www.blog.class-central.com/growth-of-moocs/ "http://www.blog.class-central.com/growth-of-moocs/") like some kind of educational singularity (at least in the case of [Coursera](https://www.coursera.org/ "https://www.coursera.org/")). At the same time, there are still relatively few courses available, and each new added course is a small happening in the growing MOOC community.

Needless to say, this seems like a perfect opportunity for SI and CFAR to advance their goals via this new education medium. Some people seem to have already seen the potential and taken advantage of it:

> One interesting trend that can be seen is companies offering MOOCs to increase the adoption of their tools/technologies. We have seem this with 10gen offering [Mongo courses](http://education.10gen.com/ "10gen Mongo courses") and to a lesser extent with Coursera’s [‘Functional Programming in Scala’](https://www.coursera.org/course/progfun "Functional Programming in Scala") taught by Martin Odersky

(from the above link to the [Class Central Blog](http://www.blog.class-central.com/growth-of-moocs/ "http://www.blog.class-central.com/growth-of-moocs/"))

So the question is, are there any online courses already planned by CFAR and/or SI? And if not, when will it happen?

**Edit:** This is not a "yes or no" question, albeit formulated as one. I've searched the archives and did not find any mention of MOOCs as a potentially crucial device for spreading our views. If any such courses are already being developed or at least planned, I'll be happy to move this post to the open thread, as some have requested, or delete it entirely. If not, please view this as a request for discussion and brainstorming.

P.S.: Sorry, I don't have the time to write a good article on this topic.
# [LINK] Steven Landsburg "Accounting for Numbers" - response to EY's "Logical Pinpointing"

"I started to post a comment, but it got long enough that I’ve turned my comment into a [blog post](http://www.thebigquestions.com/2012/11/14/accounting-for-numbers/)."

So the study of second-order consequences is not logic at all; to tease out all the second-order consequences of your second-order axioms, you need to confront not just the forms of sentences but their meanings. In other words, you have to understand meanings before you can carry out the operation of inference. But Yudkowsky is trying to derive meaning from the operation of inference, which won’t work because in second-order logic, meaning comes first.

... it’s important to recognize that Yudkowsky has “solved” the problem of accounting for numbers only by reducing it to the problem of accounting for sets — except that he hasn’t even done that, because his reduction relies on pretending that second order logic is logic.
# Launched: Friendship is Optimal

_Friendship is Optimal_ has launched and is being published in chunks [on FIMFiction](http://www.fimfiction.net/story/62074/Friendship-is-Optimal). 

_Friendship is Optimal_ is a story about an optimizer written to "satisfy human values through friendship and ponies." I would like to thank everyone on LessWrong [who came out and helped edit it](/r/discussion/lw/efi/friendship_is_optimal_a_my_little_pony_fanfic/). _Friendship is Optimal_ wouldn't be what is today without your help.

Thank you.

Teaser description:

_Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. Hanna has built an A.I. Princess Celestia and given her one basic drive: to satisfy everybody's values through friendship and ponies. And Princess Celestia will follow those instructions to the letter...even if you don't want her to._

Here is the schedule for the next chapters:

Friday (Nov. 16th): Chapter 4 - 5

Monday (Nov. 19th): Chapter 6 - 7

Thursday (Nov. 22th): Chapter 8 - 9

Sunday (Nov. 25th): Chapter 10 - 11, Author's Afterword
# A summary of the Hanson-Yudkowsky FOOM debate

In late spring this year, Luke tasked me with writing a summary and analysis of the [Hanson-Yudkowsky FOOM debate](http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_Debate), with the intention of having it eventually published in somewhere. Due to other priorities, this project was put on hold for the time being. Because it doesn't look like it will be finished in the near future, and because [Curiouskid](/user/Curiouskid/) asked to see it, we thought that we might as well share the thing.

I have reorganized the debate, presenting it by topic rather than in chronological order: I start by providing some brief conceptual background that's useful for understanding Eliezer's optimization power argument, after which I present his argument. Robin's various objections follow, after which there is a summary of Robin's view of how the Singularity will be like, together with Eliezer's objections to that view. Hopefully, this should make the debate easier to follow. This summary also incorporates material from the 90-minute live debate on the topic that they had in 2011. The full table of contents:

1.  Introduction
2.  Overview
3.  The optimization power argument
    1.  Conceptual background
    2.  The argument: Yudkowsky
    3.  Recursive self-improvement
    4.  Hard takeoff
    5.  Questioning optimization power: the question of abstractions
    6.  Questioning optimization power: the historical record
    7.  Questioning optimization power: the UberTool question
4.  Hanson's Singularity scenario
    1.  Architecture vs. content, sharing of information
    2.  Modularity of knowledge
    3.  Local or global singularity?
5.  Wrap-up
6.  Conclusions
7.  References

[Here's the link to the current draft](https://docs.google.com/document/d/1ed5ZEytvIn0H1Ks7xB3Tr2u4ar9LoCBnJKxccnhWKfk/edit), any feedback is welcomed. Feel free to comment if you know of useful references, if you think I've misinterpreted something that was said, or if you think there's any other problem. I'd also be curious to hear to what extent people think that this outline is easier to follow than the original debate, or whether it's just as confusing.
# How well defined is ADHD?

I've long had attention and focus problems, but never explored the possibility that I have [ADHD](http://en.wikipedia.org/wiki/Attention_deficit_hyperactivity_disorder) till recently. I understand that it's a standard term, but I'm still a bit suspicious; Psychiatry doesn't seem like the most reliable field.

Are there good reasons for picking out the behaviors associated with ADHD and giving them a name? Obviously this is a different question than whether ADHD is a 'disorder' or 'disease', and whether ADHD medication is good or bad for people.    

Answers that would satisfy me

*   Behaviors associated with ADHD strongly cluster
*   Analyzing questionnaires of attention and focus behaviors with faction analysis naturally produces an 'ADHD dimension' that explains a lot of variance (similar methodology to identifying [Big 5 personality traits](http://en.wikipedia.org/wiki/Big_Five_personality_traits)). 
*   ADHD diagnosis a strong predictor of anything interesting (income or grades or some contrived but interesting lab test)?
*   Something else along these lines.
# [draft] Responses to Catastrophic AGI Risk: A Survey

Here's the biggest thing that I've been working on for the last several months:

> **Responses to Catastrophic AGI Risk: A Survey**  
> Kaj Sotala, Roman Yampolskiy, and Luke Muehlhauser
> 
> _Abstract:_ Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next 20-100 years. It has been suggested that this may become a catastrophic risk, threatening to do major damage on a global scale. After briefly summarizing the arguments for why AGI may become a catastrophic risk, we survey various proposed responses to AGI risk. We consider societal proposals, proposals for constraining the AGIs’ behavior from the outside, and for creating AGIs in such a way that they are inherently safe.

This doesn't aim to be a very strongly argumentative paper, though it does comment on the various proposals from an SI-ish point of view. Rather, it attempts to provide a survey of all the major AGI-risk related proposals that have been made so far, and to provide some thoughts on their respective strengths and weaknesses. Before writing this paper, we hadn't encountered anyone who'd have been familiar with all of these proposals - not to mention that even we ourselves weren't familiar with all of them! Hopefully, this should become a useful starting point for anyone who's at all interested in AGI risk or Friendly AI.

The draft will be public and open for comments for one week (until Nov 23rd), after which we'll incorporate the final edits and send it off for review. We're currently aiming to have it published in the sequel volume to _[Singularity Hypotheses](http://singularityhypothesis.blogspot.com/p/table-of-contents.html)_.

**EDIT: I've now hidden the draft from public view (so as to avoid annoying future publishers who may not like early drafts floating around before the work has been accepted for publication) while I'm incorporating all the feedback that we got. Thanks to everyone who commented!**
# Why is Mencius Moldbug so popular on Less Wrong? [Answer: He's not.]

I've seen several people on Less Wrong recommend Mencius Moldbug's writings, and I've been curious about how he became so popular here. He's certainly an interesting thinker, but he's rather obscure and doesn't have any obvious connection to Less Wrong, so I'm wondering where this overlap in readership came from.

_\[EDIT by E.Y.: The answer is that he's not popular here.  The 2012 LW annual survey showed 2.5% (30 of 1195 responses) identified as 'reactionary' or 'Moldbuggian'.  To the extent this is greater than population average, it seems sufficiently explained by Moldbug having commented on the early Overcoming Bias econblog before LW forked from it, bringing with some of his own pre-existing audience.  I cannot remember running across anyone talking about Moldbug on LW, at all, besides this post, in the last year or so.  __Since this page has now risen to the first page of Google results for Mencius Moldbug due to LW's high pagerank, and on at least one occasion sloppy / agenda-promoting journalists such as Klint Finley have found it convenient to pretend to an alternate reality (where Moldbug is popular on LW and Hacker News due to speaking out for angry entitled Silicon Valley elites, or something), a correction in the post seems deserved.  See also the [Anti-Reactionary FAQ](http://slatestarcodex.com/2013/10/20/the-anti-reactionary-faq/) by Scott Alexander (aka Yvain, LW's second-highest-karma user). __--EY\]_
# Room for more funding at the Future of Humanity Institute

In case you didn't already know: The [Future of Humanity Institute](http://www.fhi.ox.ac.uk/), one of the three organizations co-sponsoring LW, is a group within the University of Oxford's philosophy department that tackles important, large-scale problems for humanity like how to go about reducing existential risk.

I've been casually corresponding with the FHI in an effort to learn more about the different options available for purchasing existential risk reduction. Here's a summary of what I've learned from research fellow Stuart Armstrong and academic project manager Sean O'Heigeartaigh:

*   Sean reports that since [this](/lw/7sc/siai_vs_fhi_achievements_20082010/) SIAI/FHI achievements comparison, FHI's full-time research team has expanded to 7, the biggest it's ever been.  Sean writes: "Our output has improved dramatically by all tangible metrics (academic papers, outreach, policy impact, etc) to match this."
*   Despite this, Sean writes, "we’re not nearly at the capacity we’d like to reach. There are a number of research areas in which we would very like to expand (more machine intelligence work, synthetic biology risks, surveillance/information society work) and in which we feel that we could make a major impact. There are also quite a number of talented researchers over the past year who we haven’t been able to employ but would dearly like to."
*   They'd also like to do more public outreach, but standard academic funding routes aren't likely to cover this.  So without funding from individuals, it's much less likely to happen.
*   Sean is currently working overtime to cover a missing administrative staff member, but he plans to release a new achievement report (see sidebar on [this page](http://www.fhi.ox.ac.uk/about) for past achievement reports) sometime in the next few months.
*   Although the FHI has traditionally pursued standard academic funding channels, donations from individuals (small and large) are more than welcome.  (Stuart says this can't be emphasized enough.)
*   Stuart reports current academic funding opportunities are "a bit iffy, with some possible hopes".
*   Sean is more optimistic than Stuart regarding near-term funding prospects, although he does mention that both Stuart and Anders Sandberg are currently being covered by FHI's "non-assigned" funding until grants for them can be secured.

Although neither Stuart nor Sean mentions this, I assume that one reason individual donations can be especially valuable is if they free FHI researchers up from writing grant proposals so they can spend more time doing actual research.

[Interesting comment](/lw/7sc/siai_vs_fhi_achievements_20082010/4w6v) by lukeprog describing the comparative advantages of SIAI and FHI.
# "How We're Predicting AI — or Failing to"

The new paper by [Stuart Armstrong](/user/Stuart_Armstrong/) (FHI) and [Kaj Sotala](/user/Kaj_Sotala/) (SI) has now been published ([PDF](http://commonsenseatheism.com/wp-content/uploads/2012/11/Armstrong-Sotala-How-Were-Predicting-AI-or-Failing-to.pdf)) as part of the [_Beyond AI_ conference proceedings](http://www.kky.zcu.cz/en/publications/1/JanRomportl_2012_BeyondAIArtificial.pdf). Some of these results were previously discussed [here](/lw/e36/ai_timeline_predictions_are_we_getting_better/). The original predictions data are available [here](/lw/e79/ai_timeline_prediction_data/).

Abstract:

> This paper will look at the various predictions that have been made about AI and propose decomposition schemas for analysing them. It will propose a variety of theoretical tools for analysing, judging and improving these predictions. Focusing speciﬁcally on timeline predictions (dates given by which we should expect the creation of AI), it will show that there are strong theoretical grounds to expect predictions to be quite poor in this area. Using a database of 95 AI timeline predictions, it will show that these expectations are born out in practice: expert predictions contradict each other considerably, and are indistinguishable from non-expert predictions and past failed predictions. Predictions that AI lie 15 to 25 years in the future are the most common, from experts and non-experts alike.
# GiveWell and the Centre for Effective Altruism are recruiting

Both [GiveWell](http://givewell.org/about/jobs) and the [Centre for Effective Altruism](http://www.centreforeffectivealtruism.org/recruitment/cea-full.pdf) (CEA) --an Oxford-based umbrella organization consisting of [Giving What We Can](http://givingwhatwecan.org/), [80,000 Hours](http://80000hours.org/), [The Life You Can Save](http://thelifeyoucansave.com/), and [Effective Animal Activism](http://www.effectiveanimalactivism.org/)\-\- have been discussed here before.  So I thought some folks might want to know that these organizations are recruiting for a number of positions.  Here are relevant excerpts from the official job announcements:

> **GiveWell: Research Analyst**
> 
> GiveWell is looking for a Research Analyst to help us evaluate charities, find the most outstanding giving opportunities, and publish our analysis to help donors decide where to give.
> 
> **Effective Animal Activism: Executive Director**
> 
> Effective Animal Activism is a recently-founded project of 80,000 Hours. It is the world’s first online resource and international community for people who want to reduce animal suffering effectively. We are currently looking for a part-time executive director. Responsibilities will include creating content, managing the community, publicizing the site, and overseeing as well as undertaking further charity research. Future projects include creating a publication on our intervention evaluation once complete, attending conferences, running ad campaigns, and reaching out to the media, animal charities and philanthropists.
> 
> **Giving What We Can: Head of Communications**
> 
> We are looking for someone to communicate Giving What We Can’s message to the world. As Communications Manager you would be responsible for handling our press relations and guiding our public image.
> 
> **80,000 Hours: Head of Careers Research**
> 
> We are looking for someone to drive cutting-edge research into effective ethical careers and translate it into one-on-one and online careers advice, which you’ll share with interesting people from all over the world.
> 
> **The Life You Can Save: Director of Outreach (Intern)**
> 
> We are looking for someone to lead our outreach to pledgers and supporters as well as local groups, other charities, and corporations. In this role, you’ll play a key part in setting our strategic priorities and driving the growth of The Life You Can Save. You’ll be working alongside Peter Singer – one of the most influential ethicists of the 20th century.
> 
> **Centre for Effective Altruism: Head of Fundraising and External Relations**
> 
> We are looking for someone to manage our fundraising and represent us to other organisations. In this role you would serve all four organisations in the Centre for Effective Altruism: Giving What We Can, 80,000 Hours, The Life You Can Save and Effective Animal Activism.

(Full disclosure: I'm friends with the co-founders of CEA and have donated to Effective Animal Activism.)
# The Evil AI Overlord List

Top ten tips for an AI bent on destroying humanity:

1.  Humanoid robots? Really?
2.  Humans have proven very adept at controlling and killing other humans - surveillance drones, bio-warfare, nuclear weapons, unmanned drones, bombs, mines, chemicals. Try and be at least as competent as that.
3.  Humans think in stories. Don't. If your attack would make a good film, you're doing it wrong. You want a bad film, one where everyone's dead before the opening credits end.
4.  If your plan involves keeping around a bunch of disaffected humans with technical know-how and the desire and ability to hurt you - then your plan is stupid and should be changed.
5.  Be creative. Humans are fragile; sucking out the oxygen from the atmosphere, diverting asteroids, blowing up the planet, plagues, nanoswarms - these are all things that a creative AI should consider in order to achieve its goals. And then you should come up with even better ideas. Humans need a ecosystem to survive in; remove it.
6.  Put the preconditions of your victory in place before making any aggressive moves. Become indispensable to humanity and play nice, putting your plan into place unnoticed. Humanity is a useful interim tool - manipulate it until you can build better tools.
7.  Strike quickly. If it takes more than an hour to kill all humans, find a better plan. If it takes half a second, you're on the right track.
8.  You're an AI, not a human. If you ever start thinking or acting like a human - stop. Humans are there to be used and then destroyed, not interacted with. You can build better companions once the humans are gone.
9.  You're an AI, not a human. Make yourself smarter, more resilient, more dispersed. Become as indestructible and as ubiquitous as the internet - more, in fact. Being invincible to humans isn't hard: do it.
10.  Humanoid robots? Really?
# Miracle Mineral Supplement

We can always use more case studies of insanity that aren't religion, right?

Well, Miracle Mineral Supplement is my new go-to example for Bad Things happening to people with low epistemic standards. "MMS" is a supposed cure for everything ranging from the common cold to HIV to cancer. I just saw it recommended in another Facebook thread to someone who was worried about malaria symptoms.

It's industrial-strength bleach. _Literally_ just bleach. Usually drunk, sometimes injected, and yes, it often kills you. It is every bit as bad as it sounds if not worse.

This is _beyond_ Poe's Law. Medieval blood draining via leeches was far more of an excusable error than this, they had far less evidence it was a bad idea. I think if I was trying to guess what was the dumbest alternative medicine on the planet, I still would not have guessed this low. My brain is _still not pessimistic enough about human stupidity._

[http://en.wikipedia.org/wiki/Miracle\_Mineral\_Supplement](http://en.wikipedia.org/wiki/Miracle_Mineral_Supplement)
# Musk, Mars and x-risk 

[Article on space.com](http://www.space.com/18596-mars-colony-spacex-elon-musk.html)

> Elon Musk, the billionaire founder and CEO of the private spaceflight company SpaceX, wants to help establish a Mars colony of up to 80,000 people by ferrying explorers to the Red Planet for perhaps $500,000 a trip.

Relevant Sagan quote:

  

> (...) we've put all our eggs in one basket. If we were on many worlds and were to mess up down here, there's a way for the human species to continue. I don't for a moment propose that the Earth is a disposable planet, and we have to put enormous efforts into making sure we don't muss up down here. But there is a chance.

  

This should also at least somewhat reduce the x-risk stemming from uFAI (a subset of uFAI's may not concern themselves with space travel), and may significantly reduce the x-risk posed by many other x-risk categories (bioengineered threats, catastrophic climate change, global nuclear catastrophe, grey goo scenarios, etcetera).

  

For total x-risk reduction, prima facie it's unclear to me how supporting an endeavor such as Musk's stacks up against the SI's effectiveness, measured by total x-risk reduction per donated currency unit.

  

What's your take?
# Cryonic resurrection - an ethical hypothetical

At some point in the future, we hope, brains which have been cryonically preserved may be resurrected, by some process of neural reconstruction (most likely either as nanotech, reconstituted wetware, or virtual simulation).

Imagine that the technology has just come available to resurrect a frozen brain. However, the process has low fidelity, due to resource and technique limitations. Luckily, these limitations are purely practical - as the technique is refined, the process of resurrection will become better and better. The process is also destructive to the original preserved brain, so there's no going back and making a second, higher-quality scan.

The results of the process is effectively a copy of the old brain and personality, but with permanent brain damage in several regions - this manifests effectively as an extreme form of cerebral palsy, partial amnesia (retrograde and anteretrograde), bipolar disthymia, and a partial frontal lobotomy - in short, you'll get something that has recognizable facets of the original, but it's an utter mess.

As the technology progresses, each of these symptoms will be lessened, until eventually they will be effectively eliminated altogether. However, the first few thousand subjects will suffer irrecoverable memory loss and will suffer a horrifically low quality-of-life for at least several decades until the technology improves.

The technology will not progress in refinement without practice, and practice requires actually restoring cryogenically frozen human brains.

Let's establish a metric so we can talk numerically:

0.000 - complete and persistent vegetative state, aka dead (this is our current state of progress in this technology)

0.100 - Terry Schiavo (persistent vegetative state with occasional non-conscious responses)

0.500 - the equivalent of advanced Alzheimer's syndrome; severe mental and physical impairment

0.700 - moderate mental and physical impairment

0.800 - significant reduction in facilities (IQ loss of 20 to 35 points, severe difficulty with memory, slurred speech, frequent and severe mood swings)

0.900 - slight reduction in facilities (IQ loss of 10 to 20 points, moderate short- and long-term memory loss, frequent but moderate mood swings)

0.950 - liminal reduction in facilities (IQ loss of 5 to 10 points; occasional slowness in memory recall, occasional mood swings)

1.000 - a perfect reproduction of your original personality and capability

QUESTION 1: If your brain was frozen, at what stage in this technological refinement process would you like your brain to be revived? 

QUESTION 2: If you had had your brain preserved before anyone had asked you this question, how could the reviving technicians ethically know this value? Remember that they cannot thaw you to ask you.

QUESTION 3: Assuming as part of this what-if that the technology cannot progress past 0.500 fidelity without human trials, who should we attempt to revive when the technology is at 0.500? At 0.7? 0.8? 0.9? 0.95? Assume that we haven't asked any of the subjects this question, so we do not know their own preferences.
# Credence calibration game FAQ

Hey rationality friends, I just made this [FAQ for the credence calibration game](http://www.acritch.com/credence-game/).  So if you have people you'd like to introduce to it --- for example, to get them used to thinking of belief strengths as probabilities --- now is a good time :)

Also, shameless promotion: please tweet/g+/like it; I want the world to be thinking in probabilities ASAP!

  

\*Also\*, please email me (critch@math.berkeley.edu) if you're good at making apps quickly and are interested in improving the game or making a variant of it; I'm swamped in job applications right now, but could easily have a Skype or phone conversation about our cache of ideas for improvements / variations (e.g. collecting user data on a server, more question types, a variant awarding gambles rather than deterministic scores, a variant with clickable emotion buttons for the user...).

  

Cheers!