Machine Intelligence Research Institute Progress Report, December 2011

“I think the Machine Intelligence Research Institute has some very smart people working on the most important mission on Earth, but… what exactly are they doing these days? I’m in the dark.”
There’s a good reason I hear this comment so often. We haven’t done a good job of communicating our progress to our supporters.
Since being appointed Executive Director of the Machine Intelligence Research Institute (SI) in November, I’ve been working to change that. I gave two Q&As about SI and explained our research program with a list of open problems in AI risk research. Now, I’d like to introduce our latest effort in transparency: monthly progress reports.
We begin with last month: December 2011. What did we do in December 2011?
(From this point on I’ll refer to myself as “Luke,” for clarity.)

Winter fundraiser. We launched our winter fundraiser and have been contacting our supporters. The fundraiser has raised over $40k so far, though we still have $60k to go! (So, please donate!)
Singularity Summit 2012. Our chief operating officer, Amy Willey, worked all month on preparations for Singularity Summit 2012, with much help from Luke. As a result we have now chosen a team of professionals with which we will take the Summit to “the next level,” and we’ve already confirmed several major speakers: Ray Kurzweil, Steven Pinker, Tyler Cowen, Temple Grandin, Peter Norvig, Robin Hanson, Peter Thiel, Melanie Mitchell, Vernor Vinge, and Carl Zimmer. We have also opened negotiations with many other speakers. This is a big improvement over our preparations for Singularity Summit 2011, which effectively began in May 2011, leaving us little time to capture certain speakers and develop certain kinds of media coverage. This much progress at such an early stage, in addition to a larger budget and greater professional assistance, will allow Singularity Summit 2012 to be a major leap forward for the event. Amy has also been developing arrangements for a possible European Singularity Summit in 2012.
Rationality Org. As explained in our strategic plan, we recognize the branding confusion produced by focusing on both AI risk research and rationality education, so we are preparing to spin off a separate rationality education organization so that the Machine Intelligence Research Institute can focus on AI risk research. Internally, we are calling the rationality education organization “Rationality Org.” Anna and Eliezer, with some help from Luke, did a lot of work developing plans for the future Rationality Org. We spent even more time developing the core rationality lessons, testing versions of them on different groups of people, and iterating the content. We expect the Rationality Org to launch late this year or early next year, and we expect it to not only raise the sanity waterline but also bring significant funding toward existential risk reduction.
New website design. Our media director, Michael Anissimov, with much help from Luke, worked out the strategy and design of SI’s new website and worked with a designer to iterate the design several times. The designer is now programming the site.
New donor database. In December, our Director of Development, Louie Helm, finished setting up our new donor database, including the custom code for automatically importing data from Paypal, Google Checkout, etc. This database gives us a much better view of who our supporters are, and allows us to more effectively thank them for their support. Anissimov wrote personal thank-you notes to hundreds of past donors.
Research articles. Luke and Anna made continued progress on their overview article “Intelligence Explosion: Evidence and Import.” Carl continued work with FHI‘s Stuart Armstrong on their article “Arms Races and Intelligence Explosions,” and continued work with Nick Bostrom on their article “How Hard is Artificial Intelligence? Evolutionary Arguments and Selection Effects.”
Other articles. Luke wrote a few articles for Less Wrong: Hack Away at the Edges, Why study the cognitive science of concepts, and So You Want to Save the World. Eliezer made lots of progress on his new Bayes Theorem tutorial, including (outsourced) illustrations and much audience testing.
Eliezer’s book. Eliezer finished the book proposal for his first book (already mostly written), The Science of Changing Your Mind. We have begun looking for good agents to represent the book.
Facing the Singularity. Luke continued to develop his online book Facing the Singularity, a layman’s introduction to the Singularity, its consequences, and what we can do about it. The chapters he wrote in December 2011 were: The Crazy Robot’s Rebellion, Not Built to Think About AI, Playing Taboo with “Intelligence”, Superstition in Retreat, Plenty of Room Above Us, and Don’t Flinch Away.
Additional transparency efforts. Anissimov and Luke began work on the design and content for an annual report. They also shot and produced Luke’s video Q&A #1.
Optimal philanthropy. The optimal philanthropy movement (e.g. Giving What We Can) is growing exponentially. Carl and Anna did much collaboration and research with other members of the movement. Partly due to their work, the optimal philanthropy movement has great awareness of the case for existential risk reduction as optimal philanthropy, which should bring significant funding for existential risk reduction work in the coming years.
Meetings with advisors, supporters, and potential researchers. During December 2011, various SI staff met or spoke with dozens of advisors, supporters, and collaborators about how to build the existential risk community, along with other topics. We also met with several potential researchers to gauge their interest and abilities.
Google Adwords upgrade. For months, Louie and others have been tweaking the ads we get from $10k/month of Google Adwords donated to us by Google. By December 2011, our ads were so successful that we qualified for an upgrade, and are now receiving $40k/month of free advertising via Google Adwords.
Better financial management. In December 2011 we began to train our new treasurer, long-time donor and friend of SI, Jesse Liptrap. This means that someone outside the organization is keeping a close watch on our finances. We also began work on improving our book-keeping and accounting practices, which will allow better budgeting, forecasting, and resource management.
Unpublished research. As with most research institutes, most of our research does not end up in a published paper for 1-3 years, if ever, even though it informs our views on many things. Unpublished research in December 2011 included research on population ethics, brain-computer interfaces, optimal philanthropy, technological forecasting, nuclear extinction risks, AI architectures, anthropics, decision theories, rationality training, Oracle AI, science productivity,  and more. SI’s research associates contributed to some of this research, including the Less Wrong discussion post A model of UDT with a halting oracle.
New board member. Quixey co-founder and CEO, Tomer Kagan, was added to SI’s board of directors. Tomer is a good friend and brings a wealth of business and management experience to our team.
Much more. Of course, we worked on dozens of other, smaller projects. These include: updates to IntelligenceExplosion.com; development of contacts for Rationality Org; the organization of regular SI staff dinners, to promote coordination and friendship; speaking with donors at Peter Thiel’s “Fast Forward” party; development of a database of helpful volunteers and assistants; implementing Olark on our donate page; meetings with reporters from various media organizations; uploading old videos to Vimeo and YouTube; fixing errors and outdated content on our website; finishing our 2010 990 and sent it to Brandon Reinhart to add to his financial examination of the Machine Intelligence Research Institute, preparing a new template for SI research publications (courtesy of research associate Daniel Dewey); and much more.

The post Machine Intelligence Research Institute Progress Report, December 2011 appeared first on Machine Intelligence Research Institute.


Q&A #2 with Luke Muehlhauser, Machine Intelligence Research Institute Executive Director

Machine Intelligence Research Institute Activities
Bugmaster asks:
…what does the SIAI actually do? You don’t submit your work to rigorous scrutiny by your peers in the field… you either aren’t doing any AGI research, or are keeping it so secret that no one knows about it… and you aren’t developing any practical applications of AI, either… So, what is it that you are actually working on, other than growing the SIAI itself ?
It’s a good question, and my own biggest concern right now. Donors would like to know: Where is the visible return on investment? How can I see that I’m buying existential risk reduction when I donate to the Machine Intelligence Research Institute?
SI has a problem, here, because it has done so much invisible work lately. Our researchers have done a ton of work that hasn’t been written up and published yet; Eliezer has been writing his rationality books that aren’t yet published; Anna and Eliezer have been developing a new rationality curriculum for the future “Rationality Org” that will be spun off from the Machine Intelligence Research Institute; Carl has been doing a lot of mostly invisible work in the optimal philanthropy community; and so on. I believe this is all valuable x-risk-reducing work, but of course not all of our supporters are willing to just take our word for it that we’re doing valuable work. Our supporters want to see tangible results, and all they see is the Singularity Summit, a few papers a year, some web pages and Less Wrong posts, and a couple rationality training camps. That’s good, but not good enough!
I agree with this concern, which is why I’m focused on doing things that happen to be both x-risk-reducing and visible.
First, we’ve been working on visible “meta” work that makes the Machine Intelligence Research Institute more transparent and effective in general: a strategic plan, a donor database (“visible” to donors in the form of thank-yous), a new website (forthcoming), and an annual report (forthcoming).
Second, we’re pushing to publish more research results this year. We have three chapters forthcoming in The Singularity Hypothesis, one chapter forthcoming in The Cambridge Handbook of Artificial Intelligence, one forthcoming article on the difficulty of AI, and several other articles and working papers we’re planning to publish in 2012. I’ve also begun writing the first comprehensive outline of open problems in Singularity research, so that interested researchers from around the world can participate in solving the world’s most important problems.
Third, there is visible rationality work forthcoming. One of Eliezer’s books is now being shopped to agents and publishers, and we’re field-testing different versions of rationality curriculum material for use in Less Wrong meetups and classes.
Fourth, we’re expanding the Singularity Summit brand, an important platform for spreading the memes of x-risk reduction and AI safety.
So my answer is to the question is: “Yes, visible return on investment has been a problem lately due to our choice of projects. Even before I was made Executive Director, it was one of my top concerns to help correct that situation, and this is still the case today.”
What if?
XiXiDu asks:
What would SI do if it became apparent that AGI is at most 10 years away?
This would be a serious problem because by default, AGI will be extremely destructive, and we don’t yet know how to make AGI not be destructive.
What would we do if we thought AGI was at most 10 years away?
This depends on whether it’s apparent to a wider public that AGI is at most 10 years away, or a conclusion based only on a nonpublic analysis.
If it becomes apparent to a wide variety of folks that AGI is close, then it should be much easier to get people and support for Friendly AI work, so a big intensification of effort would be a good move. If the analysis that AGI is 10 years away leads to hundreds of well-staffed and well-funded AGI research programs and a rich public literature, then trying to outrace the rest with a Friendly AI project becomes much harder. After an intensified Friendly AI effort, one could try to build up knowledge in Friendly AI theory and practice that could be applied (somewhat less effectively) to systems not designed from the ground up for Friendliness. This knowledge could then be distributed widely to increase the odds of a project pulling through, calling in real Friendliness experts, etc. But in general, a widespread belief that AGI is only 10 years away would be a much hairier situation than the one we’re in now.
But if the basis for thinking AI was 10 years away was nonpublic (but nonetheless persuasive to supporters who have lots of resources), then it could be used to differentially attract support to a Friendly AI project, hopefully without provoking dozens of AGI teams to intensify their efforts. So if we had a convincing case that AGI was only 10 years away, we might not publicize this but would instead make the case to individual supporters that we needed to immediately intensify our efforts toward a theory of Friendly AI in a way that only much greater funding can allow.
Budget
MileyCyrus asks:
What kind of budget would be required to solve the friendly AI problem?
Large research projects always come with large uncertainties concerning how difficult they will be, especially ones that require fundamental breakthroughs in mathematics and philosophy like Friendly AI does.
Even a small, 10-person team of top-level Friendly AI researchers taking academic-level salaries for a decade would require tens of millions of dollars. And even getting to the point where you can raise that kind of money requires a slow “ramping up” of researcher recruitment and output. We need enough money to attract the kinds of mathematicians who are also being recruited by hedge funds, Google, and the NSA, and have a funded “chair” for each of them such that they can be prepared to dedicate their careers to the problem. That part alone requires tens of millions of dollars for just a few researchers.
Other efforts like the Summit, Less Wrong, outreach work, and early publications cost money, and they work toward having the community and infrastructure required to start funding chairs for top-level mathematicians to be career Friendly AI researchers. This kind of work costs between $500,000 and $3 million per year, with more money per year of course producing more progress.
Predictions
Wix asks:
How much do members’ predictions of when the singularity will happen differ within the Machine Intelligence Research Institute?
I asked some Machine Intelligence Research Institute staff members to answer a slightly different question, one pulled from the Future of Humanity Institute’s 2011 machine intelligence survey:
Assuming no global catastrophe halts progress, by what year would you assign a 10% / 50% / 90% chance of the development of human-level machine intelligence? Feel free to answer ‘never’ if you believe such a milestone will never be reached.
In short, the survey participants’ median estimates (excepting 5 outliers) for 10% / 50% / 90% were:
2028 / 2050 / 2150
Here are five of the Machine Intelligence Research Institute’s staff members’ responses, names unattached, for the years by which they would assign a 10% / 50% / 90% chance of HLAI creation, conditioning on no global catastrophe halting scientific progress:
2025 / 2073 / 2168
2030 / 2060 / 2200
2027 / 2055 / 2160
2025 / 2045 / 2100
2040 / 2080 / 2200
Those are all the answers I had time to prepare in this round; I hope they are helpful!
The post Q&A #2 with Luke Muehlhauser, Machine Intelligence Research Institute Executive Director appeared first on Machine Intelligence Research Institute.


2011 Machine Intelligence Research Institute Winter Fundraiser

Help us raise $100,000 that will go towards funding a website redesign, new research papers, and staging a bigger and better Singularity Summit 2012 in San Francisco. This fundraiser ends January 20th, so please contribute now to help us achieve our goal!
Donate now!

$0

$25K

$50K

$75K

$100K

This is your last chance to make a tax-deductible donation in 2011.
ARTIFICIAL INTELLIGENCE MORE RELEVANT THAN EVER
Recent books like Machine Ethics from Cambridge University Press and Robot Ethics from MIT Press, along with the U.S. military-funded research that resulted in Governing Lethal Behavior in Autonomous Robots show that the world is waking up to the challenges of building safe and ethical AI. But these projects focus on limited AI applications and fail to address the most important concern: how to ensure that smarter-than-human AI benefits humanity. The Machine Intelligence Research Institute has been working on that problem longer than anybody, a full decade before the Singularity landed on the cover of TIME magazine.
ACCOMPLISHMENTS IN 2011
2011 was our biggest year yet. Since the year began, we have:

Held our annual Singularity Summit in New York City, with more than 900 in attendance. Speakers included inventor and futurist Ray Kurzweil, economist Tyler Cowen, PayPal co-founder Peter Thiel, Skeptic publisher Michael Shermer, Mathematica and WolframAlpha creator Stephen Wolfram, neuroscientist Christof Koch, MIT physicist Max Tegmark, and famed Jeopardy! contestant Ken Jennings.
Held a smaller Singularity Summit in Salt Lake City.
Held a one-week Rationality Minicamp and a ten-week Rationality Boot Camp.
Created the Research Associates program, which currently has 7 researchers coordinating with Singularity Institute.
Published our Intelligence Explosion FAQ, IntelligenceExplosion.com, and Friendly-AI.com.
Wrote three chapters for Springer’s upcoming volume The Singularity Hypothesis, along with four other research papers.
Began work on a new, clearer website design with lots of new content, which should go live Q1 2012.
Began outlining open problems in Singularity research to help outside collaborators better understand our research priorities.

FUTURE PLANS YOU CAN HELP SUPPORT
In the coming year, we plan to do the following:

Hold our annual Singularity Summit, in San Francisco this year.
Improve organizational transparency by creating a simpler, easier-to-use website that includes Machine Intelligence Research Institute planning and policy documents.
Publish a document of open research problems in Singularity Research, to clarify the research space and encourage other researchers to contribute to our mission.
Add additional skilled researchers to our Research Associates program.
Publish a well-researched document making the case for existential risk reduction as optimal philanthropy.
Diversify our funding sources by applying for targeted grants and advertising our affinity credit card program.

We appreciate your support for our high-impact work.
The post 2011 Machine Intelligence Research Institute Winter Fundraiser appeared first on Machine Intelligence Research Institute.


Interview with New MIRI Research Fellow Luke Muehlhauser

Section One: Background and Core Ideas
Q1. What is your personal background?
Q2. Why should we care about artificial intelligence?
Q3. Why do you think smarter-than-human artificial intelligence is possible?
Q4. The mission of the Machine Intelligence Research Institute is to “to ensure that the creation of smarter-than- human intelligence benefits society.” How is your research contributing to that mission?
Q5. How does MIRI’s approach to making friendly AI differ from the concept of Asimov laws?
Q6. Why is it necessary to make an AI that “€œwants the same things we want”?
Q7. If dangerous AI were to develop, why couldn’€™t we just “€œpull the plug”€?
Q8. Why are you and the Machine Intelligence Research Institute focused on artificial intelligence instead of human intelligence enhancement or whole brain emulation?

Section Two: Research Area Questions
Q9. What research areas do you specifically investigate to develop “Friendliness content” for artificial intelligence?
Q10. Why are you and the Machine Intelligence Research Institute focused on artificial intelligence instead of human intelligence enhancement or whole brain emulation?
Q11. What is value extraction and how is it relevant to Friendly AI theory?
Q12. What is the psychology of concepts and how is it relevant to Friendly AI theory?
Q13. What is game theory and how is it relevant to Friendly AI theory?
Q14. What is metaethics and how is it relevant to Friendly AI theory?
Q15. What is normativity and how is it relevant to Friendly AI theory?
Q16. What is machine ethics and how is it relevant to Friendly AI theory?
Q16. What are some of those open problems in Friendly AI theory?
Q17. In late 2010 the Machine Intelligence Research Institute published “€œTimeless Decision Theory.” What is timeless decision theory and how is it relevant to Friendly AI theory?
Q18. What is reflective decision theory and how is it relevant to Friendly AI?
Q19. Can you give a concrete example of how your research has made progress towards a solution on one or more open problems in Friendly AI?
Q20. Why hasn’€™t Machine Intelligence Research Institute produced any concrete artificial intelligence code?
Section Three: Less Wrong and Rationality
Q21. You originally came to Machine Intelligence Research Institute’s attention when you gained over 10,000 karma points very quickly on Less Wrong. For those who aren’€™t familiar with it, can you tell us what Less Wrong is and what its relationship is to the Machine Intelligence Research Institute?
Q22. What originally got you interested in rationality?
Q23. You recently were a instructor at Rationality MiniCamp in Berkeley during the summer. Can you tell us a little about the Rationality camps, what people do there, what you taught, and so on?
Q24. Besides being a website, Less Wrong groups meet up around the world. If I were interested, where would I be able to get involved in one of those meetups, and what goes on at these meetups?
Q25. The recently released Strategic Plan mentions intentions to “€œspin off rationality training to another organization so that the Machine Intelligence Research Institute can focus on Friendly AI research.” Can you tell us something about that?
Section Four: Machine Intelligence Research Institute Operations
Q26. You and Louie Helm were hired on September 1st. How were the two of you hired?
Q27. What does Louie Helm do as Director of Development?
Q28. The Machine Intelligence Research Institute just raised $250,000 in our Summer Challenge Grant. What will those funds be spent on?
Q29: Carl Shulman was hired not long before you and Louie. What is his role in the organization?
Q30. What sort of new researchers is the Machine Intelligence Research Institute looking for?
Michael Anissimov: You were recently hired as a full-time employee by the Machine Intelligence Research Institute. What is your personal background?
Luke Muehlhauser: I studied psychology in university but quickly found that I learn better and faster as an autodidact. Since then, I’€™ve consumed many fields of science and philosophy, one at a time, as they were relevant to my interests. I’€™ve written dozens of articles for my blog and for Less Wrong, and I host a podcast on which I interview leading philosophers and scientists about their work. I also have an interest in the mathematics and cognitive science of human rationality, because I want the research I do to be arriving at plausibly correct answers, not just answers that make me feel good.
Michael: Why should we care about artificial intelligence?
Luke: Artificial intelligence is becoming a more powerful technology every year. We now have robots that do original scientific research, and the U.S. military is developing systems for autonomous battlefield robots that make their own decisions. Artificial intelligence will become even more important when it passes human levels of intelligence, at which points it will be able to do many things we care about better than we can — things like curing cancer and preventing disasters.
Michael: Why do you think smarter-than-human artificial intelligence is possible?
Luke: The first reason is scientific. Human intelligence is a product of information processing in a brain made of meat. But meat is not an ideal platform for intelligence; it’€™s just the first one that evolution happened to produce. Information processing on a faster, more durable, and more flexible platform like silicon should be able to surpass the abilities of an intelligence running on meat if we can figure out which information processing algorithms are required for intelligence –€” either by looking more closely at which algorithms the brain is using or by gaining new insights in mathematics.
The second reason is historical. Machines have already passed human ability in hundreds of particular tasks: playing chess or Jeopardy, searching through large databanks, and in a recent advance, reading road signs. There is little reason to suspect this trend will stop, unless scientific progress in general stops.
Michael: The mission of the Machine Intelligence Research Institute is to “to ensure that the creation of smarter-than-human intelligence benefits society.” How is your research contributing to that mission?
Luke: A smarter-than-human machine intelligence that benefits (rather than harms) society is called “Friendly AI.” My primary research focus is what we call the problem of “friendliness content.” What does it look like for an AI to be “friendly” or to “benefit” society? We all have ideas about what a good world looks like and what a bad world looks like, but when thinking about that in the context of AI you must be very precise, because an AI will only do exactly what it is programmed to do.
If we can figure out how to specify exactly what it would mean for an AI to be “friendly,” then the creation of Friendly AI could be the best thing that ever happened. An advanced artificial intelligence could do science better and faster than we can, and thereby cure cancer, cure diseases, allow for human immortality, prevent disasters, solve the problems of climate change, and allow us to extend our civilization to other planets. A Friendly AI could also discover better economic and political systems that improve conditions for everyone.
Michael: How does SIAI’s approach to making friendly AI differ from the concept of Asimov’s laws?
Luke: Asimov’s Three Laws of Robotics for governing robot behavior are widely considered to be inadequate for making sure that intelligent machines bring no harm to humans. In fact, Asimov used his stories to illustrate many of the ways in which those laws could lead to unintended consequences. SIAI’s approach is very different in that we don’t think that constraints on AI behavior will work in the long run. We need advanced AI to want the same things we want. If the AI wants something different than what we want, it will eventually find a way around whatever constraints we put on it, due to its vastly superior intelligence. But if we can make an AI want the same things we want, then it will be much more effective than we can be at bringing about the kind of world that we want – curing cancer and inventing immortality and so on.
Michael: Why is it necessary to make an AI that “wants the same things we want”€?
Luke: A powerful AI that wants something different than we do could be dangerous. For example, suppose the AI’€™s goal system is programmed to maximize pleasure. That sounds good at first, but if you tell a super-powerful AI to “maximize pleasure,” it might do something like (1) convert most of Earth’s resources into computing machinery, killing all humans in the process, so that it can (2) tile the solar system with as many small digital minds as possible, and (3) have those digital minds run a continuous cycle of the single most pleasurable experience possible. But of course, that’s not what we want! We don’t just value pleasure, we also value things like novelty and exploration. So we need to be very careful when we tell an AI precisely what it means to be “friendly.”
We must be careful not to anthropomorphize. A machine intelligence won’€™t necessarily have our “€˜common sense,”™ or our values, or even be sentient. When AI researchers talk about machine intelligence, they only mean to talk about a machine that is good at achieving its goals — whatever they are –€” in a wide variety of environments. So if you tell an AI to maximize pleasure, it will do exactly that. It’s not going to stop halfway through and “€˜realize”€™ –€” like a human might –€” that maximizing pleasure isn’€™t what was intended, and that it should do something else.
Michael: If dangerous AI were to develop, why couldn’€™t we just “pull the plug”€?
Luke: We might not know that an AI was dangerous until it was too late. An AI with a certain level of intelligence is going to realize that in order to achieve its goals it needs to avoid being turned off, and so it would hide both the level of its own intelligence and its dangerousness.
But the bigger problem is that if some AI development team has already developed an AI that is intelligent enough to be dangerous, then other teams are only a few months or years behind them. You can’€™t just unplug every dangerous AI that comes along until the end of time. We’€™ll need to develop a Friendly AI that can ensure safety much better than humans can.
Michael: Why are you and Machine Intelligence Research Institute focused on artificial intelligence instead of human intelligence enhancement or whole brain emulation?
Luke: Human intelligence enhancement is important, and may be needed to solve some of the harder problems of Friendly AI. Whole brain emulation is a particularly revolutionary kind of human intelligence enhancement that, if invented, could allow us to upload human minds into computers, run them at speeds much faster than is possible with neurons, make backup copies, and allow immortality.
Many researchers think that artificial intelligence will arrive before whole brain emulation does, but predicting the timelines of future technology can be difficult. We are very interested in whole brain emulation, and in fact that was the subject of a presentation our researcher Anna Salamon gave at a recent AI conference. One reason for us to focus on AI for the moment is that there are dozens of open problems in Friendly AI theory that we can make progress on right now without needing the vast computational resources required to make progress in whole brain emulation.
Section Two: Research Area Questions
Michael: What research areas do you specifically investigate to develop “Friendliness content” for artificial intelligence?
Luke: One relevant area of research is cognitive neuroscience, especially the subfields neuroeconomics and affective neuroscience.
Worlds are “good” or “bad” to us because of our values, and our values are stored in the brain’s neural networks. For decades, we’ve had to infer human values by observing human behavior because the brain has been a “black box” to us. But that can only take us so far because the environments in which we act are highly complex, and that makes it difficult to infer human values merely from behavior. Recently, new technologies like fMRI and TMS and optogenetics have allowed us to look into the black box and watch what the brain does. In fact, we’ve located the specific neurons that seem to encode the brain’s expected subjective value for the possible actions we are considering at a given moment. We’ve also learned a lot about the specific algorithms used by the brain to update how much we value certain things – in fact, they turned out to be a type of algorithm first discovered in computer science, called temporal difference reinforcement learning.
A second relevant area of research is choice modeling and preference elicitation. Economists use a variety of techniques, for example Willingness to Pay measures, to infer human preferences from human behavior. AI researchers also do this, usually for the purposes of designing a piece of software called a decision support system. The human brain doesn’t seem to encode a coherent preference set, so we’€™ll need to use choice modeling and preference elicitation techniques to extract a coherent preference set from whatever it is that human brains actually do.
Other fields relevant to friendliness content theory include value extrapolation, the psychology of concepts, game theory, metaethics, normativity, and machine ethics.
Michael: What is value extrapolation and how is it relevant to Friendly AI theory?
Luke: Most philosophers talk about “ideal preference theories,” but I prefer to call them “€˜value extrapolation algorithms.”€™ If we want to develop Friendly AI, we may not want to just scan human values from our brains and give those same values to an AI. I want to eat salty foods all day, but I kind of wish I didn’t want that, and I certainly don’t want an AI to feed me salty foods all day. Moreover, I would probably change my desires if I knew more and was more rational. I might learn things that would change what I want. And it’s unlikely that the human species has reached the end of moral development. So we don’t want to fix things in place by programming an AI with our current values. We want an AI to extrapolate our values so that it cares about what we would want if we knew more, were more rational, were more morally developed, and so on.
Michael: What is the psychology of concepts and how is it relevant to Friendly AI theory?
Luke: Some researchers think that part of the solution to the friendliness content problem will come from examining our intuitive concept of “€˜ought”€™ or “€˜good,”€™ and using this to inform our picture of what we think a good world would be like, and thus what the goal system of a super-powerful machine should be aimed toward. Philosophers have been examining our intuitive concepts of “€˜ought”€™ or “€˜good”€™ for centuries and made little progress, but perhaps new tools in psychology and neuroscience can help us do this conceptual analysis better than philosophers could from their armchairs.
On the other hand, psychological experiments have been undermining our classical theories about what concepts are, leading some to go so far as to conclude that concepts do not exist in any useful sense. The results of that research program in psychology and philosophy could have profound implications for any approach to friendliness content that depends on an examination of our intuitive concepts of “€˜ought”€™ or “€˜good.”€™
Michael: What is game theory and how is it relevant to Friendly AI theory?
Luke: Game theory is a highly developed field of mathematics concerned with particular scenarios (“€˜games”€™) where an agent’s success depends on the choices of others. Its models and discoveries have been applied to business, economics, political science, biology, computer science, and philosophy.
Game theory is relevant to friendliness content because many of our values result from our need to make decisions in scenarios where our success depends on the choices of others. It may also be relevant to value extrapolation algorithms, as the extrapolation process is likely to change the ways in which our values and decisions interact with the values and decisions of others.
Michael: What is metaethics and how is it relevant to Friendly AI theory?
Luke: Philosophers often divide the field of ethics into three levels. Applied ethics is the study of particular moral questions: How should we treat animals? Is lying ever acceptable? What responsibilities do corporations have concerning the environment? Normative ethics considers the principles by which we make judgments in applied ethics. Do we make one judgment over another based on which action produces the most good? Or should we be following a list of rules and respecting certain rights? Perhaps we should advocate what we would all agree to behind a veil of ignorance that kept us from knowing what our lot in life will be?
Metaethics goes one level deeper. What do terms like “€˜good”€™ and “€˜right”€™ even mean? Do moral facts exist, or is it all relative? Is there such a thing as moral progress? These questions are relevant to friendliness content because presumably, if moral facts exist, we would want an AI to respect them. Even if moral facts do not exist, our moral attitudes are part of what we value, and that is relevant to friendliness content theory.
Michael: What is normativity and how is it relevant to Friendly AI theory?
Luke: Normativity is about norms, and there are many kinds. Prudential norms concern what we ought to do to achieve our goals. Epistemic norms concern how we ought to pursue knowledge. Doxastic norms concern what we ought to believe. Moral norms concern how we ought to behave ethically. And so on.
A classic concern of normativity is the “is-ought gap.”™ Supposedly, you cannot reason from an “is”€™ statement to an “€˜ought”€™ statement. It doesn’t logically follow from “€œThe man in front of me is suffering” that “œI ought to help him.”€ Actually, it’s trivial to bridge the is-ought gap when it comes to prudential norms. “If you want Y, then you ought to do X,” is just another way of saying “Doing X will increase your chances of attaining Y.” The first sentence contains an “ought”€™ claim, but the second sentence reduces it away into a purely descriptive sentence about the natural world.
Some philosophers think that the “€˜is-ought gap”€™ can be bridged in the same way for epistemic and moral norms. Perhaps “you ought to believe X” just means “If you want true beliefs, then you ought to believe X,”€ which in turn can be reduced into the purely descriptive statement “€œBelieving X will increase your proportion of true beliefs.”€
But is there any other kind of normativity? Are there “€˜categorical”€™ oughts that do not depend on an “€œIf you want X” clause? Naturalists tend to deny this possibility, but perhaps categorical epistemic or moral oughts can be derived from the mathematics of game theory and decision theory, as naturalist Gary Drescher suggests in Good and Real. If so, it may be wise to make sure they are included in friendliness content theory, so that an AI can respect them.
Michael: What is machine ethics and how is it relevant to Friendly AI theory?
Luke: Machine ethics is one of several names for the field that studies two major questions: (1) How can we get machines to behave ethically, and (2) which types of machines can be considered genuine moral agents (in the sense of having rights or moral worth like a human might)? Most of the work in the field so far is relevant only to “€˜narrow AI”€™ machines that are not nearly as intelligent as humans are, but two directions of research that may be useful for Friendly AI are mechanized deontic logic and computational metaethics.
Unfortunately, our understanding of Friendly AI –€” and of not-yet-invented AI technologies in general –€” is so primitive that we’re not even sure which fields will turn out to matter. It seems like cognitive neuroscience, preference elicitation, value extrapolation, game theory, and several other fields are relevant to Friendly AI theory, but it might turn out that as we come to understand Friendly AI better, we’€™ll learn that some research avenues are not relevant. But the only way we can learn that is to continue to make incremental progress in the areas of research that seem to be relevant.
Michael: What are some of those open problems in Friendly AI theory?
Luke: If we think just about about the issue of Friendliness content, some of the open questions are: How does the brain choose which few possible actions it will encode expected subjective value for? How does it combine absolute value and probability estimates to make those expected subjective value computations? Where is absolute value stored, and how is it encoded? How can we extract a coherent utility function or preference set from this activity in human brains? Which algorithms should we use to extrapolate these preferences, and why? When extrapolated, will the values of two different humans converge? Will the values of all humans converge? Would the values of all sentient beings converge? Will the details of human cognitive neuroscience matter much, or will such details get “˜washed out”€™ by the higher-level mathematical structure of value systems and game theory? How can these extrapolated values be implemented in the goal system of an AI?
Friendliness content is only one area of open problems in Friendly AI theory. There are many other questions. How can an agent make optimal decisions when it is capable of directly editing its own source code, including the source code of the decision mechanism? How can we get an AI to maintain a consistent utility function throughout updates to its ontology? How do we make an AI with preferences about the external world instead of its reward signal? How can we generalize the theory of machine induction –€” called Solomonoff induction –€” so that it can use higher-order logics and reason correctly about observation selection effects? How can we approximate such ideal processes such that they are computable?
That’s a start, anyway.
Michael: In late 2010 the Machine Intelligence Research Institute published “Timeless Decision Theory.”€ What is timeless decision theory and how is it relevant to Friendly AI theory?
Luke: Decision theory is the study of how to make optimal decisions. We value different things differently, and we are uncertain about which actions will bring about what we value. One of the problems not handled well by traditional decision theories like Evidential decision theory (EDT) and Causal decision theory (CDT) is the problem of logical uncertainty –€” our uncertainty about mathematical and logical facts, for example what the nth decimal of pi is. One way to think about Timeless decision theory (TDT) is that it’s a step toward a decision theory that can handle logical uncertainty.
For an AI to be safe, its decision mechanism will have to be somewhat clear and mathematically testable for stability and safety. That probably means it will need to make decisions with decision theory, rather than through a relatively opaque neural nets mechanism. So we need to solve some fundamental problems in decision theory first, and logical uncertainty is one of the remaining fundamental problems in decision theory.
Michael: What is reflective decision theory and why is it necessary to Friendly AI?
Luke: Traditional decision theories cannot handle agents that can modify their own source code, including the source code for their decision mechanism. A reflective decision theory is one that can handle such a strongly self-modifying agent. Because an advanced AI will be intelligent enough to modify its own source code, we need to develop a reflective decision theory that will allow us to ensure that the AI will remain Friendly throughout the self-modification and self-improvement process.
Michael: Can you give a concrete example of how your research has made progress towards a solution on one or more open problems in Friendly AI?
Luke: I’ve only just begun working with the Machine Intelligence Research Institute, and making progress on open problems in Friendly AI theory is only one of the many things I do. My first contribution to friendliness content theory was to summarize some very recent advances in neuroeconomics that are relevant to the study of human values. I did that because other researchers in the field were not yet familiar with that material, and I think much of the work in friendliness content theory can be done collaboratively by a broad community of researchers if we are all well-informed.
These results from neuroeconomics appear to be relevant to friendliness content theory, though only time will tell. For example, we’ve learned that expected utility for human actions is encoded cardinally (not ordinally) in the brain, and thus avoids a limiting result from economics called Arrow’s impossibility theorem.
Michael: Why hasn’€™t the Machine Intelligence Research Institute produced any concrete artificial intelligence code?
Luke: This is a common confusion. Most of the open problems in Friendly AI theory are in math and philosophy, not in computer programming. Sometimes programmers approach us, offering to work on Friendly AI theory, and we reply: “€œWhat we need are mathematicians. Are you brilliant at math?”
As it turns out, the heroes who can save the world are not those with incredible strength or the power of flight. They are mathematicians.
Section Three: Less Wrong and Rationality
Michael: You originally came to the Machine Intelligence Research Institute’€™s attention when you gained over 10,000 karma points very quickly on Less Wrong. For those who aren’€™t familiar with it, can you tell us what Less Wrong is and what its relationship is to the Machine Intelligence Research Institute?
Luke: Less Wrong is a group blog and community devoted to the study of rationality: How to get truer beliefs and make better decisions. the Machine Intelligence Research Institute’€™s co-founder Eliezer Yudkowsky originally wrote hundreds of articles about rationality for another blog, Overcoming Bias, because he wanted to build a community of people that could think clearly about difficult problems like Friendly AI. Those articles were then used as the seed content for a new website, Less Wrong. I discovered Less Wrong because of my interest in rationality, and eventually started writing articles for the site –€” many of which became very popular.
Michael: What originally got you interested in rationality?
Luke: I was raised an enthusiastic evangelical Christian, and had a dramatic crisis of faith when I learned a few things about the historical Jesus, science, and philosophy. I was disturbed by how confidently I had believed something that was so thoroughly wrong, and I no longer trusted my intuitions. I wanted to avoid being so wrong again, so I studied the phenomena that allow human brains to be so mistaken — things like confirmation bias and the affect heuristic. I also gained an interest in the mathematics of correct thinking, like Bayesian updating and decision theory.
Michael: You recently were a instructor at Rationality MiniCamp in Berkeley during the summer. Can you tell us a little about the MiniCamp, what people did there, what you taught, and so on?
Luke: Anna and I put together the minicamp, a one-week camp full of classes and activities about rationality, social effectiveness, and existential risks. Over 20 people stayed in a large house in Berkeley, where we held the classes. Some of them came from as far away as Sweden and England.
Minicamp was a blast, mostly because the people were so great! We are still in contact, still learning and growing.
We taught things like how to update our beliefs using probability theory, how to use the principle of fungibility to better fulfill our goals, and how to use body language and fashion to improve some parts of our lives that math-heads sometimes neglect! We also taught classes on optimal philanthropy (how to get the most bang for your philanthropic buck) and existential risks (risks that could cause human extinction).
Michael: Besides being a website, Less Wrong groups meet up around the world. If I were interested, where would I be able to get involved in one of those meetups, and what goes on at these meetups?
Luke: Because of how sensitive humans are to context, surrounding yourself with other people who are learning rationality and trying to improve themselves is one of the most powerful ways to improve yourself.
The easiest way to find a Less Wrong meetup near you is probably to check for the most recent front-page post on Less Wrong with the title that begins “€œWeekly LW Meetups…”€ That post will list all the Less Wrong meetups happening that week.
Each Less Wrong meetup has different people and different activities. You can contact the meetup organizer for the meetup nearest you for more information.
Michael: The recently released Strategic Plan mentions intentions to “€œspin off rationality training to another organization so that the Machine Intelligence Research Institute can focus on Friendly AI research.”€ Can you tell us something about that?
Luke: We believe that building a large community of rationality enthusiasts is crucial to the success of our mission. The Less Wrong rationality community has been an indispensable source of human and financial capital for the Machine Intelligence Research Institute. However, we understand that it’s confusing to be an organization devoted to two such apparently different fields: advanced artificial intelligence and human rationality. That’s why we are working toward launching a new organization devoted to rationality training. The Machine Intelligence Research Institute, then, will be more solely devoted to the safety of advanced artificial intelligence.
Section Four: Machine Intelligence Research Institute Operations
Michael: You and Louie Helm were hired on September 1st. How were the two of you hired?
Luke: The Machine Intelligence Research Institute doesn’€™t hire someone unless they do quite a bit of volunteer work first. I first came to the Machine Intelligence Research Institute as a visiting fellow. During the next few months I co-organized and taught at the Rationality Minicamp, taught classes for the longer Rationality Boot Camp, wrote dozens of articles on metaethics and rationality for Less Wrong, wrote the Intelligence Explosion FAQ and IntelligenceExplosion.com, led the writing of a strategic plan for the organization, and did many smaller tasks.
Louie Helm arrived in Berkeley not long after I did. As a past visiting fellow, Louie was the one who had suggested I apply to the visiting fellows program. Louie did some teaching for Rationality Boot Camp, helped me write the strategic plan, developed a donor database so that our contact with donors is more consistent, optimized the Machine Intelligence Research Institute’s finances, did lots of fundraising, and much more.
We both produced lots of value for the organization over those months as volunteers, so the Board hired us –€” me as a research fellow and Louie as Director of Development.
Michael: What does Louie Helm do as Director of Development?
Luke: We’€™re a small team, so we all do more than our title says, and Louie is no exception. Louie raises funds, communicates with donors, applies for grants, and so on. But he also launched the Research Associates program, coordinates the Volunteer Network, helps organize the Singularity Summit, seeks out potential new researchers, and more.
Michael: The Machine Intelligence Research Institute just raised $250,000 in our Summer Challenge Grant. What will those funds be spent on?
Luke: We were very pleased by the results of the summer challenge grant. No single person gave more than $25,000, so the grant succeeded because so many different people gave. More than 40 people gave $1,000 or more, which shows a high degree of trust from our core supporters.
It costs $368,000 annually to support our lean family of eight full-time staff members, four of whom are research fellows: Eliezer Yudkowsky, Anna Salamon, Carl Shulman, and myself. The money will also be used to run the 2011 Singularity Summit, though we expect that event to be cash-positive this year. We plan to redesign the singinst.org website so that it is easier to navigate and provides greater organizational transparency. And with enough funds after the Summit, we hope to hire additional researchers.
Michael: Carl Shulman was hired not long before you and Louie. What is his role in the organization?
Luke: Carl also did quite a lot of work for the Machine Intelligence Research Institute before being hired. He has written several papers and given a few talks, many of which you can read from our publications page. He continues to work on a variety of research projects, and collaborates closely with researchers at Oxfordâ€™s Future of Humanity Institute.
Michael: What sort of new researchers is the Machine Intelligence Research Institute looking for?
Luke: Mathematicians, mostly. If you’€™re a brilliant math student and want to live and work in the Bay Area where you’ll be surrounded by smart, influential, altruistic people, please apply here.
The post Interview with New MIRI Research Fellow Luke Muehlhauser appeared first on Machine Intelligence Research Institute.


2011 Summer Matching Challenge Success!

Thanks to the effort of our donors, the 2011 Summer Singularity Challenge has been met! All $125,000 contributed will be matched dollar for dollar by our matching backers, raising a total of $250,000 to fund the Machine Intelligence Research Institute’€™s operations. We reached our goal two days early, near midnight of August 29th.
On behalf of our staff, volunteers, and entire community, I want to personally thank everyone who donated. Your dollars make the difference.
Here’€™s to a better future for the human species.
The post 2011 Summer Matching Challenge Success! appeared first on Machine Intelligence Research Institute.

2011 Summer Matching Challenge Success!


Machine Intelligence Research Institute Strategic Plan 2011

The Machine Intelligence Research Institute has prepared a Strategic Plan (PDF) to concisely describe our near-term goals, vision, and concrete actions in pursuit of those goals and vision. We welcome commentary from our supporter community on the plan. The release of this Strategic Plan is part of an overall effort to increase transparency at the Institute.
Update: This strategic plan is now obsolete. See our 2013 strategic priorities instead.
The post Machine Intelligence Research Institute Strategic Plan 2011 appeared first on Machine Intelligence Research Institute.


New Intelligence Explosion Website

We’ve put together a new website focused on the intelligence explosion concept: IntelligenceExplosion.com. The site is a “landing page” that provides an easy introduction to the topic for laymen and researchers alike.
Also see Nick Bostrom’s landing pages for anthropics, the simulation argument, and existential risk.
The post New Intelligence Explosion Website appeared first on Machine Intelligence Research Institute.

New Intelligence Explosion Website


Announcing the $125,000 Summer Singularity Challenge

Thanks to the generosity of several major donors†, every donation to the Machine Intelligence Research Institute made now until August 31, 2011 will be matched dollar-for-dollar, up to a total of $125,000.
Donate now!Now is your chance to double your impact while supporting the Machine Intelligence Research Institute and helping us raise up to $250,000 to help fund our research program and stage the upcoming Singularity Summit… for which you can register now!
† $125,000 in backing for this challenge is being generously provided by Rob Zahra, Quixey, Clippy, Luke Nosek, Edwin Evans, Rick Schwall, Brian Cartmell, Mike Blume, Jeff Bone, Johan Edsträm, Zvi Mowshowitz, John Salvatier, Louie Helm, Kevin Fischer, Emil Gilliam, Rob and Oksana Brazell, Guy Srinivasan, John Chisholm, and John Ku.

The year 2011 has been huge for Artificial Intelligence. With the IBM computer Watson defeating two top Jeopardy! champions in February, it’s clear that the field is making steady progress. Journalists like Torie Bosch of Slate have argued that “We need to move from robot-apocalypse jokes to serious discussions about the emerging technology.” We couldn’t agree more — in fact, the Machine Intelligence Research Institute has been thinking about how to create safe and ethical artificial intelligence since long before the Singularity landed on the front cover of TIME magazine.
The last 1.5 years were our biggest ever. Since the beginning of 2010, we have:

Held our annual Singularity Summit, in San Francisco. Speakers included Ray Kurzweil, James Randi, Irene Pepperberg, and many others.
Held the first Singularity Summit Australia and Singularity Summit Salt Lake City.
Held a wildly successful Rationality Minicamp.
Published seven research papers, including Yudkowsky’s much-awaited ‘Timeless Decision Theory‘.
Helped philosopher David Chalmers write his seminal paper ‘The Singularity: A Philosophical Analysis’, which has sparked broad discussion in academia, including an entire issue of Journal of Consciousness Studies and a book from Springer devoted to responses to Chalmers’ paper.
Launched the Research Associates program.
Brought MIT cosmologist Max Tegmark onto our advisory board, published our Singularity FAQ, and much more.

In the coming year, we plan to do the following:

Hold our annual Singularity Summit, in New York City this year.
Publish three chapters in the upcoming academic volume The Singularity Hypothesis, along with several other papers.
Improve organizational transparency by creating a simpler, easier-to-use website that includes Machine Intelligence Research Institute planning and policy documents.
Publish a document of open research problems related to Friendly AI, to clarify the research space and encourage other researchers to contribute to our mission.
Add additional skilled researchers to our Research Associates program.
Publish well-researched documents making the case for existential risk reduction as optimal philanthropy.
Diversify our funding sources by applying for targeted grants and advertising our affinity credit card program.

We appreciate your support for our high-impact work. As PayPal co-founder and Machine Intelligence Research Institute donor Peter Thiel said:
“I’m interested in facilitating a forum in which there can be… substantive research on how to bring about a world in which AI will be friendly to humans rather than hostile… [The Machine Intelligence Research Institute represents] a combination of very talented people with the right problem space [they’re] going after… [They’ve] done a phenomenal job… on a shoestring budget. From my perspective, the key question is always: What’s the amount of leverage you get as an investor? Where can a small amount make a big difference? This is a very leveraged kind of philanthropy.”
Donate now, and seize a better than usual chance to move our work forward. Credit card transactions are securely processed through Causes.com, Google Checkout, or PayPal. If you have questions about donating, please call Amy Willey at (586) 381-1801.
The post Announcing the $125,000 Summer Singularity Challenge appeared first on Machine Intelligence Research Institute.


Tallinn-Evans Challenge Grant Success!

Thanks to the effort of our donors, the Tallinn-Evans Singularity Challenge has been met!  All $125,000 contributed will be matched dollar for dollar by Jaan Tallinn and Edwin Evans, raising a total of $250,000 to fund the Machine Intelligence Research Institute’s operations in 2011.  On behalf of our staff, volunteers, and entire community, I want to personally thank everyone who donated.  Keep watching this blog throughout the year for updates on our activity, and sign up for our mailing list if you haven’t yet.
Here’s to a better future for the human species.
We are preparing a donor page to provide a place for everyone who donated to share some information about themselves if they wish, including their name, location, and a quote about why they donate to the Machine Intelligence Research Institute.  If you would like to be included in our public list, please email us.
Again, thank you.  The Machine Intelligence Research Institute depends entirely on contributions from individual donors to exist.  Money is indeed the unit of caring, and one of the easiest ways that anyone can contribute directly to the success of the Machine Intelligence Research Institute.  Another important way you can help is by plugging us into your networks, so please email us if you want to help.
If you’re interested in connecting with other Machine Intelligence Research Institute supporters, we encourage joining our group on Facebook.  There are also local Less Wrong meetups in cities like San Francisco, Los Angeles, New York, and London.
The post Tallinn-Evans Challenge Grant Success! appeared first on Machine Intelligence Research Institute.

Tallinn-Evans Challenge Grant Success!


Announcing the Tallinn-Evans $125,000 Singularity Challenge

Thanks to the generosity of two major donors; Jaan Tallinn, a founder of Skype and Ambient Sound Investments, and Edwin Evans, CEO of the mobile applications startup Quinly, every contribution to the Machine Intelligence Research Institute up until January 20, 2011 will be matched dollar-for-dollar, up to a total of $125,000.

$0

$31.25K

$62.5K

$93.75K

$125K

Interested in optimal philanthropy — that is, maximizing the future expected benefit to humanity per charitable dollar spent? The technological creation of greater-than-human intelligence has the potential to unleash an “intelligence explosion” as intelligent systems design still more sophisticated successors. This dynamic could transform our world as greatly as the advent of human intelligence has already transformed the Earth, for better or for worse. Thinking rationally about these prospects and working to encourage a favorable outcome offers an extraordinary chance to make a difference. The Machine Intelligence Research Institute exists to do so through its research, the Singularity Summit, and public education.
We support both direct engagements with the issues as well as the improvements in methodology and rationality needed to make better progress. Through our Visiting Fellows program, researchers from undergrads to Ph.Ds pursue questions on the foundations of Artificial Intelligence and related topics in two-to-three month stints. Our Resident Faculty, up to four researchers from three last year, pursues long-term projects, including AI research, a literature review, and a book on rationality, the first draft of which was just completed. MIRI researchers and representatives gave over a dozen presentations at half a dozen conferences in 2010. Our Singularity Summit conference in San Francisco was a great success, bringing together over 600 attendees and 22 top scientists and other speakers to explore cutting-edge issues in technology and science.
We are pleased to receive donation matching support this year from Edwin Evans of the United States, a long-time MIRI donor, and Jaan Tallinn of Estonia, a more recent donor and supporter. Jaan recently gave a talk on the Singularity and his life at a entrepreneurial group in Finland. Here’s what Jaan has to say about us:

“We became the dominant species on this planet by being the most intelligent species around. This century we are going to cede that crown to machines. After we do that, it will be them steering history rather than us. Since we have only one shot at getting the transition right, the importance of MIRI’s work cannot be overestimated. Not finding any organisation to take up this challenge as seriously as MIRI on my side of the planet, I conclude that it’s worth following them across 10 time zones.”
— Jaan Tallinn, MIRI donor
Make a lasting impact on the long-term future of humanity today — make a donation to the Machine Intelligence Research Institute and help us reach our $125,000 goal. For more detailed information on our projects and work, contact us at admin@intelligence.org or read our new organizational overview.
The post Announcing the Tallinn-Evans $125,000 Singularity Challenge appeared first on Machine Intelligence Research Institute.


2010 Singularity Research Challenge Fulfilled!

Thanks to our donors, yesterday we met our fundraising goal of $100,000 for the 2010 Singularity Research Challenge.  MIRI would like to thank the grant’s matching donors and everyone who contributed.  Every donation, however small, funds research and advocacy targeted towards maximizing the probability of a positive Singularity.
If you have any questions or comments about MIRI’s activity or would like to discuss targeted donations for future projects, please feel free to contact us anytime at admin at intelligence dot org.  We also encourage you to subscribe to this blog, if you haven’t already, to stay up-to-date on MIRI’s activity.
Again, thank you, and here’s to a productive and successful 2010!
The post 2010 Singularity Research Challenge Fulfilled! appeared first on Machine Intelligence Research Institute.


Announcing the 2010 Singularity Research Challenge

Offering unusually good philanthropic returns –€” meaning greater odds of a positive Singularity and lesser odds of human extinction — the Machine Intelligence Research Institute has launched a new challenge campaign. The sponsors, Edwin Evans, Rolf Nelson, Henrik Jonsson, Jason Joachim, and Robert Lecnik, have generously put up $100,000 of matching funds, so that every donation you make until February 28th will be matched dollar for dollar. If the campaign is successful, it will raise a full $200,000 to fund MIRI’s 2010 activities.
For almost a decade, the Machine Intelligence Research Institute has been asking questions on the future of human civilization: How can we benefit from increasingly powerful technology without succumbing to the risks, up to and including human extinction? What is the best way to handle artificial general intelligence (AGI): programs as smart as humans, or smarter?
Among MIRI’s core aims is to continue studying “Friendly AI”: AI that acts benevolently because it holds goals aligned with human values. This involves drawing on and contributing to fields like decision theory, computer science, cognitive and moral psychology, and technology forecasting.
Creating AI, especially the Friendly kind, is a difficult undertaking. We’re in it for as long as it takes, but we’ve been doing more than laying the groundwork for Friendly AI. We’ve been raising the profile of AI risk and Singularity issues in academia and elsewhere, forming communities around enhancing human rationality, and researching other avenues that promise to reduce the most severe risks the most effectively.
If you make a donation to the Machine Intelligence Research Institute, you can choose which grant proposal your donation should help to fill. Any time a grant proposal is fully funded, it goes into our “€œactive projects”€ file: it becomes a project that we have money enough to fund, and that we are publicly committed to funding. (Some of the projects will go forward even without earmarked donations, with money from the general fund –€” but many won’€™t, and since our work is limited by how much money we have available to support skilled staff and Visiting Fellows, more money allows more total projects to go forward.)
Donate now, and seize a better than usual chance to move our work forward.
The post Announcing the 2010 Singularity Research Challenge appeared first on Machine Intelligence Research Institute.


Introducing Myself

Many friends of MIRI will know that I have been a supporter of its mission since its founding, and have rendered my informal assistance, including a major role in arranging matching funds for the Institute’s 2007 Challenge Grant.
I am pleased to take a more direct role in fostering its success as President of MIRI. I have left my previous role as Founder and Chief Strategist at SirGroovy.com, a growing online music licensing firm, and have been assuming responsibility for the management of the Machine Intelligence Research Institute over the last few weeks.
Prospective volunteers, donors, and aspiring researchers should now make contact with me rather than with Tyler Emerson.
To those who I am greeting for the first time, let me introduce myself. On a professional level, I hold a Master of Business Administration from Drexel University, and am coming from a role that combined management, research, analysis, and strategy in a fast-growing music licensing firm from its founding in New York. In that capacity, in a previous role at Aon, and in my academic studies I have been enduringly interested in finance and economics, particular the economics of technology and IP. Scientifically, I earned my undergraduate degree in biochemistry and have worked in several labs, as well as serving at the National Institute of Standards and Technology, and have extensively studied the history of science and technology, as well as the potential for biological cognitive enhancement. I have served with the Peace Corps in Kazakhstan, and am splitting my time between Manhattan, where I live with my wife Aruna, and Silicon Valley.
My interest in the safety of technological development, driven by the potentially grave ethical consequences, is over a decade old. I have been particularly focused on the potential of advanced nanotechnology and artificial intelligence, participating in forums such as Transvision and Foresight conferences, the SL4 mailing list, Overcoming Bias, and organizations such as MIRI and the Center for Responsible Nanotechnology (CRN). I plan to make my relevant work available at a single site, but in the meantime I will point to a small selection. For instance, I coauthored an analysis of the risks of advanced molecular manufacturing and mitigating strategies with Robert Freitas , and contributed “Corporate Cornucopia” as a member of CRN’s Global Task Force. Those who would like to see more can view Michael Anissimov’s archive of some of my writings at his website, and some of my most recent talks, an Institute of Ethics and Emerging Technologies presentation on the political implications of different conceptions of willpower and a Convergence08 talk on decision theory for humans, are available on the web.
As President, I plan to build on MIRI’s successes, such as the Singularity Summit, while also working to increase its internal and extramural research capabilities and output. In the course of the latter, I shall pay particular attention to the publication of research that improves the quality of our thinking about the potential and safety of advanced artificial intelligence, such as MIRI Research Fellow Eliezer Yudkowsky’s two contributions to the Oxford edited volume, Global Catastrophic Risks, and thus to better communicating internal research progress to our supporters.
Some of this work will involve indirect, meta-level contributions. For instance, recent work at MIRI by Rolf Nelson, Anna Salamon, Steven Rayhawk, Thomas McCabe and others has led to the development of a software tool for combining judgments about particular future scenarios and technological developments to reveal inconsistencies and enable the adoption of a more coherent probability assignment for planning. The content and algorithms of this tool have been completed, and work is now underway to finalize the software’s interface and make it publicly available to improve the quality of reasoning about interrelated technology scenarios, including those involving artificial intelligence.
Another effort involves conducting expert elicitation research to determine the state of academic and non-academic expert opinion regarding timelines and risks for advanced artificial intelligence.
Future research along these lines may explore particular biases and psychological factors affecting attitudes and reasoning related to artificial intelligence.
Other research will be directly focused on object-level problems. I plan to work vigorously to identify more promising extramural scholars whose work can be fruitfully promoted by MIRI grants, work such as MIRI-Canada Academic Prize Recipient Shane Legg’s “Machine Super-Intelligence.” At the same time I will be working to recruit and make best use of talented Research Fellows for MIRI’s internal efforts.
I look forward to describing further directions over the coming months, and invite the advice and opinions of the friends of MIRI at admin@intelligence.org.
Yours,
Michael Vassar
The post Introducing Myself appeared first on Machine Intelligence Research Institute.

Introducing Myself


Three Major Singularity Schools

I’ve noticed that Singularity discussions seem to be splitting up into three major schools of thought: Accelerating Change, the Event Horizon, and the Intelligence Explosion.

Accelerating Change:

Core claim: Our intuitions about change are linear; we expect roughly as much change as has occurred in the past over our own lifetimes. But technological change feeds on itself, and therefore accelerates. Change today is faster than it was 500 years ago, which in turn is faster than it was 5000 years ago. Our recent past is not a reliable guide to how much change we should expect in the future.
Strong claim: Technological change follows smooth curves, typically exponential. Therefore we can predict with fair precision when new technologies will arrive, and when they will cross key thresholds, like the creation of Artificial Intelligence.
Advocates: Ray Kurzweil, Alvin Toffler(?), John Smart

Event Horizon:

Core claim: For the last hundred thousand years, humans have been the smartest intelligences on the planet. All our social and technological progress was produced by human brains. Shortly, technology will advance to the point of improving on human intelligence (brain-computer interfaces, Artificial Intelligence). This will create a future that is weirder by far than most science fiction, a difference-in-kind that goes beyond amazing shiny gadgets.
Strong claim: To know what a superhuman intelligence would do, you would have to be at least that smart yourself. To know where Deep Blue would play in a chess game, you must play at Deep Blue’s level. Thus the future after the creation of smarter-than-human intelligence is absolutely unpredictable.
Advocates: Vernor Vinge

Intelligence Explosion:

Core claim: Intelligence has always been the source of technology. If technology can significantly improve on human intelligence – create minds smarter than the smartest existing humans – then this closes the loop and creates a positive feedback cycle. What would humans with brain-computer interfaces do with their augmented intelligence? One good bet is that they’d design the next generation of brain-computer interfaces. Intelligence enhancement is a classic tipping point; the smarter you get, the more intelligence you can apply to making yourself even smarter.
Strong claim: This positive feedback cycle goes FOOM, like a chain of nuclear fissions gone critical – each intelligence improvement triggering an average of >1.000 further improvements of similar magnitude – though not necessarily on a smooth exponential pathway. Technological progress drops into the characteristic timescale of transistors (or super-transistors) rather than human neurons. The ascent rapidly surges upward and creates superintelligence (minds orders of magnitude more powerful than human) before it hits physical limits.
Advocates: I. J. Good, Eliezer Yudkowsky

The thing about these three logically distinct schools of Singularity thought is that, while all three core claims support each other, all three strong claims tend to contradict each other.
If you extrapolate our existing version of Moore’s Law past the point of smarter-than-human AI to make predictions about 2099, then you are contradicting both the strong version of the Event Horizon (which says you can’t make predictions because you’re trying to outguess a transhuman mind) and the strong version of the Intelligence Explosion (because progress will run faster once smarter-than-human minds and nanotechnology drop it into the speed phase of transistors).
I find it very annoying, therefore, when these three schools of thought are mashed up into Singularity paste. Clear thinking requires making distinctions.
But what is still more annoying is when someone reads a blog post about a newspaper article about the Singularity, comes away with none of the three interesting theses, and spontaneously reinvents the dreaded fourth meaning of the Singularity:

Apocalyptism: Hey, man, have you heard? There’s this bunch of, like, crazy nerds out there, who think that some kind of unspecified huge nerd thing is going to happen. What a bunch of wackos! It’s geek religion, man.

I’ve heard (many) other definitions of the Singularity attempted, but I usually find them to lack separate premises and conclusions. For example, the old Extropian FAQ used to define the “Singularity” as the Inflection Point, “the time when technological development will be at its fastest” and just before it starts slowing down. But what makes this an interesting point in history apart from its definition? What are the consequences of this assumption? To qualify as a school of thought or even a thesis, one needs an internal structure of argument, not just a definition.
If you’re wondering which of these is the original meaning of the term “Singularity,” it is the Event Horizon thesis of Vernor Vinge, who coined the word.
The post Three Major Singularity Schools appeared first on Machine Intelligence Research Institute.

Three Major Singularity Schools


The Power of Intelligence

In our skulls we carry around 3 pounds of slimy, wet, greyish tissue, corrugated like crumpled toilet paper. You wouldn’t think, to look at the unappetizing lump, that it was some of the most powerful stuff in the known universe. If you’d never seen an anatomy textbook, and you saw a brain lying in the street, you’d say “Yuck!” and try not to get any of it on your shoes. Aristotle thought the brain was an organ that cooled the blood. It doesn’t look dangerous.
Five million years ago, the ancestors of lions ruled the day, the ancestors of wolves roamed the night. The ruling predators were armed with teeth and claws – sharp, hard cutting edges, backed up by powerful muscles. Their prey, in self-defense, evolved armored shells, sharp horns, poisonous venoms, camouflage. The war had gone on through hundreds of eons and countless arms races. Many a loser had been removed from the game, but there was no sign of a winner. Where one species had shells, another species would evolve to crack them; where one species became poisonous, another would evolve to tolerate the poison. Each species had its private niche – for who could live in the seas and the skies and the land at once? There was no ultimate weapon and no ultimate defense and no reason to believe any such thing was possible.
Then came the Day of the Squishy Things.

They had no armor. They had no claws. They had no venoms.
If you saw a movie of a nuclear explosion going off, and you were told an Earthly life form had done it, you would never in your wildest dreams imagine that the Squishy Things could be responsible. After all, Squishy Things aren’t radioactive.
In the beginning, the Squishy Things had no fighter jets, no machine guns, no rifles, no swords. No bronze, no iron. No hammers, no anvils, no tongs, no smithies, no mines. All the Squishy Things had were squishy fingers – too weak to break a tree, let alone a mountain. Clearly not dangerous. To cut stone you would need steel, and the Squishy Things couldn’t excrete steel. In the environment there were no steel blades for Squishy fingers to pick up. Their bodies could not generate temperatures anywhere near hot enough to melt metal. The whole scenario was obviously absurd.
And as for the Squishy Things manipulating DNA – that would have been beyond ridiculous. Squishy fingers are not that small. There is no access to DNA from the Squishy level; it would be like trying to pick up a hydrogen atom. Oh, technically it’s all one universe, technically the Squishy Things and DNA are part of the same world, the same unified laws of physics, the same great web of causality. But let’s be realistic: you can’t get there from here.
Even if Squishy Things could someday evolve to do any of those feats, it would take thousands of millennia. We have watched the ebb and flow of Life through the eons, and let us tell you, a year is not even a single clock tick of evolutionary time. Oh, sure, technically a year is six hundred trillion trillion trillion trillion Planck intervals. But nothing ever happens in less than six hundred million trillion trillion trillion trillion Planck intervals, so it’s a moot point. The Squishy Things, as they run across the savanna now, will not fly across continents for at least another ten million years; no one could have that much sex.
Now explain to me again why an Artificial Intelligence can’t do anything interesting over the Internet unless a human programmer builds it a robot body.
I have observed that someone’s flinch-reaction to “intelligence” – the thought that crosses their mind in the first half-second after they hear the word “intelligence” – often determines their flinch-reaction to the Singularity. Often they look up the keyword “intelligence” and retrieve the concept booksmarts – a mental image of the Grand Master chessplayer who can’t get a date, or a college professor who can’t survive outside academia.
“It takes more than intelligence to succeed professionally,” people say, as if charisma resided in the kidneys, rather than the brain. “Intelligence is no match for a gun,” they say, as if guns had grown on trees. “Where will an Artificial Intelligence get money?” they ask, as if the first Homo sapiens had found dollar bills fluttering down from the sky, and used them at convenience stores already in the forest. The human species was not born into a market economy. Bees won’t sell you honey if you offer them an electronic funds transfer. The human species imagined money into existence, and it exists – for us, not mice or wasps – because we go on believing in it.
I keep trying to explain to people that the archetype of intelligence is not Dustin Hoffman in The Rain Man, it is a human being, period. It is squishy things that explode in a vacuum, leaving footprints on their moon. Within that grey wet lump is the power to search paths through the great web of causality, and find a road to the seemingly impossible – the power sometimes called creativity.
People – venture capitalists in particular – sometimes ask how, if the Machine Intelligence Research Institute successfully builds a true AI, the results will be commercialized. This is what we call a framing problem.
Or maybe it’s something deeper than a simple clash of assumptions. With a bit of creative thinking, people can imagine how they would go about travelling to the Moon, or curing smallpox, or manufacturing computers. To imagine a trick that could accomplish all these things at once seems downright impossible – even though such a power resides only a few centimeters behind their own eyes.  The gray wet thing still seems mysterious to the gray wet thing.
And so, because people can’t quite see how it would all work, the power of intelligence seems less real; harder to imagine than a tower of fire sending a ship to Mars. The prospect of visiting Mars captures the imagination. But if one should promise a Mars visit, and also a grand unified theory of physics, and a proof of the Riemann Hypothesis, and a cure for obesity, and a cure for cancer, and a cure for aging, and a cure for stupidity – well, it just sounds wrong, that’s all.
And well it should. It’s a serious failure of imagination to think that intelligence is good for so little. Who could have imagined, ever so long ago, what minds would someday do? We may not even know what our real problems are.
But meanwhile, because it’s hard to see how one process could have such diverse powers, it’s hard to imagine that one fell swoop could solve even such prosaic problems as obesity and cancer and aging.
Well, one trick cured smallpox and built airplanes and cultivated wheat and tamed fire. Our current science may not agree yet on how exactly the trick works, but it works anyway. If you are temporarily ignorant about a phenomenon, that is a fact about your current state of mind, not a fact about the phenomenon. A blank map does not correspond to a blank territory. If one does not quite understand that power which put footprints on the Moon, nonetheless, the footprints are still there – real footprints, on a real Moon, put there by a real power. If one were to understand deeply enough, one could create and shape that power. Intelligence is as real as electricity. It’s merely far more powerful, far more dangerous, has far deeper implications for the unfolding story of life in the universe – and it’s a tiny little bit harder to figure out how to build a generator.
The post The Power of Intelligence appeared first on Machine Intelligence Research Institute.

The Power of Intelligence