October 2015 Newsletter

Research updates

New paper: Asymptotic Logical Uncertainty and The Benford Test
New at IAFF: Proof Length and Logical Counterfactuals Revisited; Quantilizers Maximize Expected Utility Subject to a Conservative Cost Constraint

General updates

As a way to engage more researchers in mathematics, logic, and the methodology of science, Andrew Critch and Tsvi Benson-Tilsen are currently co-running a seminar at UC Berkeley on Provability, Decision Theory and Artificial Intelligence.
We have collected links to a number of the posts we wrote for our Summer Fundraiser on intelligence.org/info.
German and Swiss donors can now make tax-advantaged donations to MIRI and other effective altruist organizations through GBS Switzerland.
MIRI has received Public Benefit Organization status in the Netherlands, allowing Dutch donors to make tax-advantaged donations to MIRI as well. Our tax reference number (RSIN) is 823958644.

News and links

Tech Times reports on the AI Impacts project.
Rise of Concerns About AI: Tom Dietterich and Eric Horvitz discuss long-term AI risk. See also Luke Muehlhauser’s response.
From the Open Philanthropy Project: a general update, and a discussion of the effects of AI progress on other global catastrophic risks.
There are many new job openings at GiveWell, the Centre for Effective Altruism, and the Future of Life Institute.

The post October 2015 Newsletter appeared first on Machine Intelligence Research Institute.

October 2015 Newsletter


New paper: “Asymptotic logical uncertainty and the Benford test”

We have released a new paper on logical uncertainty, co-authored by Scott Garrabrant, Siddharth Bhaskar, Abram Demski, Joanna Garrabrant, George Koleszarik, and Evan Lloyd: “Asymptotic logical uncertainty and the Benford test.”
Garrabrant gives some background on his approach to logical uncertainty on the Intelligent Agent Foundations Forum:
The main goal of logical uncertainty is to learn how to assign probabilities to logical sentences which have not yet been proven true or false.
One common approach is to change the question, assume logical omniscience and only try to assign probabilities to the sentences that are independent of your axioms (in hopes that this gives insight to the other problem). Another approach is to limit yourself to a finite set of sentences or deductive rules, and assume logical omniscience on them. Yet another approach is to try to define and understand logical counterfactuals, so you can try to assign probabilities to inconsistent counterfactual worlds.
One thing all three of these approaches have in common is they try to allow (a limited form of) logical omniscience. This makes a lot of sense. We want a system that not only assigns decent probabilities, but which we can formally prove has decent behavior. By giving the system a type of logical omniscience, you make it predictable, which allows you to prove things about it.
However, there is another way to make it possible to prove things about a logical uncertainty system. We can take a program which assigns probabilities to sentences, and let it run forever. We can then ask about whether or not the system eventually gives good probabilities.
At first, it seems like this approach cannot work for logical uncertainty. Any machine which searches through all possible proofs will eventually give a good probability (1 or 0) to any provable or disprovable sentence. To counter this, as we give the machine more and more time to think, we have to ask it harder and harder questions.
We therefore have to analyze the machine’s behavior not on individual sentences, but on infinite sequences of sentences. For example, instead of asking whether or not the machine quickly assigns 1/10 to the probability that the 3↑↑↑↑3rd digit of π is a 5 we look at the sequence:
an:= the probability the machine assigns at timestep 2n to the n↑↑↑↑nth digit of π being 5,
and ask whether or not this sequence converges to 1/10.
Benford’s law is the observation that the first digit in base 10 of various random numbers (e.g., random powers of 3) is likely to be small: the digit 1 comes first about 30% of the time, 2 about 18% of the time, and so on; 9 is the leading digit only 5% of the time. In their paper, Garrabrant et al. pick the Benford test as a concrete example of logically uncertain reasoning, similar to the π example: a machine passes the test iff it consistently assigns the correct subjective probability to “The first digit is a 1.” for the number 3 to the power f(n), where f is a fast-growing function and f(n) cannot be quickly computed.
Garrabrant et al.’s new paper describes an algorithm that passes the Benford test in a nontrivial way by searching for infinite sequences of sentences whose truth-values cannot be distinguished from the output of a weighted coin.
In other news, the papers “Toward idealized decision theory” and “Reflective oracles: A foundation for classical game theory” are now available on arXiv. We’ll be presenting a version of the latter paper with a slightly altered title (“Reflective oracles: A foundation for game theory in artificial intelligence”) at LORI-V next month.
Update June 12, 2016: “Asymptotic logical uncertainty and the Benford test” has been accepted to AGI-16.

Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.

×

The post New paper: “Asymptotic logical uncertainty and the Benford test” appeared first on Machine Intelligence Research Institute.


September 2015 Newsletter

Research updates

New analyses: When AI Accelerates AI; Powerful Planners, Not Sentient Software
New at AI Impacts: Research Bounties; AI Timelines and Strategies
New at IAFF: Uniform Coherence 2; The Two-Update Problem
Andrew Critch, a CFAR cofounder, mathematician, and former Jane Street trader, joined MIRI as our fifth research fellow this month!
As a result of our successful fundraiser and summer workshop series, I’m happy to announce that we’re hiring two additional full-time researchers later in 2015: Mihály Bárász and Scott Garrabrant!

General updates

We’ve wrapped up our largest fundraiser ever: 258 donors brought in a total of $629,123! Thanks to you, we’ve hit our first two fundraising goals and received $129,123 that will go toward our third goal: Taking MIRI to the Next Level.
“If you agree AI matters, why MIRI?” Two new replies to this question: Assessing Our Past and Potential Impact and What Sets MIRI Apart?
We attended the Effective Altruism Global conference. Background: AI and Effective Altruism.
We’re moving to a larger office in the same building this October. If you have experience in office planning or access control systems and would like to help out with our plans, shoot Malo an email at malo@intelligence.org.

News and links

GiveWell publishes its initial review of potential risks from advanced artificial intelligence, as well as a new article on the long-term significance of reducing global catastrophic risks.
Stuart Russell explains the AI alignment problem in a San Francisco talk and in a (paywalled) interview in Science.
A much earlier discussion of the AI control problem in a 1960 Science article.
From Vox: Robots Aren’t Taking Your Jobs — And That’s the Problem.

The post September 2015 Newsletter appeared first on Machine Intelligence Research Institute.

September 2015 Newsletter


Our summer fundraising drive is complete!

Our summer fundraising drive is now finished. We raised a grand total of $631,957, from 263 donors.1 This is an incredible sum, and your support has made this the biggest fundraiser we’ve ever run.

We’ve already been hard at work growing our research team and spinning up new projects, and I’m excited to see what our research team can do this year. Thank you for making our summer fundraising drive so successful!
That total may change over the next few days if we receive contributions that were initiated before the end the fundraiser.The post Our summer fundraising drive is complete! appeared first on Machine Intelligence Research Institute.


Final fundraiser day: Announcing our new team

Today is the final day of MIRI’s summer fundraising drive, and as of this morning, our total stands at $543,373. Our donors’ efforts have made this fundraiser the biggest one we’ve ever run, and we’re hugely grateful.
As our fundraiser nears the finish line, I’d like to update you on the new shape of MIRI’s research team. We’ve been actively recruiting throughout the fundraiser, and we are taking on three new full-time researchers in 2015.
At the beginning of the fundraiser, we had three research fellows on our core team: Eliezer Yudkowsky, Benja Fallenstein, and Patrick LaVictoire. Eliezer is one of MIRI’s co-founders, and Benja joined the team a little over a year ago (in March 2014). Patrick is a newer recruit; he joined in March of 2015. He has a mathematics PhD from U.C. Berkeley, and he has industry experience from Quixey doing applied machine learning and data science. He’s responsible for some important insights into our open problems, and he’s one of the big reasons why our summer workshops have been running so smoothly.
On August 1st, Jessica Taylor became the fourth member of our core research team. She recently completed a master’s degree in computer science at Stanford, where she studied machine learning and probabilistic programming. Jessica is quite interested in AI alignment, and has been working with MIRI in her spare time for many months now. Already, she’s produced some exciting research, and I’m delighted to have her on the core research team.
Meanwhile, over the course of the fundraiser, we’ve been busy expanding the team. Today, I’m happy to announce our three newest hires!

Andrew Critch is joining our research team tomorrow, September 1. Andrew earned his PhD in mathematics at UC Berkeley studying applications of algebraic geometry to machine learning models. He cofounded the Center for Applied Rationality and SPARC, and previously worked as an algorithmic stock trader at Jane Street Capital. In addition to his impressive skills as a mathematician, Andrew Critch has a knack for explaining complex ideas. I expect that he will be an important asset as we ramp up our research program. On a personal level, I expect his infectious enthusiasm to be handy for getting members of the AI community excited about our research area.

Mihály Bárász, a former Google engineer, will be joining MIRI in the fall. Mihály has an MSc summa cum laude in mathematics from Eotvos Lorand University, Budapest. Mihály attended MIRI’s earliest workshops, and is the lead author of the paper “Robust Cooperation in the Prisoner’s Dilemma: Program Equilibrium via Provability Logic.” He’s a brilliant mathematician (with a perfect score at the International Math Olympiad) who has worked with us a number of times in the past, and we’re very excited by the prospect of having him on the core research team.

Scott Garrabrant is joining MIRI toward the end of 2015, after completing a mathematics PhD at UCLA. He is currently studying applications of theoretical computer science to enumerative combinatorics. Scott was one of the most impressive attendees of the MIRI Summer Fellows Program, and has been steadily producing a large number of new technical results on the Intelligent Agent Foundations Forum. I’m thrilled to have him working on these issues full-time.

We’ve already begun executing on some of our other fundraiser goals, as well. Over the last few weeks, we have brought Jack Gallagher on as an intern to begin formalizing in type theory certain tools that MIRI has developed (described briefly in this post). His code can be found in a few different repositories on github. We’ve also brought on another intern, Kaya Stechly, to help us write up some of the many new results that we haven’t yet had the time to polish.
I’m eager to see what this new team can do going forward. Meanwhile, there are even more recruitment opportunities and projects that we’d like to undertake, given sufficient funding. Further donations at this point would allow us to grow more quickly and more securely. Over the course of the fundraiser, we’ve laid out a number of reasons why we think MIRI’s growth is important:

Four Background Claims explains why we think AI will have an increasingly large impact as it begins to outperform humans in general reasoning tasks.
Assessing Our Past and Potential Impact and What Sets MIRI Apart? argue that MIRI is unusually well-positioned to help make the long-term impact of AI positive.
MIRI’s Approach explains why we think our technical agenda is tractable and highly important.
An Astounding Year and Why Now Matters note that the interest in AI safety work is booming, and this is a critical time for MIRI to have a big impact on early AI alignment discussions.
And Target 1, Target 2, and Target 3 detail what we would use additional funding for.

We’ve made our case, and our donors have come through in a big way. However, our funding gap isn’t closed yet, and additional donors over the next few hours can still make a difference in deciding which of our future plans we can begin executing on.
To all our supporters: Thank you for helping us make our expansion plans a reality! We owe this new growth to you. Now let’s see what we can do with one more day!

Update 12/3/15: Mihály Bárász has deferred his research fellowship, and now plans to join MIRI’s research team in 2016 instead of late 2015.

The post Final fundraiser day: Announcing our new team appeared first on Machine Intelligence Research Institute.


AI and Effective Altruism

MIRI is a research nonprofit specializing in a poorly-explored set of problems in theoretical computer science. GiveDirectly is a cash transfer service that gives money to poor households in East Africa. What kind of conference would bring together representatives from such disparate organizations — alongside policy analysts, philanthropists, philosophers, and many more?
Effective Altruism Global, which is beginning its Oxford session in a few hours, is that kind of conference. Effective altruism (EA) is a diverse community of do-gooders with a common interest in bringing the tools of science to bear on the world’s biggest problems. EA organizations like GiveDirectly, the Centre for Effective Altruism, and the charity evaluator GiveWell have made a big splash by calling for new standards of transparency and humanitarian impact in the nonprofit sector.
What is MIRI’s connection to effective altruism? In what sense is safety research in artificial intelligence “altruism,” and why do we assign a high probability to this being a critically important area of computer science in the coming decades? I’ll give quick answers to each of those questions below.

MIRI and effective altruism
Why is MIRI associated with EA? In large part because effective altruists and MIRI use the same kind of criteria in deciding what work to prioritize.
MIRI’s mission, to develop the formal tools needed to make smarter-than-human AI systems useful and safe, comes from our big-picture view that scientific and technological advances will be among the largest determiners of human welfare, as they have been historically. Automating intellectual labor is therefore likely to be a uniquely high-impact line of research — both for good and for ill. (See Four Background Claims.) Which open problems we work on then falls out of our efforts to identify tractable and neglected theoretical prerequisites for aligning the goals of AI systems with our values. (See MIRI’s Approach.)

Daniel Dewey, Nick Bostrom, Elon Musk, Nate Soares, and Stuart Russelldiscuss AI risk at the EA Global conference. Photo by Robbie Shade.

MIRI is far from the only group that uses criteria like these to identify important cause areas and interventions, and these groups have found that banding together is a useful way to have an even larger impact. Because members of these groups aren’t permanently wedded to a single cause area, and because we assign a lot of value to our common outlook in its own right, we can readily share resources and work together to promote the many exciting ideas that are springing out from this outlook. Hence the effective altruist community.
One example of this useful exchange was MIRI’s previous Executive Director, Luke Muehlhauser, leaving MIRI in June to investigate nutrition science and other areas for potential philanthropic opportunities under the Open Philanthropy Project, an offshoot of GiveWell.1 In turn, OpenPhil has helped fund a large AI grants program that MIRI participated in.
GiveWell/OpenPhil staff have given us extremely useful critical feedback in the past, and we’ve had a number of conversations with them over the years (1, 2, 3, 4, 5). Although they work on a much broader range of topics than MIRI does and they don’t share all of our views, their interest in finding interventions that are “important, tractable and relatively uncrowded” has led them to pick out AI as an important area to investigate for reasons that overlap with MIRI’s. (See OpenPhil’s March update on global catastrophic risk and their newly released overview document on potential risks from advanced artificial intelligence.)
Most EAs work on areas other than AI risk, and MIRI’s approach is far from the only plausible way to have an outsized impact on human welfare. Because we attempt to base our decisions on broadly EA considerations, however — and therefore end up promoting EA-like philosophical commitments when we explain the reasoning behind our research approach — we’ve ended up forming strong ties to many other people with an interest in identifying high-impact humanitarian interventions.
High-stakes and high-probability risks
A surprisingly common misconception about EA cause areas is that they break down into three groups: high-probability crises afflicting the global poor; medium-probability crises afflicting non-human animals; and low-probability global catastrophes. The assumption (for example, in Dylan Matthews’ recent Vox article) is that this is the argument for working on AI safety or biosecurity: there’s a very small chance of disaster occurring, but disaster would be so terrible if it did occur that it’s worth investigating just in case.
This misunderstands MIRI’s position — and, I believe, the position of people interested in technological risk at the Future of Humanity Institute and a number of other organizations. We believe that existential risk from misaligned autonomous AI systems is high-probability if we do nothing to avert it, and we base our case for MIRI on that view; if we thought that the risks from AI were very unlikely to arise, we would deprioritize AI alignment research in favor of other urgent research projects.
As a result, we expect EAs who strongly disagree with us about the likely future trajectory of the field of AI to work on areas other than AI risk. We don’t think EAs should donate to MIRI “just in case,” and we reject arguments based on “Pascal’s Mugging.” (“Pascal’s Mugging” is the name MIRI researchers coined for decision-making that mistakenly focuses on infinitesimally small probabilities of superexponentially vast benefits.)2
As Stuart Russell writes, “Improving decision quality, irrespective of the utility function chosen, has been the goal of AI research – the mainstream goal on which we now spend billions per year, not the secret plot of some lone evil genius.” Thousands of person-hours are pouring into research to increase the general capabilities of AI systems, with the aim of building systems that can outperform humans in arbitrary cognitive tasks.
We don’t know when such efforts will succeed, but we expect them to succeed eventually — possibly in the next few decades, and quite plausibly during this century. Shoring up safety guarantees for autonomous AI systems would allow us to reap many more of the benefits from advances in AI while significantly reducing the probability of a global disaster over the long term.
MIRI’s mission of making smarter-than-human AI technology reliably beneficial is ambitious, but it’s ambitious in the fashion of goals like “prevent global warming” or “abolish factory farming.” Working toward such goals usually means making incremental progress that other actors can build on — more like setting aside $x of each month’s paycheck for a child’s college fund than like buying a series of once-off $x lottery tickets.
A particular $100 is unlikely to make a large once-off impact on your child’s career prospects, but it can still be a wise investment. No single charity working against global warming is going to solve the entire problem, but that doesn’t make charitable donations useless. Although MIRI is a small organization, our work represents early progress toward more robust, transparent, and beneficial AI systems, which can then be built on by other groups and integrated into AI system design.3
Rather than saying that AI-mediated catastrophes are high-probability and stopping there, though, I would say that such catastrophes are high-probability conditional on AI research continuing on its current trajectory. Disaster isn’t necessarily high-probability if the field of AI shifts to include alignment work along with capabilities work among its key focuses.
It’s because we consider AI disasters neither unlikely nor unavoidable that we think technical work in this area is important. From the perspective of aspiring effective altruists, the most essential risks to work on will be ones that are highly likely to occur in the near future if we do nothing, but substantially less likely to occur if we work on the problem and get existing research communities and scientific institutions involved.
Principles like these apply outside the domain of AI, and although MIRI is currently the only organization specializing in long-term technical research on AI alignment, we’re one of a large and growing number of organizations that attempt to put these underlying EA principles into practice in one fashion or another. And to that extent, although effective altruists disagree about the best way to improve the world, we ultimately find ourselves on the same team.

Although effective altruism is sometimes divided into separate far-future, animal welfare, global poverty, and “meta” cause areas, this has always been a somewhat artificial division. Toby Ord, the founder of the poverty relief organization Giving What We Can, is one of the leading scholars studying existential risk and holds a position at the Future of Humanity Institute. David Pearce, one of the strongest proponents of animal activism within EA, is best known for his futurism. Peter Singer is famous for his early promotion of global poverty causes as well as his promotion of animal welfare. And Anna Salamon, the Executive Director of the “meta”-focused Center for Applied Rationality, is a former MIRI researcher.Quoting MIRI senior researcher Eliezer Yudkowsky in 2013:
I abjure, refute, and disclaim all forms of Pascalian reasoning and multiplying tiny probabilities by large impacts when it comes to existential risk.  We live on a planet with upcoming prospects of, among other things, human intelligence enhancement, molecular nanotechnology, sufficiently advanced biotechnology, brain-computer interfaces, and of course Artificial Intelligence in several guises.  If something has only a tiny chance of impacting the fate of the world, there should be something with a larger probability of an equally huge impact to worry about instead. […]
To clarify, “Don’t multiply tiny probabilities by large impacts” is something that I apply to large-scale projects and lines of historical probability.  On a very large scale, if you think FAI [Friendly AI] stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody’s dumping effort into it then you should dump more effort than currently into it.  On a smaller scale, to compare two x-risk mitigation projects in demand of money, you need to estimate something about marginal impacts of the next added effort (where the common currency of utilons should probably not be lives saved, but “probability of an ok outcome”, i.e., the probability of ending up with a happy intergalactic civilization).  In this case the average marginal added dollar can only account for a very tiny slice of probability, but this is not Pascal’s Wager.  Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginally increased probabilities of success per added small unit of effort.  It would only be Pascal’s Wager if the whole route-to-an-OK-outcome were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.
Nick Bostrom made a similar point at EA Global: that AI is an important cause even though any one individual’s actions are unlikely to make a decisive difference. In a panel on artificial superintelligence, Bostrom said that he thought people had a “low” (as opposed to “high” or “medium”) probability of making a difference on AI risk, which Matthews and a number of others appear to have taken to mean that Bostrom thinks AI is a speculative cause area. When I asked Bostrom about his intended meaning myself, however, he elaborated:
The point I was making in the EA global comment was the probability that you (for any ‘you’ in the audience) will save the world from an AI catastrophe is very small, not that the probability of AI catastrophe is very small. Thus working on AI risk is similar to volunteering for a presidential election campaign.
The post AI and Effective Altruism appeared first on Machine Intelligence Research Institute.

AI and Effective Altruism


Powerful planners, not sentient software

Over the past few months, some major media outlets have been spreading concern about the idea that AI might spontaneously acquire sentience and turn against us. Many people have pointed out the flaws with this notion, including Andrew Ng, an AI scientist of some renown:
I don’t see any realistic path from the stuff we work on today—which is amazing and creating tons of value—but I don’t see any path for the software we write to turn evil.
He goes on to say, on the topic of sentient machines:
Computers are becoming more intelligent and that’s useful as in self-driving cars or speech recognition systems or search engines. That’s intelligence. But sentience and consciousness is not something that most of the people I talk to think we’re on the path to.
I say, these objections are correct. I endorse Ng’s points wholeheartedly — I see few pathways via which software we write could spontaneously “turn evil.”
I do think that there is important work we need to do in advance if we want to be able to use powerful AI systems for the benefit of all, but this is not because a powerful AI system might acquire some “spark of consciousness” and turn against us. I also don’t worry about creating some Vulcan-esque machine that deduces (using cold mechanic reasoning) that it’s “logical” to end humanity, that we are in some fashion “unworthy.” The reason to do research in advance is not so fantastic as that. Rather, we simply don’t yet know how to program intelligent machines to reliably do good things without unintended consequences.
The problem isn’t Terminator. It’s “King Midas.” King Midas got exactly what he wished for — every object he touched turned to gold. His food turned to gold, his children turned to gold, and he died hungry and alone.
Powerful intelligent software systems are just that: software systems. There is no spark of consciousness which descends upon sufficiently powerful planning algorithms and imbues them with feelings of love or hatred. You get only what you program.1

To build a powerful AI software system, you need to write a program that represents the world somehow, and that continually refines this world-model in response to percepts and experience. You also need to program powerful planning algorithms that use this world-model to predict the future and find paths that lead towards futures of some specific type.
The focus of our research at MIRI isn’t centered on sentient machines that think or feel as we do. It’s aimed towards improving our ability to program software systems to execute plans leading towards very specific types of futures.
A machine programmed to build a highly accurate world-model and employ powerful planning algorithms could yield extraordinary benefits. Scientific and technological innovation have had great impacts on quality of life around the world, and if we can program machines to be intelligent in the way that humans are intelligent — only faster and better — we can automate scientific and technological innovation. When it comes to the task of improving human and animal welfare, that would be a game-changer.
To build a machine that attains those benefits, the first challenge is to do this world-modeling and planning in a highly reliable fashion: you need to ensure that it will consistently pursue its goal, whatever that is. If you can succeed at this, the second challenge is making that goal a safe and useful one.
If you build a powerful planning system that aims at futures in which cancer is cured, then it may well represent all of the following facts in its world-model: (a) The fastest path to a cancer cure involves proliferating robotic laboratories at the expense of the biosphere and kidnapping humans for experimentation; (b) once you realize this, you’ll attempt to shut it down; and (c) if you shut it down, it will take a lot longer for cancer to be cured. The system may then execute a plan which involves deceiving you until it is able to resist and then proliferating robotic laboratories and kidnapping humans. This is, in fact, what you asked for.
We can avoid this sort of outcome, if we manage to build machines that do what we mean rather than what we said. That sort of behavior doesn’t come for free: you have to program it in.
A superhuman planning algorithm with an extremely good model of the world could find solutions you never imagined. It can make use of patterns you haven’t noticed and find shortcuts you didn’t recognize. If you follow a plan generated by a superintelligent search process, it could have disastrous unintended consequences. To quote professor Stuart Russell (author of the leading AI textbook):
The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:
1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.
Humans have a lot of fiddly little constraints akin to “oh, and don’t kidnap any humans while you’re curing cancer”. Programming in a full description of human values and human norms by hand, in a machine-readable format, doesn’t seem feasible. If we want the plans generated by superhuman planning algorithms to respect all of our complicated unspoken constraints and desires, then we’ll need to develop new tools for predicting and controlling the behavior of general-purpose autonomous agents. There’s no two ways about it.

Many people, when they first encounter this problem, come up with a reflexive response about why the problem won’t be as hard as it seems. One common one is “If a powerful planner starts running amok, we can just unplug it” — an objection which is growing obsolete in the era of cloud computing, and which fails completely if the system has access to the internet or any other network where it can copy itself onto other machines.
Another common one is “Why not have the system output a plan rather than having it execute the plan?” — but if we direct a powerful planning procedure to generate plans such that (a) humans who examine the plan approve of it and (b) executing it leads to cancer being cured, then the plan may well be one that looks good but which exploits some predictable oversight in the verification procedure and kidnaps people anyway.
Or you could say, “How about we just make systems which only answer questions?” But how exactly do you direct a superhuman planning procedures towards “answering questions”? Will you program it to output text that it predicts will cause you to press the “highly satisfied” button after the answer has been output? Because in that case, the system may well output text that constitutes a particularly deceptive answer. Or, if you add a constraint that the answer must be accurate, it may output text that manipulates you into asking easier questions in the future.
Maybe you reply, “Well, perhaps instead I’ll direct the planner to move toward futures where its output is measured by this clever metric where…,” and now you’ve been drawn in. How exactly could we build powerful planers that search for beneficial futures? It looks like it’s possible to build systems that somehow learn the user’s intentions or values and act according to them, but actually doing so is not trivial. You’ve got to think hard to build systems that figure out all the intricacies of your intentions without deceiving or manipulating you while acquiring that information. That doesn’t happen for free: ambitious, long-term software projects are still ultimately software projects, and we have to figure out how to actually write the required code.
If we can figure out how to build smarter-than-human machines aligned with our interests, the benefits could be extraordinary. Like Phil Libin (founder of Evernote) says, AI could be “one of the greatest forces for good the universe has ever seen.” It’s possible to get there, but it’s going to require some work.

You could likely program an AI system to be conscious, which would greatly complicate the situation — for then the system itself would be a moral patient, and its preferences would weigh into our considerations. As Ng notes, however, “consciousness” is not the same thing as “intelligence.”The post Powerful planners, not sentient software appeared first on Machine Intelligence Research Institute.


What Sets MIRI Apart?

Last week, we received several questions from the effective altruist community in response to our fundraising post. Here’s Maxwell Fritz:
[…] My snap reaction to MIRI’s pitches has typically been, “yeah, AI is a real concern. But I have no idea whether MIRI are the right people to work on it, or if their approach to the problem is the right one” [… I]f you agree AI matters, why MIRI?
And here are two more questions in a similar vein, added by Tristan Tager:
[… W]hat can MIRI do? Why should I expect that the MIRI vision and the MIRI team are going to get things done? What exactly can I expect them to get done? […]
But the second and much bigger question is, what would MIRI do that Google wouldn’t? Google has a ton of money, a creative and visionary staff, the world’s best programmers, and a swath of successful products that incorporate some degree of AI — and moreover they recently acquired several AI businesses and formed an AI ethics board. It seems like they’re approaching the same big problem directly rather than theoretically, and have deep pockets, keen minds, and a wealth of hands-on experience.
These are great questions. My answer to “Why MIRI?”, in short, is that MIRI has a brilliant team of researchers focused on the fundamental theoretical research that almost nobody else is pursuing. We’re focused entirely on aligning smarter-than-human AI systems with humane values, for the long haul.
Most academics aren’t working on AI alignment problems yet, and none are doing it full-time. Most industry folks aren’t working on these problems yet, either. I know this because I’m in conversations with a number of them. (The field is large, but it isn’t that large.)
There are quite a few good reasons why academics and industry professionals aren’t working on these problems yet, and I’ll touch on a few of them in turn.

1. Most AI scientists focus predominantly on the short and medium term.
This makes sense: the field of AI has been burned more than once in the past by over-promising and under-delivering, and history shows that it’s often easier to do AI work when one keeps an eye on goals that are achievable within the next few years.
Many AI scientists have incentives to stick to practical work, look to the short term, and make only realistic promises. (This is evidenced in part by the field’s recent focus on “machine learning,” which has a narrower and more immediate focus than the “old-fashioned” field of AI.) This has paid off, and industry and academia both have made continual incremental breakthroughs that have led to some amazing new technologies. However, it has also made long-term considerations somewhat toxic, and “artificial general intelligence” has been a taboo topic of sorts in recent years. This sentiment is starting to shift, but most academics are still loath to work on foundational problems pertaining to artificial general intelligence, preferring research that improves the capabilities of practical systems today.

2. We’re solving a different sort of problem.
In MIRI’s Approach I spoke of two different classes of computer science problem. Class 1 problems involve figuring out how to do, in practice and with reasonable amounts of computing power, things which we know how to do in principle. Class 2 problems involve figuring out how to do in principle things that we can’t even do in principle yet.
Our current approach to alignment research is to try to move problems from Class 2 to Class 1. This kind of research has been pursued successfully in other areas in the past, and in the context of AI alignment I believe that it deserves significantly more attention than it is receiving.
Industry is traditionally best suited for the first problem class. Academia, too, also often focuses on the first class of problems instead of the second class — especially in the field of AI, for reasons related to point 1. It is common for academics to take some formalization of something like probability theory and then explore and extend the framework, figuring out where it applies and developing practical approximations of intractable algorithms and so on. It’s much rarer for academics to create theoretical foundations for problems that cannot yet be solved even in principle, and this tends to happen only when someone is searching for new theoretical foundations on purpose. For reasons discussed above, most academics aren’t attempting this sort of research yet when it comes to AI alignment.
This is what MIRI brings to the table: a laser focus on the relevant technical challenges.

3. We don’t have competing interests or priorities.
Academic research tends to follow the contours of tractability and curiosity, wherever they may lead. Industry research tends to follow short- and medium-term profits. It is not obvious that either of these will zero in on the AI alignment problems we raise in our technical agenda.1
By contrast, MIRI researchers focus full-time on the most important AI alignment problems they can identify, without taking breaks to teach classes, pursue other interesting theoretical questions, or tackle more immediately profitable or publishable topics. We aren’t going to switch to a different set of problems in order to win a grant; and when we win a three-year grant, we aren’t going to switch to a different set of problems as soon as the grant expires. We will simply continue working on the most important technical problems we can find.
Our mission is to solve the hard technical problems of AI alignment, and our current approach is to zero in on the open problems laid out in our research agenda, without distraction.

The takeaway is that at least in the near term, solutions to the specific problems we’re looking at — generated by the question “what would we still be unable to solve, even if the alignment problem were simpler?” — are most likely to come from MIRI and our collaborators.

Tristan asked: “What would MIRI do that Google wouldn’t?” The answer is: the fundamental theoretical research. I talk to industry folks fairly regularly about what they’re working on and about what we’re working on. Over and over, the reaction I get to our work is something along the lines of “Ah, yes, those are very important questions. We aren’t working on those, but it does seem like we’re missing some useful tools there. Let us know if you find some answers.”
Or, just as often, the response we get is some version of “Well, yes, that tool would be awesome, but getting it sounds impossible,” or “Wait, why do you think we actually need that tool?”2 Regardless, the conversations I’ve had tend to end the same way for all three groups: “That would be a useful tool if you can develop it; we aren’t working on that; let us know if you find some answers.”
Industry is interested in pushing modern practical systems towards increasingly general applications. They’re focused on problems such as hierarchical planning, continual learning, and transfer learning. When they work on safety research (and they often do!), they work on tools that improve the transparency of deep neural networks, and they work on improving the calibration and decreasing the error rates in their systems. Our relations with industry AI scientists are good, and we keep tabs on each other. But major industry groups simply aren’t competing with MIRI in AI alignment research; they aren’t doing our type of work at the moment.
It’s also worth keeping in mind that MIRI researchers tend to be agnostic about AI timelines. Last year, the question du jour was “why do you think your research can matter, when we’re so far off from smarter-than-human AI that it’s impossible to know what architecture will be used for machine intelligence?” This year, the question du jour is “why do you think your research can matter, when Google is obviously going to develop smarter-than-human AI using artificial neural networks?”
My answer has remained similar in both cases: we’re developing basic conceptual tools, akin to probability theory, that are likely to improve the transparency, verifiability, stability, and overall safety of AGI systems across a variety of different architectures. My timelines are about the same today as they were last year: I don’t expect to see smarter-than-human AI systems in the next decade, and I do expect to see them this century. It is not obvious to me when smarter-than-human AI systems will be developed, or how, or by whom.
A decade is a lot of time in industry, and a century is an eternity. The industry leaders today may look quite strong, and some groups may be developing a strong lead over others. However, I’m not nearly confident enough in any one group to put all our chips on their success.
MIRI is doing foundational research to develop tools that will help all industry groups design safer systems. Our goal is to make it easier to design robust and beneficial general-purpose reasoners from the ground up, and I think it’s valuable for those sorts of tools to be developed by an impartial third party that answers to the public rather than to its shareholders.

Superintelligence probably isn’t just around the corner. In the interim, we’re working on a different set of problems than existing groups in industry and academia; most teams aren’t focused on foundational theoretical research on long-term AI alignment problems.
MIRI is. The open problems we work on are underserved, and we’re well-positioned to make inroads on the most critical technical obstacles lying ahead. Our hope is that numerous brilliant and skilled organizations will take on these technical challenges as well, and that our work on these topics will help the research area grow more quickly. I would deeply appreciate more competition in this space. Until then, if you want there to be at least one group whose mission is tackling the “Class 2” technical research, MIRI is the team to fund.

Indeed, the problems in the technical agenda were chosen in part because they seem important but don’t appear to be on the default path.There isn’t much correlation between which of our open problems gets which of these three answers, across AI scientists.The post What Sets MIRI Apart? appeared first on Machine Intelligence Research Institute.

What Sets MIRI Apart?


Assessing our past and potential impact

We’ve received several thoughtful questions in response to our fundraising post to the Effective Altruism Forum and our new FAQ. From quant trader Maxwell Fritz:
My snap reaction to MIRI’s pitches has typically been, “yeah, AI is a real concern. But I have no idea whether MIRI are the right people to work on it, or if their approach to the problem is the right one”.
Most of the FAQ and pitch tends to focus on the “does this matter” piece. It might be worth selling harder on the second component – if you agree AI matters, why MIRI?
At that point, there’s two different audiences – one that has the expertise in the field to make a reasoned assessment based on the quality of your existing work, and a second that doesn’t have a clue (me) and needs to see a lot of corroboration from unaffiliated, impressive sources (people in that first group).
The pitches tend to play up famous people who know their shit and corroborate AI as a concern – but should especially make it clear when those people believe in MIRI. That’s what matters for the “ok, why you?” question. And the natural follow up is if all of these megarich people are super on board with the concern of AI, and experts believe MIRI should lead the charge, why aren’t you just overflowing with money already?
And from mathematics grad student Tristan Tager:
I would guess that “why MIRI”, rather than “who’s MIRI” or “why AI”, is the biggest marketing hurdle you guys should address.
For me, “why MIRI” breaks down into two questions. The first and lesser question is, what can MIRI do? Why should I expect that the MIRI vision and the MIRI team are going to get things done? What exactly can I expect them to get done? Most importantly in addressing this question, what have they done already and why is it useful? The Technical Agenda is vague and mostly just refers to the list of papers. And the papers don’t help much — those who don’t know much about academia need something more accessible, and those who do know more about academia will be skeptical about MIRI’s self-publishing and lack of peer review.
But the second and much bigger question is, what would MIRI do that Google wouldn’t? Google has tons of money, a creative and visionary staff, the world’s best programmers, and a swath of successful products that incorporate some degree of AI — and moreover they recently acquired several AI businesses and formed an AI ethics board. It seems like they’re approaching the same big problem directly rather than theoretically, and have deep pockets, keen minds, and a wealth of hands-on experience.
There are a number of good questions here. Later this week, Nate plans to post a response to Tristan’s last question: Why is MIRI currently better-positioned to work on this problem than AI groups in industry or academia? (Update February 17: Link here.)
Here, I want to reply to several other questions Tristan and Maxwell raised:

How can non-specialists assess MIRI’s research agenda and general competence?
What kinds of accomplishments can we use as measures of MIRI’s past and future success?
And lastly: If a lot of people take this cause seriously now, why is there still a funding gap?

General notes
When we make our case for MIRI, we usually focus on the reasons to consider AI an important point of leverage on existential risk (Nate’s four background claims, Eliezer’s five theses) and for thinking that early theoretical progress is possible in this area (MIRI’s Approach).
We focus on these big-picture arguments because the number of people working on this topic is still quite small. The risk scenario MIRI works on has only risen to national attention in the last 6-12 months; at the moment, MIRI is the only research organization I know of that is even claiming to specialize in early technical research on the alignment problem.
There are multiple opportunities to support technical research into long-term AI safety at the scale of “funding individual researcher X to work on discrete project Y.” Some recipients of the 2015 Future of Life Institute (FLI) grants fall into this category, e.g., Stuart Russell and Paul Christiano.
However, there aren’t multiple opportunities in this area at the scale of “funding an AI group to pursue a large-scale or long-term program,” and there aren’t many direct opportunities to bring in entirely new people and grow and diversify the field. MIRI would love to have organizational competitors (and collaborators) in this space, but they don’t yet exist. We expect this to change eventually, and one of our expansion goals is to make this happen faster, by influencing other math and computer science research groups to take on more AI alignment work and by recruiting highly qualified specialists from outside the existential risk community to become career AI alignment researchers.
The upside of getting started early is that we have a chance to have a larger impact in a less crowded space. The downside is that there are fewer authoritative sources to appeal to when outsiders want to verify that our research agenda is on the right track.
For the most part, those authoritative sources will probably need to wait a year or two. Academia is slow, and at this stage many computer scientists have only been aware of this area for a few months. Our mission is seen as important by a growing number of leaders in science and industry, and our technical agenda is seen by a number of AI specialists as promising and deserving of more attention — hence its inclusion in the FLI research priorities document. But we don’t expect the current state of the evidence to be universally convincing to our most skeptical potential donors.
For those who want detailed independent assessments of MIRI’s output, our advice is to wait a bit for the wider academic community to respond. (We’re also looking into options for directly soliciting public feedback from independent researchers regarding our research agenda and early results.)
In the interim, however, “Why Now Matters” notes reasons donations to MIRI are likely to have a much larger impact now than they would several years down the line. For donors who are skeptical (or curious), but are not so skeptical that they require a fine-grained evaluation of our work by the scholarly community, I’ll summarize some of the big-picture reasons to think MIRI’s work is likely to be high-value.1
Influence on the AI conversation
In the absence of in-depth third-party evaluations of our work, interested non-specialists can look at our publication history and the prominence of our analyses in scholarly discussions of AI risk.
MIRI’s most important accomplishments fall into three categories: writing up top-priority AI alignment problems; beginning early work on these problems; and getting people interested in our research and our mission. Our review of the last year discusses our recent progress in formalizing alignment problems, as well as our progress in getting the larger AI community interested in long-term AI safety. Our publications list gives a more detailed and long-term picture of our output.
MIRI co-founder and senior researcher Eliezer Yudkowsky and Future of Humanity Institute (FHI) founding director Nick Bostrom are responsible for much of the early development and popularization of ideas surrounding smarter-than-human AI risk. Eliezer’s ideas are cited prominently in the 2009 edition of Artificial Intelligence: A Modern Approach (AI:MA), the leading textbook in the field of AI, and also in Bostrom’s 2014 book Superintelligence.
Credit for more recent success in popularizing long-term AI safety research is shared between MIRI and a number of actors: Nick Bostrom and FHI, Max Tegmark and FLI, Stuart Russell (co-author of AI:MA), Jaan Tallinn, and others. Many people in this existential-risk-conscious cluster broadly support MIRI’s efforts and are in regular contact with us about our decisions. Bostrom, Tegmark, Russell, and Tallinn are all MIRI advisors, and Tallinn, a co-founder of FLI and of the Cambridge Centre for the Study of Existential Risk (CSER), cites MIRI as a key source for his views on AI risk.
Writing in early 2014, Russell and Tegmark, together with Stephen Hawking and Frank Wilczek, noted in The Huffington Post that “little serious research is devoted to these issues [of long-term AI risk] outside small non-profit institutes such as the Cambridge Center for Existential Risk, the Future of Humanity Institute, the Machine Intelligence Research Institute, and the Future of Life Institute.”2 Of these organizations, MIRI is the one that currently specializes in the technical obstacles to designing safe, beneficial smarter-than-human AI systems. FHI, CSER, and FLI do important work on a broader set of issues, including forecasting and strategy work, outreach, and investigations into other global risks.
Publications
Tristan raised the concern that MIRI’s technical agenda was self-published. Reviewing our past publications, MIRI has self-published many of its results. More than once, we’ve had the experience of seeing papers rejected with comments that the results are interesting but the AI motivation is just too strange. We’ve begun submitting stripped-down papers and putting the full versions on arXiv, but figuring out the best way to get these results published took some trial and error.
Part of the underlying problem is that the AI field has been repeatedly burned by “winters” when past generations over-promised and under-delivered. Members of the field are often uncomfortable looking too far ahead, and have historically been loathe to talk about general intelligence.
Our approach is exactly the opposite: we focus directly on trying to identify basic aspects of general reasoning that are still not well-understood, while explicitly avoiding safety research focused on present-day AI systems (which is more crowded and more likely to occur regardless of our efforts). This means that our work often lacks direct practical application today, while also broaching the unpopular subject of general intelligence.
We’re getting more work published these days in part because the topic of smarter-than-human AI is no longer seen as academically illegitimate to the extent it was in the past, and in part because we’ve pivoted in recent years from being an organization that primarily worked on movement growth (via Singularity Summits, writings on rationality, etc.) and some forecasting research, to an organization that focuses solely on novel technical research.
Our seven-paper technical agenda was initially self-published, but this was primarily in order to make it available early enough to be read and cited by attendees of the “Future of AI” conference in January. Since then, a shorter version of the technical agenda paper “Toward Idealized Decision Theory” has been accepted to AGI-15 (the full version is on arXiv), and we’ve presented the full technical agenda paper “Corrigibility” at AAAI-15, a leading academic conference in AI. The overview paper, “Aligning Superintelligence with Human Interests,” is forthcoming in a Springer anthology on the technological singularity.
The other four technical agenda papers aren’t going through peer review because they’re high-level overview papers that are long on explanation and background, but short on new results. We’ve been putting associated results through peer review instead. We published Vingean reflection results in the AGI-14 proceedings (“Problems of Self-Reference in Self-Improving Space-Time Embedded Intelligence”), and other results have been accepted to ITP 2015 (“Proof-Producing Reflection for HOL“). We have two peer-reviewed papers related to both logical uncertainty and realistic world-models: one we presented two weeks ago at AGI-15 (“Reflective Variants of Solomonoff Induction and AIXI”) and another we’re presenting at LORI-V later this year (“Reflective Oracles: A Foundation for Classical Game Theory”). We also presented relevant decision theory results at AAAI-14.
The “Value Learning” paper is the only paper in the research agenda suite that hasn’t had associated work go through peer review yet. It’s the least technical part of the agenda, so it may be a little while before we have technical results to put through peer review on this topic.
(Update July 15: “The Value Learning Problem” has now been peer reviewed and presented at the IJCAI 2016 Ethics for Artificial Intelligence workshop.)
By publishing in prominent journals and conference proceedings, we hope to get many more researchers interested in our work. A useful consequence of this is that there will be more non-MIRI evaluations of (and contributions to) the basic research questions we’re working on. In the nearer future, we also have a few blog posts in the works that are intended to explain some of the more technical parts of our research agenda, such as Vingean reflection.
In all, we’ve published seven peer-reviewed papers since Nate and Benja came on in early 2014. In response to recent industry progress and new work by MIRI and the existential risk community, it’s become much easier to publish papers that wrestle directly with open AI alignment problems, and we expect it to become even easier over the next few years.
What does success look like?
Success for MIRI partly means delivering on our Summer Fundraiser targets: growing our research team and taking on additional projects conditional on which funding targets we’re able to hit. Our current plan this year is to focus on producing a few high-quality publications in elite venues. If our fundraiser goes well, this should impact how effectively we can execute on that plan and how quickly we can generate and publish new results.
A more direct measure of success is our ability to make progress on the specific technical problems we’ve chosen to focus on, as assessed by MIRI researchers and the larger AI community. In “MIRI’s Approach,” Nate distinguishes two types of problems: ones for which we know the answer in principle, but lack practical algorithms; and ones that are not yet well-specified enough for us to know how to construct an answer even in principle. At this point, large-scale progress for MIRI looks like moving important and neglected AI problems from the second category to the first category.
Maxwell raised one more question: if MIRI is receiving more mainstream attention and approval, why does it still have a funding gap?
Part of the answer, sketched out in “Why Now Matters,” is that we do think there’s a good chance that large grants or donations could close our funding gap in the coming years. However, large donors and grantmakers can be slow to act, and whether or not our funding gap is closed five or ten years down the line, it’s very valuable for us to be able to expand and diversify our activities now.
Elon Musk and the Open Philanthropy Project recently awarded $7M in grants to AI safety research, and MIRI’s core research program received a large project grant. This is a wonderful infusion of funding into the field, and means that many more academics will be able to start focusing on AI alignment research. However, given the large number of high-quality grant recipients, the FLI grants aren’t enough to make the most promising research opportunities funding-saturated. MIRI received the fourth-largest project grant, which amounts to $83,000 per year for three years.3 This is a very generous grant, and it will significantly bolster our efforts to support researchers and run workshops, but it’s nowhere near enough to close our funding gap.
Since this is the first fundraiser we’ve run in 2015, it’s a bit early to ask why the newfound attention and approval our work has received this year hasn’t yet closed the gap. The FLI grants and our ongoing fundraiser are part of the mechanism by which the funding gap shrinks. It is shrinking, but the process isn’t instantaneous — and part of the process is making our case to new potential supporters. Our hope is that if we make our case for MIRI clearer to donors, we can close our funding gap faster and thereby have a bigger impact on the early scholarly conversation about AI safety.
We’re very grateful for all the support we’ve gotten so far — and in particular for the support we received before we had mainstream computer science publications, a fleshed-out research agenda, or a track record of impacting the discourse around the future of AI. The support we received early on was critical in getting us to where we are today, and as our potential as an organization becomes clearer through our accomplishments, we hope to continue to attract a wider pool of supporters and collaborators.

As always, you can also shoot us more specific questions that aren’t addressed here.Quoting Russell, Tegmark, Hawking, and Wilczek:
Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all.
So, facing possible futures of incalculable benefits and risks, the experts are surely doing everything possible to ensure the best outcome, right? Wrong. If a superior alien civilization sent us a text message saying, “We’ll arrive in a few decades,” would we just reply, “OK, call us when you get here — we’ll leave the lights on”? Probably not — but this is more or less what is happening with AI. Although we are facing potentially the best or worst thing ever to happen to humanity, little serious research is devoted to these issues outside small non-profit institutes such as the Cambridge Center for Existential Risk, the Future of Humanity Institute, the Machine Intelligence Research Institute, and the Future of Life Institute.We additionally received about $50,000 for the AI Impacts project, and will receive some fraction of the funding from two other grants where our researchers are secondary investigators, “Inferring Human Values” and “Applying Formal Verification to Reflective Reasoning.”The post Assessing our past and potential impact appeared first on Machine Intelligence Research Institute.

Assessing our past and potential impact


Target 3: Taking It To The Next Level

One week ago, we hit our first fundraising target. I’m thrilled to announce that we’re now closing in on our second target: our fundraising total passed $400,000 today!
As we approach target number two, we’re already taking active steps to grow our team. Jessica Taylor joined our core research team on August 1; another research fellow will be coming on in September; and a third researcher has just signed on to join our team in the near future — details forthcoming. These three new recruits will increase the size of our team to six full-time researchers.
We’re courting a few other researchers who may be able to join us later in the year. Meanwhile, we’re running a workshop on logical uncertainty, and we’ve started onboarding a new intern with the aim of helping us with our writing bottleneck.
We’re already growing quickly — but we could still make use of additional funds to pursue a much more ambitious growth plan. Given that we’re only halfway through our fundraiser, this is a good time to start thinking big.
At present, we’re recruiting primarily from a small but dedicated pool of mathematicians and computer scientists who come to us on their own initiative. If our fundraiser successfully passes target number two, any further funds will enable us to pivot toward recruiting top talent more broadly — including highly qualified mathematicians and computer scientists who have never heard of us before.
We have a strong pitch: we’re working on some of the most interesting and important problems in the world, on a research topic which is still in its infancy. There is lots of low-hanging fruit to be picked, and the first papers on these topics will end up defining this new paradigm of research. Researchers at MIRI have a rare opportunity to make groundbreaking discoveries that may play a critical role in AI progress over the next few decades.
Moreover, MIRI researchers don’t have to teach classes, and they aren’t under a “publish or perish” imperative. Their job is just to focus on the most important technical problems they can identify, while leaving the mundane inconveniences of academic research to our operations team. When we make it our priority to recruit the world’s top math talent, we’ll be able to put together a pretty tempting offer!
This is what we’ll do at funding target number three: Take MIRI’s growth to the next level. At this level, we’ll start stepping up our recruitment efforts to build our AI alignment dream team.

Target number three is significantly more ambitious than targets one and two; target one was $250k, and target two was $500k, whereas our third funding target is $1.5M. The more money we raise beyond the $500k level, the more of our next-level growth plans we’ll be able to execute on in the near future. We would execute on some of the below plans at (e.g.) the $750k level, but not all of them.
There are five reasons we can put significantly more funding to good use at this level:
First, to expand our recruiting efforts beyond the pool of people who came to us, we would hire dedicated staff focused specifically on active recruiting. At this level, we hire a research steward with a lot of technical ability who is skilled at engaging with rising stars in AI and mathematics and getting them excited about our research. Dedicated staff could also make it a priority to run seminars at Berkeley, give talks to the graduating classes at places like MIT and Stanford, and start building relationships with professional AI scientists and mathematicians around the world who can point interested talent our way.
Second, if we’re going to start actively recruiting, we’ll need to be confident that we have enough money to actually offer the best people jobs. Increased funding now gives us confidence that we will actually be able to hire on more people and continue to support them over the coming years.
Third, as we reach beyond the community of people who found us on their own, we will have to start offering more competitive salaries. Higher salaries aren’t sufficient to win the best people, but they often are necessary: in the past, we’ve lost potential researchers who wanted to support a family on a single income, or who felt social pressure to stay in the same income bracket as their peers. The market for top talent is competitive, and if we want to pull in the best people, we need to be able to offer market rates.
Fourth, as we increase our growth rate and widen our recruiting pool, we may need to start moving to larger offices, providing better benefits, and building up a bit more runway. When hiring the very best people, it’s important to offer them comfort and job security, especially if academia is the alternative we have to tempt them away from.
Fifth, at a higher funding level we can implement several new administrative projects that will allow us to recruit more effectively, such as by building a map of where all the research talent comes from and goes to, experimenting with different types of talks and seminars to see what works for us, and improving our branding and communication.
Up to now, we’ve been recruiting passively, by publishing ideas on the internet, filtering people who come to us, and building relationships with the very brightest of them. As we approach target three we’ll start recruiting actively, by engaging with a much broader pool of potential researchers.
Our plan is to do this in a way that will also benefit the wider existential risk reduction community. As we get better at recruiting and outreach, we’ll meet more and more bright young people who are interested in addressing the challenges that lie between us and beneficial machine intelligence. In all likelihood, most of them won’t quite fit into the vacancies we have at MIRI, and we’ll point a host of them towards the Future of Humanity Institute at Oxford or the Centre for Study of Existential Risk at Cambridge, both of which will also be hiring aggressively this year.
Hitting target three means growing our existing research program as fast as we feel we sustainably can — and that’s an incredibly exciting proposition. If you can help us get there between now and August 30, this could be a massive opportunity for MIRI, and for the wider field of AI alignment research.

The post Target 3: Taking It To The Next Level appeared first on Machine Intelligence Research Institute.

Target 3: Taking It To The Next Level


When AI Accelerates AI

Last week, Nate Soares outlined his case for prioritizing long-term AI safety work:
1. Humans have a fairly general ability to make scientific and technological progress. The evolved cognitive faculties that make us good at organic chemistry overlap heavily with the evolved cognitive faculties that make us good at economics, which overlap heavily with the faculties that make us good at software engineering, etc.
2. AI systems will eventually strongly outperform humans in the relevant science/technology skills. To the extent these faculties are also directly or indirectly useful for social reasoning, long-term planning, introspection, etc., sufficiently powerful and general scientific reasoners should be able to strongly outperform humans in arbitrary cognitive tasks.
3. AI systems that are much better than humans at science, technology, and related cognitive abilities would have much more power and influence than humans. If such systems are created, their decisions and goals will have a decisive impact on the future.
4. By default, smarter-than-human AI technology will be harmful rather than beneficial. Specifically, it will be harmful if we exclusively work on improving the scientific capability of AI agents and neglect technical work that is specifically focused on safety requirements.
To which I would add:

Intelligent, autonomous, and adaptive systems are already challenging to verify and validate; smarter-than-human scientific reasoners present us with extreme versions of the same challenges.
Smarter-than-human systems would also introduce qualitatively new risks that can’t be readily understood in terms of our models of human agents or narrowly intelligent programs.

None of this, however, tells us when smarter-than-human AI will be developed. Soares has argued that we are likely to be able to make early progress on AI safety questions; but the earlier we start, the larger is the risk that we misdirect our efforts. Why not wait until human-equivalent decision-making machines are closer at hand before focusing our efforts on safety research?
One reason to start early is that the costs of starting too late are much worse than the costs of starting too early. Early work can also help attract more researchers to this area, and give us better models of alternative approaches. Here, however, I want to focus on a different reason to start work early: the concern that a number of factors may accelerate the development of smarter-than-human AI.
AI speedup thesis. AI systems that can match humans in scientific and technological ability will probably be the cause and/or effect of a period of unusually rapid improvement in AI capabilities.

If general scientific reasoners are invented at all, this probably won’t be an isolated event. Instead, it is likely to directly feed into the development of more advanced AI. Similar considerations suggest that such systems may be the result of a speedup in intelligence growth rates, as measured in the cognitive and technological output of humans and machines.
When AI capabilities work is likely to pick up speed more than AI safety work does, putting off safety work raises larger risks (because we may be failing to account for future speedup effects that give us less time than is apparent) and is less useful (because we have a shorter window of time between ‘we have improved AI algorithms we can use to inform our safety work’ and ‘our safety work needs to be ready for implementation’).
I’ll note four broad reasons to expect speedups:
1. Overlap between accelerators of AI progress and enablers/results of AI progress. In particular, progress in automating science and engineering work can include progress in automating AI work.
2. Overall difficulty of AI progress. If smarter-than-human AI is sufficiently difficult, its invention may require auxiliary technologies that effect a speedup. Alternatively, even if such technologies aren’t strictly necessary for AI, they may appear before AI if they are easier to develop.
3. Discontinuity of AI progress. Plausibly, AI development won’t advance at a uniform pace. There will sometimes be very large steps forward, such as new theoretical insights that resolve a number of problems in rapid succession. If a software bottleneck occurs while hardware progress continues, we can expect a larger speedup when a breakthrough occurs: Shulman and Sandberg argue that the availability of cheap computing resources in this scenario would make it much easier to quickly copy and improve on advanced AI software.
4. Increased interest in AI. As AI software increases in capability, we can expect increased investment in the field, especially if a race dynamic develops.
Intelligence explosion is an example of a speedup of the first type. In an intelligence explosion scenario, the ability of AI systems to innovate within the field of AI leads to a positive feedback loop of accelerating progress resulting in superintelligence.
Intelligence explosion and other forms of speedup are often conflated with the hypothesis that smarter-than-human AI is imminent; but some reasons to expect speedups (e.g., ‘overall difficulty of AI progress’ and ‘discontinuity of AI progress’) can equally imply that smarter-than-human AI systems are further off than many researchers expect.
Are there any factors that could help speed up safety work relative to capabilities work? Some have suggested that interest in safety is likely to increase as smarter-than-human AI draws nearer. However, this might coincide with a compensatory increase in AI capabilities investment. Since systems approaching superintelligence will have incentives to appear safe, it is also possible that safety work will erroneously appear less necessary when AI systems approach humans in intelligence, as in Nick Bostrom’s treacherous turn scenario.
We could also imagine outsourcing AI safety work to sufficiently advanced AI systems, just as we might outsource AI capabilities work. However, it is likely to take a special effort to reach the point where we can (safely) delegate a variety of safety tasks before we can delegate a comparable amount of capabilities work.
On the whole, capabilities speedup effects make it more difficult to make robust predictions about AI timelines. If rates of progress are discontinuous, highly capable AI systems may continue to appear about equally far off until shortly before their invention. This suggests that it would be unwise to wait until advanced AI appears to be near to begin investing in basic AI safety research.

The post When AI Accelerates AI appeared first on Machine Intelligence Research Institute.

When AI Accelerates AI


August 2015 Newsletter

Research updates

We’ve rewritten the first and last sections of the main paper summarizing our research program. This version of the paper will also be published with minor changes in the Springer anthology The Technological Singularity.
New analyses: Four Background Claims; MIRI’s Approach
New at AI Impacts: Conversation with Steve Potter; Costs of Human-Level Hardware
New at IAFF: Waterfall Truth Predicates; A Counterexample to an Informal Conjecture…; An Idea for Corrigible, Recursively Improving Math Oracles
Jessica Taylor has joined MIRI’s full-time research team! A fifth MIRI researcher will be coming on in September.

General updates

Our 2015 Summer Fundraiser is live! Our announcement of this on the Effective Altruism Forum makes the case for donating now more explicit.
139 people have collectively donated $283,167 over the last two weeks! This means that we’ve just hit our first of five funding goals! Donations over the next four weeks will now go to our second goal: Accelerating MIRI.
MIRI has a new Frequently Asked Questions page! If you have questions you’d like answered here or on the blog, shoot me an email.
An Astounding Year summarizes recent events in AI risk mitigation.
Cornell computer scientist Bart Selman has become a MIRI research advisor.

News and links

We’re at day three of the Effective Altruism Global conference! You can watch a selection of talks on the livestream.
Thousands sign an open letter by the Future of Life Institute advocating “a ban on offensive autonomous weapons beyond meaningful human control.”

The post August 2015 Newsletter appeared first on Machine Intelligence Research Institute.

August 2015 Newsletter


A new MIRI FAQ, and other announcements

MIRI is at Effective Altruism Global! A number of the talks can be watched online at the EA Global Livestream.
We have a new MIRI Frequently Asked Questions page, which we’ll be expanding as we continue getting new questions over the next four weeks. Questions covered so far include “Why is safety important for smarter-than-human AI?” and “Do researchers think AI is imminent?”
We’ve also been updating other pages on our website. About MIRI now functions as a short introduction to our mission, and Get Involved has a new consolidated application form for people who want to collaborate with us on our research program.
Finally, an announcement: just two weeks into our six-week fundraiser, we have hit our first major fundraising target! We extend our thanks to the donors who got us here so quickly. Thanks to you, we now have the funds to expand our core research team to 6–8 people for the coming year.
New donations we receive at https://intelligence.org/donate will now go toward our second target: “Accelerated Growth.” If we hit this second target ($500k total), we will be able to expand to a ten-person core team and take on a number of important new projects. More details on our plans if we hit our first two fundraiser targets: Growing MIRI.
The post A new MIRI FAQ, and other announcements appeared first on Machine Intelligence Research Institute.

A new MIRI FAQ, and other announcements