A new MIRI research program with a machine learning focus

I’m happy to announce that MIRI is beginning work on a new research agenda, “value alignment for advanced machine learning systems.” Half of MIRI’s team — Patrick LaVictoire, Andrew Critch, and I — will be spending the bulk of our time on this project over at least the next year. The rest of our time will be spent on our pre-existing research agenda.
MIRI’s research in general can be viewed as a response to Stuart Russell’s question for artificial intelligence researchers: “What if we succeed?” There appear to be a number of theoretical prerequisites for designing advanced AI systems that are robust and reliable, and our research aims to develop them early.
Our general research agenda is agnostic about when AI systems are likely to match and exceed humans in general reasoning ability, and about whether or not such systems will resemble present-day machine learning (ML) systems. Recent years’ impressive progress in deep learning suggests that relatively simple neural-network-inspired approaches can be very powerful and general. For that reason, we are making an initial inquiry into a more specific subquestion: “What if techniques similar in character to present-day work in ML succeed in creating AGI?”.
Much of this work will be aimed at improving our high-level theoretical understanding of task-directed AI. Unlike what Nick Bostrom calls “sovereign AI,” which attempts to optimize the world in long-term and large-scale ways, task AI is limited to performing instructed tasks of limited scope, satisficing but not maximizing. Our hope is that investigating task AI from an ML perspective will help give information about both the feasibility of task AI and the tractability of early safety work on advanced supervised, unsupervised, and reinforcement learning systems.
To this end, we will begin by investigating eight relevant technical problems:

1. Inductive ambiguity detection.
How can we design a general methodology for ML systems (such as classifiers) to identify when the classification of a test instance is underdetermined by training data?
For example: If an ambiguity-detecting classifier is designed to distinguish images of tanks from images of non-tanks, and the training set only contains images of tanks on cloudy days and non-tanks on sunny days, this classifier ought to detect that the classification of an image of a tank on a sunny day is ambiguous, and pose some query for its operators to disambiguate it and avoid errors.
While past and current work in active learning and statistical learning theory more broadly has made progress towards this goal, more work is necessary to establish realistic statistical bounds on the error rates and query rates of real-world systems in advance of their deployment in complex environments.
2. Informed oversight.
How might we train a reinforcement learner to output both an action and a “report” comprising information to help a human evaluate its action?
For example: If a human is attempting to train a reinforcement learner to output original stories, then in evaluating the story, the human will want to know some information about the story (such as whether it has been plagiarized from another story) that may be difficult to determine by looking at the story itself.
3. Safe training procedures for human-imitators.
How might we design a ML system that imitates humans performing some task that involves rich outputs (such as answering questions in natural language), to the best of the ML system’s abilities?
While there are existing approaches to imitation learning and generative models, these have some theoretical shortcomings that prevent them from fully solving the general problem. In particular, a generative adversarial model trained on human actions only has an incentive to imitate aspects of the human that the adversary can detect; thus, issues similar to the plagiarism problem from (2) can arise.
4. Conservative concepts.
How might we design a system that, given some positive examples of a concept, can synthesize new instances of the concept without synthesizing edge cases of it?
For example: If we gave the system detailed information about 100 human-created burritos as training data, it should manufacture additional burritos while avoiding edge cases such as extremely small burritos (even though these could still be considered burritos).
By default, most objective functions will lead to such edge cases (say, because small burritos are cheaper to manufacture). Can we develop a general technique for avoiding this problem?
5. Specifying environmental goals using sensory data.
The goals of most ML systems are defined in terms of sensory data, such as discounted reward over time. While these sensory goals can be useful proxies for environmental goals, they are not identical to them: tricking your own sensors into perceiving a strawberry in the room is not the the same as actually having one there. How might we create systems that directly pursue useful goals in their environment, such as “make there be a strawberry in this room”, in a way that is robust to opportunities for the system to interfere with the proxies we used to specify the goal?
6. Low-impact agents.
We would prefer a highly intelligent task-oriented system to generally avoid creating large unintended-by-us side effects in pursuit of its goal, and also to notify us of any large impacts that might result from achieving its goal. For example, if we ask it to build a house for a homeless family, it should know implicitly that it should avoid destroying nearby houses for materials — a large side effect — while still having the desirable large follow-on effect of improving the family’s socioeconomic situation. For any specific task we can specify ad-hoc cost functions for side effects like nearby house destruction, but since we cannot always anticipate such costs in advance, we want a quantitative understanding of how to generally limit an AI’s side effects (without also limiting its ability to have large positive intended impacts).
7. Mild optimization.
In the event that the low impact criteria we employ in (6) are flawed or missing some key considerations, we would like some way to tell our system “don’t try too hard” or “don’t exercise too much optimization power”, since intuitively, large impacts are more likely to result from large efforts (although the correspondence is imperfect).
Less capable AI systems tend to be less likely to have large unintended side effects, and we would like to know how to make even highly capable systems safe in a similar way, by having them apply less “effort”. With a satisfactory encoding of our notion of “effort”, we could then program a system to impose absolute limits on its total effort, or to use only as much effort as is needed to achieve its objective with reasonably high probability, thus potentially limiting impacts that we did not manage to measure in (6).
8. Averting instrumental pressures.
How might we design a system that is not motivated to pursue certain convergent instrumental goals — such as gaining additional resources — even when pursuing these goals would help it achieve its main objective?
In particular, we may wish to build a system that has no incentive to cause or prevent its own shutdown/suspension. This relates to (6) and (7) in that instrumental pressures like “ensure my continued operation” can incentivize large impacts/efforts. However, this is a distinct agenda item because it may be possible to completely eliminate certain instrumental incentives in a way that would apply even before solutions to (6) and (7) would take effect.

Having identified these topics of interest, we expect our work on this agenda to be timely. The idea of “robust and beneficial” AI has recently received increased attention as a result of the new wave of breakthroughs in machine learning. The kind of theoretical work in this project has more obvious connections to the leading paradigms in AI and ML than, for example, our recent work in logical uncertainty or in game theory, and therefore lends itself better to collaborations with AI/ML researchers in the near future.

Thanks to Eliezer Yudkowsky and Paul Christiano for seeding many of the initial ideas for these research directions, to Patrick LaVictoire, Andrew Critch, and other MIRI researchers for helping develop these ideas, and to Chris Olah, Dario Amodei, and Jacob Steinhardt for valuable discussion.
The post A new MIRI research program with a machine learning focus appeared first on Machine Intelligence Research Institute.


New papers dividing logical uncertainty into two subproblems

I’m happy to announce two new technical results related to the problem of logical uncertainty, perhaps our most significant results from the past year. In brief, these results split the problem of logical uncertainty into two distinct subproblems, each of which we can now solve in isolation. The remaining problem, in light of these results, is to find a unified set of methods that solve both at once.
The solutions for each subproblem are available in two new papers, based on work spearheaded by Scott Garrabrant: “Inductive coherence”1 and “Asymptotic convergence in online learning with unbounded delays.”2
To give some background on the problem: Modern probability theory models reasoners’ empirical uncertainty, their uncertainty about the state of a physical environment, e.g., “What’s behind this door?” However, it can’t represent reasoners’ logical uncertainty, their uncertainty about statements like “this Turing machine halts” or “the twin prime conjecture has a proof that is less than a gigabyte long.”3
Roughly speaking, if you give a classical probability distribution variables for statements that could be deduced in principle, then the axioms of probability theory force you to put probability either 0 or 1 on those statements, because you’re not allowed to assign positive probability to contradictions. In other words, modern probability theory assumes that all reasoners know all the consequences of all the things they know, even if deducing those consequences is intractable.
We want a generalization of probability theory that allows us to model reasoners that have uncertainty about statements that they have not yet evaluated. Furthermore, we want to understand how to assign “reasonable” probabilities to claims that are too expensive to evaluate.
Imagine an agent considering whether to use quicksort or mergesort to sort a particular dataset. They might know that quicksort typically runs faster than mergesort, but that doesn’t necessarily apply to the current dataset. They could in principle figure out which one uses fewer resources on this dataset, by running both of them and comparing, but that would defeat the purpose. Intuitively, they have a fair bit of knowledge that bears on the claim “quicksort runs faster than mergesort on this dataset,” but modern probability theory can’t tell us which information they should use and how.4

What does it mean for a reasoner to assign “reasonable probabilities” to claims that they haven’t computed, but could compute in principle? Without probability theory to guide us, we’re reduced to using intuition to identify properties that seem desirable, and then investigating which ones are possible. Intuitively, there are at least two properties we would want logically non-omniscient reasoners to exhibit:
1. They should be able to notice patterns in what is provable about claims, even before they can prove or disprove the claims themselves. For example, consider the claims “this Turing machine outputs an odd number” and “this Turing machine outputs an even number.” A good reasoner thinking about those claims should eventually recognize that they are mutually exclusive, and assign them probabilities that sum to at most 1, even before they can run the relevant Turing machine.
2. They should be able to notice patterns in sentence classes that are true with a certain frequency. For example, they should assign roughly 10% probability to “the 10100th digit of pi is a 7” in lieu of any information about the digit, after observing (but not proving) that digits of pi tend to be uniformly distributed.
MIRI’s work on logical uncertainty this past year can be very briefly summed up as “we figured out how to get these two properties individually, but found that it is difficult to get both at once.”
“Inductive coherence,” which I co-authored with Garrabrant, Benya Fallenstein, and Abram Demski, shows how to get the first property. The abstract reads:
While probability theory is normally applied to external environments, there has been some recent interest in probabilistic modeling of the outputs of computations that are too expensive to run. Since mathematical logic is a powerful tool for reasoning about computer programs, we consider this problem from the perspective of integrating probability and logic.
Recent work on assigning probabilities to mathematical statements has used the concept of coherent distributions, which satisfy logical constraints such as the probability of a sentence and its negation summing to one. Although there are algorithms which converge to a coherent probability distribution in the limit, this yields only weak guarantees about finite approximations of these distributions. In our setting, this is a significant limitation: Coherent distributions assign probability one to all statements provable in a specific logical theory, such as Peano Arithmetic, which can prove what the output of any terminating computation is; thus, a coherent distribution must assign probability one to the output of any terminating computation.
To model uncertainty about computations, we propose to work with approximations to coherent distributions. We introduce inductive coherence, a strengthening of coherence that provides appropriate constraints on finite approximations, and propose an algorithm which satisfies this criterion.
Given a series of provably mutually exclusive sentences, or a series of sentences where each provably implies the next, an inductively coherent predictor’s probabilities eventually start respecting this pattern. This is true even if the predictor hasn’t been able to prove that the pattern holds yet; if it would be possible in principle to eventually prove each instance of the pattern, then the inductively coherent predictor will start recognizing it “before too long,” in a specific technical sense, even if the proofs themselves are very long.
“Asymptotic convergence in online learning with unbounded delays,” which I co-authored with Garrabrant and Jessica Taylor, describes an algorithm with the second property. The abstract reads:
We study the problem of predicting the results of computations that are too expensive to run, via the observation of the results of smaller computations. We model this as an online learning problem with delayed feedback, where the length of the delay is unbounded, which we study mainly in a stochastic setting. We show that in this setting, consistency is not possible in general, and that optimal forecasters might not have average regret going to zero. However, it is still possible to give algorithms that converge asymptotically to Bayes-optimal predictions, by evaluating forecasters on specific sparse independent subsequences of their predictions. We give an algorithm that does this, which converges asymptotically on good behavior, and give very weak bounds on how long it takes to converge. We then relate our results back to the problem of predicting large computations in a deterministic setting.
The first property is about recognizing patterns about logical relationships between claims — saying “claim A implies claim B, so my probability on B must be at least my probability on A.” By contrast, the second property is about recognizing frequency patterns between similar claims — saying “I lack the resources to tell whether this claim is true, but 90% of similar claims have been true, so the base rate is 90%” (where part of the problem is figuring out what counts as a “similar claim”).
In this technical report, we model the latter task as an online learning problem, where a predictor observes the behavior of many small computations and has to predict the behavior of large computations. We give an algorithm that eventually assigns the “right” probabilities to every predictable subsequence of observations, in a specific technical sense.

Each paper is interesting in its own right, but for us, the exciting result is that we have teased apart and formalized two separate notions of what counts as “good reasoning” under logical uncertainty, both of which are compelling.
Furthermore, our approaches to formalizing these two notions are very different. “Inductive coherence” frames the problem in the traditional “unify logic with probability” setting, whereas “Asymptotic convergence in online learning with unbounded delays” fits more naturally into the online machine learning framework. The methods we found for solving the first problem don’t appear to help with the second problem, and vice versa. In fact, the two isolated solutions appear quite difficult to reconcile. The problem that these two papers leave open is: Can we get one algorithm that satisfies both properties at once?

Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.

×

This work was originally titled “Uniform coherence”. This post has been updated to reflect the new terminology.Garrabrant’s IAFF forum posts provide a record of how these results were originally developed, as a response to Ray Solomonoff’s theory of algorithmic probability. Concrete Failure of the Solomonoff Approach and The Entangled Benford Test lay groundwork for the “Asymptotic convergence…” problem, a limited early version of which was featured in the “Asymptotic logical uncertainty and the Benford test” report. Inductive coherence is defined in Uniform Coherence 2, and an example of an inductively coherent predictor is identified in The Modified Demski Prior is Uniformly Coherent.This type of uncertainty is called “logical uncertainty” mainly for historical reasons. I think of it like this: We care about agents’ ability to reason about software systems, e.g., “this program will halt.” Those claims can be expressed in sentences of logic. The question “what probability does the agent assign to this machine halting?” then becomes “what probability does this agent assign to this particular logical sentence?” The truth of these statements could be determined in principle, but the agent may not have the resources to compute the answers in practice.For more background on logical uncertainty, see Gaifman’s “Concerning measures in first-order calculi,” Garber’s “Old evidence and logical omniscience in Bayesian confirmation theory,” Hutter, Lloyd, Ng, and Uther’s “Probabilities on sentences in an expressive logic,” and Aaronson’s “Why philosophers should care about computational complexity.”The post New papers dividing logical uncertainty into two subproblems appeared first on Machine Intelligence Research Institute.


April 2016 Newsletter

Research updates

A new paper: “Parametric Bounded Löb’s Theorem and Robust Cooperation of Bounded Agents“
New at IAFF: What Does it Mean for Correct Operation to Rely on Transfer Learning?; Virtual Models of Virtual AIs in Virtual Worlds

General updates

We’re currently accepting applicants to two programs we’re running in June: our 2016 Summer Fellows program (details), and a new Colloquium Series on Robust and Beneficial AI (details).
MIRI has a new second-in-command: Malo Bourgon.
We’re hiring! Apply here for our new research position in type theory.
AI Impacts is asking for examples of concrete tasks AI systems can’t yet achieve. You can also submit these tasks to Phil Tetlock, who is making the same request for Good Judgment Open.
MIRI senior researcher Eliezer Yudkowsky discusses his core AI concerns with Bryan Caplan. (See Caplan’s response and Yudkowsky’s follow-up.)
Yudkowsky surveys lessons from game-playing AI.

News and links

Google DeepMind’s AlphaGo software defeats leading Go player Lee See-dol 4-1. GoGameGuru provides excellent commentary on each game (1, 2, 3, 4, 5). Lee’s home country of South Korea responds with an AI funding push.
In other Google news: The New York Times reports on an AI platform war; Alphabet’s head of moonshots rejects AI risk concerns; and Alphabet jettisons its main robotics division.
The UK Parliament is launching an inquiry into “social, legal, and ethical issues” raised by AI, and invites written submissions of relevant evidence and arguments.
The White House’s Council of Economic Advisers predicts the widespread automation of low-paying jobs. Related: How Machines Destroy (And Create!) Jobs.
CGP Grey, who discussed automation in Humans Need Not Apply (video), has a thoughtful conversation about Nick Bostrom’s Superintelligence (audio).
Amitai and Oren Etzioni call for the development of guardian AI, “second-order AI software that will police AI.”
In a new paper, Bostrom weighs the pros and cons of openness in AI.
Bostrom argues for scalable AI control methods at RSA Conference (video).
The Open Philanthropy Project, a collaboration between GiveWell and Good Ventures, awards $100,000 to the Future of Life Institute.
The Center for Applied Rationality is seeking participants for two free programs: a Workshop on AI Safety Strategy and EuroSPARC, a math summer camp.

The post April 2016 Newsletter appeared first on Machine Intelligence Research Institute.

April 2016 Newsletter


New paper on bounded Löb and robust cooperation of bounded agents

MIRI Research Fellow Andrew Critch has written a new paper on cooperation between software agents in the Prisoner’s Dilemma, available on arXiv: “Parametric bounded Löb’s theorem and robust cooperation of bounded agents.” The abstract reads:
Löb’s theorem and Gödel’s theorem make predictions about the behavior of systems capable of self-reference with unbounded computational resources with which to write and evaluate proofs. However, in the real world, systems capable of self-reference will have limited memory and processing speed, so in this paper we introduce an effective version of Löb’s theorem which is applicable given such bounded resources. These results have powerful implications for the game theory of bounded agents who are able to write proofs about themselves and one another, including the capacity to out-perform classical Nash equilibria and correlated equilibria, attaining mutually cooperative program equilibrium in the Prisoner’s Dilemma. Previous cooperative program equilibria studied by Tennenholtz and Fortnow have depended on tests for program equality, a fragile condition, whereas “Löbian” cooperation is much more robust and agnostic of the opponent’s implementation.
Tennenholtz (2004) showed that cooperative equilibria exist in the Prisoner’s Dilemma between agents with transparent source code. This suggested that a number of results in classical game theory, where it is a commonplace that mutual defection is rational, might fail to generalize to settings where agents have strong guarantees about each other’s conditional behavior.
Tennenholtz’s version of program equilibrium, however, only established that rational cooperation was possible between agents with identical source code. Patrick LaVictoire and other researchers at MIRI supplied the additional result that more robust cooperation was possible between non-computable agents, and that it is possible to efficiently determine the outcomes of such games. However, some readers objected to the infinitary nature of the methods (for example, the use of halting oracles) and worried that not all of the results would carry over to finite computations.
Critch’s report demonstrates that robust cooperative equilibria exist for bounded agents. In the process, Critch proves a new generalization of Löb’s theorem, and therefore of Gödel’s second incompleteness theorem. This parametric version of Löb’s theorem holds for proofs that can be written out in n or fewer characters, where the parameter n can be set to any number. For more background on the result’s significance, see LaVictoire’s “Introduction to Löb’s theorem in MIRI research.”
The new Löb result shows that bounded agents face obstacles to self-referential reasoning similar to those faced by unbounded agents, and can also reap some of the same benefits. Importantly, this lemma will likely allow us to discuss many other self-referential phenomena going forward using finitary examples rather than infinite ones.

Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.

×

The post New paper on bounded Löb and robust cooperation of bounded agents appeared first on Machine Intelligence Research Institute.


MIRI has a new COO: Malo Bourgon

I’m happy to announce that Malo Bourgon, formerly a program management analyst at MIRI, has taken on a new leadership role as our chief operating officer.
As MIRI’s second-in-command, Malo will be taking over a lot of the hands-on work of coordinating our day-to-day activities: supervising our ops team, planning events, managing our finances, and overseeing internal systems. He’ll also be assisting me in organizational strategy and outreach work.
Prior to joining MIRI, Malo studied electrical, software, and systems engineering at the University of Guelph in Ontario. His professional interests included climate change mitigation, and during his master’s, he worked on a project to reduce waste through online detection of inefficient electric motors. Malo started working for us shortly after completing his master’s in early 2012, which makes him MIRI’s longest-standing team member next to Eliezer Yudkowsky.

Until now, I’ve generally thought of Malo as our secret weapon — a smart, practical efficiency savant. While Luke Muehlhauser (our previous executive director) provided the vision and planning that transformed us into a mature research organization, Malo was largely responsible for the implementation. Behind the scenes, nearly every system or piece of software MIRI uses has been put together by Malo, or in a joint effort by Malo and Alex Vermeer — a close friend of Malo’s from the University of Guelph who now works as a MIRI program management analyst. Malo’s past achievements at MIRI include:

coordinating MIRI’s first research workshops and establishing our current recruitment pipeline.
establishing MIRI’s immigration workflow, allowing us to hire Benya Fallenstein, Katja Grace, and a number of other overseas researchers and administrators.
running MIRI’s (presently inactive) volunteer program.
standardizing MIRI’s document production workflow.
developing and leading the execution of our 2014 SV Gives strategy. This resulted in MIRI receiving $171,575 in donations and prizes (at least $61,330 of which came from outside our usual donor pool) over a 24-hour span.
designing MIRI’s fundraising infrastructure, including our live graphs.

More recently, Malo has begun representing MIRI in meetings with philanthropic organizations, government agencies, and for-profit AI groups.
Malo has been an invaluable asset to MIRI, and I’m thrilled to have him take on more responsibilities here. As one positive consequence, this will free up more of my time to work on strategy, recruiting, fundraising, and research.
In other news, MIRI’s head of communications, Rob Bensinger, has been promoted to the role of research communications manager. He continues to be the best person to contact at MIRI if you have general questions about our work and mission.
Lastly, Katja Grace, the primary contributor to the AI Impacts project, has been promoted to our list of research staff. (Katja is not part of our core research team, and works on questions related to AI strategy and forecasting rather than on our technical research agenda.)
My thanks and heartfelt congratulations to Malo, Rob, and Katja for all the work they’ve done, and all they continue to do.
The post MIRI has a new COO: Malo Bourgon appeared first on Machine Intelligence Research Institute.

MIRI has a new COO: Malo Bourgon


Announcing a new colloquium series and fellows program

The Machine Intelligence Research Institute is accepting applicants to two summer programs: a three-week AI robustness and reliability colloquium series (co-run with the Oxford Future of Humanity Institute), and a two-week fellows program focused on helping new researchers contribute to MIRI’s technical agenda (co-run with the Center for Applied Rationality).

The Colloquium Series on Robust and Beneficial AI (CSRBAI), running from May 27 to June 17, is a new gathering of top researchers in academia and industry to tackle the kinds of technical questions featured in the Future of Life Institute’s long-term AI research priorities report and project grants, including transparency, error-tolerance, and preference specification in software systems.
The goal of the event is to spark new conversations and collaborations between safety-conscious AI scientists with a variety of backgrounds and research interests. Attendees will be invited to give and attend talks at MIRI’s Berkeley, California offices during Wednesday/Thursday/Friday colloquia, to participate in hands-on Saturday/Sunday workshops, and to drop by for open discussion days:

Scheduled speakers include Stuart Russell (May 27), UC Berkeley Professor of Computer Science and co-author of Artificial Intelligence: A Modern Approach, Tom Dietterich (May 27), AAAI President and OSU Director of Intelligent Systems, and Bart Selman (June 3), Cornell Professor of Computer Science.
Apply here to attend any portion of the event, as well as to propose a talk or discussion topic:

Application Form

The 2016 MIRI Summer Fellows program, running from June 19 to July 3, doubles as a workshop for developing new problem-solving skills and mathematical intuitions, and a crash course on MIRI’s active research projects.
This is a smaller and more focused version of the Summer Fellows program we ran last year, which resulted in multiple new hires for us. As such, the program also functions as a high-intensity research retreat where MIRI staff and potential collaborators can get to know each other and work together on important open problems in AI. Apply here to attend the program:

Application Form

Both programs are free of charge, including free room and board for all MIRI Summer Fellows program participants, free lunches and dinners for CSRBAI participants, and additional partial accommodations and travel assistance for select attendees. For additional information, see the CSRBAI event page and the MIRI Summer Fellows event page.
The post Announcing a new colloquium series and fellows program appeared first on Machine Intelligence Research Institute.


Seeking Research Fellows in Type Theory and Machine Self-Reference

The Machine Intelligence Research Institute (MIRI) is accepting applications for a full-time research fellow to develop theorem provers with self-referential capabilities, beginning by implementing a strongly typed language within that very language. The goal of this research project will be to help us understand autonomous systems that can prove theorems about systems with similar deductive capabilities. Applicants should have experience programming in functional programming languages, with a preference for languages with dependent types, such as Agda, Coq, or Lean.
MIRI is a mathematics and computer science research institute specializing in long-term AI safety and robustness work. Our offices are in Berkeley, California, near the UC Berkeley campus.

Type Theory in Type Theory
Our goal with this project is to build tools for better modeling reflective reasoning in software systems, as with our project modeling the HOL4 proof assistant within itself. There are Gödelian reasons to think that self-referential reasoning is not possible in full generality. However, many real-world tasks that cannot be solved in full generality admit of effective mostly-general or heuristic approaches. Humans, for example, certainly succeed in trusting their own reasoning in many contexts.
There are a number of tools missing in modern-day theorem provers that would be helpful for studying self-referential reasoning. First among these is theorem provers that can construct proofs about software systems that make use of a very similar theorem prover. To build these tools in a strongly typed programming language, we need to start by writing programs and proofs that can make reference to the type of programs and proofs in the same language.
Type theory in type theory has recently received a fair amount of attention. James Chapman’s work is pushing in a similar direction to what we want, as is Matt Brown and Jens Palsberg’s, but these projects don’t yet give us the tools we need. (F-omega is too weak a logic for our purposes, and methods like Chapman’s don’t get us self-representations.)
This is intended to be an independent research project, though some collaborations with other researchers may occur. Our expectation is that this will be a multi-year project, but it is difficult to predict exactly how difficult this task is in advance. It may be easier than it looks, or substantially more difficult.
Depending on how the project goes, researchers interested in continuing to work with us after this project’s completion may be able to collaborate on other parts of our research agenda or propose their own additions to our program.
Working at MIRI
We try to make working at MIRI a great experience. Here’s how we operate:

Modern Work Spaces. Many of us have adjustable standing desks with large external monitors. We consider workspace ergonomics important, and try to rig up work stations to be as comfortable as possible. Free snacks, drinks, and meals are also provided at our office.
Flexible Hours. This is a salaried position. We don’t have strict office hours, and we don’t limit employees’ vacation days. Our goal is to make quick progress on our research agenda, and we would prefer that researchers take a day off than that they extend tasks to fill an extra day.
Living in the Bay Area. MIRI’s office is located in downtown Berkeley, California. From our office, you’re a 30-second walk to the BART (Bay Area Rapid Transit), which can get you around the Bay Area; a 3-minute walk to UC Berkeley campus; and a 30-minute BART ride to downtown San Francisco.
Travel Assistance. Visa assistance is available if needed. If you are moving to the Bay Area, we’ll cover up to $3,500 in moving expenses. We also provide a public transit pass with a large monthly allowance.

The salary for this position is negotiable, and comes with top-notch health and dental benefits.
About MIRI
MIRI is a Berkeley-based research nonprofit studying foundational questions in artificial intelligence. Our goal is to ensure that high-quality decision-making systems have a positive global impact in coming decades. MIRI Research Advisor and AI pioneer Stuart Russell outlines several causes for concerns about high-capability AI software:

The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.
Our focus is on systems too complex and autonomous for software developers to anticipate all possible states the system and its environment might evolve into. To employ such systems safely, software engineers will need to understand their behavior on a deep level, and be able to use machine analysis and verification techniques to confirm high-level system properties.
MIRI is the primary organization specializing in this line of technical research. Our work is cited in the Research Priorities for Robust and Beneficial Artificial Intelligence report, and has been discussed in Nick Bostrom’s Superintelligence and Russell and Norvig’s Artificial Intelligence: A Modern Approach.
Long-term AI safety is a rapidly growing field of research that has recently received significant public attention. Our current research is intended to provide this new field with theoretical foundations that will help guide the direction of future research.
How to Apply
(Update 2022: We are not actively seeking to fill this role at this time.)

The post Seeking Research Fellows in Type Theory and Machine Self-Reference appeared first on Machine Intelligence Research Institute.


March 2016 Newsletter

Research updates

A new paper: “Defining Human Values for Value Learners“
New at IAFF: Analysis of Algorithms and Partial Algorithms; Naturalistic Logical Updates; Notes from a Conversation on Act-Based and Goal-Directed Systems; Toy Model: Convergent Instrumental Goals
New at AI Impacts: Global Computing Capacity
A revised version of “The Value Learning Problem” (pdf) has been accepted to a AAAI spring symposium.

General updates

MIRI and other Future of Life Institute (FLI) grantees participated in a AAAI workshop on AI safety this month.
MIRI researcher Eliezer Yudkowsky discusses Ray Kurzweil, the Bayesian brain hypothesis, and an eclectic mix of other topics in a new interview.
Alexei Andreev and Yudkowsky are seeking investors for Arbital, a new technology for explaining difficult topics in economics, mathematics, computer science, and other disciplines. As a demo, Yudkowsky has written a new and improved guide to Bayes’s Rule.

News and links

Should We Fear or Welcome the Singularity? (video): a conversation between Kurzweil, Stuart Russell, Max Tegmark, and Harry Shum.
The Code That Runs Our Lives (video): Deep learning pioneer Geoffrey Hinton expresses his concerns about smarter-than-human AI (at 10:00).
The State of AI (video): Russell, Ya-Qin Zhang, Matthew Grob, and Andrew Moore share their views on a range of issues at Davos, including superintelligence (at 21:09).
Bill Gates discusses AI timelines.
Paul Christiano proposes a new AI alignment approach: algorithm learning by bootstrapped approval-maximization.
Robert Wiblin asks the effective altruism community: If tech progress might be bad, what should we tell people about it?
FLI collects introductory resources on AI safety research.
Raising for Effective Giving, a major fundraiser for MIRI and other EA organizations, is seeking a Director of Growth.
Murray Shanahan answers questions about the new Leverhulme Centre for the Future of Intelligence. Leverhulme CFI is presently seeking an Executive Director.

The post March 2016 Newsletter appeared first on Machine Intelligence Research Institute.

March 2016 Newsletter


John Horgan interviews Eliezer Yudkowsky

 Scientific American writer John Horgan recently interviewed MIRI’s senior researcher and co-founder, Eliezer Yudkowsky. The email interview touched on a wide range of topics, from politics and religion to existential risk and Bayesian models of rationality.
Although Eliezer isn’t speaking in an official capacity in the interview, a number of the questions discussed are likely to be interesting to people who follow MIRI’s work. We’ve reproduced the full interview below.

John Horgan: When someone at a party asks what you do, what do you tell her?

Eliezer Yudkowsky: Depending on the venue: “I’m a decision theorist”, or “I’m a cofounder of the Machine Intelligence Research Institute”, or if it wasn’t that kind of party, I’d talk about my fiction.

John: What’s your favorite AI film and why?

Eliezer: AI in film is universally awful. Ex Machina is as close to being an exception to this rule as it is realistic to ask.

John: Is college overrated?

Eliezer: It’d be very surprising if college were underrated, given the social desirability bias of endorsing college. So far as I know, there’s no reason to disbelieve the economists who say that college has mostly become a positional good, and that previous efforts to increase the volume of student loans just increased the cost of college and the burden of graduate debt.

John: Why do you write fiction?

Eliezer: To paraphrase Wondermark, “Well, first I tried not making it, but then that didn’t work.”
Beyond that, nonfiction conveys knowledge and fiction conveys experience. If you want to understand a proof of Bayes’s Rule, I can use diagrams. If I want you to feel what it is to use Bayesian reasoning, I have to write a story in which some character is doing that.

John: Are you religious in any way?

Eliezer: No. When you make a mistake, you need to avoid the temptation to go defensive, try to find some way in which you were a little right, look for a silver lining in the cloud. It’s much wiser to just say “Oops”, admit you were not even a little right, swallow the whole bitter pill in one gulp, and get on with your life. That’s the attitude humanity should take toward religion.

John: If you were King of the World, what would top your “To Do” list?

Eliezer: I once observed, “The libertarian test is whether, imagining that you’ve gained power, your first thought is of the laws you would pass, or the laws you would repeal.” I’m not an absolute libertarian, since not everything I want would be about repealing laws and softening constraints. But when I think of a case like this, I imagine trying to get the world to a condition where some unemployed person can offer to drive you to work for 20 minutes, be paid five dollars, and then nothing else bad happens to them. They don’t have their unemployment insurance phased out, have to register for a business license, lose their Medicare, be audited, have their lawyer certify compliance with OSHA rules, or whatever. They just have an added $5.
I’d try to get to the point where employing somebody was once again as easy as it was in 1900. I think it can make sense nowadays to have some safety nets, but I’d try to construct every safety net such that it didn’t disincent or add paperwork to that simple event where a person becomes part of the economy again.
I’d try to do all the things smart economists have been yelling about for a while but that almost no country ever does. Replace investment taxes and income taxes with consumption taxes and land value tax. Replace minimum wages with negative wage taxes. Institute NGDP level targeting regimes at central banks and let the too-big-to-fails go hang. Require loser-pays in patent law and put copyright back to 28 years. Eliminate obstacles to housing construction. Copy and paste from Singapore’s healthcare setup. Copy and paste from Estonia’s e-government setup. Try to replace committees and elaborate process regulations with specific, individual decision-makers whose decisions would be publicly documented and accountable. Run controlled trials of different government setups and actually pay attention to the results. I could go on for literally hours.
All this might not matter directly from the perspective of two hundred million years later. But the goodwill generated by the resulting economic boom might stand my government in good stead when I tried to figure out what the heck to do about Artificial Intelligence. The obvious thing, I guess, would be a Manhattan Project on an island somewhere, with pay competitive with top hedge funds, where people could collaborate on researching parts of the Artificial General Intelligence problem without the publication of their work automatically moving us closer to the end of the world. We’d still be working to an unknown deadline, and I wouldn’t feel relaxed at that point. Unless we postulate that I have literally magical powers or an utterly unshakeable regime, I don’t see how any law I could reasonably decree could delay AI timelines for very long on a planet where computers are already ubiquitous.
All of this is an impossible thought experiment in the first place, and I see roughly zero hope of it ever coming to pass in real life.

John: What’s so great about Bayes’s Theorem?

Eliezer: For one thing, Bayes’s Theorem is incredibly deep. So it’s not easy to give a brief answer to that.
I might answer that Bayes’s Theorem is a kind of Second Law of Thermodynamics for cognition. If you obtain a well-calibrated posterior belief that some proposition is 99% probable, whether that proposition is milk being available at the supermarket or global warming being anthropogenic, then you must have processed some combination of sufficiently good priors and sufficiently strong evidence. That’s not a normative demand, it’s a law. In the same way that a car can’t run without dissipating entropy, you simply don’t get an accurate map of the world without a process that has Bayesian structure buried somewhere inside it, even if the process doesn’t explicitly represent probabilities or likelihood ratios. You had strong-enough evidence and a good-enough prior or you wouldn’t have gotten there.
On a personal level, I think the main inspiration Bayes has to offer us is just the fact that there are rules, that there are iron laws that govern whether a mode of thinking works to map reality. Mormons are told that they’ll know the truth of the Book of Mormon through feeling a burning sensation in their hearts. Let’s conservatively set the prior probability of the Book of Mormon at one to a billion (against). We then ask about the likelihood that, assuming the Book of Mormon is false, someone would feel a burning sensation in their heart after being told to expect one. If you understand Bayes’s Rule you can see at once that the improbability of the evidence is not commensurate with the improbability of the hypothesis it’s trying to lift. You don’t even have to make up numbers to see that the numbers don’t add up — as Philip Tetlock found in his study of superforecasters, superforecasters often know Bayes’s Rule but they rarely make up specific probabilities. On some level, it’s harder to be fooled if you just realize on a gut level that there is math, that there is some math you’d do to arrive at the exact strength of the evidence and whether it sufficed to lift the prior improbability of the hypothesis. That you can’t just make stuff up and believe what you want to believe because that doesn’t work.

John: Does the Bayesian-brain hypothesis impress you?

Eliezer: I think some of the people in that debate may be talking past each other. Asking whether the brain is a Bayesian algorithm is like asking whether a Honda Accord runs on a Carnot heat engine. If you have one person who’s trying to say, “Every car is a thermodynamic process that requires fuel and dissipates waste heat” and the person on the other end hears, “If you draw a diagram of a Carnot heat engine and show it to a mechanic, they should agree that it looks like the inside of a Honda Accord” then you are going to have some fireworks.
Some people will also be really excited when they open up the internal combustion engine and find the cylinders and say, “I bet this converts heat into pressure and helps drive the car forward!” And they’ll be right, but then you’re going to find other people saying, “You’re focusing on what’s merely a single component in a much bigger library of car parts; the catalytic converter is also important and that doesn’t appear anywhere on your diagram of a Carnot heat engine. Why, sometimes we run the air conditioner, which operates in the exact opposite way of how you say a heat engine works.”
I don’t think it would come as much of a surprise that I think the people who adopt a superior attitude and say, “You are clearly unfamiliar with modern car repair; you need a toolbox of diverse methods to build a car engine, like spark plugs and catalytic converters, not just these thermodynamic processes you keep talking about” are missing a key level of abstraction.
But if you want to know whether the brain is literally a Bayesian engine, as opposed to doing cognitive work whose nature we can understand in a Bayesian way, then my guess is “Heck, no.” There might be a few excitingly Bayesian cylinders in that engine, but a lot more of it is going to look like weird ad-hoc seat belts and air conditioning. None of which is going to change the fact that to correctly identify an apple based on sensory evidence, you need to do something that’s ultimately interpretable as resting on an inductive prior that can learn the apple concept, and updating on evidence that distinguishes apples from nonapples.

John: Can you be too rational?

Eliezer: You can run into what we call “The Valley of Bad Rationality.” If you were previously irrational in multiple ways that balanced or canceled out, then becoming half-rational can leave you worse off than before. Becoming incrementally more rational can make you incrementally worse off, if you choose the wrong place to invest your skill points first.
But I would not recommend to people that they obsess over that possibility too much. In my experience, people who go around talking about cleverly choosing to be irrational strike me as, well, rather nitwits about it, to be frank. It’s hard to come up with a realistic non-contrived life situation where you know that it’s a good time to be irrational and you don’t already know the true answer. I think in real life, you just tell yourself the truth as best you know it, and don’t try to be clever.
On an entirely separate issue, it’s possible that being an ideal Bayesian agent is ultimately incompatible with living the life best-lived from a fun-theoretic perspective. But we’re a long, long, long way from that being a bigger problem than our current self-destructiveness.

John: How does your vision of the Singularity differ from that of Ray Kurzweil?

Eliezer:

I don’t think you can time AI with Moore’s Law. AI is a software problem.
I don’t think that humans and machines “merging” is a likely source for the first superhuman intelligences. It took a century after the first cars before we could even begin to put a robotic exoskeleton on a horse, and a real car would still be faster than that.
I don’t expect the first strong AIs to be based on algorithms discovered by way of neuroscience any more than the first airplanes looked like birds.
I don’t think that nano-info-bio “convergence” is probable, inevitable, well-defined, or desirable.
I think the changes between 1930 and 1970 were bigger than the changes between 1970 and 2010.
I buy that productivity is currently stagnating in developed countries.
I think extrapolating a Moore’s Law graph of technological progress past the point where you say it predicts smarter-than-human AI is just plain weird. Smarter-than-human AI breaks your graphs.
Some analysts, such as Illka Tuomi, claim that Moore’s Law broke down in the ’00s. I don’t particularly disbelieve this.
The only key technological threshold I care about is the one where AI, which is to say AI software, becomes capable of strong self-improvement. We have no graph of progress toward this threshold and no idea where it lies (except that it should not be high above the human level because humans can do computer science), so it can’t be timed by a graph, nor known to be near, nor known to be far. (Ignorance implies a wide credibility interval, not being certain that something is far away.)
I think outcomes are not good by default — I think outcomes can be made good, but this will require hard work that key actors may not have immediate incentives to do. Telling people that we’re on a default trajectory to great and wonderful times is false.
I think that the “Singularity” has become a suitcase word with too many mutually incompatible meanings and details packed into it, and I’ve stopped using it.

John: Do you think you have a shot at becoming a superintelligent cyborg?

Eliezer: The conjunction law of probability theory says that P(A&B) ≤ P(A) — the probability of both A and B happening is less than the probability of A alone happening. Experimental conditions that can get humans to assign P(A&B) > P(A) for some A&B are said to exhibit the “conjunction fallacy” — for example, in 1982, experts at the International Congress on Forecasting assigned higher probability to “A Russian invasion of Poland, and a complete breakdown of diplomatic relations with the Soviet Union” than a separate group did for “A complete breakdown of diplomatic relations with the Soviet Union”. Similarly, another group assigned higher probability to “An earthquake in California causing a flood that causes over a thousand deaths” than another group assigned to “A flood causing over a thousand deaths somewhere in North America.” Even though adding on additional details necessarily makes a story less probable, it can make the story sound more plausible. I see understanding this as a kind of Pons Asinorum of serious futurism — the distinction between carefully weighing each and every independent proposition you add to your burden, asking if you can support that detail independently of all the rest, versus making up a wonderful vivid story.
I mention this as context for my reply, which is, “Why the heck are you tacking on the ‘cyborg’ detail to that? I don’t want to be a cyborg.” You’ve got to be careful with tacking on extra details to things.

John: Do you have a shot at immortality?

Eliezer: What, literal immortality? Literal immortality seems hard. Living significantly longer than a few trillion years requires us to be wrong about the expected fate of the expanding universe. Living longer than, say, a googolplex years, requires us to be wrong about the basic character of physical law, not just the details.
Even if some of the wilder speculations are true and it’s possible for our universe to spawn baby universes, that doesn’t get us literal immortality. To live significantly past a googolplex years without repeating yourself, you need computing structures containing more than a googol elements, and those won’t fit inside a single Hubble volume.
And a googolplex is hardly infinity. To paraphrase Martin Gardner, Graham’s Number is still relatively small because most finite numbers are very much larger. Look up the fast-growing hierarchy if you really want to have your mind blown, well, eternity is longer than that. Only weird and frankly terrifying anthropic theories would let you live long enough to gaze, perhaps knowingly and perhaps not, upon the halting of the longest-running halting Turing machine with 100 states.
But I’m not sure that living to look upon the 100th Busy Beaver Number feels to me like it matters very much on a deep emotional level. I have some imaginative sympathy with myself a subjective century from now. That me will be in a position to sympathize with their future self a subjective century later. And maybe somewhere down the line is someone who faces the prospect of their future self not existing at all, and they might be very sad about that; but I’m not sure I can imagine who that person will be. “I want to live one more day. Tomorrow I’ll still want to live one more day. Therefore I want to live forever, proof by induction on the positive integers.” Even my desire for merely physical-universe-containable longevity is an abstract want by induction; it’s not that I can actually imagine myself a trillion years later.

John: I’ve described the Singularity as an “escapist, pseudoscientific” fantasy that distracts us from climate change, war, inequality and other serious problems. Why am I wrong?

Eliezer: Because you’re trying to forecast empirical facts by psychoanalyzing people. This never works.
Suppose we get to the point where there’s an AI smart enough to do the same kind of work that humans do in making the AI smarter; it can tweak itself, it can do computer science, it can invent new algorithms. It can self-improve. What happens after that — does it become even smarter, see even more improvements, and rapidly gain capability up to some very high limit? Or does nothing much exciting happen?
It could be that, (A), self-improvements of size δ tend to make the AI sufficiently smarter that it can go back and find new potential self-improvements of size k ⋅ δ and that k is greater than one, and this continues for a sufficiently extended regime that there’s a rapid cascade of self-improvements leading up to superintelligence; what I. J. Good called the intelligence explosion. Or it could be that, (B), k is less than one or that all regimes like this are small and don’t lead up to superintelligence, or that superintelligence is impossible, and you get a fizzle instead of an explosion. Which is true, A or B? If you actually built an AI at some particular level of intelligence and it actually tried to do that, something would actually happen out there in the empirical real world, and that event would be determined by background facts about the landscape of algorithms and attainable improvements.
You can’t get solid information about that event by psychoanalyzing people. It’s exactly the sort of thing that Bayes’s Theorem tells us is the equivalent of trying to run a car without fuel. Some people will be escapist regardless of the true values on the hidden variables of computer science, so observing some people being escapist isn’t strong evidence, even if it might make you feel like you want to disaffiliate with a belief or something.
There is a misapprehension, I think, of the nature of rationality, which is to think that it’s rational to believe “there are no closet goblins” because belief in closet goblins is foolish, immature, outdated, the sort of thing that stupid people believe. The true principle is that you go in your closet and look. So that in possible universes where there are closet goblins, you end up believing in closet goblins, and in universes with no closet goblins, you end up disbelieving in closet goblins.
It’s difficult but not impossible to try to sneak peeks through the crack of the closet door, to ask the question, “What would look different in the universe now if you couldn’t get sustained returns on cognitive investment later, such that an AI trying to improve itself would fizzle? What other facts should we observe in a universe like that?”
So you have people who say, for example, that we’ll only be able to improve AI up to the human level because we’re human ourselves, and then we won’t be able to push an AI past that. I think that if this is how the universe looks in general, then we should also observe, e.g., diminishing returns on investment in hardware and software for computer chess past the human level, which we did not in fact observe. Also, natural selection shouldn’t have been able to construct humans, and Einstein’s mother must have been one heck of a physicist, et cetera.
You have people who say, for example, that it should require more and more tweaking to get smarter algorithms and that human intelligence is around the limit. But this doesn’t square up with the anthropological record of human intelligence; we can know that there were not diminishing returns to brain tweaks and mutations producing improved cognitive power. We know this because population genetics says that mutations with very low statistical returns will not evolve to fixation at all.
And hominids definitely didn’t need exponentially vaster brains than chimpanzees. And John von Neumann didn’t have a head exponentially vaster than the head of an average human.
And on a sheerly pragmatic level, human axons transmit information at around a millionth of the speed of light, even when it comes to heat dissipation each synaptic operation in the brain consumes around a million times the minimum heat dissipation for an irreversible binary operation at 300 Kelvin, and so on. Why think the brain’s software is closer to optimal than the hardware? Human intelligence is privileged mainly by being the least possible level of intelligence that suffices to construct a computer; if it were possible to construct a computer with less intelligence, we’d be having this conversation at that level of intelligence instead.
But this is not a simple debate and for a detailed consideration I’d point people at an old informal paper of mine, “Intelligence Explosion Microeconomics“, which is unfortunately probably still the best source out there. But these are the type of questions one must ask to try to use our currently accessible evidence to reason about whether or not we’ll see what’s colloquially termed an “AI FOOM” — whether there’s an extended regime where δ improvement in cognition, reinvested into self-optimization, yields greater than δ further improvements.
As for your question about opportunity costs:
There is a conceivable world where there is no intelligence explosion and no superintelligence. Or where, a related but logically distinct proposition, the tricks that machine learning experts will inevitably build up for controlling infrahuman AIs carry over pretty well to the human-equivalent and superhuman regime. Or where moral internalism is true and therefore all sufficiently advanced AIs are inevitably nice. In conceivable worlds like that, all the work and worry of the Machine Intelligence Research Institute comes to nothing and was never necessary in the first place, representing some lost number of mosquito nets that could otherwise have been bought by the Against Malaria Foundation.
There’s also a conceivable world where you work hard and fight malaria, where you work hard and keep the carbon emissions to not much worse than they are already (or use geoengineering to mitigate mistakes already made). And then it ends up making no difference because your civilization failed to solve the AI alignment problem, and all the children you saved with those malaria nets grew up only to be killed by nanomachines in their sleep. (Vivid detail warning! I don’t actually know what the final hours will be like and whether nanomachines will be involved. But if we’re happy to visualize what it’s like to put a mosquito net over a bed, and then we refuse to ever visualize in concrete detail what it’s like for our civilization to fail AI alignment, that can also lead us astray.)
I think that people who try to do thought-out philanthropy, e.g., Holden Karnofsky of GiveWell, would unhesitatingly agree that these are both conceivable worlds we prefer not to enter. The question is just which of these two worlds is more probable as the one we should avoid. And again, the central principle of rationality is not to disbelieve in goblins because goblins are foolish and low-prestige, or to believe in goblins because they are exciting or beautiful. The central principle of rationality is to figure out which observational signs and logical validities can distinguish which of these two conceivable worlds is the metaphorical equivalent of believing in goblins.
I think it’s the first world that’s improbable and the second one that’s probable. I’m aware that in trying to convince people of that, I’m swimming uphill against a sense of eternal normality — the sense that this transient and temporary civilization of ours that has existed for only a few decades, that this species of ours that has existed for only an eyeblink of evolutionary and geological time, is all that makes sense and shall surely last forever. But given that I do think the first conceivable world is just a fond dream, it should be clear why I don’t think we should ignore a problem we’ll predictably have to panic about later. The mission of the Machine Intelligence Research Institute is to do today that research which, 30 years from now, people will desperately wish had begun 30 years earlier.

John: Does your wife Brienne believe in the Singularity?

Eliezer: Brienne replies:
If someone asked me whether I “believed in the singularity”, I’d raise an eyebrow and ask them if they “believed in” robotic trucking. It’s kind of a weird question. I don’t know a lot about what the first fleet of robotic cargo trucks will be like, or how long they’ll take to completely replace contemporary ground shipping. And if there were a culturally loaded suitcase term “robotruckism” that included a lot of specific technological claims along with whole economic and sociological paradigms, I’d be hesitant to say I “believed in” driverless trucks. I confidently forecast that driverless ground shipping will replace contemporary human-operated ground shipping, because that’s just obviously where we’re headed if nothing really weird happens. Similarly, I confidently forecast an intelligence explosion. That’s obviously where we’re headed if nothing really weird happens. I’m less sure of the other items in the “singularity” suitcase.
To avoid prejudicing the result, Brienne composed her reply without seeing my other answers. We’re just well-matched.

John: Can we create superintelligences without knowing how our brains work?

Eliezer: Only in the sense that you can make airplanes without knowing how a bird flies. You don’t need to be an expert in bird biology, but at the same time, it’s difficult to know enough to build an airplane without realizing some high-level notion of how a bird might glide or push down air with its wings. That’s why I write about human rationality in the first place — if you push your grasp on machine intelligence past a certain point, you can’t help but start having ideas about how humans could think better too.

John: What would superintelligences want? Will they have anything resembling sexual desire?

Eliezer: Think of an enormous space of possibilities, a giant multidimensional sphere. This is Mind Design Space, the set of possible cognitive algorithms. Imagine that somewhere near the bottom of that sphere is a little tiny dot representing all the humans who ever lived — it’s a tiny dot because all humans have basically the same brain design, with a cerebral cortex, a prefrontal cortex, a cerebellum, a thalamus, and so on. It’s conserved even relative to chimpanzee brain design. Some of us are weird in little ways, you could say it’s a spiky dot, but the spikes are on the same tiny scale as the dot itself; no matter how neuroatypical you are, you aren’t running on a different cortical algorithm.
Asking “what would superintelligences want” is a Wrong Question. Superintelligences are not this weird tribe of people who live across the water with fascinating exotic customs. “Artificial Intelligence” is just a name for the entire space of possibilities outside the tiny human dot. With sufficient knowledge you might be able to reach into that space of possibilities and deliberately pull out an AI that wanted things that had a compact description in human wanting-language, but that wouldn’t be because this is a kind of thing that those exotic superintelligence people naturally want, it would be because you managed to pinpoint one part of the design space.
When it comes to pursuing things like matter and energy, we may tentatively expect partial but not total convergence — it seems like there should be many, many possible superintelligences that would instrumentally want matter and energy in order to serve terminal preferences of tremendous variety. But even there, everything is subject to defeat by special cases. If you don’t want to get disassembled for spare atoms, you can, if you understand the design space well enough, reach in and pull out a particular machine intelligence that doesn’t want to hurt you.
So the answer to your second question about sexual desire is that if you knew exactly what you were doing and if you had solved the general problem of building AIs that stably want particular things as they self-improve and if you had solved the general problem of pinpointing an AI’s utility functions at things that seem deceptively straightforward to human intuitions, and you’d solved an even harder problem of building an AI using the particular sort of architecture where ‘being horny’ or ‘sex makes me happy’ makes sense in the first place, then you could perhaps make an AI that had been told to look at humans, model what humans want, pick out the part of the model that was sexual desire, and then want and experience that thing too.
You could also, if you had a sufficiently good understanding of organic biology and aerodynamics, build an airplane that could mate with birds.
I don’t think this would have been a smart thing for the Wright Brothers to try to do in the early days. There would have been absolutely no point.
It does seem a lot wiser to figure out how to reach into the design space and pull out a special case of AI that will lack the default instrumental preference to disassemble us for spare atoms.

John: I like to think superintelligent beings would be nonviolent, because they will realize that violence is stupid. Am I naive?

Eliezer: I think so. As David Hume might have told you, you’re making a type error by trying to apply the ‘stupidity’ predicate to an agent’s terminal values or utility function. Acts, choices, policies can be stupid given some set of preferences over final states of the world. If you happen to be an agent that has meta-preferences you haven’t fully computed, you might have a platform on which to stand and call particular guesses at the derived object-level preferences as ‘stupid’.
A paperclip maximizer is not making a computational error by having a preference order on outcomes that prefers outcomes with more paperclips in them. It is not standing from within your own preference framework and choosing blatantly mistaken acts, nor is it standing within your meta-preference framework and making mistakes about what to prefer. It is computing the answer to a different question than the question that you are asking when you ask, “What should I do?” A paperclip maximizer just outputs the action leading to the greatest number of expected paperclips.
The fatal scenario is an AI that neither loves you nor hates you, because you’re still made of atoms that it can use for something else. Game theory, and issues like cooperation in the Prisoner’s Dilemma, don’t emerge in all possible cases. In particular, they don’t emerge when something is sufficiently more powerful than you that it can disassemble you for spare atoms whether you try to press Cooperate or Defect. Past that threshold, either you solved the problem of making something that didn’t want to hurt you, or else you’ve already lost.

John: Will superintelligences solve the “hard problem” of consciousness?

Eliezer: Yes, and in retrospect the answer will look embarrassingly obvious from our perspective.

John: Will superintelligences possess free will?

Eliezer: Yes, but they won’t have the illusion of free will.

John: What’s your utopia?

Eliezer: I refer your readers to my nonfiction Fun Theory Sequence, since I have not as yet succeeded in writing any novel set in a fun-theoretically optimal world.

The original interview can be found at AI Visionary Eliezer Yudkowsky on the Singularity, Bayesian Brains and Closet Goblins.
Other conversations that feature MIRI researchers have included: Yudkowsky on “What can we do now?”; Yudkowsky on logical uncertainty; Benya Fallenstein on the Löbian obstacle to self-modifying systems; and Yudkowsky, Muehlhauser, Karnofsky, Steinhardt, and Amodei on MIRI strategy.
The post John Horgan interviews Eliezer Yudkowsky appeared first on Machine Intelligence Research Institute.


New paper: “Defining human values for value learners”

MIRI Research Associate Kaj Sotala recently presented a new paper, “Defining Human Values for Value Learners,” at the AAAI-16 AI, Society and Ethics workshop.
The abstract reads:
Hypothetical “value learning” AIs learn human values and then try to act according to those values. The design of such AIs, however, is hampered by the fact that there exists no satisfactory definition of what exactly human values are. After arguing that the standard concept of preference is insufficient as a definition, I draw on reinforcement learning theory, emotion research, and moral psychology to offer an alternative definition. In this definition, human values are conceptualized as mental representations that encode the brain’s value function (in the reinforcement learning sense) by being imbued with a context-sensitive affective gloss. I finish with a discussion of the implications that this hypothesis has on the design of value learners.
Economic treatments of agency standardly assume that preferences encode some consistent ordering over world-states revealed in agents’ choices. Real-world preferences, however, have structure that is not always captured in economic models. A person can have conflicting preferences about whether to study for an exam, for example, and the choice they end up making may depend on complex, context-sensitive psychological dynamics, rather than on a simple comparison of two numbers representing how much one wants to study or not study.
Sotala argues that our preferences are better understood in terms of evolutionary theory and reinforcement learning. Humans evolved to pursue activities that are likely to lead to certain outcomes — outcomes that tended to improve our ancestors’ fitness. We prefer those outcomes, even if they no longer actually maximize fitness; and we also prefer events that we have learned tend to produce such outcomes.
Affect and emotion, on Sotala’s account, psychologically mediate our preferences. We enjoy and desire states that are highly rewarding in our evolved reward function. Over time, we also learn to enjoy and desire states that seem likely to lead to high-reward states. On this view, our preferences function to group together events that lead on expectation to similarly rewarding outcomes for similar reasons; and over our lifetimes we come to inherently value states that lead to high reward, instead of just valuing such states instrumentally. Rather than directly mapping onto our rewards, our preferences map onto our expectation of rewards.
Sotala proposes that value learning systems informed by this model of human psychology could more reliably reconstruct human values. On this model, for example, we can expect human preferences to change as we find new ways to move toward high-reward states. New experiences can change which states my emotions categorize as “likely to lead to reward,” and they can thereby modify which states I enjoy and desire. Value learning systems that take these facts about humans’ psychological dynamics into account may be better equipped to take our likely future preferences into account, rather than optimizing for our current preferences alone.
The post New paper: “Defining human values for value learners” appeared first on Machine Intelligence Research Institute.


February 2016 Newsletter

Research updates

New at IAFF: Thoughts on Logical Dutch Book Arguments; Another View of Quantilizers: Avoiding Goodhart’s Law; Another Concise Open Problem

General updates

Fundraiser and grant successes: MIRI will be working with AI pioneer Stuart Russell and a to-be-determined postdoctoral researcher on the problem of corrigibility, thanks to a $75,000 grant by the Center for Long-Term Cybersecurity.

News and links

In a major break from trend in Go progress, DeepMind’s AlphaGo software defeats the European Go champion 5-0. A top Go player analyzes AlphaGo’s play.
NYU hosted a Future of AI symposium this month, with a number of leading thinkers in AI and existential risk reduction in attendance.
Marvin Minsky, one of the early architects of the field of AI, has passed away.
Learning and Logic: Paul Christiano writes on the challenge of “pursuing symbolically defined goals” without known observational proxies.
OpenAI, a new Elon-Musk-backed AI research nonprofit, answers questions on Reddit. (MIRI senior researcher Eliezer Yudkowsky also chimes in.)
Victoria Krakovna argues that people concerned about AI safety should consider becoming AI researchers.
The Centre for Effective Altruism is accepting applicants through Feb. 14 to the Pareto Fellowship, a new three-month training program for ambitious altruists.

The post February 2016 Newsletter appeared first on Machine Intelligence Research Institute.

February 2016 Newsletter


End-of-the-year fundraiser and grant successes

Our winter fundraising drive has concluded. Thank you all for your support!
Through the month of December, 175 distinct donors gave a total of $351,298. Between this fundraiser and our summer fundraiser, which brought in $630k, we’ve seen a surge in our donor base; our previous fundraisers over the past five years had brought in on average $250k (in the winter) and $340k (in the summer). We additionally received about $170k in 2015 grants from the Future of Life Institute, and $150k in other donations.
In all, we’ve taken in about $1.3M in grants and contributions in 2015, up from our $1M average over the previous five years. As a result, we’re entering 2016 with a team of six full-time researchers and over a year of runway.
Our next big push will be to close the gap between our new budget and our annual revenue. In order to sustain our current growth plans — which are aimed at expanding to a team of approximately ten full-time researchers — we’ll need to begin consistently taking in close to $2M per year by mid-2017.
I believe this is an achievable goal, though it will take some work. It will be even more valuable if we can overshoot this goal and begin extending our runway and further expanding our research program. On the whole, I’m very excited to see what this new year brings.

In addition to our fundraiser successes, we’ve begun seeing new grant-winning success. In collaboration with Stuart Russell at UC Berkeley, we’ve won a $75,000 grant from the Berkeley Center for Long-Term Cybersecurity. The bulk of the grant will go to funding a new postdoctoral position at UC Berkeley under Stuart Russell. The postdoc will collaborate with Russell and MIRI Research Fellow Patrick LaVictoire on the problem of AI corrigibility, as described in the grant proposal:
Consider a system capable of building accurate models of itself and its human operators. If the system is constructed to pursue some set of goals that its operators later realize will lead to undesirable behavior, then the system will by default have incentives to deceive, manipulate, or resist its operators to prevent them from altering its current goals (as that would interfere with its ability to achieve its current goals). […]
We refer to agents that have no incentives to manipulate, resist, or deceive their operators as “corrigible agents,” using the term as defined by Soares et al. (2015). We propose to study different methods for designing agents that are in fact corrigible.
This postdoctoral position has not yet been filled. Expressions of interest can be emailed to alex@intelligence.org using the subject line “UC Berkeley expression of interest.”
The post End-of-the-year fundraiser and grant successes appeared first on Machine Intelligence Research Institute.


January 2016 Newsletter

Research updates

A new paper: “Proof-Producing Reflection for HOL”
A new analysis: Safety Engineering, Target Selection, and Alignment Theory
New at IAFF: What Do We Need Value Learning For?; Strict Dominance for the Modified Demski Prior; Reflective Probability Distributions and Standard Models of Arithmetic; Existence of Distributions That Are Expectation-Reflective and Know It; Concise Open Problem in Logical Uncertainty

General updates

Our Winter Fundraiser is over! A total of 176 people donated $351,411, including some surprise matching donors. All of you have our sincere thanks.
Jed McCaleb writes on why MIRI matters, while Andrew Critch writes on the need to scale MIRI’s methods.
We attended NIPS, which hosted a symposium on the “social impacts of machine learning” this year. Viktoriya Krakovna summarizes her impressions.
We’ve moved to a new, larger office with the Center for Applied Rationality (CFAR), a few floors up from our old one.
Our paper announcements now have their own MIRI Blog category.

News and links

“The 21st Century Philosophers”: AI safety research gets covered in OZY.
Sam Altman and Elon Musk have brought together leading AI researchers to form a new $1 billion nonprofit, OpenAI. Andrej Karpathy explains OpenAI’s plans (link), and Altman and Musk provide additional background (link).
Alphabet chairman Eric Schmidt and Google Ideas director Jared Cohen write on the need to “establish best practices to avoid undesirable outcomes” from AI.
A new Future of Humanity Institute (FHI) paper: “Learning the Preferences of Ignorant, Inconsistent Agents.”
Luke Muehlhauser and The Telegraph signal-boost FHI’s AI safety job postings (deadline Jan. 6). The Global Priorities Project is also seeking summer interns (deadline Jan. 10).
CFAR is running a matching fundraiser through the end of January.

The post January 2016 Newsletter appeared first on Machine Intelligence Research Institute.

January 2016 Newsletter