February 2018 Newsletter

Updates

New at IAFF: An Untrollable Mathematician
New at AI Impacts: 2015 FLOPS Prices
We presented “Incorrigibility in the CIRL Framework” at the AAAI/ACM Conference on AI, Ethics, and Society.
From MIRI researcher Scott Garrabrant: Sources of Intuitions and Data on AGI

News and links

In “Adversarial Spheres,” Gilmer et al. investigate the tradeoff between test error and vulnerability to adversarial perturbations in many-dimensional spaces.
Recent posts on Less Wrong: Critch on “Taking AI Risk Seriously” and Ben Pace’s background model for assessing AI x-risk plans.
“Solving the AI Race“: GoodAI is offering prizes for proposed responses to the problem that “key stakeholders, including [AI] developers, may ignore or underestimate safety procedures, or agreements, in favor of faster utilization”.
The Open Philanthropy Project is hiring research analysts in AI alignment, forecasting, and strategy, along with generalist researchers and operations staff.

The post February 2018 Newsletter appeared first on Machine Intelligence Research Institute.

February 2018 Newsletter


January 2018 Newsletter

Our 2017 fundraiser was a huge success, with 341 donors contributing a total of $2.5 million!
Some of the largest donations came from Ethereum inventor Vitalik Buterin, bitcoin investors Christian Calderon and Marius van Voorden, poker players Dan Smith and Tom and Martin Crowley (as part of a matching challenge), and the Berkeley Existential Risk Initiative. Thank you to everyone who contributed!
Research updates

The winners of the first AI Alignment Prize include Scott Garrabrant’s Goodhart Taxonomy and recent IAFF posts: Vanessa Kosoy’s Why Delegative RL Doesn’t Work for Arbitrary Environments and More Precise Regret Bound for DRL, and Alex Mennen’s Being Legible to Other Agents by Committing to Using Weaker Reasoning Systems and Learning Goals of Simple Agents.
New at AI Impacts: Human-Level Hardware Timeline; Effect of Marginal Hardware on Artificial General Intelligence
We’re hiring for a new position at MIRI: ML Living Library, a specialist on the newest developments in machine learning.

General updates

From Eliezer Yudkowsky: A Reply to Francois Chollet on Intelligence Explosion.
Counterterrorism experts Richard Clarke and R. P. Eddy profile Yudkowsky in their new book Warnings: Finding Cassandras to Stop Catastrophes.
There have been several recent blog posts recommending MIRI as a donation target: from Ben Hoskin, Zvi Mowshowitz, Putanumonit, and the Open Philanthropy Project’s Daniel Dewey and Nick Beckstead.

News and links

A generalization of the AlphaGo algorithm, AlphaZero, achieves rapid superhuman performance on Chess and Shogi.
Also from Google DeepMind: “Specifying AI Safety Problems in Simple Environments.”
Viktoriya Krakovna reports on NIPS 2017: “This year’s NIPS gave me a general sense that near-term AI safety is now mainstream and long-term safety is slowly going mainstream. […] There was a lot of great content on the long-term side, including several oral / spotlight presentations and the Aligned AI workshop.”
80,000 Hours interviews Phil Tetlock and investigates the most important talent gaps in the EA community.
From Seth Baum: “A Survey of AGI Projects for Ethics, Risk, and Policy.” And from the Foresight Institute: “AGI: Timeframes & Policy.”
The Future of Life Institute is collecting proposals for a second round of AI safety grants, due February 18.

The post January 2018 Newsletter appeared first on Machine Intelligence Research Institute.

January 2018 Newsletter


Fundraising success!

Our 2017 fundraiser is complete! We’ve had an incredible month, with, by far, our largest fundraiser success to date. More than 300 distinct donors gave just over $2.5M1, doubling our third fundraising target of $1.25M. Thank you!

Target 1$625,000CompletedTarget 2$850,000CompletedTarget 3$1,250,000Completed

$2,504,625 raised in total!
358 donors contributed

×
Target Descriptions

Target 1
Target 2
Target 3

$625k: Basic target
At this funding level, we’ll be in a good position to pursue our mainline hiring goal in 2018, although we will likely need to halt or slow our growth in 2019.

$850k: Mainline-growth target
At this level, we’ll be on track to fully fund our planned expansion over the next few years, allowing us to roughly double the number of research staff over the course of 2018 and 2019.

$1.25M: Rapid-growth target
At this funding level, we will be on track to maintain a 1.5-year runway even if our hiring proceeds a fair amount faster than our mainline prediction. We’ll also have greater freedom to pay higher salaries to top-tier candidates as needed.

Our largest donation came toward the very end of the fundraiser in the form of an Ethereum donation worth $763,970 from Vitalik Buterin, the inventor and co-founder of Ethereum. Vitalik’s donation represents the third-largest single contribution we’ve received to date, after a $1.25M grant disbursement from the Open Philanthropy Project in October, and a $1.01M Ethereum donation in May.
In our mid-fundraiser update, we noted that MIRI was included in a large Matching Challenge: In partnership with Raising For Effective Giving, professional poker players Dan Smith, Tom Crowley and Martin Crowley announced they would match all donations to MIRI and nine other organizations through the end of December. Donors helped get us to our matching cap of $300k within 2 weeks, resulting in a $300k match from Dan, Tom, and Martin (thanks guys!). Other big winners from the Matching Challenge, which raised $4.5m (match included) in less than 3 weeks, include GiveDirectly ($588k donated) and the Good Food Institute ($416k donated).
Other big donations we received in December included:

$367,575 from Christian Calderon
$100,000 from the Berkeley Existential Risk Institute
$59,251 from Marius van Voorden

We also received substantial support from medium-sized donors: a total of $631,595 from the 42 donors who gave $5,000–$50,000 and a total of $113,556 from the 75 who gave $500–$5,000 (graph). We also are grateful to donors who leveraged their employers’ matching generosity, donating a combined amount of over $100,000 during December.
66% of funds donated during this fundraiser were in the form of cryptocurrency (mainly Bitcoin and Ethereum), including Vitalik, Marius, and Christian’s donations, along with Dan, Tom, and Martin’s matching contributions.
Overall, we’ve had an amazingly successful month and a remarkable year! I’m extremely grateful for all the support we’ve received, and excited about the opportunity this creates for us to grow our research team more quickly. For details on our growth plans, see our fundraiser post.
The exact total might increase slightly over the coming weeks as we process donations initiated in December 2017 that arrive in January 2018.The post Fundraising success! appeared first on Machine Intelligence Research Institute.

Fundraising success!


End-of-the-year matching challenge!

Update 2017-12-27: We’ve blown past our 3rd and final target, and reached the matching cap of $300,000 for the Matching Challenge! Thanks so much to everyone who supported us!
All donations made before 23:59 PST on Dec 31st will continue to be counted towards our fundraiser total. The fundraiser total includes projected matching funds from the Challenge.

Professional poker players Martin Crowley, Tom Crowley, and Dan Smith, in partnership with Raising for Effective Giving, have just announced a $1 million Matching Challenge and included MIRI among the 10 organizations they are supporting!
Give to any of the organizations involved before noon (PST) on December 31 for your donation to be eligible for a dollar-for-dollar match, up to the $1 million limit!
The eligible organizations for matching are:

Animal welfare — Effective Altruism Funds’ animal welfare fund, The Good Food Institute
Global health and development — Against Malaria Foundation, Schistosomiasis Control Initiative, Helen Keller International’s vitamin A supplementation program, GiveDirectly
Global catastrophic risk — MIRI
Criminal justice reform — Brooklyn Community Bail Fund, Massachusetts Bail Fund, Just City Memphis

The Matching Challenge’s website lists two options for MIRI donors to get matched: (1) donating on 2017charitydrive.com, or (2) donating directly on MIRI’s website and sending the receipt to receiptsforcharity@gmail.com. We recommend option 2, particularly for US tax residents (because MIRI is a 501(c)(3) organization) and those looking for a wider array of payment methods.

In other news, we’ve hit our first fundraising target ($625,000)!
We’re also happy to announce that we’ve received a $368k bitcoin donation from Christian Calderon, a cryptocurrency enthusiast, and also a donation worth $59k from early bitcoin investor Marius van Voorden.
In total, so far, we’ve received donations valued $697,638 from 137 distinct donors, 76% of it in the form of cryptocurrency (48% if we exclude Christian’s donation). Thanks as well to Jacob Falkovich for his fundraiser/matching post whose opinion distribution curves plausibly raised over $27k for MIRI this week, including his match.
Our funding drive will be continuing through the end of December, along with the Matching Challenge. Current progress (updated live):

Donate

Correction December 17: I previously listed GiveWell as one of the eligible organizations for matching, which is not correct.
The post End-of-the-year matching challenge! appeared first on Machine Intelligence Research Institute.

End-of-the-year matching challenge!


ML Living Library Opening

Update Jan. 2021: We’re no longer seeking applications for this position.

The Machine Intelligence Research Institute is looking for a very specialized autodidact to keep us up to date on developments in machine learning—a “living library” of new results.
ML is a fast-moving and diverse field, making it a challenge for any group to stay updated on all the latest and greatest developments. To support our AI alignment research efforts, we want to hire someone to read every interesting-looking paper about AI and machine learning, and keep us abreast of noteworthy developments, including new techniques and insights.
We expect that this will sound like a very fun job to a lot of people! However, this role is important to us, and we need to be appropriately discerning—we do not recommend applying if you do not already have a proven ability in this or neighboring domains.
Our goal is to hire full-time, ideally for someone who would be capable of making a multi-year commitment—we intend to pay you to become an expert on the cutting edge of machine learning, and don’t want to make the human capital investment unless you’re interested in working with us long-term.

About the Role
We’d like to fill this position as soon as we find the right candidate. Our hiring process tends to involve a lot of sample tasks and probationary hires, so if you are interested, we encourage you to apply early.
This is a new position for a kind of work that isn’t standard. Although we hope to find someone who can walk in off the street and perform well, we’re also interested in candidates who think they might take three months of training to meet the requirements.
Examples of the kinds of work you’ll do:

Read through archives and journals to get a sense of literally every significant development in the field, past and present.
Track general trends in the ML space—e.g., “Wow, there sure is a lot of progress being made on Dota 2!”—and let us know about them.
Help an engineer figure out why their code isn’t working—e.g., “Oh, you forgot the pooling layer in your convolutional neural network.”
Answer/research MIRI staff questions about ML techniques or the history of the field.
Share important developments proactively; researchers who haven’t read the same papers as you often won’t know the right questions to ask unprompted!

The Ideal Candidate
Some qualities of the ideal candidate:

Extensive breadth and depth of machine learning knowledge, including the underlying math.
Familiarity with ideas related to AI alignment.
Delight at the thought of getting paid to Know All The Things.
Programming capability—for example, you’ve replicated some ML papers.
Ability to deeply understand and crisply communicate ideas. Someone who just recites symbols to our team without understanding them isn’t much better than a web scraper.
Enthusiasm about the prospect of working at MIRI and helping advance the field’s understanding of AI alignment.
Residence in (or willingness to move to) the Bay Area. This job requires high-bandwidth communication with the research team, and won’t work as a remote position.
Ability to work independently with minimal supervision, and also to work in team/group settings.

Working at MIRI
We strive to make working at MIRI a rewarding experience.

Modern Work Spaces — Many of us have adjustable standing desks with large external monitors. We consider workspace ergonomics important, and try to rig up workstations to be as comfortable as possible. Free snacks, drinks, and meals are also provided at our office.
Flexible Hours — We don’t have strict office hours, and we don’t limit employees’ vacation days. Our goal is to make rapid progress on our research agenda, and we would prefer that staff take a day off rather than extend tasks to fill an extra day.
Living in the Bay Area — MIRI’s office is located in downtown Berkeley, California. From our office, you’re a 30-second walk to the BART (Bay Area Rapid Transit), which can get you around the Bay Area; a 3-minute walk to UC Berkeley campus; and a 30-minute BART ride to downtown San Francisco.

EEO & Employment Eligibility
MIRI is an equal opportunity employer. We are committed to making employment decisions based on merit and value. This commitment includes complying with all federal, state, and local laws. We desire to maintain a work environment free of harassment or discrimination due to sex, race, religion, color, creed, national origin, sexual orientation, citizenship, physical or mental disability, marital status, familial status, ethnicity, ancestry, status as a victim of domestic violence, age, or any other status protected by federal, state, or local laws

Apply
If interested, click here to apply. For questions or comments, email Buck (buck@intelligence.org).
The post ML Living Library Opening appeared first on Machine Intelligence Research Institute.

ML Living Library Opening


A reply to Francois Chollet on intelligence explosion

This is a reply to Francois Chollet, the inventor of the Keras wrapper for the Tensorflow and Theano deep learning systems, on his essay “The impossibility of intelligence explosion.”
In response to critics of his essay, Chollet tweeted:

If you post an argument online, and the only opposition you get is braindead arguments and insults, does it confirm you were right? Or is it just self-selection of those who argue online?
And he earlier tweeted:

Don’t be overly attached to your views; some of them are probably incorrect. An intellectual superpower is the ability to consider every new idea as if it might be true, rather than merely checking whether it confirms/contradicts your current views.
Chollet’s essay seemed mostly on-point and kept to the object-level arguments.  I am led to hope that Chollet is perhaps somebody who believes in abiding by the rules of a debate process, a fan of what I’d consider Civilization; and if his entry into this conversation has been met only with braindead arguments and insults, he deserves a better reply.  I’ve tried here to walk through some of what I’d consider the standard arguments in this debate as they bear on Chollet’s statements.
As a meta-level point, I hope everyone agrees that an invalid argument for a true conclusion is still a bad argument.  To arrive at the correct belief state we want to sum all the valid support, and only the valid support.  To tally up that support, we need to have a notion of judging arguments on their own terms, based on their local structure and validity, and not excusing fallacies if they support a side we agree with for other reasons.
My reply to Chollet doesn’t try to carry the entire case for the intelligence explosion as such.  I am only going to discuss my take on the validity of Chollet’s particular arguments.  Even if the statement “an intelligence explosion is impossible” happens to be true, we still don’t want to accept any invalid arguments in favor of that conclusion.
Without further ado, here are my thoughts in response to Chollet.

The basic premise is that, in the near future, a first “seed AI” will be created, with general problem-solving abilities slightly surpassing that of humans. This seed AI would start designing better AIs, initiating a recursive self-improvement loop that would immediately leave human intelligence in the dust, overtaking it by orders of magnitude in a short time.
I agree this is more or less what I meant by “seed AI” when I coined the term back in 1998.  Today, nineteen years later, I would talk about a general question of “capability gain” or how the power of a cognitive system scales with increased resources and further optimization.  The idea of recursive self-improvement is only one input into the general questions of capability gain; for example, we recently saw some impressively fast scaling of Go-playing ability without anything I’d remotely consider as seed AI being involved.  That said, I think that a lot of the questions Chollet raises about “self-improvement” are relevant to capability-gain theses more generally, so I won’t object to the subject of conversation.

Proponents of this theory also regard intelligence as a kind of superpower, conferring its holders with almost supernatural capabilities to shape their environment
A good description of a human from the perspective of a chimpanzee.
From a certain standpoint, the civilization of the year 2017 could be said to have “magic” from the perspective of 1517. We can more precisely characterize this gap by saying that we in 2017 can solve problems using strategies that 1517 couldn’t recognize as a “solution” if described in advance, because our strategies depend on laws and generalizations not known in 1517.  E.g., I could show somebody in 1517 a design for a compressor-based air conditioner, and they would not be able to recognize this as “a good strategy for cooling your house” in advance of observing the outcome, because they don’t yet know about the temperature-pressure relation.  A fancy term for this would be “strong cognitive uncontainability”; a metaphorical term would be “magic” although of course we did not do anything actually supernatural.  A similar but much larger gap exists between a human and a smaller brain running the previous generation of software (aka a chimpanzee).
It’s not exactly unprecedented to suggest that big gaps in cognitive ability correspond to big gaps in pragmatic capability to shape the environment.  I think a lot of people would agree in characterizing intelligence as the Human Superpower, independently of what they thought about the intelligence explosion hypothesis.

— as seen in the science-fiction movie Transcendence (2014), for instance.

I agree that public impressions of things are things that someone ought to be concerned about.  If I take a ride-share and I mention that I do anything involving AI, half the time the driver says, “Oh, like Skynet!”  This is an understandable reason to be annoyed.  But if we’re trying to figure out the sheerly factual question of whether an intelligence explosion is possible and probable, it’s important to consider the best arguments on all sides of all relevant points, not the popular arguments.  For that purpose it doesn’t matter if Deepak Chopra’s writing on quantum mechanics has a larger readership than any actual physicist.
Thankfully Chollet doesn’t spend the rest of the essay attacking Kurzweil in particular, so I’ll leave this at that.

 The intelligence explosion narrative equates intelligence with the general problem-solving ability displayed by individual intelligent agents — by current human brains, or future electronic brains.
I don’t see what work the word “individual” is doing within this sentence.  From our perspective, it matters little whether a computing fabric is imagined to be a hundred agents or a single agency, if it seems to behave in a coherent goal-directed way as seen from outside.  The pragmatic consequences are the same.  I do think it’s fair to say that I think about “agencies” which from our outside perspective seem to behave in a coherent goal-directed way.

 The first issue I see with the intelligence explosion theory is a failure to recognize that intelligence is necessarily part of a broader system — a vision of intelligence as a “brain in jar” that can be made arbitrarily intelligent independently of its situation.
I’m not aware of myself or Nick Bostrom or another major technical voice in this field claiming that problem-solving can go on independently of the situation/environment.
That said, some systems function very well in a broad variety of structured low-entropy environments.  E.g. the human brain functions much better than other primate brains in an extremely broad set of environments, including many that natural selection did not explicitly optimize for.  We remain functional on the Moon, because the Moon has enough in common with the Earth on a sufficiently deep meta-level that, for example, induction on past experience goes on functioning there.  Now if you tossed us into a universe where the future bore no compactly describable relation to the past, we would indeed not do very well in that “situation”—but this is not pragmatically relevant to the impact of AI on our own real world, where the future does bear a relation to the past.

In particular, there is no such thing as “general” intelligence. On an abstract level, we know this for a fact via the “no free lunch” theorem — stating that no problem-solving algorithm can outperform random chance across all possible problems.
Scott Aaronson’s reaction:  “Citing the ‘No Free Lunch Theorem’—i.e., the (trivial) statement that you can’t outperform brute-force search on random instances of an optimization problem—to claim anything useful about the limits of AI, is not a promising sign.”
It seems worth spelling out an as-simple-as-possible special case of this point in mathy detail, since it looked to me like a central issue given the rest of Chollet’s essay. I expect this math isn’t new to Chollet, but reprise it here to establish common language and for the benefit of everyone else reading along.
Laplace’s Rule of Succession, as invented by Thomas Bayes, gives us one simple rule for predicting future elements of a binary sequence based on previously observed elements.  Let’s take this binary sequence to be a series of “heads” and “tails” generated by some sequence generator called a “coin”, not assumed to be fair.  In the standard problem setup yielding the Rule of Succession, our state of prior ignorance is that we think there is some frequency \(\theta\) that a coin comes up heads, and for all we know \(\theta\) is equally likely to take on any real value between \(0\) and and \(1\).  We can do some Bayesian inference and conclude that after seeing \(M\) heads and \(N\) tails, we should predict that the odds for heads : tails on the next coinflip are:

$$\frac{M + 1}{M + N + 2} : \frac{N + 1}{M + N + 2}$$

(See Laplace’s Rule of Succession for the proof.)
This rule yields advice like:  “If you haven’t yet observed any coinflips, assign 50-50 to heads and tails” or “If you’ve seen four heads and no tails, assign 1/6 probability rather than 0 probability to the next flip being tails” or “If you’ve seen the coin come up heads 150 times and tails 75 times, assign around 2/3 probability to the coin coming up heads next time.”
Now this rule does not do super-well in any possible kind of environment.  In particular, it doesn’t do any better than the maximum-entropy prediction “the next flip has a 50% probability of being heads, or tails, regardless of what we have observed previously” if the environment is in fact a fair coin.  In general, there is “no free lunch” on predicting arbitrary binary sequences; if you assign greater probability mass or probability density to one binary sequence or class of sequences, you must have done so by draining probability from other binary sequences.  If you begin with the prior that every binary sequence is equally likely, then you never expect any algorithm to do better on average than maximum entropy, even if that algorithm luckily does better in one particular random draw.
On the other hand, if you start from the prior that every binary sequence is equally likely, you never notice anything a human would consider an obvious pattern.  If you start from the maxentropy prior, then after observing a coin come up heads a thousand times, and tails never, you still predict 50-50 on the next draw; because on the maxentropy prior, the sequence “one thousand heads followed by tails” is exactly as likely as “one thousand heads followed by heads”.
The inference rule instantiated by Laplace’s Rule of Succession does better in a generic low-entropy universe of coinflips.  It doesn’t start from specific knowledge; it doesn’t begin from the assumption that the coin is biased heads, or biased tails.  If the coin is biased heads, Laplace’s Rule learns that; if the coin is biased tails, Laplace’s Rule will soon learn that from observation as well.  If the coin is actually fair, then Laplace’s Rule will rapidly converge to assigning probabilities in the region of 50-50 and not do much worse per coinflip than if we had started with the max-entropy prior.
Can you do better than Laplace’s Rule of Succession?  Sure; if the environment’s probability of generating heads is equal to 0.73 and you start out knowing that, then you can guess on the very first round that the probability of seeing heads is 73%.  But even with this non-generic and highly specific knowledge built in, you do not do very much better than Laplace’s Rule of Succession unless the first coinflips are very important to your future survival.  Laplace’s Rule will probably figure out the answer is somewhere around 3/4 in the first dozen rounds, and get to the answer being somewhere around 73% after a couple of hundred rounds, and if the answer isn’t 0.73 it can handle that case too.
Is Laplace’s Rule the most general possible rule for inferring binary sequences?  Obviously not; for example, if you saw the initial sequence…
$$HTHTHTHTHTHTHTHT…$$

…then you would probably guess with high though not infinite probability that the next element generated would be \(H\).  This is because you have the ability to recognize a kind of pattern which Laplace’s Rule does not, i.e., alternating heads and tails.  Of course, your ability to recognize this pattern only helps you in environments that sometimes generate a pattern like that—which the real universe sometimes does.  If we tossed you into a universe which just as frequently presented you with ‘tails’ after observing a thousand perfect alternating pairs, as it did ‘heads’, then your pattern-recognition ability would be useless.  Of course, a max-entropy universe like that will usually not present you with a thousand perfect alternations in the initial sequence to begin with!
One extremely general but utterly intractable inference rule is Solomonoff induction, a universal prior which assigns probabilities to every computable sequence (or computable probability distribution over sequences) proportional to algorithmic simplicity, that is, in inverse proportion to the exponential of the size of the program required to specify the computation.  Solomonoff induction can learn from observation any sequence that can be generated by a compact program, relative to a choice of universal computer which has at most a bounded effect on the amount of evidence required or the number of mistakes made.  Of course a Solomonoff inductor will do slightly-though-not-much-worse than the max-entropy prior in a hypothetical structure-avoiding universe in which algorithmically compressible sequences are less likely; thankfully we don’t live in a universe like that.
It would then seem perverse not to recognize that for large enough milestones we can see an informal ordering from less general inference rules to more general inference rules, those that do well in an increasingly broad and complicated variety of environments of the sort that the real world is liable to generate:
The rule that always assigns probability 0.73 to heads on each round, performs optimally within the environment where each flip has independently a 0.73 probability of coming up heads.
Laplace’s Rule of Succession will start to do equally well as this, given a couple of hundred initial coinflips to see the pattern; and Laplace’s Rule also does well in many other low-entropy universes besides, such as those where each flip has 0.07 probability of coming up heads.
A human is more general and can also spot patterns like \(HTTHTTHTTHTT\) where Laplace’s Rule would merely converge to assigning probability 1/3 of each flip coming up heads, while the human becomes increasingly certain that a simple temporal process is at work which allows each succeeding flip to be predicted with near-certainty.
If anyone ever happened across a hypercomputational device and built a Solomonoff inductor out of it, the Solomonoff inductor would be more general than the human and do well in any environment with a programmatic description substantially smaller than the amount of data the Solomonoff inductor could observe.
None of these predictors need do very much worse than the max-entropy prediction in the case that the environment is actually max-entropy.  It may not be a free lunch, but it’s not all that expensive even by the standards of hypothetical randomized universes; not that this matters for anything, since we don’t live in a max-entropy universe and therefore we don’t care how much worse we’d do in one.
Some earlier informal discussion of this point can be found in No-Free-Lunch Theorems Are Often Irrelevant.

If intelligence is a problem-solving algorithm, then it can only be understood with respect to a specific problem.
Some problems are more general than other problems—not relative to a maxentropy prior, which treats all problem subclasses on an equal footing, but relative to the low-entropy universe we actually live in, where a sequence of a million observed heads is on the next round more liable to generate H than T.  Similarly, relative to the problem classes tossed around in our low-entropy universe, “figure out what simple computation generates this sequence” is more general than a human which is more general than “figure out what is the frequency of heads or tails within this sequence.”
Human intelligence is a problem-solving algorithm that can be understood with respect to a specific problem class that is potentially very, very broad in a pragmatic sense.

In a more concrete way, we can observe this empirically in that all intelligent systems we know are highly specialized. The intelligence of the AIs we build today is hyper specialized in extremely narrow tasks — like playing Go, or classifying images into 10,000 known categories. The intelligence of an octopus is specialized in the problem of being an octopus. The intelligence of a human is specialized in the problem of being human.
The problem that a human solves is much more general than the problem an octopus solves, which is why we can walk on the Moon and the octopus can’t.  We aren’t absolutely general—the Moon still has a certain something in common with the Earth.  Scientific induction still works on the Moon.  It is not the case that when you get to the Moon, the next observed charge of an electron has nothing to do with its previously observed charge; and if you throw a human into an alternate universe like that one, the human stops working.  But the problem a human solves is general enough to pass from oxygen environments to the vacuum.

What would happen if we were to put a freshly-created human brain in the body of an octopus, and let in live at the bottom of the ocean? Would it even learn to use its eight-legged body? Would it survive past a few days? … The brain has hardcoded conceptions of having a body with hands that can grab, a mouth that can suck, eyes mounted on a moving head that can be used to visually follow objects (the vestibulo-ocular reflex), and these preconceptions are required for human intelligence to start taking control of the human body.
It could be the case that in this sense a human’s motor cortex is analogous to an inference rule that always predicts heads with 0.73 probability on each round, and cannot learn to predict 0.07 instead.  It could also be that our motor cortex is more like a Laplace inductor that starts out with 72 heads and 26 tails pre-observed, biased toward that particular ratio, but which can eventually learn 0.07 after another thousand rounds of observation.
It’s an empirical question, but I’m not sure why it’s a very relevant one.  It’s possible that human motor cortex is hyperspecialized—not just jumpstarted with prior knowledge, but inductively narrow and incapable of learning better—since in the ancestral environment, we never got randomly plopped into octopus bodies.  But what of it?  If you put some humans at a console and gave them a weird octopus-like robot to learn to control, I’d expect their full deliberate learning ability to do better than raw motor cortex in this regard.  Humans using their whole intelligence, plus some simple controls, can learn to drive cars and fly airplanes even though those weren’t in our ancestral environment.
We also have no reason to believe human motor cortex is the limit of what’s possible.  If we sometimes got plopped into randomly generated bodies, I expect we’d already have motor cortex that could adapt to octopodes.  Maybe MotorCortex Zero could do three days of self-play on controlling randomly generated bodies and emerge rapidly able to learn any body in that class.  Or, humans who are allowed to use Keras could figure out how to control octopus arms using ML.  The last case would be most closely analogous to that of a hypothetical seed AI.

Empirical evidence is relatively scarce, but from what we know, children that grow up outside of the nurturing environment of human culture don’t develop any human intelligence. Feral children raised in the wild from their earliest years become effectively animals, and can no longer acquire human behaviors or language when returning to civilization.
Human visual cortex doesn’t develop well without visual inputs.  This doesn’t imply that our visual cortex is a simple blank slate, and that all the information to process vision is stored in the environment, and the visual cortex just adapts to that from a blank slate; if that were true, we’d expect it to easily take control of octopus eyes.  The visual cortex requires visual input because of the logic of evolutionary biology: if you make X an environmental constant, the species is liable to acquire genes that assume the presence of X.  It has no reason not to.  The expected result would be that the visual cortex contains a large amount of genetic complexity that makes it better than generic cerebral cortex at doing vision, but some of this complexity requires visual input during childhood to unfold correctly.
But if in the ancestral environment children had grown up in total darkness 10% of the time, before seeing light for the first time on adulthood, it seems extremely likely that we could have evolved to not require visual input in order for the visual cortex to wire itself up correctly.  E.g., the retina could have evolved to send in simple hallucinatory shapes that would cause the rest of the system to wire itself up to detect those shapes, or something like that.
Human children reliably grow up around other humans, so it wouldn’t be very surprising if humans evolved to build their basic intellectual control processes in a way that assumes the environment contains this info to be acquired.  We cannot thereby infer how much information is being “stored” in the environment or that an intellectual control process would be too much information to store genetically; that is not a problem evolution had reason to try to solve, so we cannot infer from the lack of an evolved solution that such a solution was impossible.
And even if there’s no evolved solution, this doesn’t mean you can’t intelligently design a solution. Natural selection never built animals with steel bones or wheels for limbs, because there’s no easy incremental pathway there through a series of smaller changes, so those designs aren’t very evolvable; but human engineers still build skyscrapers and cars, etcetera.
Among humans, the art of Go is stored in a vast repository of historical games and other humans, and future Go masters among us grow up playing Go as children against superior human masters rather than inventing the whole art from scratch. You would not expect even the most talented human, reinventing the gameplay all on their own, to be able to win a competition match with a first-dan pro.
But AlphaGo was initialized on this vast repository of played games in stored form, rather than it needing to actually play human masters.
And then less than two years later, AlphaGo Zero taught itself to play at a vastly human-superior level, in three days, by self-play, from scratch, using a much simpler architecture with no ‘instinct’ in the form of precomputed features.
Now one may perhaps postulate that there is some sharp and utter distinction between the problem that AlphaGo Zero solves, and the much more general problem that humans solve, whereby our vast edifice of Go knowledge can be surpassed by a self-contained system that teaches itself, but our general cognitive problem-solving abilities can neither be compressed into a database for initialization, nor taught by self-play.  But why suppose that?  Human civilization taught itself by a certain sort of self-play; we didn’t learn from aliens.  More to the point, I don’t see a sharp and utter distinction between Laplace’s Rule, AlphaGo Zero, a human, and a Solomonoff inductor; they just learn successively more general problem classes.  If AlphaGo Zero can waltz past all human knowledge of Go, I don’t see a strong reason why AGI Zero can’t waltz past the human grasp of how to reason well, or how to perform scientific investigations, or how to learn from the data in online papers and databases.
This point could perhaps be counterargued, but it hasn’t yet been counterargued to my knowledge, and it certainly isn’t settled by any theorem of computer science known to me.

If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, then you cannot hope to arbitrarily increase the intelligence of an agent merely by tuning its brain — no more than you can increase the throughput of a factory line by speeding up the conveyor belt. Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment.
It’s not obvious to me why any of this matters.  Say an AI takes three days to learn to use an octopus body.  So what?
That is:  We agree that it’s a mathematical truth that you need “some amount” of experience to go from a broadly general prior to a specific problem.  That doesn’t mean that the required amount of experience is large for pragmatically important problems, or that it takes three decades instead of three days.  We cannot casually pass from “proven: some amount of X is required” to “therefore: a large amount of X is required” or “therefore: so much X is required that it slows things down a lot”.  (See also: Harmless supernova fallacy: bounded, therefore harmless.)

If the gears of your brain were the defining factor of your problem-solving ability, then those rare humans with IQs far outside the normal range of human intelligence would live lives far outside the scope of normal lives, would solve problems previously thought unsolvable, and would take over the world — just as some people fear smarter-than-human AI will do.
“von Neumann? Newton? Einstein?” —Scott Aaronson
More importantly:  Einstein et al. didn’t have brains that were 100 times larger than a human brain, or 10,000 times faster.  By the logic of sexual recombination within a sexually reproducing species, Einstein et al. could not have had a large amount of de novo software that isn’t present in a standard human brain.  (That is:  An adaptation with 10 necessary parts, each of which is only 50% prevalent in the species, will only fully assemble 1 out of 1000 times, which isn’t often enough to present a sharp selection gradient on the component genes; complex interdependent machinery is necessarily universal within a sexually reproducing species, except that it may sometimes fail to fully assemble.  You don’t get “mutants” with whole new complex abilities a la the X-Men.)
Humans are metaphorically all compressed into one tiny little dot in the vastness of mind design space.  We’re all the same make and model of car running the same engine under the hood, in slightly different sizes and with slightly different ornaments, and sometimes bits and pieces are missing.  Even with respect to other primates, from whom we presumably differ by whole complex adaptations, we have 95% shared genetic material with chimpanzees.   Variance between humans is not something that thereby establishes bounds on possible variation in intelligence, unless you import some further assumption not described here.
The standard reply to anyone who deploys e.g. the Argument from Gödel to claim the impossibility of AGI is to ask, “Why doesn’t your argument rule out humans?”
Similarly, a standard question that needs to be answered by anyone who deploys an argument against the possibility of superhuman general intelligence is, “Why doesn’t your argument rule out humans exhibiting pragmatically much greater intellectual performance than chimpanzees?”
Specialized to this case, we’d ask, “Why doesn’t the fact that the smartest chimpanzees aren’t building rockets let us infer that no human can walk on the Moon?”
No human, not even John von Neumann, could have reinvented the gameplay of Go on their own and gone on to stomp the world’s greatest Masters. AlphaGo Zero did so in three days. It’s clear that in general, “We can infer the bounds of cognitive power from the bounds of human variation” is false. If there’s supposed to be some special case of this rule which is true rather than false, and forbids superhuman AGI, that special case needs to be spelled out.

Intelligence is not a superpower; exceptional intelligence does not, on its own, confer you with proportionally exceptional power over your circumstances.
…said the Homo sapiens, surrounded by countless powerful artifacts whose abilities, let alone mechanisms, would be utterly incomprehensible to the organisms of any less intelligent Earthly species.

A high-potential human 10,000 years ago would have been raised in a low-complexity environment, likely speaking a single language with fewer than 5,000 words, would never have been taught to read or write, would have been exposed to a limited amount of knowledge and to few cognitive challenges. The situation is a bit better for most contemporary humans, but there is no indication that our environmental opportunities currently outpace our cognitive potential.
Does this imply that technology should be no more advanced 100 years from today, than it is today?  If not, in what sense have we taken every possible opportunity of our environment?
Is the idea that opportunities can only be taken in sequence, one after another, so that today’s technology only offers the possibilities of today’s advances?  Then why couldn’t a more powerful intelligence run through them much faster, and rapidly build up those opportunities?

A smart human raised in the jungle is but a hairless ape. Similarly, an AI with a superhuman brain, dropped into a human body in our modern world, would likely not develop greater capabilities than a smart contemporary human. If it could, then exceptionally high-IQ humans would already be displaying proportionally exceptional levels of personal attainment; they would achieve exceptional levels of control over their environment, and solve major outstanding problems— which they don’t in practice.
It can’t eat the Internet? It can’t eat the stock market?  It can’t crack the protein folding problem and deploy arbitrary biological systems?  It can’t get anything done by thinking a million times faster than we do?  All this is to be inferred from observing that the smartest human was no more impressive than John von Neumann?
I don’t see the strong Bayesian evidence here.  It seems easy to imagine worlds such that you can get a lot of pragmatically important stuff done if you have a brain 100 times the size of John von Neumann’s, think a million times faster, and have maxed out and transcended every human cognitive talent and not just the mathy parts, and yet have the version of John von Neumann inside that world be no more impressive than we saw.  How then do we infer from observing John von Neumann that we are not in such worlds?
We know that the rule of inferring bounds on cognition by looking at human maximums doesn’t work on AlphaGo Zero. Why does it work to infer that “An AGI can’t eat the stock market because no human has eaten the stock market”?

However, these billions of brains, accumulating knowledge and developing external intelligent processes over thousand of years, implement a system — civilization — which may eventually lead to artificial brains with greater intelligence than that of a single human. It is civilization as a whole that will create superhuman AI, not you, nor me, nor any individual. A process involving countless humans, over timescales we can barely comprehend. A process involving far more externalized intelligence — books, computers, mathematics, science, the internet — than biological intelligence…
Will the superhuman AIs of the future, developed collectively over centuries, have the capability to develop AI greater than themselves? No, no more than any of us can.
The premise is that brains of a particular size and composition that are running a particular kind of software (human brains) can only solve a problem X (which in this case is equal to “build an AGI”) if they cooperate in a certain group size N and run for a certain amount of time and build Z amount of external cognitive prostheses.  Okay.  Humans were not especially specialized on the AI-building problem by natural selection.  Why wouldn’t an AGI with larger brains, running faster, using less insane software, containing its own high-speed programmable cognitive hardware to which it could interface directly in a high-bandwidth way, and perhaps specialized on computer programming in exactly the way that human brains aren’t, get more done on net than human civilization?  Human civilization tackling Go devoted a lot of thinking time, parallel search, and cognitive prostheses in the form of playbooks, and then AlphaGo Zero blew past it in three days, etcetera.
To sharpen this argument:
We may begin from the premise, “For all problems X, if human civilization puts a lot of effort into X and gets as far as W, no single agency can get significantly further than W on its own,” and from this premise deduce that no single AGI will be able to build a new AGI shortly after the first AGI is built.
However, this premise is obviously false, as even Deep Blue bore witness.  Is there supposed to be some special case of this generalization which is true rather than false, and says something about the ‘build an AGI’ problem which it does not say about the ‘win a chess game’ problem?  Then what is that special case and why should we believe it?
Also relevant:  In the game of Kasparov vs. The World, the world’s best player Garry Kasparov played a single game against thousands of other players coordinated in an online forum, led by four chess masters.  Garry Kasparov’s brain eventually won, against thousands of times as much brain matter.  This tells us something about the inefficiency of human scaling with simple parallelism of the nodes, presumably due to the inefficiency and low bandwidth of human speech separating the would-be arrayed brains. It says that you do not need a thousand times as much processing power as one human brain to defeat the parallel work of a thousand human brains.  It is the sort of thing that can be done even by one human who is a little more talented and practiced than the components of that parallel array.  Humans often just don’t agglomerate very efficiently.

However, future AIs, much like humans and the other intelligent systems we’ve produced so far, will contribute to our civilization, and our civilization, in turn, will use them to keep expanding the capabilities of the AIs it produces.
This takes in the premise “AIs can only output a small amount of cognitive improvement in AI abilities” and reaches the conclusion “increase in AI capability will be a civilizationally diffuse process.”  I’m not sure that the conclusion follows, but would mostly dispute that the premise has been established by previous arguments.  To put it another way, this particular argument does not contribute anything new to support “AI cannot output much AI”, it just tries to reason further from that as a premise.

Our problem-solving abilities (in particular, our ability to design AI) are already constantly improving, because these abilities do not reside primarily in our biological brains, but in our external, collective tools. The recursive loop has been in action for a long time, and the rise of “better brains” will not qualitatively affect it — no more than any previous intelligence-enhancing technology.
From Arbital’s Harmless supernova fallacy page:

Precedented, therefore harmless: “Really, we’ve already had supernovas around for a while: there are already devices that produce ‘super’ amounts of heat by fusing elements low in the periodic table, and they’re called thermonuclear weapons. Society has proven well able to regulate existing thermonuclear weapons and prevent them from being acquired by terrorists; there’s no reason the same shouldn’t be true of supernovas.” (Noncentral fallacy / continuum fallacy: putting supernovas on a continuum with hydrogen bombs doesn’t make them able to be handled by similar strategies, nor does finding a category such that it contains both supernovas and hydrogen bombs.)

Our brains themselves were never a significant bottleneck in the AI-design process.
A startling assertion.  Let’s say we could speed up AI-researcher brains by a factor of 1000 within some virtual uploaded environment, not permitting them to do new physics or biology experiments, but still giving them access to computers within the virtual world.  Are we to suppose that AI development would take the same amount of sidereal time?  I for one would expect the next version of Tensorflow to come out much sooner, even taking into account that most individual AI experiments would be less grandiose because the sped-up researchers would need those experiments to complete faster and use less computing power.  The scaling loss would be less than total, just like adding CPUs a thousand times as fast to the current research environment would probably speed up progress by at most a factor of 5, not a factor of 1000.  Similarly, with all those sped-up brains we might see progress increase only by a factor of 50 instead of 1000, but I’d still expect it to go a lot faster.
Then in what sense are we not bottlenecked on the speed of human brains in order to build up our understanding of AI?

Crucially, the civilization-level intelligence-improving loop has only resulted in measurably linear progress in our problem-solving abilities over time.
I obviously don’t consider myself a Kurzweilian, but even I have to object that this seems like an odd assertion to make about the past 10,000 years.

Wouldn’t recursively improving X mathematically result in X growing exponentially? No — in short, because no complex real-world system can be modeled as `X(t + 1) = X(t) * a, a > 1)`.
This seems like a really odd assertion, refuted by a single glance at world GDP.  Note that this can’t be an isolated observation, because it also implies that every necessary input into world GDP is managing to keep up, and that every input which isn’t managing to keep up has been economically bypassed at least with respect to recent history.

We don’t have to speculate about whether an “explosion” would happen the moment an intelligent system starts optimizing its own intelligence. As it happens, most systems are recursively self-improving. We’re surrounded with them… Mechatronics is recursively self-improving — better manufacturing robots can manufacture better manufacturing robots. Military empires are recursively self-expanding — the larger your empire, the greater your military means to expand it further. Personal investing is recursively self-improving — the more money you have, the more money you can make.
If we define “recursive self-improvement” to mean merely “causal process containing at least one positive loop” then the world abounds with such, that is true.  It could still be worth distinguishing some feedback loops as going much faster than others: e.g., the cascade of neutrons in a nuclear weapon, or the cascade of information inside the transistors of a hypothetical seed AI.  This seems like another instance of “precedented therefore harmless” within the harmless supernova fallacy.

Software is just one cog in a bigger process — our economies, our lives — just like your brain is just one cog in a bigger process — human culture. This context puts a hard limit on the maximum potential usefulness of software, much like our environment puts a hard limit on how intelligent any individual can be — even if gifted with a superhuman brain.
“A chimpanzee is just one cog in a bigger process—the ecology.  Why postulate some kind of weird superchimp that can expand its superchimp economy at vastly greater rates than the amount of chimp-food produced by the current ecology?”
Concretely, suppose an agent is smart enough to crack inverse protein structure prediction, i.e., it can build its own biology and whatever amount of post-biological molecular machinery is permitted by the laws of physics.  In what sense is it still dependent on most of the economic outputs of the rest of human culture?  Why wouldn’t it just start building von Neumann machines?

Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as bottlenecks. Antagonistic processes will arise in response to recursive self-improvement and squash it .
Smart agents will try to deliberately bypass these bottlenecks and often succeed, which is why the world economy continues to grow at an exponential pace instead of having run out of wheat in 1200 CE.  It continues to grow at an exponential pace despite even the antagonistic processes of… but I’d rather not divert this conversation into politics.
Now to be sure, the smartest mind can’t expand faster than light, and its exponential growth will bottleneck on running out of atoms and negentropy if we’re remotely correct about the character of physical law.  But to say that this is therefore no reason to worry would be the “bounded, therefore harmless” variant of the harmless supernova fallacy.  A supernova isn’t infinitely hot, but it’s pretty darned hot and you can’t survive one just by wearing a Nomex jumpsuit.

When it comes to intelligence, inter-system communication arises as a brake on any improvement of underlying modules — a brain with smarter parts will have more trouble coordinating them;
Why doesn’t this prove that humans can’t be much smarter than chimps?
What we can infer about the scaling laws that were governing human brains from the evolutionary record is a complicated topic.  On this particular point I’d refer you to section 3.1, “Returns on brain size”, pp. 35–39, in my semitechnical discussion of returns on cognitive investment.  The conclusion there is that we can infer from the increase in equilibrium brain size over the last few million years of hominid history, plus the basic logic of population genetics, that over this time period there were increasing marginal returns to brain size with increasing time and presumably increasingly sophisticated neural ‘software’.  I also remark that human brains are not the only possible cognitive computing fabrics.

It is perhaps not a coincidence that very high-IQ people are more likely to suffer from certain mental illnesses.
I’d expect very-high-IQ chimps to be more likely to suffer from some neurological disorders than typical chimps.  This doesn’t tell us that chimps are approaching the ultimate hard limit of intelligence, beyond which you can’t scale without going insane.  It tells us that if you take any biological system and try to operate under conditions outside the typical ancestral case, it is more likely to break down.  Very-high-IQ humans are not the typical humans that natural selection has selected-for as normal operating conditions.

Yet, modern scientific progress is measurably linear. I wrote about this phenomenon at length in a 2012 essay titled “The Singularity is not coming”. We didn’t make greater progress in physics over the 1950–2000 period than we did over 1900–1950 — we did, arguably, about as well. Mathematics is not advancing significantly faster today than it did in 1920. Medical science has been making linear progress on essentially all of its metrics, for decades.
I broadly agree with respect to recent history.  I tend to see this as an artifact of human bureaucracies shooting themselves in the foot in a way that I would not expect to apply within a single unified agent.
It’s possible we’re reaching the end of available fruit in our finite supply of physics.  This doesn’t mean our present material technology could compete with the limits of possible material technology, which would at the very least include whatever biology-machine hybrid systems could be rapidly manufactured given the limits of mastery of biochemistry.

As scientific knowledge expands, the time and effort that have to be invested in education and training grows, and the field of inquiry of individual researchers gets increasingly narrow.
Our brains don’t scale to hold it all, and every time a new human is born you have to start over from scratch instead of copying and pasting the knowledge.  It does not seem to me like a slam-dunk to generalize from the squishy little brains yelling at each other to infer the scaling laws of arbitrary cognitive computing fabrics.

Intelligence is situational — there is no such thing as general intelligence. Your brain is one piece in a broader system which includes your body, your environment, other humans, and culture as a whole.
True of chimps; didn’t stop humans from being much smarter than chimps.

No system exists in a vacuum; any individual intelligence will always be both defined and limited by the context of its existence, by its environment.
True of mice; didn’t stop humans from being much smarter than mice.
Part of the argument above was, as I would perhaps unfairly summarize it, “There is no sense in which a human is absolutely smarter than an octopus.”  Okay, but pragmatically speaking, we have nuclear weapons and octopodes don’t.  A similar pragmatic capability gap between humans and unaligned AGIs seems like a matter of legitimate concern.  If you don’t want to call that an intelligence gap then call it what you like.

Currently, our environment, not our brain, is acting as the bottleneck to our intelligence.
I don’t see what observation about our present world licenses the conclusion that speeding up brains tenfold would produce no change in the rate of technological advancement.

Human intelligence is largely externalized, contained not in our brain but in our civilization. We are our tools — our brains are modules in a cognitive system much larger than ourselves.
What about this fact is supposed to imply slower progress by an AGI that has a continuous, high-bandwidth interaction with its own onboard cognitive tools?

A system that is already self-improving, and has been for a long time.
True if we redefine “self-improving” as “any positive feedback loop whatsoever”.  A nuclear fission weapon is also a positive feedback loop in neutrons triggering the release of more neutrons.  The elements of this system interact on a much faster timescale than human neurons fire, and thus the overall process goes pretty fast on our own subjective timescale.  I don’t recommend standing next to one when it goes off.

Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions arising from the broader context in which they exist, cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement.
Falsified by a graph of world GDP on almost any timescale.

In particular, this is the case for scientific progress — science being possibly the closest system to a recursively self-improving AI that we can observe.
I think we’re mostly just doing science wrong, but that would be a much longer discussion.
Fits-on-a-T-Shirt rejoinders would include “Why think we’re at the upper bound of being-good-at-science any more than chimps were?”

Recursive intelligence expansion is already happening — at the level of our civilization. It will keep happening in the age of AI, and it progresses at a roughly linear pace.
If this were to be true, I don’t think it would be established by the arguments given.
Much of this debate has previously been reprised by myself and Robin Hanson in the “AI Foom Debate.”  I expect that even Robin Hanson, who was broadly opposing my side of this debate, would have a coughing fit over the idea that progress within all systems is confined to a roughly linear pace.
For more reading I recommend my own semitechnical essay on what our current observations can tell us about the scaling of cognitive systems with increasing resources and increasing optimization, “Intelligence Explosion Microeconomics.”
The post A reply to Francois Chollet on intelligence explosion appeared first on Machine Intelligence Research Institute.


December 2017 Newsletter

Our annual fundraiser is live. Discussed in the fundraiser post:

News  — What MIRI’s researchers have been working on lately, and more.
Goals — We plan to grow our research team 2x in 2018–2019. If we raise $850k this month, we think we can do that without dipping below a 1.5-year runway.
Actual goals — A bigger-picture outline of what we think is the likeliest sequence of events that could lead to good global outcomes.

Our funding drive will be running until December 31st.

Research updates

New at IAFF: Reward Learning Summary; Reflective Oracles as a Solution to the Converse Lawvere Problem; Policy Selection Solves Most Problems
We ran a workshop on Paul Christiano’s research agenda.
We’ve hired the first members of our new engineering team, including math PhD Jesse Liptrap and former Quixey lead architect Nick Tarleton! If you’d like to join the team, apply here!
I’m also happy to announce that Blake Borgeson has come on in an advisory role to help establish our engineering program. Blake is a Nature-published computational biologist who co-founded Recursion Pharmaceuticals, where he leads the biotech company’s machine learning work.

General updates

“Security Mindset and Ordinary Paranoia”: a new dialogue from Eliezer Yudkowsky. See also part 2.
The Open Philanthropy Project has awarded MIRI a three-year, $3.75 million grant!
We received $48,132 in donations during Facebook’s Giving Tuesday event, of which $11,371 — from speedy donors who made the event’s 85-second cutoff! — will be matched by the Bill and Melinda Gates Foundation.
MIRI Staff Writer Matthew Graves has published an article in Skeptic magazine: “Why We Should Be Concerned About Artificial Superintelligence.”
Yudkowsky’s new book Inadequate Equilibria is out! See other recent discussion of modest epistemology and inadequacy analysis by Scott Aaronson, Robin Hanson, Abram Demski, Gregory Lewis, and Scott Alexander.

The post December 2017 Newsletter appeared first on Machine Intelligence Research Institute.

December 2017 Newsletter


MIRI’s 2017 Fundraiser

Update 2017-12-27: We’ve blown past our 3rd and final target, and reached the matching cap of $300,000 for the $2 million Matching Challenge! Thanks so much to everyone who supported us!
All donations made before 23:59 PST on Dec 31st will continue to be counted towards our fundraiser total. The fundraiser total includes projected matching funds from the Challenge.

MIRI’s 2017 fundraiser is live through the end of December! Our progress so far (updated live):

Target 1$625,000CompletedTarget 2$850,000CompletedTarget 3$1,250,000Completed

$2,504,625 raised in total!
358 donors contributed

×
Target Descriptions

Target 1
Target 2
Target 3

$625k: Basic target
At this funding level, we’ll be in a good position to pursue our mainline hiring goal in 2018, although we will likely need to halt or slow our growth in 2019.

$850k: Mainline-growth target
At this level, we’ll be on track to fully fund our planned expansion over the next few years, allowing us to roughly double the number of research staff over the course of 2018 and 2019.

$1.25M: Rapid-growth target
At this funding level, we will be on track to maintain a 1.5-year runway even if our hiring proceeds a fair amount faster than our mainline prediction. We’ll also have greater freedom to pay higher salaries to top-tier candidates as needed.

Donate Now

MIRI is a research nonprofit based in Berkeley, California with a mission of ensuring that smarter-than-human AI technology has a positive impact on the world. You can learn more about our work at “Why AI Safety?” or via MIRI Executive Director Nate Soares’ Google talk on AI alignment.
In 2015, we discussed our interest in potentially branching out to explore multiple research programs simultaneously once we could support a larger team. Following recent changes to our overall picture of the strategic landscape, we’re now moving ahead on that goal and starting to explore new research directions while also continuing to push on our agent foundations agenda. For more on our new views, see “There’s No Fire Alarm for Artificial General Intelligence” and our 2017 strategic update. We plan to expand on our relevant strategic thinking more in the coming weeks.
Our expanded research focus means that our research team can potentially grow big, and grow fast. Our current goal is to hire around ten new research staff over the next two years, mostly software engineers. If we succeed, our point estimate is that our 2018 budget will be $2.8M and our 2019 budget will be $3.5M, up from roughly $1.9M in 2017.1
We’ve set our fundraiser targets by estimating how quickly we could grow while maintaining a 1.5-year runway, on the simplifying assumption that about 1/3 of the donations we receive between now and the beginning of 2019 will come during our current fundraiser.2
Hitting Target 1 ($625k) then lets us act on our growth plans in 2018 (but not in 2019); Target 2 ($850k) lets us act on our full two-year growth plan; and in the case where our hiring goes better than expected, Target 3 ($1.25M) would allow us to add new members to our team about twice as quickly, or pay higher salaries for new research staff as needed.
We discuss more details below, both in terms of our current organizational activities and how we see our work fitting into the larger strategy space.

            What’s new at MIRI              |              Fundraising goals              |              Strategic background

What’s new at MIRI
New developments this year have included:

The release of Eliezer Yudkowsky’s Inadequate Equilibria: Where and How Civilizations Get Stuck, a book on systemic failure, outperformance, and epistemology.

New introductory material on decision theory: “Functional Decision Theory,” “Cheating Death in Damascus,” and “Decisions Are For Making Bad Outcomes Inconsistent.”

Extremely generous new support for our research in the form of a one-time $1.01 million donation from a cryptocurrency investor and a three-year $3.75 million grant from the Open Philanthropy Project.3

Thanks in part to this major support, we’re currently in a position to scale up the research team quickly if we can find suitable hires. We intend to explore a variety of new research avenues going forward, including making a stronger push to experiment and explore some ideas in implementation.4 This means that we’re currently interested in hiring exceptional software engineers, particularly ones with machine learning experience.
The two primary things we’re looking for in software engineers are programming ability and value alignment. Since we’re a nonprofit, it’s also worth noting explicitly that we’re generally happy to pay excellent research team applicants with the relevant skills whatever salary they would need to work at MIRI. If you think you’d like to work with us, apply here!
In that vein, I’m pleased to announce that we’ve made our first round of hires for our engineer positions, including:

Jesse Liptrap, who previously worked on the Knowledge Graph at Google for four years, and as a bioinformatician at UC Berkeley. Jesse holds a PhD in mathematics from UC Santa Barbara, where he studied category-theoretic underpinnings of topological quantum computing.
Nick Tarleton, former lead architect at the search startup Quixey. He previously studied computer science and decision science at Carnegie Mellon University, and Nick worked with us at the first iteration of our summer fellows program, studying consequences of proposed AI goal systems.

On the whole, our initial hiring efforts have gone quite well, and I’ve been very impressed with the high caliber of our hires and of our pool of candidates.
On the research side, our recent work has focused heavily on open problems in decision theory, and on other questions related to naturalized agency. Scott Garrabrant divides our recent work on the agent foundations agenda into four categories, tackling different AI alignment subproblems:

Decision theory — Traditional models of decision-making assume a sharp Cartesian boundary between agents and their environment. In a naturalized setting in which agents are embedded in their environment, however, traditional approaches break down, forcing us to formalize concepts like “counterfactuals” that can be left implicit in AIXI-like frameworks.

  More

Recent focus areas:

As Rob noted in April, “a common thread in our recent work is that we’re using probability and topological fixed points in settings where we used to use provability. This means working with (and improving) logical inductors and reflective oracles.” Examples of applications of logical induction to decision theory include logical inductor evidential decision theory (“Prediction Based Robust Cooperation,” “Two Major Obstacles for Logical Inductor Decision Theory“) and asymptotic decision theory (“An Approach to Logically Updateless Decisions,” “Where Does ADT Go Wrong?”).
Unpacking the notion of updatelessness into pieces that we can better understand, e.g., in “Conditioning on Conditionals,” “Logical Updatelessness as a Robust Delegation Problem,” “The Happy Dance Problem.”
The relationship between decision theories that rely on Bayesian conditionalization on the one hand (e.g., evidential decision theory and Wei Dai’s updateless decision theory), and ones that rely on counterfactuals on the other (e.g., causal decision theory, timeless decision theory, and the version of functional decision theory discussed in Yudkowsky and Soares (2017)): “Smoking Lesion Steelman,” “Comparing LICDT and LIEDT.”
Lines of research relating to correlated equilibria, such as “A Correlated Analogue of Reflective Oracles” and “Smoking Lesion Steelman II.”
The Converse Lawvere Problem (1, 2, 3): “Does there exist a topological space X (in some convenient category of topological spaces) such that there exists a continuous surjection from X to the space [0,1]X (of continuous functions from X to [0,1])?”
Multi-agent coordination problems, often using the “Cooperative Oracles” framework.

Naturalized world-models — Similar issues arise for formalizing how systems model the world in the absence of a sharp agent/environment boundary. Traditional models leave implicit aspects of “good reasoning” such as causal and multi-level world-modeling, reasoning under deductive limitations, and agents modeling themselves.

  More

Recent focus areas:

Kakutani’s fixed-point theorem and reflective oracles: “Hyperreal Brouwer.”
Transparency and merging of opinions in logical inductors.
Ontology merging, a possible approach to reasoning about ontological crises and transparency.
Attempting to devise a variant of logical induction that is “Bayesian” in the sense that its belief states can be readily understood as conditionalized prior probability distributions.

Subsystem alignment — A key reason that agent/environment boundaries are unhelpful for thinking about AGI is that a given AGI system may consist of many different subprocesses optimizing many different goals or subgoals. The boundary between different “agents” may be ill-defined, and a given optimization process is likely to construct subprocesses that pursue many different goals. Addressing this risk requires limiting the ways in which new optimization subprocesses arise in the system.

  More

Recent focus areas:

Benign induction: “Maximally Efficient Agents Will Probably Have an Anti-Daemon Immune System.”
Work related to KWIK learning: “Some Problems with Making Induction Benign, and Approaches to Them” and “How Likely Is A Random AGI To Be Honest?”

Robust delegation — In cases where it’s desirable to delegate to another agent (e.g. an AI system or a successor), it’s critical that the agent be well-aligned and trusted to perform specified tasks. The value learning problem and most of the AAMLS agenda fall in this category.

  More

Recent focus areas:

Goodhart’s Curse, “the combination of the Optimizer’s Curse and Goodhart’s Law” stating that “a powerful agent neutrally optimizing a proxy measure U that we hoped to align with true values V, will implicitly seek out upward divergences of U from V”: “The Three Levels of Goodhart’s Curse.”
Corrigibility: “Corrigibility Thoughts,” “All the Indifference Designs.”
Value learning and inverse reinforcement learning: “Incorrigibility in the CIRL Framework,” “Reward Learning Summary.”
The reward hacking problem: “Stable Pointers to Value: An Agent Embedded In Its Own Utility Function.”

Additionally, we ran several research workshops, including one focused on Paul Christiano’s research agenda.

Fundraising goals
To a first approximation, we view our ability to make productive use of additional dollars in the near future as linear in research personnel additions. We don’t expect to run out of additional top-priority work we can assign to highly motivated and skilled researchers and engineers. This represents an important shift from our past budget and team size goals.5
Growing our team as much as we hope to is by no means an easy hiring problem, but it’s made significantly easier by the fact that we’re now looking for top software engineers who can help implement experiments we want to run, and not just productive pure researchers who can work with a high degree of independence. (In whom we are, of course, still very interested!) We therefore think we can expand relatively quickly over the next two years (productively!), funds allowing.
In our mainline growth scenario, our reserves plus next year’s $1.25M installment of the Open Philanthropy Project’s 3-year grant would leave us with around 9 months of runway going into 2019. However, we have substantial uncertainty about exactly how quickly we’ll be able to hire additional researchers and engineers, and therefore about our 2018–2019 budgets.
Our 2018 budget breakdown in the mainline success case looks roughly like this:

2018 Budget Estimate (Mainline Growth)

To determine our fundraising targets this year, we estimated the support levels (above the Open Philanthropy Project’s support) that would make us reasonably confident that we can maintain a 1.5-year runway going into 2019 in different growth scenarios, assuming that our 2017 fundraiser looks similar to next year’s fundraiser and that our off-fundraiser donor support looks similar to our on-fundraiser support:

Basic target — $625,000. At this funding level, we’ll be in a good position to pursue our mainline hiring goal in 2018, although we will likely need to halt or slow our growth in 2019.

Mainline-growth target — $850,000. At this level, we’ll be on track to fully fund our planned expansion over the next few years, allowing us to roughly double the number of research staff over the course of 2018 and 2019.

Rapid-growth target — $1,250,000. At this funding level, we will be on track to maintain a 1.5-year runway even if our hiring proceeds a fair amount faster than our mainline prediction. We’ll also have greater freedom to pay higher salaries to top-tier candidates as needed.

Beyond these growth targets: if we saw an order-of-magnitude increase in MIRI’s funding in the near future, we have several ways we believe we can significantly accelerate our recruitment efforts to grow the team faster. These include competitively paid trial periods and increased hiring outreach across venues and communities where we expect to find high-caliber candidates. Funding increases beyond the point where we could usefully use the money to hire faster would likely cause us to spin off new initiatives to address the problem of AI x-risk from other angles; we wouldn’t expect them to go to MIRI’s current programs.
On the whole, we’re in a very good position to continue expanding, and we’re enormously grateful for the generous support we’ve already received this year. Relative to our present size, MIRI’s reserves are much more solid than they have been in the past, putting us in a strong position going into 2018.
Given our longer runway, this may be a better year than usual for long-time MIRI supporters to consider supporting other projects that have been waiting in the wings. That said, we don’t personally know of marginal places to put additional dollars that we currently view as higher-value than MIRI, and we do expect our fundraiser performance to affect our growth over the next two years, particularly if we succeed in growing the MIRI team as fast as we’re hoping to.

Donate Now

Strategic background
Taking a step back from our immediate organizational plans: how does MIRI see the work we’re doing as tying into positive long-term, large-scale outcomes?
A lot of our thinking on these issues hasn’t yet been written up in any detail, and many of the issues involved are topics of active discussion among people working on existential risk from AGI. In very broad terms, however, our approach to global risk mitigation is to think in terms of desired outcomes, and to ask: “What is the likeliest way that the outcome in question might occur?” We then repeat this process until we backchain to interventions that actors can take today.
Ignoring a large number of subtleties, our view of the world’s strategic situation currently breaks down as follows:

1. Long-run good outcomes. Ultimately, we want humanity to figure out the best possible long-run future and enact that kind of future, factoring in good outcomes for all sentient beings. However, there is currently very little we can say with confidence about what desirable long-term outcomes look like, or how best to achieve them; and if someone rushes to lock in a particular conception of “the best possible long-run future,” they’re likely to make catastrophic mistakes both in how they envision that goal and in how they implement it.
	In order to avoid making critical decisions in haste and locking in flawed conclusions, humanity needs:

2. A stable period during which relevant actors can accumulate whatever capabilities and knowledge are required to reach robustly good conclusions about long-run outcomes. This might involve decisionmakers developing better judgment, insight, and reasoning skills in the future, solving the full alignment problem for fully autonomous AGI systems, and so on.
	Given the difficulty of the task, we expect a successful stable period to require:

3. A preceding end to the acute risk period. If AGI carries a significant chance of causing an existential catastrophe over the next few decades, this forces a response under time pressure; but if actors attempt to make irreversible decisions about the long-term future under strong time pressure, we expect the result to be catastrophically bad. Conditioning on good outcomes, we therefore expect a two-step process where addressing acute existential risks takes temporal priority.
	To end the acute risk period, we expect it to be necessary for actors to make use of:

4. A risk-mitigating technology. On our current view of the technological landscape, there are a number of plausible future technologies that could be leveraged to end the acute risk period.
	We believe that the likeliest way to achieve a technology in this category sufficiently soon is through:

5. AGI-empowered technological development carried out by task-directed AGI systems. Depending on early AGI systems’ level of capital-intensiveness, on whether AGI is a late-paradigm or early-paradigm invention, and on a number of other factors, AGI might be developed by anything from a small Silicon Valley startup to a large-scale multinational collaboration. Regardless, we expect AGI to be developed before any other (meta)technology that can be employed to end the acute risk period, and if early AGI systems can be used safely at all, then we expect it to be possible for an AI-empowered project to safely automate a reasonably small set of concrete science and engineering tasks that are sufficient for ending the risk period. This requires:

6. Construction of minimal aligned AGI. We specify “minimal” because we consider success much more likely if developers attempt to build systems with the bare minimum of capabilities for ending the acute risk period. We expect AGI alignment to be highly difficult, and we expect additional capabilities to add substantially to this difficulty.
 Added: “Minimal aligned AGI” means “aligned AGI that has the minimal necessary capabilities”; be sure not to misread it as “minimally aligned AGI”. Rob Bensinger adds: “The MIRI view isn’t ‘rather than making alignment your top priority and working really hard to over-engineer your system for safety, try to build a system with the bare minimum of capabilities’. It’s: ‘in addition to making alignment your top priority and working really hard to over-engineer your system for safety, also build the system to have the bare minimum of capabilities’.”
	If an aligned system of this kind were developed, we would expect two factors to be responsible:

7a. A technological edge in AGI by an operationally adequate project. By “operationally adequate” we mean a project with strong opsec, research closure, trustworthy command, a commitment to the common good, security mindset, requisite resource levels, and heavy prioritization of alignment work. A project like this needs to have a large enough lead to be able to afford to spend a substantial amount of time on safety measures, as discussed at FLI’s Asilomar conference.

7b. A strong white-boxed system understanding on the part of the operationally adequate project during late AGI development. By this we mean that developers go into building AGI systems with a good understanding of how their systems decompose and solve particular cognitive problems, of the kinds of problems different parts of the system are working on, and of how all of the parts of the system interact.
	On our current understanding of the alignment problem, developers need to be able to give a reasonable account of how all of the AGI-grade computation in their system is being allocated, similar to how secure software systems are built to allow security professionals to give a simple accounting of why the system has no unforeseen vulnerabilities. See “Security Mindset and Ordinary Paranoia” for more details.
Developers must be able to explicitly state and check all of the basic assumptions required for their account of the system’s alignment and effectiveness to hold. Additionally, they need to design and modify AGI systems only in ways that preserve understandability — that is, only allow system modifications that preserve developers’ ability to generate full accounts of what cognitive problems any given slice of the system is solving, and why the interaction of all of the system’s parts is both safe and effective.
Our view is that this kind of system understandability will in turn require:

8. Steering toward alignment-conducive AGI approaches. Leading AGI researchers and developers need to deliberately direct research efforts toward ensuring that the earliest AGI designs are relatively easy to understand and align.
	We expect this to be a critical step, as we do not expect most approaches to AGI to be alignable after the fact without long, multi-year delays.

We plan to say more in the future about the criteria for operationally adequate projects in 7a. We do not believe that any project meeting all of these conditions currently exists, though we see various ways that projects could reach this threshold.
The above breakdown only discusses what we view as the “mainline” success scenario.6 If we condition on good long-run outcomes, the most plausible explanation we can come up with cites an operationally adequate AI-empowered project ending the acute risk period, and appeals to the fact that those future AGI developers maintained a strong understanding of their system’s problem-solving work over the course of development, made use of advance knowledge about which AGI approaches conduce to that kind of understanding, and filtered on those approaches.
For that reason, MIRI does research to intervene on 8 from various angles, such as by examining holes and anomalies in the field’s current understanding of real-world reasoning and decision-making. We hope to thereby reduce our own confusion about alignment-conducive AGI approaches and ultimately help make it feasible for developers to construct adequate “safety-stories” in an alignment setting. As we improve our understanding of the alignment problem, our aim is to share new insights and techniques with leading or up-and-coming developer groups, who we’re generally on good terms with.
A number of the points above require further explanation and motivation, and we’ll be providing more details on our view of the strategic landscape in the near future.
Further questions are always welcome at contact@intelligence.org, regarding our current organizational activities and plans as well as the long-term role we hope to play in giving AGI developers an easier and clearer shot at making the first AGI systems robust and safe. For more details on our fundraiser, including corporate matching, see our Donate page.

Note that this $1.9M is significantly below the $2.1–2.5M we predicted for the year in April. Personnel costs are MIRI’s most significant expense, and higher research staff turnover in 2017 meant that we had fewer net additions to the team this year than we’d budgeted for. We went under budget by a relatively small margin in 2016, spending $1.73M versus a predicted $1.83M.
Our 2018–2019 budget estimates are highly uncertain, with most of the uncertainty coming from substantial uncertainty about how quickly we’ll be able to take on new research staff.This is roughly in line with our experience in previous years, when excluding expected grants and large surprise one-time donations. We’ve accounted for the former in our targets but not the latter, since we think it unwise to bank on unpredictable windfalls.
Note that in previous years, we’ve set targets based on maintaining a 1-year runway. Given the increase in our size, I now think that a 1.5-year runway is more appropriate.Including the $1.01 million donation and the first $1.25 million from the Open Philanthropy Project, we have so far raised around $3.16 million this year, overshooting the $3 million goal we set earlier this year!We emphasize that, as always, “experiment” means “most things tried don’t work.” We’d like to avoid setting expectations of immediate success for this exploratory push.Our previous goal was to slowly ramp up to the $3–4 million level and then hold steady with around 13–17 research staff. We now expect to be able to reach (and surpass) that level much more quickly.There are other paths to good outcomes that we view as lower-probability, but still sufficiently high-probability that the global community should allocate marginal resources to their pursuit.The post MIRI’s 2017 Fundraiser appeared first on Machine Intelligence Research Institute.

MIRI’s 2017 Fundraiser