Josef Urban on Machine Learning and Automated Reasoning

 Josef Urban is a postdoctoral researcher at the Institute for Computing and Information Sciences of the Radboud University in Nijmegen, the Netherlands. His main interest is development of combined inductive and deductive AI methods over large formal (fully semantically specified) knowledge bases, such as large corpora of formally stated mathematical definitions, theorems and proofs. This involves deductive AI fields such as Automated Theorem Proving (ATP) and inductive AI fields such as Machine Learning. In 2003, he made the largest corpus of formal mathematics – the Mizar library – available to ATP and AI methods, and since then has been developing the first approaches, systems and benchmarks for AI and automated reasoning over such corpora. His Machine Learner for Automated Reasoning (MaLARea) has in 2013 solved by a large margin most problems in the mathematically oriented Large Theory division of the CADE Automated Systems Competition (CASC). The system is based on several positive-feedback loops between deducing new proofs and learning how to guide the reasoning process based on successful proofs. Such AI methods are also being deployed as systems that assist interactive theorem proving and formal verification in systems like Mizar, Isabelle and HOL Light.
He received his PhD in Computers Science from the Charles University in Prague in 2004, and MSc in Mathematics at the same university in 1998. In 2003, he co-founded the Prague Automated Reasoning Group, and from 2006 to 2008 he was a Marie-Curie fellow at the University of Miami.

Luke Muehlhauser: In Urban & Vyskocil (2013) and other articles, you write about the state of the art in both the formalization of mathematics and automated theorem proving. Before I ask about machine learning in this context, could you quickly sum up for us what we can and can’t do at this stage, when it comes to automated theorem proving?

Josef Urban: The best-known result is the proof of the Robbins conjecture found automatically by Bill McCune’s system EQP in 1996. Automated theorem proving (ATP) can be used today for solving some open equational algebraic problems, for example in the quasigroup and loop theory. Proofs of such open problems may be thousands of inferences long, and quite different from the proofs that mathematicians produce. Solving such problems with ATP is often not a “push-button” matter. It may require specific problem reformulation (usually involving only a few axioms) suitable for ATP, exploration of suitable parameters for the ATP search, and sometimes also guiding the final proof attempt by lemmas (“hints”) that were useful for solving simpler related problems.
ATP is today also used for proving small lemmas in large formal libraries developed with interactive theorem provers (ITPs) such as Mizar, Isabelle and HOL Light. This is called “large-theory” ATP. In the past ten years, we have found that even with the large mathematical libraries (thousands of definitions and theorems) available in these ITPs, it is often possible to automatically select the relevant previous knowledge from such libraries and find proofs of small lemmas automatically without any manual pre-processing. Such proofs are usually much shorter than the long proofs of the equational algebraic problems, but even that already helps a lot to the people who use ITPs to formally prove more advanced theorems.
An illustrative example is the Flyspeck (“Formal Proof of Kepler”) project led by Tom Hales, who proved the Kepler conjecture in 1998. The proof is so involved that the only way how to ensure its correctness is formal verification with an ITP. Tom Hales has recently written a 300-page book about the proof, with the remarkable property that most of the concepts, lemmas and proofs have a computer-understandable “Flyspeck” counterpart that is stated and verified in HOL Light. Hales estimates that this has taken about 20 man-years. Recently, we have shown that the proofs of about 47% of the 14000 small Flyspeck lemmas could be found automatically by large-theory ATP methods. This means that the only required mental effort is to formally state these lemmas in HOL Light. Again, the caveat is that the lemmas have to be quite easy to prove from other lemmas that are already in the library. On the other hand, this means that once we have such large corpus of formally-stated everyday mathematics, it is possible to use the corpus together with the large-theory ATP methods to solve many simple formally-stated mathematical problems.
Another interesting recent development are ATP methods that efficiently combine “reasoning” and “computing”. This has been a long-time challenge for ATP systems, particularly when used for software and hardware verification. Recent SMT (Satisfiability Modulo Theories) systems such as Z3 integrate a number of decision procedures, typically about linear and nonlinear arithmetics, bitvectors, etc. I am not an expert in this area, but I get the impression that the growing strength of such systems is making the full formal verification of complex hardware and software systems more and more realistic.

Luke: As you see it, what are the most important factors limiting the success and scope of ATP algorithms?

Josef: (1) Their low reasoning power, particularly in large and advanced theories, and (2) lack of computer understanding of current human-level (math) knowledge.
The main cause of (1) is that so far the general (“universal”) reasoning algorithms often use too much brute force, without much smart guidance, specific methods for specific subproblems, and methods for self-improvement. Even on mathematical problems that are considered quite easy, such brute-force algorithms then often explode. (2) is a major obstacle because even if the strength of ATP algorithms could be already useful for simple query-answering applications in general mathematics, the systems still do not understand problems posed using the human-level mathematical language and concepts. (1) and (2) are connected: human math prose can be arbitrarily high-level, requiring the discovery of long reasoning chains just for parsing and filling the justification gaps.
One path to improving (1) leads through better (and preferably automated) analysis of how mathematicians think. In other words, through mining and understanding what they know, how they use and produce it, and how they improve their knowledge and reasoning methods. If that is correct, then it should be useful to have large corpora of mathematical knowledge in a (possibly “annotated”) format that can be at least correctly parsed, and then further analysed and perhaps used for building various self-improving ATP algorithms. To some extent, this has been started in the last decade, even though manual (“theory-driven”) engineering of ATP algorithms still largely prevails. Unfortunately, large “ATP-parsable” (or annotated) corpora are still quite rare and expensive to produce today, exactly because of (2), i.e., the gaps between human-level and computer-level understanding and reasoning. Needless to say that dually annotated human-computer corpora of mathematics should be also (together with decent ATP) useful for a “bootstrapped” solving of (2), i.e., for learning more and more computer understanding of human-level mathematical texts. Thanks to the fact that mathematics is the only field with deep formal semantics, study of such corpora might also eventually lead us to better semantic understanding of natural language and strong AI for other sciences.
In software and hardware verification, (1) is probably the main problem, although, for example, semantic annotation of large code libraries is also a major effort. Solving (1) is still arbitrarily hard, but it seems that “code” is generally not so diverse as “mathematics”, and it might be possible to get quite far with a smaller number of manually programmed techniques.

Luke: How much of human mathematical knowledge has been formalized such that automated theorem provers can use it? At what pace is that base of formalized mathematics growing?

Josef: One metric that we have is the list of 100 theorems created in 1999 and tracked for formal systems by Freek Wiedijk. Currently, 88% of these 100 theorems have been proved in at least one proof assistant, 87% in HOL Light alone. A lot of the undergraduate curriculum is probably already covered to a good extent: the basics of real/complex calculus, measure theory, algebra and linear algebra, general topology, set theory, category theory, combinatorics and graph theory, logic, etc. But I think the PhD-level coverage is still quite far.
There are some advanced mathematical formalizations like the recent formal proof of the Feit-Thompson theorem in Coq, the already mentioned Flyspeck project done in HOL Light, formalization of more than a half of the Compendium of Continuous Lattices in Mizar, and many smaller interesting projects. Some overview pages of the theorems and theories formalized in the systems are the MML Query (Mizar), the Archive of Formal Proofs (Isabelle), the Coq contribs, and the 100-theorems project in HOL Light.
The rate of growth of formal math is not very high, but also the number of authors is quite low. My very rough estimate is about 10k-20k top-level lemmas per year with some 100-300 more important theorems among them. But there is quite a lot of repetition and duplication between various systems, their various libraries, and sometimes even inside one library. People are trying various approaches to formalization, and sometimes prefer to design their own formalization from scratch rather than trying to re-use the work of somebody else. This is quite similar to code libraries. Making the libraries as re-usable as possible is a nontrivial effort.
Strictly speaking, only Mizar, Isabelle and HOL Light are currently accessible to large-theory ATP systems that really try to use the whole libraries at once to prove new conjectures. Coq’s logic is more different from the logics used by the strongest ATP systems and there is so far no sufficiently complete large-theory link between them. But there is also a lot of smaller-scale programmable proof automation inside Coq (and also other interactive provers) already.

Luke: The prospect of having large databases of formalized mathematical knowledge naturally leads to the question: “Might we be able to use machine learning algorithms for improved automated theorem proving?” What is the current state of machine learning in the context of automated theorem proving?

Josef: Most of the machine learning applications are today in the context of large-theory ATP. The task that has the most developed learning methods is selection of the most relevant theorems and definitions from a large library when proving a new conjecture. This is important for two reasons: (i) ATPs can today often get quite far when given the right axioms that are sufficient, not very redundant, and reasonably close to the conjecture, and (ii) adding irrelevant axioms to the ATP search space often quickly diminishes the chances of finding the proof within reasonable time. This selection problem became known as “premise selection” and a number of learning and non-learning algorithms have been tried for it already and combined in various ways. The oldest one tried since 2003 is naive Bayes, but more recently we tried also kernel methods, various versions of distance-weighted k-nearest neighbor, random forests, and some basic ensemble methods.
A major issue is the choice of good characterization of mathematical objects such as formulas and proofs for the learning algorithms. In a large diverse library, just using the set of symbols as formula features is already useful. This has been extended by methods ranging from purely syntactic n-gram analogs (terms and subterms and their parts, normalized in various ways), to more semantic but still quite syntactic features such as addition of type information and type hierarchies, to strongly semantic features such as the validity of formulas in a large pool of diverse models. Feature preprocessing methods such as TF-IDF and latent semantic analysis help a lot, and quite likely more can be done still.
Another major issue are the proofs used for training. Some proofs are easy for humans, but hard for today’s ATPs, so it is often better to find an ATP proof if possible and learn from it, instead of trying to learn from formally correct but ATP-infeasible proofs. This gives rise to exploratory AI systems that interleave deductive proof-finding and learning from proofs. Positive feedback loops appear: the more ATP proofs (and also counter-examples) we find, the better the next learning iteration, and the more ATP proofs we usually find in the next iteration of learning-advised proof finding. In some sense, one cannot easily “just do inductive AI (learning)” or “just do deductive AI (ATP)”: the strongest approach to large-theory ATP is to really interleave theorem proving with learning from the proofs. This should not be too surprising, because neither humans do science just by pure induction or just by pure deduction. But to me it seems really useful to have today at least one AI domain that allows such combined techniques and pragmatic experiments with them.
A bit similar ideas apply when we stop treating ATPs as black boxes and start to look inside them. Instead of evolving an inductive/deductive system that is good at selecting relevant premises for conjectures (MaLARea is an example), we may try to evolve a system that has a number of specialized problem-solving techniques (ATP strategies) for common classes of problems, and some intuition about which techniques to use for which problem classes. An example is the Blind Strategymaker, which on a large library of different problems co-evolves a population of ATP strategies (treated as sets of search-directing parameters) with a population of solvable training problems, with the ultimate goal of having a reasonably small set of strategies that together solve as many of the problems as possible. The initial expert strategies (“predators”) are mutated and quickly evaluated on their easy prey (the easy training problems they specialize in), and if the mutations show promise (faster solutions) on such training subset, they undergo a more expensive (more time allowed) evaluation on a much wider set of problems, possibly solving some previously unsolvable ones and making them into further training targets. Again, just random mutating on the so-far-unsolved problems is quite inefficient, so one really needs the intuition about which training data the mutations should be grown on. This again yields a fast “inductive” training phase, followed (if successful) by a slower “hard thinking” phase, in which the newly trained strategies attempt to solve some more problems, making them into further training data. The intuition and the deductive capability co-evolve again; doing just one of them does not work so well.
And going again from black-box to white-box, one can also ask what are really the ATP strategies and deductive techniques, and how to steer/program them more than just by finding good combinations of predefined search parameters for problem classes. There are several learning methods that go beyond such parameter settings, the most prominent and successful is probably the “hints” method by Bob Veroff (a bit differently done also by Stephan Schulz), that directs the proof search towards the lemmas (hints) that were found useful in proofs of related problems. I believe that learning such internal guidance is still quite an unexplored territory that we should be working on. One might not only direct the search towards previously useful lemmas, but also look at methods that suggest completely new lemmas and concepts, based on analogies with other problems, deeper knowledge about the problem state, etc. Depending on one’s idea about what is and what is not machine learning, one could also relate here to Douglas Lenat’s seminal work on concept and lemma-finding systems such as the Automated Mathematician and Eurisko.

Luke: What trends in this area do you expect in the next 20 years? Do you expect that in 20 years a much greater share of research mathematicians will use ATPs, at least in a highly interactive way (ala Lenat with Eurisko)? Do you expect learning (rather than non-learning) ATPs to predominate in the future? Do you expect things to advance so far that the jobs of some research mathematicians will be effectively replaced, just as some paralegal jobs have (supposedly) been replaced by better software for searching through legal databases?

Josef: First, AI predictions usually happen, but often take longer than expected. 20 years is not really much in these low-profile fields: Flyspeck itself has taken about 20 person-years and Mizar took 40 years from a visionary talk by Andrzej Trybulec in 1973 to the current 1200 articles in the Mizar library. There are not many large projects in these areas and often they are pushed only by extreme determination of single researchers.
One AI breakthrough that I believe is quite reachable within 20 years is deep automated semantic understanding (formalization, “machine reading”) of most of LaTeX-written mathematical textbooks. This has been for long time blocked by three factors: (i) lack of annotated formal/informal corpora to train such semantic parsing on, (ii) lack of sufficiently large repository of background mathematical knowledge needed for “obvious-knowledge gap-filling”, and (iii) lack of sufficiently strong large-theory ATP that could fill the reasoning gaps using the large repository of background knowledge. This has changed a lot recently and unless we are really lazy I believe it will mostly happen. It might not be immediately perfect and on the level of manual formalization, but it will gradually improve both by using more data and by better algorithms, a bit similarly to what has been happening with Google Translate between languages like English, Spanish and German. Once it starts happening automatically, it might snowball faster and faster, again thanks to the positive feedback loop between more “reasoning data” becoming accessible and the strength of the data-driven large-theory ATP and AI methods trained on such data.
I don’t think that mathematicians have any reason to “fear AI” (at least at the moment), quite the contrary: more and more are wondering about the current low level of computer understanding and assistance of math. Mathematicians like von Neumann, Turing and Goedel are the reason that we have computers and AI as a discipline in the first place. One of their motivations was Hilbert’s program and the Leibniz-style dreams of machines assisting mathematics and scientific reasoning. It is even a bit of a shame for computer science and AI that in the hundred years since Turing’s birth they have done so little for their “parent” science and that the progress is so slow. At this moment, anybody who tried writing formal mathematics prays for better automation. Without it, formalization will remain a slow, costly and marginal effort. So it is more the opposite: in order for deep semantic treatment of math and sciences to take off, automation has to improve. But once such deep semantic treatment of sciences starts to develop, it means also easier access for mathematicians. Law is just one example; I really like John McCarthy’s futuristic note on formal proof becoming a strong criterion for all kinds of policy making. Complex mathematical proofs that require formal checking are just a tip of the iceberg: today we are building and relying on more and more complex machinery, hardware, software, legal, political and economical systems, and they are all buggy and easy to hack and break. Fixing this all and allowing us to correctly implement much more complex designs is a great future opportunity and market for mathematically thinking people equipped with automation tools that allow them to make fast progress without sacrificing correctness.
I do expect that ATPs will in general have better pattern-detection capabilities in 20 years than they have now, and that they will be able to better accumulate, process and efficiently use previous knowledge and better combine brute-force search with all kinds of guidance on various levels. Specifically in more advanced mathematics, high-level heuristic human-inspired proving/thinking methods might start to be more developed. One way to try to get them automatically is again through basic computer understanding of LaTeX-written mathematical texts, and learning what high-level concepts like “by analogy” and “using diagonalization” exactly semantically mean in various contexts. This is also related to the ability to reformulate problems and map them to a setting (for current ATPs, the best is purely equational) where the theorem proving becomes more easy. And another related work that needs to be done is “explaining back” the long ATP proofs using an understandable mathematical presentation.
The word “learning” has itself many meanings. A recent breakthrough in SAT solving (propositional ATP) is called “conflict-driven clause learning” (CDCL), but it is just a correct (resolution) inference of a new lemma implied by the axioms. The generalization aspect of “learning” is quite limited there. I don’t really know how good the ATPs will be in twenty years, improvements may come from various sides, not just by using “learning”. The main ATP calculi are so far so simple and brute-force, that it is also possible that somebody will come up with a completely new approach which will considerably improve the strength of ATP systems. For example, instantiation-based ATPs like iProver relying on the CDCL-extended SAT solvers have been improved quite a lot in the recent years.
Another trend I expect is more “manual” work on all kinds of targeted decision procedures (dealing with numbers, lists, bitvectors, etc.), particularly when useful for software and hardware verification, and their integration with ATP calculi. I also hope that we will start to be able to detect such targeted algorithms automatically from the successful reasoning patterns done by ATPs. In some sense, we need better automated compilation of “search” into “computing” (we could call it “learning of decision procedures”). There is no rigid distinction between these two: in ATP-inspired programming languages like Prolog, the two things largely coincide. The better we know how to direct the proof search in certain (e.g. numerical) problem classes, the more the proof search turns into efficient computation.
Finally, I might venture one concrete and refutable prediction about the strength of large-theory ATP in 20 years: on the Flyspeck resp. Mizar libraries, learning-assisted large-theory ATP can today prove 47% resp. 40% of the toplevel lemmas. In 20 years, using the same hardware and resources (i.e., not relying on Moore’s Law), we will be able to prove automatically 80% of both (measured on the same versions of the libraries).

Luke: Thanks, Josef!
The post Josef Urban on Machine Learning and Automated Reasoning appeared first on Machine Intelligence Research Institute.


2013 in Review: Operations

It’s been 8 months since our big strategic shift to a focus on Friendly AI research. What has worked well? What hasn’t? What have we learned? In this and several upcoming posts, I’ll provide a qualitative and personal self-review of MIRI in 2013.
This review will be mostly qualitative because we haven’t yet found quantitative metrics that do a decent job of tracking what we actually care about (see below). The review will be personal because MIRI staff have a variety of perspectives, so I will usually be speaking for myself rather than for the organization as a whole.
I’ll split my review into six posts: operations (this post), outreach, strategic and expository research, Friendly AI research, fundraising, and our mid-2014 strategic plan.

Operations in 2013

2013 was a big year for capacity-building at MIRI — not so much by hiring more people but by doing more with the people we have.
That said, this progress cost lots of executive time, and I should have been more aggressive about finding ways to accomplish this with fewer executive hours.
As a result of (2), we did not make as much progress on metrics and systematic experimentation as I would have liked to make in 2013.

Capacity-building at MIRI in 2013
Here I’ll explain how our operations capacity — our ability to efficiently execute a wide range of programs and projects — has grown in 2013.1 Our research capacity will be discussed in a later post.
In 2013, our operations capacity grew not by hiring more operations staff, but by making our processes more efficient and thereby doing more with a roughly-stable number of staff.2
Below is a partial list of operational improvements we made in 2013.3 I suspect other nonprofit executives can benefit from some of them.

We moved to a larger and more centrally-located office in downtown Berkeley, 2.5 blocks from the UC Berkeley campus and 0.5 blocks from a BART subway entrance. As expected, this has made it easier for people to work together on projects, easier to meet with visitors, easier to host research workshops, etc.
We streamlined our Board meetings and reduced their frequency by providing the Board with a monthly “snapshot” update in PDF, and by giving all but the highest-level decisions to the executives, or to committees composed of particular board members and executives. (We still need to do more of this, though.)
We won an exemption to the USA’s annual cap on H-1B visas. This makes it easier for us to bring in foreign talent to work with us in Berkeley, thus greatly expanding our pool of potential hires. Using this cap exemption, we won H-1B visas for Malo Bourgon and Alex Vermeer. Alex has already joined us in Berkeley and realized the expected productivity and efficiency gains. Malo will join us in Berkeley next year.
Many operational improvements came from automating  processes with software. For example, we tested several time-tracking solutions and settled on Harvest, which makes it easy to track staff and contractor hours, and also to estimate the time costs of each major project. We also manage some projects, especially those involving contractors, via Basecamp.
With the help of a few custom scripts, we now have a pretty accurate donations database (via DonorTools) that is integrated with our favorite CRM (HighRise) and some web pages (e.g. the live progress bar for our ongoing fundraiser). This makes it easier for us to track donations, and to stay in touch with donors, potential workshop participants, and other contacts.
After struggling to efficiently use popular, broken payroll systems like Paychex and Wells Fargo Payroll, we made the jump to ZenPayroll and discovered it to be exactly what it claims to be: a painless, modern payroll system with responsive customer support.
Lots of other little tweaks: we consolidated our web domains and web hosting, we record expenses by emailing smartphone photos of receipts to an email address, we use EchoSign for signing contracts online instead of faxing or mailing them, we switched to MailChimp for newsletter management and analytics, we switched to Disqus for blog comment management, etc.

Compared to mid-2012, it is noticeably easier to just get things done at MIRI. In mid-2012, I would often find that doing X required that Y and Z be in place, and to set up Y and Z we needed four different documents, and we didn’t have any of those documents yet because MIRI hadn’t previously invested much in operations, and it was hard to outsource the production of those documents because we didn’t have a central database for that kind of organizational information, so I’d have to pull it all from my own head and make phone calls to other people, and I couldn’t fill in line 6 of document 3 because the payroll system once again wouldn’t let me load that report, and I couldn’t estimate the project costs requested by document 2 because there had been no costs tracking, and… well, these kinds of tasks are much easier now, and much easier to outsource, which means that MIRI’s executives (Louie and I) can spend more time on organizational strategy, fundraising, and building the research program.

These operational improvements consumed too much executive time
For the past few months, these operational improvements have largely been tested and implemented by Malo Bourgon and (to a lesser extent) Alex Vermeer. But for most of 2013, these improvements required much personal effort from MIRI’s executives, which carries an especially high opportunity cost. In retrospect, I think I could have conserved executive time better:

I could have, by mid-2012, recognized our ongoing need for technically-savvy operations support, and recruited more aggressively for that role. That said, I’m not sure more recruiting effort here would have paid off. (See here.)
I could have further reduced the executive time needed to make these improvements, by giving greater responsibility to Malo Bourgon earlier, and by helping Malo outsource his other responsibilities sooner.4
I could have done more to free up executive time in other ways, by delegating tasks and projects more aggressively. For example, I personally ran onsite logistics for some of this year’s research workshops, to “make sure everything went well,” but I should have delegated that role more fully to others, even if it wouldn’t have been executed quite as well. That said, we did outsource quite a lot of work (2500+ contractor hours) in 2013.

As a result, we didn’t make much progress on metrics and experimentation
To improve MIRI’s effectiveness via tight feedback loops, we need quantitative measures of the kinds of progress we care about, and we need to run frequent experiments to test hypotheses about which strategies will work best for operational efficiency, outreach success, research progress, and fundraising.
MIRI is a quickly-evolving organization with hard-to-measure goals, and thus it requires significant high-level investment from MIRI’s executives to figure out how to measure variables we care about (instead of variables that are easy to measure), via experimentation and other means. But because so much executive time in 2013 was consumed by operational improvements, we made little progress on metrics and experimentation until very recently. Right now we’re experimenting with the “OKRs and KPIs” framework used at Intel, Google, Quixey, and other companies, but we have a lot more work to do.
I think another reason for our lack of progress on metrics and experimentation, besides the lack of executive time, was that I failed to set up deadlines for concrete progress on developing metrics and experiments. In part this was because “make progress figuring out how to measure progress” felt too abstract for useful deadlines, and in part it was because MIRI’s object-level work usually felt more urgent to me than the “meta-level” work on metrics and experimentation. But these were poor reasons to avoid putting deadlines on an important project.

What I’m doing to improve
Here’s what I’m doing to improve our progress on metrics and experimentation:

I’m more aggressively delegating and outsourcing tasks and projects, and helping Louie, Malo, and Alex to do the same.
I’m prioritizing the development of metrics and experimentation over some object-level projects that could otherwise consume staff time.
I’m setting up regular deadlines for concrete increments of progress on the development of metrics and experiments.

I do not, however, intend to hire additional full-time staff to achieve our operations goals, unless we significantly expand our operations in 2014 in a way that requires more operations staff. We agree with GiveWell that growing capacity by hiring staff is difficult and risky — in this case, unnecessarily so.
In the next part of my 2013 review, I’ll review our outreach activities in 2013.
Some operations work in 2013 wasn’t related to improving our operations capacity per se. For example, we also wrote and passed some important policies that are (e.g.) required to get certain insurance policies. We also setup full offsite backups, made sure our passwords are protected with 1Password, etc. For communication security, we signed up for a trial of Silent Circle (one month before the Snowden leaks), but unfortunately Silent Circle’s phone solution was buggy and their email solution never materialized.Alex Vermeer became a full-time team member during 2013, and Malo Bourgon went full-time near the end of 2012, but Malo was already a near-full-time contractor before that, and Alex’s rising hours in the latter half of 2013 quantitatively substituted for Ioven’s hours from the first third of 2013, which were lost when Ioven departed in mid-March.In other cases, it’s not clear that the implemented change improved our efficiency. In particular, we set up MIRIvolunteers.org (using YouTopia) to manage and incentivize volunteers, but it’s not clear whether this has been worth it. We have 450+ volunteers signed up on the site, and the clickthrough rate for volunteer task announcements is surprisingly high (19%), but there were only about 25hrs/mo of volunteer labor throughout 2013, which meant that many volunteer projects saw slow progress, or none at all. Given the time cost of setting up the YouTopia site and maintaining engagement with volunteers, it’s at least not obvious that MIRIvolunteers.org increases our efficiency. We remain curious to see how much volunteer engagement we get with the Sequences ebook project, and also whether engagement improves as the YouTopia team implements new features.I couldn’t have done this so easily with Alex Vermeer, given the projects he was running at the time.The post 2013 in Review: Operations appeared first on Machine Intelligence Research Institute.

2013 in Review: Operations


New Paper: “Why We Need Friendly AI”

A new paper by Luke Muehlhauser and Nick Bostrom, “Why We Need Friendly AI,” has been published in the latest issue of the Cambridge journal Think: Philosophy for Everyone.
Abstract:
Humans will not always be the most intelligent agents on Earth, the ones steering the future. What will happen to us when we no longer play that role, and how can we prepare for this transition?
This paper is copyrighted by Cambridge University Press, and is reproduced on MIRI’s website with permission. The paper appears in Volume 13, Issue 36.
The post New Paper: “Why We Need Friendly AI” appeared first on Machine Intelligence Research Institute.

New Paper: “Why We Need Friendly AI”


MIRI’s December 2013 Newsletter

Dear friends,We’re still experimenting with our new, ultra-brief newsletter style. Please tell us what you think of it, by replying to this email. Thanks!
MIRI’s winter 2013 matching fundraiser is on! All donations made by January 15th will be matched by Peter Thiel, with 3x matching for “new large donors”: if you’ve given less than $5,000 total in the past, and now give or pledge $5,000 or more, your impact will be quadrupled! Reply to this email if you’d like to ask whether you qualify for 3x matching.
Here is a recent snapshot of our fundraising progress so far:

Other News Updates

Louie Helm’s review of Our Final Invention has been published at Singularity Hub.
Volunteers are needed to proofread a large ebook of Eliezer’s LessWrong.com writings.

Research Updates

Our 6th research workshop is ongoing. See details on all our workshops here.
New paper: “Racing to the Precipice.” A game-theoretic model of AI development.
New paper: “Predicting AGI: What can we say when we know so little?” Argues for a Pareto-distributed prior over the time until we can see that AGI is within reach.
New interview: Scott Aaronson (MIT) on philosophical progress.

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director

The post MIRI’s December 2013 Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s December 2013 Newsletter


Scott Aaronson on Philosophical Progress

Scott Aaronson is an Associate Professor of Electrical Engineering and Computer Science at MIT. Before that, he did a PhD in computer science at UC Berkeley, as well as postdocs at the Institute for Advanced Study, Princeton, and the University of Waterloo. His research focuses on the capabilities and limits of quantum computers, and more generally on the connections between computational complexity and physics. Aaronson is known for his blog as well as for founding the Complexity Zoo (an online encyclopedia of complexity classes); he’s also written about quantum computing for Scientific American and the New York Times. His first book, Quantum Computing Since Democritus, was published this year by Cambridge University Press. He’s received the Alan T. Waterman Award, the PECASE Award, and MIT’s Junior Bose Award for Excellence in Teaching.

Luke Muehlhauser: Though you’re best known for your work in theoretical computer science, you’ve also produced some pretty interesting philosophical work, e.g. in Quantum Computing Since Democritus, “Why Philosophers Should Care About Computational Complexity,” and “The Ghost in the Quantum Turing Machine.” You also taught a fall 2011 MIT class on Philosophy and Theoretical Computer Science.
Why are you so interested in philosophy? And what is the social value of philosophy, from your perspective?

Scott Aaronson: I’ve always been reflexively drawn to the biggest, most general questions that it seemed possible to ask. You know, like are we living in a computer simulation? if not, could we upload our consciousnesses into one? are there discrete “pixels” of spacetime? why does it seem impossible to change the past? could there be different laws of physics where 2+2 equaled 5? are there objective facts about morality? what does it mean to be rational? is there an explanation for why I’m alive right now, rather than some other time? What are explanations, anyway? In fact, what really perplexes me is when I meet a smart, inquisitive person—let’s say a mathematician or scientist—who claims NOT to be obsessed with these huge issues! I suspect many MIRI readers might feel drawn to such questions the same way I am, in which case there’s no need to belabor the point.
From my perspective, then, the best way to frame the question is not: “why be interested in philosophy?” Rather it’s: “why be interested in anything else?”
But I think the latter question has an excellent answer. A crucial thing humans learned, starting around Galileo’s time, is that even if you’re interested in the biggest questions, usually the only way to make progress on them is to pick off smaller subquestions: ideally, subquestions that you can attack using math, empirical observation, or both. For again and again, you find that the subquestions aren’t nearly as small as they originally looked! Much like with zooming in to the Mandelbrot set, each subquestion has its own twists and tendrils that could occupy you for a lifetime, and each one gives you a new perspective on the big questions. And best of all, you can actually answer a few of the subquestions, and be the first person to do so: you can permanently move the needle of human knowledge, even if only by a minuscule amount. As I once put it, progress in math and science — think of natural selection, Godel’s and Turing’s theorems, relativity and quantum mechanics — has repeatedly altered the terms of philosophical discussion, as philosophical discussion itself has rarely altered them! (Of course, this is completely leaving aside math and science’s “fringe benefit” of enabling our technological civilization, which is not chickenfeed either.)

On this view, philosophy is simply too big and too important to be confined to philosophy departments! Of course, the word “philosophy” used to mean the entire range of fundamental inquiry, from epistemology and metaphysics to physics and biology (which were then called “natural philosophy”), rather than just close textual analysis, or writing papers with names like “A Kripkean Reading of Wittgenstein’s Reading of Frege’s Reading of Kant.” And it seems clear to me that there’s enormous scope today for “philosophy” in the former sense — and in particular, for people who love working on the subquestions, on pushing the frontiers of neuroscience or computer science or physics or whatever else, but who also like to return every once in a while to the “deep” philosophical mysteries that motivated them as children or teenagers. Admittedly, there have been many great scientists who didn’t care at all about philosophy, or who were explicitly anti-philosophy. But there were also scientists like Einstein, Schrodinger, Godel, Turing, or Bell, who not only read lots of philosophy but (I would say) used it as a sort of springboard into science — in their cases, a wildly successful one. My guess would be that science ultimately benefits from both the “pro-philosophical” and the “anti-philosophical” temperaments, and even from the friction between them.
As for the “social value” of philosophy, I suppose there are a few things to say. First, the world needs good philosophers, if for no other reason than to refute bad philosophers! (This is similar to why the world needs lawyers, politicians, and soldiers.) Second, the Enlightenment seems like a pretty big philosophical success story. Philosophers like Locke and Spinoza directly influenced statesmen like Thomas Jefferson, in ways you don’t have to squint to see. Admittedly, philosophers’ positive influence on humankind’s moral progress is probably less today than in the 1700s (to put it mildly). And also, most of the philosophical questions that have obsessed me personally have been pretty thin in their moral implications. But that brings me to the third point: namely, to whatever extent you see social value in popularizing basic science — that is, in explaining the latest advances in cosmology, quantum information, or whatever else to laypeople — to that extent I think you also need to see social value in philosophy. For the popularizer doesn’t have the luxury of assuming the importance of the particular subquestion on which progress has been made. Instead, he or she constantly needs to say what the little tendrils currently being explored do (or just as importantly, don’t) imply about the whole fractal — and when you’re zooming out like that, it’s hard to avoid talking about philosophy.

Luke: You write that “usually the only way to make progress on [the big questions] is to pick off smaller subquestions: ideally, subquestions that you can attack using math, empirical observation, or both.” This is an idea you wrote about at greater length in one of your papers — specifically, in this passage:
whenever it’s been possible to make definite progress on ancient philosophical problems, such progress has almost always involved a [kind of] “bait-and-switch.” In other words: one replaces an unanswerable philosophical riddle Q by a “merely” scientific or mathematical question Q′, which captures part of what people have wanted to know when they’ve asked Q. Then, with luck, one solves Q′.
Of course, even if Q′ is solved, centuries later philosophers might still be debating the exact relation between Q and Q′! And further exploration might lead to other scientific or mathematical questions — Q′′, Q′′′, and so on — which capture aspects of Q that Q′ left untouched. But from my perspective, this process of “breaking oﬀ” answerable parts of unanswerable riddles, then trying to answer those parts, is the closest thing to philosophical progress that there is.
…A good replacement question Q′ should satisfy two properties: (a) Q′ should capture some aspect of the original question Q — so that an answer to Q′ would be hard to ignore in any subsequent discussion of Q, [and] (b) Q′ should be precise enough that one can see what it would mean to make progress on Q′: what experiments one would need to do, what theorems one would need to prove, etc.
What are some of your favorite examples of illuminating Q-primes that were solved within your own field, theoretical computer science?

Scott: It’s hard to know where to begin with this question! In fact, my 59-page essay Why Philosophers Should Care About Computational Complexity was largely devoted to cataloging the various “Q-primes” on which I think theoretical computer science has made progress. However, let me mention four of my favorites, referring readers to the essay for details:
(1) One of the biggest, oldest questions in the philosophy of science could be paraphrased as: “why is Occam’s Razor justified? when we find simple descriptions of past events, why do we have any grounds whatsoever to expect those descriptions to predict future events?” This, I think, is the core of Hume’s “problem of induction.” Now, I think theoretical computer science has contributed large insights to this question — including Leslie Valiant’s Probably Approximately Correct (PAC) learning model, for which he recently won the Turing Award; the notion of Vapnik-Chernonenkis (VC) dimension; and the notion of the universal prior from algorithmic information theory. In essence, these ideas all give you various formal models where Occam’s Razor provably works — where you can give “simplicity” a precise definition, and then see exactly why simple hypotheses are more likely to predict the future than complicated ones. Of course, a skeptic about induction could still ask: OK, but why are the assumptions behind these formal models justified? But to me, this represents progress! The whole discussion can now start from a more sophisticated place than before.
(2) One of the first questions anyone asks on learning quantum mechanics is, “OK, but do all these branches of the wavefunction really exist? or are they just mathematical constructs used to calculate probabilities?” Roughly speaking, Many-Worlders would say they do exist, while Copenhagenists would say they don’t. Of course, part of what makes the question slippery is that it’s not even completely clear what we mean by words like “exist”! Now, I’d say that quantum computing theory has sharpened the question in many ways, and actually answered some of the sharpened versions — but interestingly, sometimes the answer goes one way and sometimes it goes the other! So for example, we have strong evidence that quantum computers can solve certain specific problems in polynomial time that would require exponential time to solve using a classical computer. Some Many-Worlders, most notably David Deutsch, have seized on the apparent exponential speedups for problems like factoring, as the ultimate proof that the various branches of the wavefunction must literally exist: “if they don’t exist,” they ask, “then where was this huge number factored? where did the exponential resources to solve the problem come from?” The trouble is, we’ve also learned that a quantum computer could NOT solve arbitrary search problems exponentially faster than a classical computer could solve them — something you’d probably predict a QC could do, if you thought of all the branches of the wavefunction as just parallel processors. If you want a quantum speedup, then your problem needs a particular structure, which (roughly speaking) lets you choreograph a pattern of constructive and destructive interference involving ALL the branches. You can’t just “fan out” and have one branch try each possible solution — twenty years of popular articles notwithstanding, that’s not how it works! We also know today that you can’t encode more than about n classical bits into n quantum bits (qubits), in such a way that you can reliably retrieve any one of the bits afterward. And we have all lots of other results that make quantum-mechanical amplitudes feel more like “just souped-up versions of classical probabilities,” and quantum superposition feel more like just a souped-up kind of potentiality. I love how the mathematician Boris Tsirelson summarized the situation: he said that “a quantum possibility is more real than a classical possibility, but less real than a classical reality.” It’s an ontological category that our pre-mathematical, pre-quantum intuitions just don’t have a good name for.
(3) Many interesting philosophical puzzles boil down to what it means to know something: and in particular, to the difference between knowing something “explicitly” and knowing it only “implicitly.” For example, I mentioned in my essay the example of the largest “known” prime number. According to the Great Internet Mersenne Prime Search, that number is currently 2^57885161 – 1. The question is, why can’t I reply immediately that I know a bigger prime number: namely, “the first prime larger than 2^57885161 – 1”? I can even give you an algorithm to find my number, which provably halts: namely, starting from 2^57885161, try each number one by one until you hit a prime! Theoretical computer science has given us the tools to sharpen a huge number of questions of this sort, and sometimes answer them. Namely, we can say that to know a thing “explicitly” means, not merely to have ANY algorithm to generate the thing, but to have a provably polynomial-time algorithm. That gives us a very clear sense in which, for example, 2^57885161 – 1 is a “known” prime number while the next prime after it is not. And, in many cases where mathematicians vaguely asked for an “explicit construction” of something, we can sharpen the question to whether or not some associated problem has a polynomial-time algorithm. Then, sometimes, we can find such an algorithm or give evidence against its existence!
(4) One example that I didn’t discuss in the essay — but a wonderful one, and one where there’s actually been huge progress in the last few years — concerns the question of how we can ever know for sure that something is “random.” I.e., even if a string of bits passes every statistical test for randomness that we throw it at, how could we ever rule out that there’s some complicated regularity that we simply failed to find? In the 1960s, the theory of Kolmogorov complexity offered one possible answer to that question, but a rather abstract and inapplicable one: roughly speaking, it said we can consider a string “random enough for our universe” if it has no computable regularities, if there’s no program to output the string shorter than the string itself. More recently, a much more practical answer has come from the Bell inequality — and in particular, from the realization that the experimental violation of that inequality can be used to produce so-called “Einstein-certified random numbers.” These are numbers that are provably random, assuming only (a) that they were produced by two separated devices that produced such-and-such outputs in response to challenges, and (b) there was no faster-than-light communication between the devices. But it’s only within the last few years that computer scientists figured out how to implement this striking idea, in such a way that you get out more randomness than you put in. (Recently, two MIT grad students proved that, starting from a fixed “seed” of, let’s say, 100 random bits, you can produce unlimited additional random bits in this Einstein-certified way — see Infinite Randomness Expansion and Amplification with a Constant Number of Devices) And the experimental demonstration of these ideas is just getting started now. Anyway, I’m working on an article for American Scientist magazine about these developments, so rather than cannibalize the article, I’ll simply welcome people to read it when it’s done!

Luke: What do you think about philosophy the field — work published by people in philosophy departments, who publish mostly in philosophy journals like Mind and Noûs, who are writing mostly for other philosophers?
I’ve previously called philosophy a “diseased discipline,” for many reasons. For one thing, people working in philosophy-the-field tend to know strikingly little about the philosophical progress made in other fields, e.g. computer science or cognitive neuroscience. For another, books on the history of philosophy seem to be about the musings of old dead guys who were wrong about almost everything because they didn’t have 20th century science or math, rather than about actual philosophical progress, which is instead recounted in books like The Information.
Do you wish people in other fields would more directly try to use the tools of their discipline to make philosophical progress on The Big Questions? Do you wish philosophy-the-field would be reformed in certain ways? Would you like to see more crosstalk between disciplines about philosophical issues? Do you think that, as Clark Glymour suggested, philosophy departments should be defunded unless they produce work that is directly useful to other fields (as is the case with Glymour’s department)?

Scott: Well, let’s start with the positives of academic philosophy!
(1) I liked the philosophy of math and science courses that I took in college. Sure, I sometimes got frustrated by the amount of time spent on what felt like Talmudic exegesis, but on the other hand, those courses offered a scope for debating big, centuries-old questions that my math and science courses hardly ever did.
(2) These days, I go maybe once a year to conferences where I meet professional philosophers of science, and I’ve found my interactions with them stimulating and fun. Philosophers often listen to what you say more carefully than other scientists do, and they’re incredibly good at spotting hidden assumptions, imprecise use of language, that sort of thing. Also, philosophers of science tend to double in practice as science historians: they often know much, much more about what, let’s say, Einstein or Bohr or Godel or Turing wrote and believed than physicists and mathematicians themselves know.
(3) While my own reading of the philosophical classics has been woefully incomplete, I don’t feel like the time I spent with (say) Hume or J. S. Mill or William James or Bertrand Russell was wasted at all. You’re right that these “old dead guys” didn’t know all the math and science we know today, but then again, neither did Shakespeare or Dostoyevsky! I mean, sure, the central questions of philosophy have changed over time, and the human condition has changed as well: we no longer get confused over Zeno’s paradoxes or the divine right of kings, and we now have global telecommunications and the Pill. I just don’t think either human nature or human philosophical concerns have changed quickly enough for great literature on them written centuries ago to have ceased being great.
Having said all that, from what I’ve seen of academic philosophy, I do pretty much agree with your diagnoses of its “diseases.” By far the most important disease, I’d say, is the obsession with interpreting and reinterpreting the old masters, rather than moving beyond them. Back in college, after we’d spent an hour debating why this passage of Frege seemed to contradict that one, I’d sometimes want to blurt out: “so maybe he was having a bad day! I mean, he was also a raving misogynist and antisemite; he believed all kinds of things. Look, we’ve read Frege, we’ve learned from Frege, now can’t we just give the old dude a rest and debate the ground truth about the problems he was trying to solve?” Likewise, when I read books about the philosophy of physics or computing, it sometimes feels like I’m stuck in a time warp, as the contributors rehash certain specific debates from the 1930s over and over (say, about the Church-Turing Thesis or the Einstein-Podolsky-Rosen paradox). I want to shout, “enough already! why not help clarify some modern scientific debates—-say, about quantum computing, or string theory, or the black-hole firewall problem, ones where we don’t already know how everything turns out?” To be fair, today there are philosophers of science who are doing exactly that, and who have interesting and insightful things to say. That’s a kind of philosophy that I’d love to see more of, at the expense of the hermeneutic kind.
Now, regarding Clark Glymour’s suggestion that philosophy departments be defunded unless they produce work useful to other fields — from what I understand, something not far from that is already happening! As bad as our funding woes in the sciences might be, I think the philosophers have it a hundred times worse, with like a quadrillion applicants for every tenure-track opening. So it seems to me like the right question is not how much further those poor dudes should be defunded, but rather: what can philosophy departments do to make themselves more vibrant, places that scientists regularly turn to for clarifying insights, and that deans and granting agencies get excited about wanting to expand? As a non-philosopher, I hesitate to offer unsolicited “advice” about such matters, but I guess I already did in the previous paragraph.
One final note: none of the positive or hopeful things that I said about philosophy apply to the postmodern or Continental kinds. As far as I can tell, the latter aren’t really “philosophy” at all, but more like pretentious brands of performance art that fancy themselves politically subversive, even as they cultivate deliberate obscurity and draw mostly on the insights of Hitler and Stalin apologists. I suspect I won’t ruffle too many feathers here at MIRI by saying this.

Luke: Suppose a mathematically and analytically skilled student wanted to make progress, in roughly the way you describe, on the Big Questions of philosophy. What would you recommend they study? What should they read to be inspired? What skills should they develop? Where should they go to study?

Scott: The obvious thing to say is that, as a student, you should follow your talents and passions, rather than following the generic advice of some guy on the Internet who doesn’t even know you personally!
Having said that, I would think broadly about which fields can give you enough scope to address the “Big Questions of Philosophy.”  You can philosophize from math, computer science, physics, economics, cognitive science, neuroscience, and probably a bunch of other fields too.  (My colleague Seth Lloyd philosophizes left and right, from his perch in MIT’s Mechanical Engineering department.)  Furthermore, all of these fields have the crucial advantage that they’ll offer you a steady supply of “fresh meat”: that is, new and exciting empirical or theoretical discoveries in which you can participate, and that will give you something to philosophize ABOUT (not to mention, something to do when you’re not philosophizing).  If I were working in a philosophy department, I feel like I’d have to make a conscious and deliberate effort to avoid falling into a “hermeneutic trap,” where I’d spend all my time commenting on what other philosophers had said about the works of yet other philosophers, and where I’d seal myself off from anything that had happened in the world of science since (say) Godel’s Theorem or special relativity.  (Once again, though, if you find that your particular talents and passions are best served in an academic philosophy department, then don’t let some guy on the Internet stop you!)
Regardless of your major, I recommend taking a huge range of courses as an undergrad: math, computer science (both applied and theoretical), physics, humanities, history, writing, and yes, philosophy. Looking back on my own undergrad years, the most useful courses I took were probably my math courses, and that’s despite the fact that most of them were poorly taught!  Things like linear algebra, group theory, and probability have so many uses throughout science that learning them is like installing a firmware upgrade to your brain — and even the math you don’t use will stretch you in helpful ways.  After math courses, the second most useful courses I took were writing seminars — the kind where a small group of students reads and critiques one another’s writing, and the professor functions mostly as a moderator.  It was in such a seminar that I wrote my essay “Who Can Name the Bigger Number?“, which for better or worse, continues to attract more readers than anything else I’ve written in the fifteen years since.  One writing seminar, if it’s good, can easily be worth the whole cost of a college tuition.
If you’re the kind of person for whom this advice is intended, then you probably don’t have to be told to read widely and voraciously, anything you get curious about.  Don’t limit yourself to one genre, don’t limit yourself to stuff you agree with, and certainly don’t limit yourself to the assigned reading for your courses.  When I was an adolescent, my favorites were just what a nerd stereotyper might expect: science fiction (especially Isaac Asimov), books about programming and the software industry, and math puzzle books (especially Martin Gardner).  A few years later, I became obsessed with reading biographies of scientists, like Feynman, Ramanujan, Einstein, Schrodinger, Turing, Godel, von Neumann, and countless lesser luminaries.  I was interested in every aspect of their lives — in their working habits, their hobbies, their views on social and philosophical issues, their love lives — but, I confess, I was particularly interested in what they were doing as teenagers, so that I could compare to what I was doing and sort of see how I measured up.  At the same time, my reading interests were broadening to include politics, history, philosophy, psychology, and some contemporary fiction (I especially like Rebecca Goldstein).  It was only in grad school that I felt I’d sufficiently recovered from high-school English to tackle “real literature” like Shakespeare — but when I did, it was worth it.
As for where to study, well, the “tautological” answer is wherever will give you the best opportunities!  There are certain places, like Boston or the Bay Area, that are famous for having high concentrations of intellectual opportunity, but don’t go somewhere just because of what you’ve heard about the general atmosphere or prestige: particularly for graduate school, go where the particular people or programs are that resonate for you.  In quantum computing, for example, one of the centers of the world for the last decade has been Waterloo, Canada — a place many people hadn’t even heard of when I did my postdoc there eight years ago (though that’s changing now).  And one of the intellectually richest years of my life came when I attended The Clarkson School, a program that lets high-school students live and take courses at Clarkson University in Potsdam, NY.  (I went there when I was 15, and was looking for something less prison-like than high school.)  If, for what you personally want to do, there are better opportunities in Topeka, Kansas than at Harvard, go to Topeka.

Luke: Finally, I’d like to ask about which object-level research tactics — more specific than your general “bait and switch” strategy — you suspect are likely to help with philosophical research, or perhaps with theoretical research of any kind.
For example, some of the tactics we’ve found helpful at MIRI include:

When you’re confused about a fuzzy, slippery concept, try to build a simple formal model and push on it with the new tools then available to you. Even if the model doesn’t capture the complexity of the world, pushing things into the mathematical realm can lead to progress. E.g. the VNM axioms don’t exactly capture “rationality,” but it sure is easier to think clearly about rationality once you have them. Or: we’re confused about how to do principled reflective reasoning within an agent, so even though advanced AIs are unlikely to literally run into a “Löbian obstacle” to self-reflection, setting up the problem that way (in mathematical logic) can lead to some interesting insights in (e.g.) probabilistic metamathematics for reflective reasoning.
Look for tools from other fields that appear to directly map onto the phenomena you’re studying. E.g. model moral judgment as an error process amenable to Bayesian curve fitting.
Try to think of how your concept could be instantiated with infinite computing power. If you can’t do that, your concept might be fundamentally confused.
If you’re pretty familiar with modern psychology, then… When using your intuitions to judge between options, try to think about which cognitive algorithms could be generating those intuitions, and whether they are cognitive algorithms whose outputs you reflectively endorse.
To make the thing you’re studying clearer, look just next to it, and around it. Foer (2009) explains this nicely in the context of thinking about one’s values and vegetarianism: “A simple trick from the backyard astronomer: if you are having trouble seeing something, look slightly away from it. The most light-sensitive parts of our eyes (those we need to see dim objects) are on the edges of the region we normally use for focusing. Eating animals has an invisible quality. Thinking about dogs, and their relationship to the animals we eat, is one way of looking askance and making something invisible visible.”

Which object-level thinking tactics, at roughly this level of specificity, do you use in your own theoretical (especially philosophical) research? Are there tactics you suspect might be helpful, which you haven’t yet used much yourself?

Scott: As far as I can remember, I’ve never set out to do “philosophical research,” so I can’t offer specific advice about that. What I have often done is research in quantum computing and complexity theory that was motivated by some philosophical issue, usually in foundations of quantum mechanics. (I’ve also written a few philosophical essays, but I don’t really count those as “research.”) Anyway, I can certainly offer advice about doing the kind of research I like to do!
(1) Any time you find yourself in a philosophical disagreement with a fellow scientist, don’t be content just to argue philosophically — even if you’re sure you can win the argument! Instead, think hard about whether you can go further, and find a concrete technical question that captures some little piece of what you’re disagreeing about. Then see if you can answer that technical question. Of course, any time you do this, you have to be prepared for the possibility that the answer will go your opponent’s way, rather than yours! But what’s nice is that you get to publish a paper even then. (One of the best ways to tell whether a given enterprise is scientific at all, rather than ideological, is by asking whether the participants will opportunistically “go to bat for the opposing side” whenever they find a novel truth on that side.) I’d estimate that up to half the papers I’ve written had their origin in my reading or overhearing some claim — for example, “Grover’s algorithm obviously can’t work for searching actual physical databases, since the speed of light is finite,” or “the quantum states arising in Shor’s algorithm are obviously completely different from anything anyone has ever seen in the lab,” or “the interactive proof results obviously make oracle separations completely irrelevant” — and getting annoyed, either because I thought the claim was false, or because I simply didn’t think it had been adequately justified. The cases where my annoyance paid off are precisely the ones where, rather than just getting mad, I managed to get technical!
(2) Often, the key to research is figuring out how to redefine failure as success. Some stories: when Alan Turing published his epochal 1936 paper on Turing machines, he did so with great disappointment: he had recently learned that Alonzo Church had independently arrived at similar results using lambda calculus, and he didn’t know whether anyone would still be interested in his alternative, machine-based approach. In the early 1970s, Leonid Levin delayed publishing about NP-completeness for several years: apparently, his “real” goal was to prove graph isomorphism was NP-complete (something we now know is almost certainly false), and in his mind, he had failed. Instead, he merely had a few “trivialities,” like the definitions of P, NP, and NP-completeness, and the proof that satisfiability was NP-complete. And Levin’s experience is far from unique: again and again in mathematical research, you’ll find yourself saying something like: “goddammit, I’ve been trying for six months to prove Y, but I can only prove the different/weaker statement X! And every time I think I can bridge the gap between X and Y, yet another difficulty rears its head!” Any time that happens to you, think hard about whether you can write a compelling paper that begins: “Y has been a longstanding open problem. In this work, we introduce a new idea: to make progress on Y by shifting attention to the more tractable X.” More broadly, experience has shown that scientists are terrible judges of which of their ideas will be interesting or important to others. Pick any scientist’s most cited paper, and there’s an excellent chance that the scientist herself, at one point, considered it a “little recreational throwaway project” that was barely worth writing up. After you’ve seen enough examples of that, you learn you should always err on the side of publishing, and let posterity sort out which of your ideas are most important. (Yet another advantage of this approach is that, the more ideas you publish, the less emotionally invested you are in any one of them, so the less crushed you are when a few turn out to be wrong or trivial or already known.)
(3) Sometimes, when you set out to prove some mathematical conjecture, your first instinct is just to throw an arsenal of theory at it. “Hey, what if I try a topological fixed-point theorem? What if I translate the problem into a group-theoretic language? If neither of those works, what if I try both at once?” Sometimes, you rise so quickly this way into a stratosphere of generality that the original problem is barely a speck on the ground. And yes, some problems can be beaten into submission using high-powered theory. But in my experience, there are two enormous risks with this approach. First, you’re liable to get lost on a wild goose chase, where you get so immersed in theory and techniques that you lose sight of your original goal. It’s as if your efforts to break into a computer network lead you to certain complicated questions about the filesystem, which in turn lead you to yet more complicated questions about the kernel… and in the meantime someone else breaks in by guessing people’s birthdays for their passwords. Second, you’re also liable to fool yourself this way into thinking you’ve solved the problem when you haven’t. When you let high-powered machinery take the place of hands-on engagement with the problem, a single mistake in applying the machinery can creep in unbelievably easily. These risks are why I’ve learned over time to work in an extremely different way. Rather than looking for “general frameworks,” I look for easy special cases and simple sanity checks, for stuff I can try out using high-school algebra or maybe a five-line computer program, just to get a feel for the problem. Even more important, when I’m getting started, I don’t think about proof techniques at all: I think instead about obstructions. That is, I ask myself, “what would the world have to be like for the conjecture to be false? what goes wrong if I try to invent a simple counterexample? does anything go wrong? it does? OK then, what obstruction keeps me from proving this conjecture in the simplest, dumbest way imaginable?” I find that, after you’ve felt out the full space of obstructions and counterexamples, and really honestly convinced yourself of why the conjecture should be true, finding the proof techniques by which to convince everyone else is often a more-or-less routine exercise.
Finally, you ask about tactics that I suspect might be helpful, but that I haven’t used much myself. One that springs to mind is to really master a tool like Mathematica, MATLAB, Maple, or Magma — that is, to learn it so well that I can code as fast as I think, and just let it take over all the routine / calculational / example-checking parts of my work. As it is, I use pretty much the same antiquated tools that I learned as an adolescent, and I rely on students whenever there’s a need for better tools. A large part of the problem is that, as a “tenured old geezer,” I no longer have the time or patience to learn new tools just for the sake of learning them: I’m always itching just to solve the problem at hand with whatever tools I know. (The same issue has kept me from learning new mathematical tools, like representation theory, even when I can clearly see that they’d benefit me.)

Luke: Thanks, Scott!
The post Scott Aaronson on Philosophical Progress appeared first on Machine Intelligence Research Institute.


2013 Winter Matching Challenge

Thanks to Peter Thiel, every donation made to MIRI between now and January 15th, 2014 will be matched dollar-for-dollar!
Also, gifts from “new large donors” will be matched 3x! That is, if you’ve given less than $5k to SIAI/MIRI ever, and you now give or pledge $5k or more, Thiel will donate $3 for every dollar you give or pledge.
We don’t know whether we’ll be able to offer the 3:1 matching ever again, so if you’re capable of giving $5k or more, we encourage you to take advantage of the opportunity while you can. Remember that:

If you prefer to give monthly, no problem! If you pledge 6 months of monthly donations, your full 6-month pledge will be the donation amount to be matched. So if you give monthly, you can get 3:1 matching for only $834/mo (or $417/mo if you get matching from your employer).
We accept Bitcoin (BTC) and Ripple (XRP), both of which have recently jumped in value. If the market value of your Bitcoin or Ripple is $5k or more on the day you make the donation, this will count for matching.
If your employer matches your donations at 1:1 (check here), then you can take advantage of Thiel’s 3:1 matching by giving as little as $2,500 (because it’s $5k after corporate matching).

Please email malo@intelligence.org if you intend on leveraging corporate matching or would like to pledge 6 months of monthly donations, so that we can properly account for your contributions towards the fundraiser.
Thiel’s total match is capped at $250,000. The total amount raised will depend on how many people take advantage of 3:1 matching. We don’t anticipate being able to hit the $250k cap without substantial use of 3:1 matching — so if you haven’t given $5k thus far, please consider giving/pledging $5k or more during this drive. (If you’d like to know the total amount of your past donations to MIRI, just ask malo@intelligence.org.)

$0

$62.5K

$125K

$187.5k

$250k

We have reached our matching total of $250,000!
Now is your chance to double or quadruple your impact in funding our research program.

 Donate Today

Accomplishments Since Our July 2013 Fundraiser Launched:

Held three research workshops, including our first European workshop.
Talks at MIT and Harvard, by Eliezer Yudkowsky and Paul Christiano.
Yudkowsky is blogging more Open Problems in Friendly AI… on Facebook! (They’re also being written up in a more conventional format.)
New papers: (1) Algorithmic Progress in Six Domains; (2) Embryo Selection for Cognitive Enhancement; (3) Racing to the Precipice; (4) Predicting AGI: What can we say when we know so little?
New ebook: The Hanson-Yudkowsky AI-Foom Debate.
New analyses: (1) From Philosophy to Math to Engineering; (2) How well will policy-makers handle AGI? (3) How effectively can we plan for future decades? (4) Transparency in Safety-Critical Systems; (5) Mathematical Proofs Improve But Don’t Guarantee Security, Safety, and Friendliness; (6) What is AGI? (7) AI Risk and the Security Mindset; (8) Richard Posner on AI Dangers; (9) Russell and Norvig on Friendly AI.
New expert interviews: Greg Morrisett (Harvard), Robin Hanson (GMU), Paul Rosenbloom (USC), Stephen Hsu (MSU), Markus Schmidt (Biofaction), Laurent Orseau (AgroParisTech), Holden Karnofsky (GiveWell), Bas Steunebrink (IDSIA), Hadi Esmaeilzadeh (GIT), Nick Beckstead (Oxford), Benja Fallenstein (Bristol), Roman Yampolskiy (U Louisville), Ben Goertzel (Novamente), and James Miller (Smith College).
With Leverage Research, we held a San Francisco book launch party for James Barratt’s Our Final Invention, which discusses MIRI’s work at length. (If you live in the Bay Area and would like to be notified of local events, please tell malo@intelligence.org!)

 Donate Today

How Will Marginal Funds Be Used?

Hiring Friendly AI researchers, identified through our workshops, as they become available for full-time work at MIRI.
Running more workshops (next one begins Dec. 14th), to make concrete Friendly AI research progress, to introduce new researchers to open problems in Friendly AI, and to identify candidates for MIRI to hire.
Describing more open problems in Friendly AI. Our current strategy is for Yudkowsky to explain them as quickly as possible via Facebook discussion, followed by more structured explanations written by others in collaboration with Yudkowsky.
Improving humanity’s strategic understanding of what to do about superintelligence. In the coming months this will include (1) additional interviews and analyses on our blog, (2) a reader’s guide for Nick Bostrom’s forthcoming Superintelligence book, and (3) an introductory ebook currently titled Smarter Than Us.

Other projects are still being surveyed for likely cost and impact.
We appreciate your support for our work! Donate now, and seize a better than usual chance to move our work forward. If you have questions about donating, please contact Louie Helm at (510) 717-1477 or louie@intelligence.org.
The post 2013 Winter Matching Challenge appeared first on Machine Intelligence Research Institute.

2013 Winter Matching Challenge


New Paper: “Predicting AGI: What can we say when we know so little?”

MIRI research associate Benja Fallenstein and UC Berkeley student Alex Mennen have released a new working paper titled “Predicting AGI: What can we say when we know so little?”
From the introduction:
This analysis does not attempt to predict when AGI will actually be achieved, but instead, to predict when this epistemic state with respect to AGI will change, such that we will have a clear idea of how much further progress is needed before we reach AGI. Metaphorically speaking, instead of predicting when AI takes off, we predict when it will start taxiing to the runway.
The paper argues for a Pareto distribution for “time to taxi,” and concludes:
in general, a Pareto distribution suggests that we should put a much greater emphasis on short-term strategies than a less skewed distribution (e.g. a normal distribution) with the same median would.
The post New Paper: “Predicting AGI: What can we say when we know so little?” appeared first on Machine Intelligence Research Institute.


New Paper: “Racing to the Precipice”

During his time as a MIRI research fellow, Carl Shulman contributed to a paper now available as an FHI technical report: Racing to the Precipice: a Model of Artificial Intelligence Development.
Abstract:
This paper presents a simple model of an AI arms race, where several development teams race to build the first AI. Under the assumption that the first AI will be very powerful and transformative, each team is incentivized to finish first — by skimping on safety precautions if need be. This paper presents the Nash equilibrium of this process, where each team takes the correct amount of safety precautions in the arms race. Having extra development teams and extra enmity between teams can increase the danger of an AI-disaster, especially if risk taking is more important than skill in developing the AI. Surprisingly, information also increases the risks: the more teams know about each others’ capabilities (and about their own), the more the danger increases.
Update: As of July 2015, this paper has been published in the journal AI & Society.
The post New Paper: “Racing to the Precipice” appeared first on Machine Intelligence Research Institute.

New Paper: “Racing to the Precipice”


MIRI’s November 2013 Newsletter

Dear friends,

We’re experimenting with a new, ultra-brief newsletter style. To let us know what you think of it, simply reply to this email. Thanks!

News Updates

You can now support MIRI for free by shopping at smile.amazon.com instead of amazon.com. Update your bookmarks!
Louie Helm will be onsite for the Nov. 24th Marin county screening of Doug Wolens’ new documentary The Singularity. Details on this and other screenings are here.

Research Updates

New paper: “Embryo Selection for Cognitive Enhancement.” (Including a description of iterated embryo selection.)
New analysis: “From Philosophy to Math to Engineering.” (How we see research progress.)
Eliezer Yudkowsky gave a talk at MIT, but unfortunately MIT failed to record the video as scheduled. A later delivery of the talk may be recorded and released in the next few months.
Paul Christiano gave a talk at Harvard about probabilistic metamathematics. Video here.
Two surveys of what others have written about risks from AGI: Richard Posner (the most-cited legal scholar of the 20th century) and Stuart Russell & Peter Norvig (authors of the #1 AI textbook).
Several new expert interviews: Ben Goertzel (Novamente) on AGI as a field, Hadi Esmaeilzadeh (GIT) on dark silicon’s threat to Moore’s Law, Bas Steunebrink (IDSIA) on self-reflective programming, Markus Schmidt (Biofaction) on risks from novel biotechnologies, Robin Hanson (GMU) on “serious futurism,” and Greg Morrisett (Harvard) on secure and reliable systems.

Other Updates

Many of our friends have said Louie Helm’s Rockstar Research is among their favorite new sources of news; check it out!
Know an exceptionally bright, ambitious person younger than 20? Tell them to apply for The Thiel Fellowship! $100,000 to skip college and develop one’s skills and ideas, with an incredible network of mentors in the Bay Area.
CFAR has upcoming rationality workshops in February (Melbourne), March (Bay Area), and April (NYC). Tell your friends!

Cheers,

Luke Muehlhauser
Executive Director

The post MIRI’s November 2013 Newsletter appeared first on Machine Intelligence Research Institute.

MIRI’s November 2013 Newsletter


Support MIRI by Shopping at AmazonSmile

If you shop at the new AmazonSmile, Amazon donates 0.5% of the price of your eligible purchases to a charitable organization of your choosing.
MIRI is an eligible charitable organization, so the next time you consider purchasing something through Amazon, support MIRI by shopping at AmazonSmile!
If you get to Amazon.com via an affiliate link, remember to change “amazon.com” to “smile.amazon.com” in the address bar before making your purchase. Or, even easier, use the SmileAlways Chrome extension.
The post Support MIRI by Shopping at AmazonSmile appeared first on Machine Intelligence Research Institute.

Support MIRI by Shopping at AmazonSmile


Greg Morrisett on Secure and Reliable Systems

 Greg Morrisett is the Allen B. Cutting Professor of Computer Science at Harvard University. He received his B.S. in Mathematics and Computer Science from the University of Richmond in 1989, and his Ph.D. from Carnegie Mellon in 1995. In 1996, he took a position at Cornell University, and in the 2003-04 academic year, he took a sabbatical and visited the Microsoft European Research Laboratory. In 2004, he moved to Harvard, where he has served as Associate Dean for Computer Science and Engineering, and where he currently heads the Harvard Center for Research on Computation and Society.
Morrisett has received a number of awards for his research on programming languages, type systems, and software security, including a Presidential Early Career Award for Scientists and Engineers, an IBM Faculty Fellowship, an NSF Career Award, and an Alfred P. Sloan Fellowship.
He served as Chief Editor for the Journal of Functional Programming and as an associate editor for ACM Transactions on Programming Languages and Systems and Information Processing Letters. He currently serves on the editorial board for The Journal of the ACM and as co-editor-in-chief for the Research Highlights column of Communications of the ACM. In addition, Morrisett has served on the DARPA Information Science and Technology Study (ISAT) Group, the NSF Computer and Information Science and Engineering (CISE) Advisory Council, Microsoft Research’s Technical Advisory Board, and Microsoft’s Trusthworthy Computing Academic Advisory Board.

Luke Muehlhauser: One of the interesting projects in which you’re involved is SAFE, a DARPA-funded project “focused on a clean slate design for resilient and secure systems.” What is the motivation for this project, and in particular for its “clean slate” approach?

Greg Morrisett: I think that from DARPA’s perspective, the primary force is that what we’ve been doing so far in terms of securing systems isn’t working.  I suspect that some high-profile problems with successful attacks from both nation states and other organizations has made them desperate to try new and different things.
At the same time, there was a growing recognition that legacy software (and potentially hardware) was a big part of the problem. If, for instance, all of our critical software were coded in type-safe languages, then a lot of low-level problems (e.g., buffer overruns) would be gone.  So I think this was the thinking behind the whole CRASH program and there are something like 20 different teams trying a range of ideas.
Our particular project (SAFE) took a very radical approach in wanting to re-think everything:  the languages, the operating system, and even the hardware, as well as an emphasis on formal methods for trying to gain some assurance.  The team has designed some very interesting hardware that (a) effectively enforces type-safety at the machine level (i.e., object-level integrity, pointers as capabilities, etc.), and (b) provides novel support for doing fine-grained end-to-end information flow tracking.  We’ve also designed a high-level, dynamically-typed language (Breeze) that has pervasive information flow tracking built in.  This means that we can tell which principal actors may have influenced the construction of any value in the system. We’re now building compiler support for Breeze and systems-level software (e.g., scheduler, garbage collector, etc.) to bridge the high-level code and the hardware.
At the same time, we’ve been trying to apply formal models and reasoning to try to prove key properties about models of the system.  For example, we’ve had machine-checked proofs of security for the (core of the) language from the very beginning, and are currently building core models of the hardware and trying to establish similar properties.

Luke: Could you explain, preferably with examples, what you mean by “type-safety at the machine level” and “fine-grained end-to-end information flow tracking”?

Greg: So let’s first start with “type-safety”. In this context, I simply mean that we make a distinction between different values in memory and the machine will only allow appropriate operations to be applied to appropriate values. For instance, we make a distinction between pointers to objects (of a given size) and integers, and the machine will only let you load/store relative to a pointer (and within the bounds of the object to which it points). It will trap if you try to load/store relative to an integer. Similarly, we make a distinction between references to code and data, so you’re only allowed to jump-to or call references to code, and not data. These basic integrity properties block a whole range of basic attacks that rely upon violating abstractions (e.g., code injection, buffer overruns, etc.)
With respect to “information flow control” (IFC): we want to enforce policies like “unauthorized principal actors, say a web site with whom I’m communicating, should never learn my private information, say my users’ social security number of password”. The hard part is the “learn no information” as opposed to literally passing the data. If, for instance, I reverse your password and send it out, or take every other letter from your password, we want those to be seen as violations as well as explicitly copying the data. (There are much more subtle ways to communicate information as well.)
Formally, the policy we aim for is that the observations of the “unauthorized principal actors” should be the same if I change the password to an arbitrary string. The technical term for this is “non-interference”.
The way we implement IFC is by labeling every piece of data with an information flow control policy. When you combine two pieces of data (say add two numbers), we have rules for computing a new policy to place on the newly generated value. For example, if you add a “secret” and a “public” number, then the result is conservatively classified as “secret”.

Luke: In a blog post, I warned readers that no system is “provably secure” or “provably safe” in the sense of a 100% guarantee of security or safety, because e.g. a security/safety definition might be subtly wrong, or a particular security reduction might have an error in it. These things have happened before. I said that “Rather, these approaches are meant to provide more confidence than we could otherwise have, all else equal, that a given system is secure [or safe].”
Do you agree with that characterization? And, on the terminological issue: do you think terms like “provably secure” and “provably safe” are too misleading, or are they just another inevitable example of specialist terminology that is often but unavoidably interpreted incorrectly by non-specialists?

Greg: Yes, I completely agree with your blog post! All “proofs of correctness” for a system will be subject to modeling the environment, and specifying what we mean by correct. For real systems, modeling will never be sufficient, and formalizing the notion of correctness may be as complicated as reimplementing the system.
I do think that building explicit models of the environment, and writing formal specifications and constructing formal proofs has a lot of value though. As you noted, it really does help to increase confidence. And, it helps identify assumptions. Many times, an attacker will look at your assumptions and try to violate them to gain purchase. So identifying assumptions through modeling is a great exercise.
Going through the process of constructing a formal proof also leads to a lot of good architectural insight. For instance, when we were trying to construct a proof of correctness for Google’s Native Client checker, we realized it would be too hard with the tools we had at hand. So we came up with a simpler algorithm and architecture which was easier to prove correct. It also turned out to be faster! This is detailed in a technical paper: RockSalt: better, faster, stronger SFI for the x86.
Finally, all of this is a great push-back on creeping complexity in design. I think a lot of problems arise because software is so soft, and we write a lot of code without thinking deeply about what it should (not) be doing.
Back to your question as to whether we should use the term “provably secure”: probably not. I would like to use the term “certified code” because we are going through a very rigorous certification process. But unfortunately, the term “certified” has come to mean that we’ve followed some bureaucratic set of development processes that don’t actually provide much in the way of security or reliability. So I’m open to suggestions for better terms.

Luke: Harper (2000) quotes an old message by Ken Frith from the Safety Critical Mailing List:
The thought of having to apply formal proofs to intelligent systems leaves me cold. How do you provide satisfactory assurance for something that has the ability to change itself during a continuous learning process?
How do you think about the opportunities and limitations for applying formal methods to increasingly intelligent, autonomous systems?

Greg: I’ll be honest and say that I haven’t given it much thought. Perhaps the closest thing that I’ve worked on is trying to formalize information-theoretic security for cryptographic schemes and construct formal proofs in that setting. This is an area where I think the crypto/theory community has provided enough structure, we can hope to make progress, but even here, we have all sorts of shaky assumptions at the foundations.
I suspect that for intelligent and autonomous systems work (especially in the cyber-physical realm), the first real challenge is constructing appropriate models that are over-approximations of potential failure modes, and try to find something that is tractable to work with, similar to the way that crypto tries to model adversaries.

Luke: In general, for systems that need to meet a high standard of security, to what degree do you think we can take existing systems that were mostly designed to be effective, and “bolt security onto them,” versus needing to build things from the ground up to be secure? Perhaps some specific examples inform your view?

Greg: I think re-architecting and re-coding things will almost always lead to a win in terms of security, when compared to bolt-on approaches. A good example is qmail which was designed as a secure alternative to sendmail, and in comparison, had a good architecture that made it much easier to get most things right, including and especially, configuration. As far as I know, there’s only been one bug in qmail (an integer overflow) that could potentially lead to a security problem, whereas there are many, many issues with sendmail (I’ve lost count.)
But, in the end, security is at best a secondary consideration for almost all systems. And as much as I wish we could throw away all of the code and take the time to re-architect and re-code lots of things (especially using better language technology), it’s too expensive. For instance, Windows XP was over 50 million lines of code (I don’t know how big Win7 or Win8 are) so re-coding Windows just isn’t a realistic possibility.
What we need instead are tools and techniques that people can buy into now, and help block problems with legacy systems. Ideally, these same tools should give you a “pay-as-you-go” kind of property: a little bit of effort buys you some assurance, and more effort buys you a lot more.
A good example is the kind of static analysis tools that we’re starting to see for legacy C/C++ code, such as Coverity, HP/Fortify, or Microsoft’s Prefast/Prefix/SLAM/etc. These are tools that automatically identify potential errors which in turn could lead to vulnerabilities. Today, if you run say these tools on a random pile of C/C++ code, you’ll find that it gives lots of false positives. But if you work to minimize those warnings, then you’ll find that the tool becomes much more effective. And, for new code, it works quite well as long as you pay attention to the warnings and work to get rid of them.

Luke: Thanks, Greg!
The post Greg Morrisett on Secure and Reliable Systems appeared first on Machine Intelligence Research Institute.


From Philosophy to Math to Engineering

For centuries, philosophers wondered how we could learn what causes what. Some argued it was impossible, or possible only via experiment. Others kept hacking away at the problem, clarifying ideas like counterfactual and probability and correlation by making them more precise and coherent.
Then, in the 1990s, a breakthrough: Judea Pearl and others showed that, in principle, we can sometimes infer causal relations from data even without experiment, via the mathematical machinery of probabilistic graphical models.
Next, engineers used this mathematical insight to write software that can, in seconds, infer causal relations from a data set of observations.
Across the centuries, researchers had toiled away, pushing our understanding of causality from philosophy to math to engineering.

And so it is with Friendly AI research. Current progress on each sub-problem of Friendly AI lies somewhere on a spectrum from philosophy to math to engineering.

We began with some fuzzy philosophical ideas of what we want from a Friendly AI (FAI). We want it to be benevolent and powerful enough to eliminate suffering, protect us from natural catastrophes, help us explore the universe, and otherwise make life awesome. We want FAI to allow for moral progress, rather than immediately reshape the galaxy according to whatever our current values happen to be. We want FAI to remain beneficent even as it rewrites its core algorithms to become smarter and smarter. And so on.
Small pieces of this philosophical puzzle have been broken off and turned into math, e.g. Pearlian causal analysis and Solomonoff induction. Pearl’s math has since been used to produce causal inference software that can be run on today’s computers, whereas engineers have thus far succeeded in implementing (tractable approximations of) Solomonoff induction only for very limited applications.
Toy versions of two pieces of the “stable self-modification” problem were transformed into math problems in de Blanc (2011) and Yudkowsky & Herreshoff (2013), though this was done to enable further insight via formal analysis, not to assert that these small pieces of the philosophical problem had been solved to the level of math.
Thanks to Patrick LaVictoire and other MIRI workshop participants,1 Douglas Hofstadter’s FAI-relevant philosophical idea of “superrationality” seems to have been, for the most part, successfully transformed into math, and a bit of the engineering work has also been done.
I say “seems” because, while humans are fairly skilled at turning math into feats of practical engineering, we seem to be much less skilled at turning philosophy into math, without leaving anything out. For example, some very sophisticated thinkers have claimed that “Solomonoff induction solves the problem of inductive inference,” or that “Solomonoff has successfully invented a perfect theory of induction.” And indeed, it certainly seems like a truly universal induction procedure. However, it turns out that Solomonoff induction doesn’t fully solve the problem of inductive inference, for relatively subtle reasons.2
Unfortunately, philosophical mistakes like this could be fatal when humanity builds the first self-improving AGI (Yudkowsky 2008).3 FAI-relevant philosophical work is, as Nick Bostrom says, “philosophy with a deadline.”
And before them, Moshe Tennenholtz.Yudkowsky plans to write more about how to improve on Solomonoff induction, later.This is a specific instance of a problem Peter Ludlow described like this: “the technological curve is pulling away from the philosophy curve very rapidly and is about to leave it completely behind.”The post From Philosophy to Math to Engineering appeared first on Machine Intelligence Research Institute.

From Philosophy to Math to Engineering