# From Philosophy to Math to Engineering

Cross-posted from the [MIRI blog](http://intelligence.org/2013/11/04/from-philosophy-to-math-to-engineering/).

For centuries, philosophers wondered how we could learn what causes what. Some argued it was impossible, or possible only via experiment. Others kept [hacking away](/lw/8ns/hack_away_at_the_edges/) at the problem, [clarifying ideas](http://commonsenseatheism.com/wp-content/uploads/2013/10/Pearl-The-Art-and-Science-of-Cause-and-Effect.pdf) like _counterfactual_ and _probability_ and _correlation_ by making them more precise and coherent.

Then, in the 1990s, a breakthrough: Judea Pearl and others [showed](http://www.amazon.com/Causality-ebook/dp/B00AKE1VYK/) that, in principle, we can sometimes infer causal relations from data even without experiment, via the mathematical machinery of probabilistic graphical models.

Next, engineers used this mathematical insight to write [software](http://cran.r-project.org/web/packages/pcalg/index.html) that can, in seconds, infer causal relations from a data set of observations.

Across the centuries, researchers had toiled away, pushing our understanding of causality from philosophy to math to engineering.

[![From Philosophy to Math to Engineering (small)](https://intelligence.org/wp-content/uploads/2013/10/From-Philosophy-to-Math-to-Engineering-small.jpg)](https://intelligence.org/wp-content/uploads/2013/10/From-Philosophy-to-Math-to-Engineering.jpg)

And so it is with [Friendly AI](http://en.wikipedia.org/wiki/Friendly_artificial_intelligence) research. Current progress on each sub-problem of Friendly AI lies somewhere on a spectrum from philosophy to math to engineering.

We began with some fuzzy philosophical ideas of what we want from a Friendly AI (FAI). We want it to be benevolent and powerful enough to eliminate suffering, protect us from natural catastrophes, help us explore the universe, and otherwise make life _awesome_. We want FAI to allow for moral progress, rather than immediately reshape the galaxy according to whatever our current values happen to be. We want FAI to remain beneficent even as it rewrites its core algorithms to become smarter and smarter. And so on.

Small pieces of this philosophical puzzle have been broken off and [turned into](/lw/hok/link_scott_aaronson_on_free_will/9546?context=1#comments) math, e.g. [Pearlian causal analysis](http://www.amazon.com/Causality-ebook/dp/B00AKE1VYK/) and [Solomonoff induction](/lw/dhg/an_intuitive_explanation_of_solomonoff_induction/). Pearl's math has since been used to produce causal inference software that can be run on today's computers, whereas engineers have thus far succeeded in implementing (tractable approximations of) Solomonoff induction only for [very limited applications](http://arxiv.org/pdf/0909.0801v2.pdf).

Toy versions of two pieces of the "stable self-modification" problem were transformed into math problems in [de Blanc (2011)](http://intelligence.org/files/OntologicalCrises.pdf) and [Yudkowsky & Herreshoff (2013)](http://intelligence.org/files/TilingAgents.pdf), though this was done to enable further insight via formal analysis, not to assert that these small pieces of the philosophical problem had been _solved_ to the level of math.

Thanks to Patrick LaVictoire and other [MIRI workshop](http://intelligence.org/get-involved/#workshop) participants,^1^ Douglas Hofstadter's FAI-relevant philosophical idea of "[superrationality](http://en.wikipedia.org/wiki/Superrationality)" seems to have been, for the most part, successfully [transformed](http://intelligence.org/files/RobustCooperation.pdf) into math, and a bit of the engineering work has also [been done](https://github.com/klao/provability/blob/master/modal.hs).

I say "seems" because, while humans are fairly skilled at turning math into feats of practical engineering, we seem to be [much](/lw/4zs/philosophy_a_diseased_discipline/) [_less_](http://commonsenseatheism.com/wp-content/uploads/2012/10/Brennan-Scepticism-about-philosophy.pdf) [skilled](http://consc.net/papers/progress.pdf) at turning philosophy into math, without leaving anything out. For example, some very sophisticated thinkers have [claimed](http://commonsenseatheism.com/wp-content/uploads/2013/10/Rathmanner-Hutter-A-philosophical-treatise-of-universal-induction.pdf) that "Solomonoff induction solves the problem of inductive inference," or [that](http://www.amazon.com/Introduction-Kolmogorov-Complexity-Applications-Computer/dp/0387339981/) "Solomonoff has successfully invented a perfect theory of induction." And indeed, it certainly _seems_ like a truly universal induction procedure. However, it [turns out](/lw/cw1/open_problems_related_to_solomonoff_induction/) that Solomonoff induction _doesn't_ fully solve the problem of inductive inference, for relatively subtle reasons.^2^

Unfortunately, philosophical mistakes like this could be fatal when humanity builds the first self-improving AGI ([Yudkowsky 2008](http://intelligence.org/files/AIPosNegFactor.pdf)).^3^ FAI-relevant philosophical work is, as Nick Bostrom says, "philosophy with a deadline."

^1^ And before them, [Moshe Tennenholtz](http://mechroom.technion.ac.il/~moshet/progeqnote4.pdf).

^2^ Yudkowsky plans to write more about how to improve on Solomonoff induction, later.

^3^ This is a specific instance of a problem Peter Ludlow described like [this](http://leiterreports.typepad.com/blog/2013/10/progress-in-philosophy-revisited.html): "the technological curve is pulling away from the philosophy curve very rapidly and is about to leave it completely behind."