# Overcoming the Loebian obstacle using evidence logic

In this post I intend to:

*   Briefly explain the [Loebian obstacle](http://intelligence.org/files/TilingAgents.pdf) and it's relevance to AI (_feel free to skip it if you know what the Loebian obstacle is_).
*   Suggest a solution in the form a formal system which assigns _probabilities_ (more generally probability intervals) to mathematical sentences (and which admits a form of "Loebian" self-referential reasoning). The method is well-defined both for consistent and inconsistent axiomatic systems, the later being important in analysis of logical counterfactuals like in [UDT](http://wiki.lesswrong.com/wiki/Updateless_decision_theory).

Background
==========

Logic
-----

When can we consider a mathematical theorem to be established? The obvious answer is: when we proved it. Wait, proved it in what theory? Well, that's debatable. [ZFC](http://en.wikipedia.org/wiki/Zermelo%E2%80%93Fraenkel_set_theory) is popular choice for mathematicians, but how do we know it is consistent (let alone sound, i.e. that it only proves true sentences)? All those spooky infinite sets, how do you know it doesn't break somewhere along the line? There's lots of empirical evidence, but we can't _prove_ it, and it's _proofs_ we're interesting in, not mere evidence, right?

[Peano arithmetic](http://en.wikipedia.org/wiki/Peano_axioms) seems like a safer choice. After all, if the natural numbers don't make sense, what does? Let's go with that. Suppose we have a sentence **s** in the language of PA. If someone presents us with a proof **p** in PA, we believe **s** is true. Now consider the following situations: instead of giving you a proof of **s**, someone gave you a PA-proof **p**~1~ that **p** exists.After all, PA admits defining "PA-proof" in PA language. Common sense tells us that **p**~1~is a sufficient argument to believe **s**. Maybe, we can prove it _within PA_? That is, if we have a proof of "if a proof of **s** exists then **s**" and a proof of R(**s**)="a proof of **s** exists" then we just proved **s**. That's just [modus ponens](http://en.wikipedia.org/wiki/Modus_ponens). 

There are two problems with that.

First, there's no way to prove the sentence L:="_for all_ **s** if R(**s**) then **s**", since _it's not a PA-sentence at all_. The problem is that "for all **s**" references **s** as a natural number _encoding_ a sentence. On the other hand, "then **s**" references **s** as the _truth-value_ of the sentence. Maybe we can construct a PA-formula T(**s**) which means "the sentence encoded by the number **s** is true"? Nope, that would get us in trouble with the [liar paradox](http://en.wikipedia.org/wiki/Liar_paradox) (it would be possible to construct a sentence saying "this sentence is false").

Second, [Loeb's theorem](http://en.wikipedia.org/wiki/L%C3%B6b's_theorem) says that if we can prove L(**s**):="if R(**s**) exists then **s**" for a given **s**, then we can prove **s**. This is a problem since it means there can be no way to prove L(**s**) for all **s** in any sense, since it's unprovable for **s** which are unprovable. In other words, if you proved not-**s**, there is no way to conclude that "no proof of **s** exists".

What if we add an _inference rule_ Q to our logic allowing to go from R(**s**) to **s**? Let's call the new formal system PA~1~. **p**~1~ appended by a Q-step becomes an honest proof of **s** in PA~1~. Problem solved? Not really! Now someone can give you a proof of   
R~1~(**s**):="a PA~1~-proof of **s** exists". Back to square one! Wait a second, what if we add a new rule Q~1~ allowing to go from R~1~(**s**) to **s**? OK, but now we got R~2~(**s**):="a PA~2~-proof of **s** exists". Hmm, what if add an _infinite_ number of rules Q~k~? Fine, but now we got R~ω~(**s**):="a PA~ω~-proof of **s** exists". And so on, and so forth, the [recursive ordinals](http://en.wikipedia.org/wiki/Recursive_ordinal) are a plenty...

Bottom line, Loeb's theorem works for _any_ theory containing PA, so we're stuck.

AI
--

Suppose you're trying to build a self-modifying AGI called "Lucy". Lucy works by considering possible actions and looking for formal proofs that taking one of them will increase expected utility. In particular, it has self-modifying actions in its strategy space. A self-modifying action creates essentially a new agent: Lucy~2~. How can Lucy decide that becoming Lucy~2~ is a good idea? Well, a good step in this direction would be proving that Lucy~2 ~would only take actions that are "good". I.e., we would like Lucy to reason as follows "Lucy~2 ~uses the same formal system as I, so if she decides to take action **a**, it's because she has a proof **p** of the sentence **s**(**a**) that '**a** increases expected utility'. Since such a proof exits, **a** does increase expected utility, which is good news!" Problem: Lucy is using L in there, applied to _her own_ formal system! That cannot work! So, Lucy would have a hard time self-modifying in a way which doesn't make its formal system _weaker_. 

As another example where this poses a problem, suppose Lucy observes another agent called "Kurt". Lucy knows, by analyzing her sensory evidence, that Kurt proves theorems using the same formal system as Lucy. Suppose Lucy found out that Kurt proved theorem **s**, but she doesn't know how. We would like Lucy to be able to conclude **s** is, in fact, true (at least with the probability that her model of physical reality is correct). Alas, she cannot.

_See [MIRI's paper](http://intelligence.org/files/TilingAgents.pdf) for more discussion._

Evidence Logic
==============

[Here](/lw/imz/notes_on_logical_priors_from_the_miri_workshop/), cousin_it explains a method to assign probabilities to sentences in an _inconsistent_ theory T. It works as follows. Consider sentence **s**. Since T is inconsistent, there are T-proofs both of **s** and of not-**s**. Well, in a courtroom both sides are allowed to have arguments, why not try the same approach here? Let's _weight_ the proofs as a function of their length, analogically to weighting hypotheses in [Solomonoff induction](http://wiki.lesswrong.com/wiki/Solomonoff_induction). That is, suppose we have a prefix-free encoding of proofs as bit sequences. Then, it makes sense to consider a random bit sequence and ask whether it is a proof of something. Define the probability of **s** to be

P(**s**) := (probability of a random sequence to be a proof of **s**) / (probability of a random sequence to be a proof of **s** or not-**s**)

Nice, but it doesn't solve the Loebian obstacle yet.

I will now formulate an extension of this idea that allows assigning an _interval_ of probabilities \[P~min~(**s**), P~max~(**s**)\] to any sentence **s**. This interval is a sort of "[Knightian uncertainty](http://en.wikipedia.org/wiki/Knightian_uncertainty)". I have some speculations how to extract a single number from this interval in the general case, but even without that, I believe that P~min~(**s**) = P~max~(**s**) in many interesting cases.

First, the general setting:

*   With every sentence **s**, there are certain texts **v** which are considered to be "evidence relevant to **s**". These are divided into "negative" and "positive" evidence. We define sgn(**v**) := +1 for positive evidence, sgn(**v**) := -1 for negative evidence.
*   Each piece of evidence **v** is associated with the strength of the evidence str~**s**~(**v**) which is a number in \[0, 1\]
*   Each piece of evidence **v** is associated with an "energy" function e~**s**,**v**~ : \[0, 1\] -> \[0, 1\]. It is a continuous convex function.
*   The "total energy" associated with **s** is defined to b e**~s~** := ∑~**v**~ 2^-l(**v**) ^e~**s**,**v**~ where l(**v**) is the length of **v**.
*   Since **e~**s**,**v**~ **are continuous convex, so is e~s~. Hence it attains its minimum on a closed interval which is   
    \[P~min~(**s**), P~max~(**s**)\] by definition.

Now, the details:

*   A piece of evidence **v** for **s** is defined to be one of the following:
    *   a proof of **s**
        *   sgn(**v**) := +1
        *   str~**s**~(**v**) := 1
        *   **e~**s**,**v**~**(q) := (1 - q)^2^
    *   a proof of not-**s**
        *   sgn(**v**) := -1
        *   str~**s**~(**v**) := 1
        *   **e~**s**,**v**~**(q) := q^2^
    *   a piece of positive evidence for the sentence R~-+~(**s**, p) := "P~min~(**s**) >= p"
        *   sgn(**v**) := +1
        *   str~**s**~(**v**) := str~R-+(**s**, p)~(**v**) p
        *   **e~**s**,**v**~**(q) := 0 for q > p; str~R-+(**s**, p)~(**v**) (q - p)^2^ for q < p
    *   a piece of negative evidence for the sentence R~--~(**s**, p) := "P~min~(**s**) < p"
        *   sgn(**v**) := +1
        *   str~**s**~(**v**) := str~R--(**s**, p)~(**v**) p
        *   **e~**s**,**v**~**(q) := 0 for q > p; str~R--(**s**, p)~(**v**) (q - p)^2^ for q < p
    *   a piece of negative evidence for the sentence R~++~(**s**, p) := "P~max~(**s**) \> p"
        *   sgn(**v**) := -1
        *   str~**s**~(**v**) := str~R++(**s**, p)~(**v**) (1 - p)
        *   **e~**s**,**v**~**(q) := 0 for q < p; str~R-+(**s**, p)~(**v**) (q - p)^2^ for q > p
    *   a piece of positive evidence for the sentence R~+-~(**s**, p) := "P~max~(**s**) <= p"
        *   sgn(**v**) := -1
        *   str~**s**~(**v**) := str~R+-(**s**, p)~(**v**) (1 - p)
        *   **e~**s**,**v**~**(q) := 0 for q < p; str~R-+(**s**, p)~(**v**) (q - p)^2^ for q > p

_Technicality:_ I suggest that for our purposes, a "proof of **s**" is allowed to be a proof of sentence equivalent to **s** in 0-th order logic (e.g. not-not-**s**). This ensures that our probability intervals obey the properties we'd like them to obey wrt propositional calculus.

  

Now, consider again our self-modifying agent Lucy. Suppose she makes her decisions according to a system of evidence logic like above. She can now reason along the lines of "Lucy~2 ~uses the same formal system as I. If she decides to take action **a**, it's because she has strong evidence for the sentence **s**(**a**) that '**a** increases expected utility'. I just proved that there would be strong evidence for the expected utility increasing. Therefore, the expected utility would have a high value with high logical probability. But evidence for high logical probability of a sentence is evidence for the sentence itself. Therefore, I now have evidence that expected utility will increase!"

  

This analysis is very sketchy, but I think it lends hope that the system leads to the desired results.