Abstract: I apologize for all of the exclamation points in the previous paragraph; in my defense, if any result deserves exclamation points, it's Gödel's First Incompleteness Theorem! Also, there's a sleight of hand here in the demonstration that G never halts. We'll get further into that in Section 4.

An Introduction to Löb's Theorem in MIRI Research

Introduction This expository note is devoted to answering the following question: why do many MIRI research papers cite a 1955 theorem of Martin Löb  [12] , and indeed, why does MIRI focus so heavily on mathematical logic? The short answer is that this theorem illustrates the basic kind of self-reference involved when an algorithm considers its own output as part of the universe, and it is thus germane to many kinds of research involving self-modifying agents, especially when formal verification is involved or when we want to cleanly prove things in model problems. For a longer answer, well, welcome! I'll assume you have some background doing mathematical proofs and writing computer programs, but I won't assume any background in mathematical logic beyond knowing the usual logical operators, nor that you've even heard of Löb's Theorem before. To motivate the mathematical sections that follow, let's consider a toy problem. Say that we've designed Deep Thought 1.0, an AI that reasons about its possible actions and only takes actions that it can show to have good consequences on balance. One such action is designing a successor, Deep Thought 2.0, which has improved deductive abilities. But if Deep Thought 1.0 (hereafter called DT1) is to actually build Deep Thought 2.0 (DT2), DT1 must first conclude that building DT2 will have good consequences on balance. There's an immediate difficulty-the consequences of building DT2 include the actions that DT2 takes; but since DT2 has increased deductive powers, DT1 can't actually figure out what actions DT2 is going to take. Naively, it seems as if it should be enough for DT1 to know that DT2 has the same goals as DT1, that DT2's deductions are reliable, and that DT2 only takes actions that it deduces to have good consequences on balance. Unfortunately, the straightforward way of setting up such a model fails catastrophically on the innocent-sounding step "DT1 knows that DT2's deductions are reliable". If we try and model DT1 and DT2 as proving statements in two formal systems (one stronger than the other), then the only way that DT1 can make such a statement about DT2's reliability is if DT1 (and thus both) are in fact unreliable! This counterintuitive roadblock is best explained by reference to Löb's theorem, and so we turn to the background of that theorem. 2 Crash Course in Löb's Theorem
 Gödelian self-reference and quining programs Löb's Theorem makes use of the machinery of Kurt Gödel's incompleteness theorems  [10] , so we will discuss those first. Informally, Gödel found a way to import self-reference into a mathematical system that was simply trying to talk about properties of natural numbers, and then pointed out the odd consequences of a mathematical statement that asserts its own unprovability. One (anachronistic) way of stating Gödel's key insight is that you can use computer programs to search for proofs, and you can prove statements about computer programs. If we think about any conjecture in mathematics that can be stated in terms of arithmetic, you can write a rather simple program that loops over all possible strings, checks whether any of them is a valid proof of the conjecture, and halts if and only if it finds one. Thus, instead of trying to prove that conjecture directly, we could instead try to show (or prove) that the program halts. Now, this generally doesn't make things easier at all, since we're just restating the same problem in a more complicated way, and actually looping over all strings is basically the worst way to try and find proofs of theorems. However, this reformulation makes it more intuitive that we can embed self-reference in mathematics, because we can embed self-reference in computer code! The kind of self-reference I'm talking about is called a quine: a program that is able to reproduce its own source code without taking any inputs. One way to do this is for the program to include a string with a variable, and for it to replace that variable with the original string itself, such that the resulting expanded string is the entire source code of the program itself. Exercise. Write a quining program in your favorite language. No fair copying one from the Wikipedia article, and no fair making calls to external programs or the filesystem. In addition to quines that merely print their source code, programs can be made which perform arbitrary tasks using their own source code. Indeed, one prominent example of a quining program that does other tasks is Ken Thompson's famous C compiler Trojan horse  [15] , which uses the quining trick to avoid ever being visible outside of machine code. Thus we can have a program G which refers to itself in this way, and searches for proofs in arithmetic related to its own source code. In particular, we consider G which searches for a proof of the statement "G runs forever", and halts if and only if it succeeds at finding one. Now, we claim that G never finds a proof, and also that we can never prove that G runs forever! For if we could prove that G ran forever, then G would find that proof, and it would halt-and we could prove that it halted, and thereby produce a contradiction! On the other hand, if G actually halted after some finite number of steps, then the number at which it halts encodes a proof that G runs forever, which again leads to a contradiction!  1  It seems silly to ask whether we could prove that G halts, given that G actually runs forever. But it actually wouldn't be a contradiction if we asserted that G actually halted, so long as we didn't say anything about how long it took! It's only a claim like "G halts in fewer than a googolplex steps" that would be an actual contradiction. It turns out that we could add either "G never halts" or "G halts" (but not both!) as a new axiom of arithmetic, and not introduce a contradiction by doing so. We express this by saying that G is undecidable, and therefore we've proved Theorem 2.1 (Gödel's First Incompleteness Theorem) If the theory of arithmetic is consistent, then there exist undecidable statements in the theory of arithmetic. Remark. Gödel's Theorem is a bit different from the Halting Problem; the latter shows there's no one program X that can tell whether every program Y halts or not, but of course there may be particular programs X that tell you this fact for particular programs Y . But this is saying that there's no program X that can tell you definitively whether the program G halts or runs forever. So Gödel found that we can write undecidable statements about properties of natural numbers, and furthermore showed that adding new axioms won't fix it, since you can repeat the process with the new rule system for whether a statement is a theorem. (There's only one loophole, and it's not a very exciting one: if we use inconsistent axioms, then everything is a theorem, thanks to the Principle of Explosion, and therefore everything is decidable). Now we're not here to bury mathematical logic, but to use it. So from Gödel's Theorem we move on to...
 Löb's Theorem Löb's Theorem  [12]  takes the same framework as Gödel's First Incompleteness Theorem, and constructs programs of the following sort: • Let X be any logical statement (the sort of thing that could be proved or disproved). • Now let ProofSeeker(X) be the program that searches all possible proofs, and halts if and only if one of them is a valid proof of the statement X. • Finally, let L(X) be the statement "if ProofSeeker(X) halts, then X". Let's ponder for a moment whether L(X) should be true or not, using three different kinds of statements X. • If X is a provable statement, for example "2 + 2 = 4", then ProofSeeker(X) halts, and L(X) is "if [true thing], then [true thing]", which is a valid statement. • If X is disprovable, for example the statement "2 + 2 = 5", then ProofSeeker(X) does not halt, and L(X) is "if [false thing], then [false thing]", which is also a valid statement. • If X is neither provable nor disprovable, for example Gödel's statement G, then again ProofSeeker(X) does not halt, and L(X) is "if [false thing], then [maybe true thing]", which is also a true statement (remember your propositional calculus). So it seems like L(X) is always true. And it would certainly be handy if L(X) were provable for every X: for instance, you could use the second case above to show that mathematics proves no contradictions! Because if we could prove "if ProofSeeker("2 + 2 = 5") halts, then 2+2 = 5", then we would have the contrapositive "if 2+2 = 5, then ProofSeeker("2+2 = 5") never halts", and since we can prove 2 + 2 = 5, then we could prove that there is no contradictory proof of 2 + 2 = 5. Alas, that proof of mathematical consistency is too good to be true. As in the case of Gödel's Theorem, something being true is no guarantee of it being provable. And in fact, we find that L(X) is only provable in the first of the three cases above: Theorem 2.2 (Löb's Theorem) For all statements X, if L(X) is provable, then X is provable. (There is a neatly presented formal proof of this theorem at The Cartoon Guide to Löb's Theorem  [16] .) So in particular, if we could prove that mathematics would never prove a contradiction, then in fact mathematics would prove that contradiction! (Note that by the Principle of Explosion, this is indeed a possible state of affairs: if your mathematical axioms lead to a contradiction, then they can prove every statement in the language, including the statement that your mathematical axioms don't lead to a contradiction!) Thus the only systems that prove their own consistency are the inconsistent ones; incidentally, this is precisely Gödel's Second Incompleteness Theorem, although he originally proved it without Löb's Theorem. Remark. If you're interested in going deeper on these topics, Computability and Logic by Boolos, Burgess, and Jeffrey  [5]  is a good reference. Remark. There is also a version of Löb's Theorem for bounded proof searches, in the sense of "look through all formulas of length ≤ N and see if any of them are a proof of φ", and it controls the length of the proof of φ in terms of the length of the proof of φ → φ. In the limit of arbitrarily large computational resources, the phenomena we care about happen in the same way that they do in the case of infinite computation (i.e. access to halting oracles), and so we will generally discuss the latter case because the proofs are simpler and clearer.
 Direct Uses of Löb's Theorem in MIRI Research We can now exhibit three simple cases where Löb's Theorem comes up in MIRI research topics: one where it forms an unexpected obstacle to justifying self-modifications, one where it neatly enables mutual cooperation in a nonstandard Prisoner's Dilemma setup, and one where it frustrates a naive decision algorithm.
 "The Löbstacle" Let's return to the problem we discussed in the Introduction: Deep Thought 1.0 wants to verify that switching on its successor, Deep Thought 2.0, will have good consequences. Because DT2 has better deductive capacities, DT1 cannot deduce exactly what actions DT2 will take, but it does know that DT2 has the same utility function, and that it too will only take actions it deduces to be good. Intuitively, this should be enough for DT1 to "trust" DT2, to say that whatever DT2 does, DT2 must have deduced that to be good, and therefore it must actually be good. But that last clause is analogous to the Löbian statement L(X): "if the action is deduced to be good, then it must actually be good"! And therefore DT1 cannot generally prove that clause (unless its reasoning is inconsistent), since then it could prove that every action is good. This is not just an analogy; when we consider a simple mathematical model of a selfmodifying agent that uses proofs in some consistent formal system to justify its actions, that agent has precisely this problem. In models that presume infinite computing power and represent different deductive powers with different axiom systems, a simple agent with a utility function will only create successors whose formal systems are strictly weaker than its own, since only those are fully trusted by the current system. There are a number of partial and potential remedies to this "Löbstacle", some of them more appealing than others. For more details, see the MIRI preprints Tiling Agents for Self-Modifying AI, and the Löbian Obstacle  [18]  and Problems of self-reference in self-improving space-time embedded intelligence  [9] .
 Löbian cooperation The second topic concerns a variation on the usual Prisoner's Dilemma. Rather than playing directly, you write an algorithm to play against other algorithms on your behalf, as in Robert Axelrod's famous algorithmic Prisoner's Dilemma tournament  [1] . However, instead of that setup, in which the algorithms play iterated games against one another, in this case your algorithm and theirs get to read the opponent's source code, calculate for as long as they like, and then play only once. Using quining (recall Section 2.1), one can write a program that cooperates if and only if the opponent's source code is identical to its own. However, there are many ways to write such programs, and different ones will not cooperate with each other; this is therefore a fragile form of mutual cooperation. There's a different algorithm which (at the cost of using lots of computation) avoids that fragility. We call this agent FairBot, and it operates as follows: • Both FairBot and its opponent X are functions of one argument, which take the opponent's source code and output either C or D. • When FairBot() is called on the input X, it searches through all proofs of length ≤ N to see whether any are valid proofs of the statement "X(FairBot)= C". If yes, then it outputs C; if no, then it outputs D. (Here N is a parameter that doesn't depend on X; we'll think of it as some extremely large number. The only reason we have that parameter at all is so that our algorithm does in fact always return an output in finite time.) Some things are clear from the definition of FairBot. One is that it is capable of cooperation: if X is a simple algorithm that always returns C, then FairBot will return C as well. Another is that (unless arithmetic is inconsistent) FairBot will never be exploited (cooperate while its opponent defects). What is less immediately clear is the outcome when FairBot plays against itself. Intuitively, it seems like both mutual cooperation and mutual defection are stable fixed points of the situation. However, a Löbian statement breaks the deadlock in favor of cooperation! To see this, ignore for the moment the parameter N . Then if we consider the statement L("FairBot(FairBot)= C"), we find that it follows directly from the code of FairBot. For if there is a proof that FairBot(FairBot)= C, then FairBot will discover that proof and output C. But then, by Löb's Theorem, there must be an actual proof of the statement "FairBot(FairBot)= C"! 2 As mentioned in Section 2.2, there is a quantitatively bounded version of Löb's Theorem such that this works even with the parameter N , for sufficiently large values of N . Furthermore, this form of cooperation is much more robust: two such programs don't need to figure out that they are functionally identical, they only need to search for proofs about one another's output. Moreover, even algorithms that are not functionally identical to FairBot can achieve mutual cooperation with it. The FairBot algorithm is due to Vladimir Slepnev; for more on Löbian cooperation, see the MIRI paper Program Equilibrium in the Prisoner's Dilemma via Löb's Theorem  [11] . We will return to this topic in Section 7.1 when we have developed a few more tools.
 Spurious counterfactuals The third topic concerns an unexpected issue that comes up when the agent is a part of the same universe that the agent is proving theorems about. The setting is a pure computational universe U () which takes no input, and which just runs and eventually outputs a number. Within that universe there is an agent A() which knows the code for U (), which does some computations as part of the natural order of the universe and then outputs something. This may seem like a rather extreme setup, and it is: we want to see what we can say about agents that know the source code of the universe and wield arbitrarily large computational resources, because it is often easier to show what happens with such agents than what happens with more realistic, messy, and bounded agents. Moreover, any obstacles we discover in the ideal setting are likely to correspond in the real setting to obstacles that can't be overcome merely by adding more knowledge and computing power to the system. Let's first consider a really simple sort of universe (Algorithm 1), where A() must try to figure out whether to open door number 1, door number 2, or neither. Algorithm 1: U () if A() = 1 then return 100 else if A() = 2 then return 1 else return −∞ end This requires us to specify the function A(). Now of course we could consider a function A() hard-coded to return 1, but that's not especially interesting. Instead, we'll write one (Algorithm 2) which tries to deduce which door (if either) to open, by looking through proofs of arithmetical statements. It seems obvious that there should be proofs of A() = 1 → U () = 100 and A() = 2 → U () = 1 whose Gödel numbers are smaller than a googolplex, and therefore A should correctly prove those statements and choose Door 1. Algorithm 2: A() Door1 ← −∞; Door2 ← −∞; n = 1; while Door1 + Door2 = −∞ and n < 10 10 100 do if n encodes a valid proof of "A() = 1 → U () = c" for some c then Door1 ← c; if n encodes a valid proof of "A() = 2 → U () = c" for some c then Door2 ← c; n+ = 1; end if Door1 ≥ Door2 then return 1 else return 2 end But there's something very strange that can happen instead: A could prove A() = 2 → U () = 1, and then prove a "spurious counterfactual" such as A() = 1 → U () = −1, which then makes A() stop looking early and select door number 2! This wouldn't be a contradiction, because if A() = 2, then A() = 1 → U () = −1 is formally true  3  . But why should it actually happen? It's clearest to see in the case where I can make one innocent-seeming change to A(): I ask that before it starts searching through all n < 10 10 100 to see if they encode proofs, I want it to check two particular proofs. The first is the simple proof that A() = 2 → U () = 1, and the second is a statement φ. Now if φ really is a proof of the spurious counterfactual A() = 1 → U () = −1, it's clear from the rest of the source code of A() that it will break the while loop and choose Door 2. In fact, we can formalize that reasoning: there's a proof that if φ is a proof of A() = 1 → U () = −1, then in fact it's true that A() = 2 and thus A() = 1 → U () = −1. But this is almost a Löbian statement, just with the addition of a specific φ rather than an existential quantifier! By the machinery of quining, we can find a φ that serves our purposes, so it works just as before: since the Löbian statement is provable, so is the conclusion A() = 1 → U () = −1, by means of that particular φ. Even without that malicious change, these kinds of agents and universes are susceptible to spurious counterfactuals. In Section 7.2, we'll consider a similar setup, but with a more careful ordering of which proofs it pays attention to, and that agent can be shown to make the correct decisions (given enough deductive power). The idea was originally due to Benja Fallenstein; Tsvi Benson-Tilson's paper UDT with Known Search Order  [3]  discusses spurious counterfactuals and a different response to them.
 Crash Course in Model Theory For the later topics, we'll need some additional background in mathematical logic. Let's start out by just talking formally about the natural numbers with addition and multiplication.
 Axioms and theories We begin with a language of symbols, and we don't start out assuming anything about what they mean. We'll take some symbols of the propositional calculus, set theory, arithmetic, and three special symbols: {(, ), ∧, ∨, ¬, →, ↔, ∈, ∀, ∃, =, +, •, O, S, N} Here we'll think of N as standing in for the set of natural numbers, O as standing in for zero, and S as standing in for the successor (so that SO represents 1, SSO represents 2, etc.). That way we don't need rules for dealing with infinitely many number symbols, or rules for dealing with digits. We'll also add an infinite family of symbols like x and y for variables (so long as they're quantified over with ∀ or ∃). Finally, we'll use letters like φ to stand in for a formula within the language, but note that φ is not itself part of the language, so we can't say things like ∀φ. We'll import the standard rules of the propositional calculus and set theory as applied to the appropriate symbols (for instance, whenever φ ∧ ψ is a theorem, then both φ and ψ are theorems), but we won't yet assume any properties of +, •, O, S, or N. That's because we'll want to pick our own axioms about arithmetic on the natural numbers! This creates a theory, the set of all theorems that follow from the axioms; different sets of axioms can give rise to different theories. So let's take some true statements about the natural numbers, make them axioms of our logical system, and see what kind of theory we get. (Parenthetical comments are included for our intuition, but are not actually a part of the theory.) A theory can contain contradictions, of course: some theorem φ such that ¬φ is also a theorem. So how can we tell whether the theory built on these axioms contains any contradictions? We could try proving things and see if we ever find a contradiction, but as there are infinitely many theorems, we couldn't be sure that we just hadn't found a contradiction yet. 1. O ∈ N (zero is a natural number) 2. ∀x ∈ N Sx ∈ N (if x is a natural number, so is its successor) 3. ∀x ∈ N∀y ∈ N (x + y ∈ N) ∧ (x • y ∈ N) (the But there's another thing we could do: exhibit an example of a set for which all of the axioms hold! That serves as conclusive evidence that the theory does not contain contradictions, because φ is either actually true or actually false for that specific set, and so at most one of φ and ¬φ can actually be a theorem. We need to do a little more than exhibit an object: we need to have a correspondence between the symbols in the language and the parts of the object. So for instance, we identify the symbol O with the number 0, SO with 1, and so on 4 ; we identify N with the full set of natural numbers N; and we identify addition and multiplication with the usual operations on N. We call such an object a model, and such a correspondence an interpretation of the theory. However, the same theory can have many models, some of them not at all what you were thinking of when you made the axioms...
 Alternative and nonstandard models For instance, an alternate model of the theory above is the set with a single element "Alice"; we identify O with Alice, and then SO with Alice as well (so SO = O), and so on; we declare that Alice+Alice=Alice•Alice=Alice; and then we identify N with the set {Alice}. Note that all of the axioms above are true of this alternate model, even if their intuitive meanings aren't! Fine, let's patch this. We can't add an axiom directly saying it's not this particular model, since Alice isn't part of the language of the theory itself. But we can add an axiom saying SO = O, which is true of the model of the natural numbers we want, but not true of the Alice model. Well, we excluded that alternate model, but there are still others we haven't excluded. In particular, we can consider arithmetic mod n as a model for any n! In order to exclude these, we could add axioms saying SSO = O, SSSO = O, etc, or we could be more economical and just use the axiom ∀x ∈ N Sx = O. (The resulting theory is known as Robinson arithmetic, and is interesting in its own right.) Have we eliminated all alternate models from this theory? Hardly! We're just getting started. Consider the model where N refers to the natural numbers combined with one additional entity, "Bob"; we identify S(Bob)=Bob, Bob•0 = 0, Bob•x =Bob for all x = 0, and Bob+x =Bob for all x. Then we again see that this satisfies all of our axioms so far. We could patch this with the axiom ∀x ∈ N Sx = x, but again we can build more and more complicated alternate models. (One particularly interesting model for Robinson arithmetic consists of all polynomials with integer coefficients whose leading term is nonnegative.) It turns out that what we're missing from the natural numbers is the principle of induction. (Indeed, with Robinson arithmetic you can't even prove that addition is commutative!) In order to express induction in the language (which doesn't have variables for properties, only for numbers), we must resort to an infinite family of new axioms: for every logical formula φ(x) (with no free variables except for x), we want to add the axiom (φ(0) ∧ (∀x ∈ N φ(x) → φ(Sx))) → ∀y ∈ N φ(y).  5  With all of that completed, we've got the axioms for the theory of Peano Arithmetic. Now are we done with alternative models? Of course not! Remember how Gödel's self-referential formula G was undecidable? There are models of Peano arithmetic where G holds, and other models where G fails to hold. The models where G holds include our standard intuitive model of the natural numbers, since as we discussed before, G definitely does not halt at any finite number. But then, what kind of model of Peano Arithmetic could G fail to halt on? It may help to think of the model of N with Bob. If we do Gödel's construction in Robinson Arithmetic, there will be no contradiction if we assert that the special extra number Bob in fact satisfies the property that the formula G is checking (the property which represents "this is a valid proof that no number is a valid proof of G"). This is okay, because Bob doesn't actually have a representation in terms of lots of S's and an O, so this assertion can't be used to actually construct a finite sequence of logical formulas that comprise a proof that no number is a valid proof of G. This loophole suffices to maintain consistency for the new system! We're using Peano Arithmetic, though, so our nonstandard models will be weirder than the natural numbers plus Bob. The key to understanding these is that G never halts at any finite number, but we can't actually define in our formal language what "finite" means. The nonstandard models of Peano Arithmetic are those which have all the usual numbers but also lots of extra numbers that are "too large" to ever be written as lots of S's followed by an O, but which nonetheless are swept along in any inductive statement. (I still don't know quite how to visualize this model, but you can rigorously construct it using set theory.) And again, a "number" that can't actually be written in S's and an O can be asserted to represent a valid proof, but that proof can't be extracted from the number and used against itself, so it all works out consistently. Remark. Since nonstandard models of arithmetic contain numbers larger than any of the usual natural numbers, they can be used in other areas of mathematics. If you take the reciprocal of one of those numbers, you get an infinitesimal; thus you can use these to define the concepts of calculus without using limits! This is known as nonstandard analysis. And it doesn't end there! As I mentioned before, we can add either G or ¬G as an extra axiom of Peano Arithmetic, and since adding a new axiom changes the rules for what counts as a valid proof, we can redo Gödel's construction in our new formal system, and have statements that are undecidable given the new rules. Or we can directly talk about the consistency of Peano Arithmetic (by making a statement that asserts that there is no proof in PA that 0 = 1), and the consistency of the system with all the rules of Peano Arithmetic plus the axiom that PA is consistent, and so on. (Ponder for a moment the formal system which has all the axioms of PA, plus the axiom that PA is consistent, plus the axiom that "the system which has all the axioms of PA, plus the axiom that PA is consistent" is inconsistent. As it turns out, this is a perfectly consistent system 6 !) We'll work with a particular hierarchy of such formal systems in Section 6, but before then, we'll discuss another MIRI paper that relates to theories and models. Remark. Again, Computability and Logic by Boolos, Burgess, and Jeffrey  [5]  covers all of this material at a much deeper level.
 Uses of Model Theory in MIRI Research
 Reflection in probabilistic logic The topic of the first MIRI mathematical workshop paper, Definability of Truth in Probabilistic Logic  [7] , isn't directly entwined with Löb's Theorem. But we've just covered the necessary background to understand it, and we'll need another ingredient before getting to the other MIRI papers in these notes, so we might as well detour here! This paper concerns probabilistic logic: assigning probabilities in [0, 1] to logical statements, rather than just assigning them the labels "provable", "disprovable 7 ", and "undecidable". We'll want to make sure we assign the value 1 to all provable statements and 0 to all disprovable statements, and that we assign values to the undecidable statements in a consistent way (so if φ and ψ are undecidable but φ → ψ is provable, then the probability we assign to φ should be less than or equal to the probability we assign to ψ). Why is this an important topic for MIRI to study? Probabilistic logic is an interesting, clean, and rich model for Bayesian reasoning and for bounded inference. Consider an AI that cannot deduce with certainty whether P = N P ; knowing it one way or the other should certainly change its actions (for instance, its cryptographic precautions against other agents), and so the most sensible way to proceed seems to be assigning it a tentative probability based on the available evidence, using that probability to do consequentialist calculations, and updating it in the light of newer evidence. So, in order to understand bounded reasoning, we might start with the nice and symmetrical (if unrealistic) case of a coherent probability assignment over all logical statements. So let's consider a Gödelian obstacle that bedevils logic, and see if it looks different within probabilistic logic. That obstacle is Tarski's undefinability of truth  [14] . OK, so we've previously established that some statements are undecidable, and in fact, undecidable statements hold in some models of a theory but not others. We might want to endorse some particular model as the "true" one (for instance, our standard model of the natural numbers, without all of those weird nonstandard numbers), and say that logical statements are true if they hold in that model and false if they don't. This truth predicate exists outside the language, and so the logical statements can't talk about the truth predicate, only about weaker notions like provability. All that is perfectly fine, thus far. The trouble comes when we try to construct a language that contains its own truth predicate T , either by finding a formula for it using the previously existing symbols, or by directly adding a new symbol with some rules of usage, such that for all φ, it's true that T (φ) ↔ φ. Either approach is doomed, by an argument that should seem familiar by now: there exists  7  By this, we mean statements whose negation is provable, not statements which cannot be proven. a sentence X that refers to itself by quining, so that it's provable that X ↔ T (¬X). And now, clearly, neither T (X) nor T (¬X) can hold.
 All right, so what changes when you incorporate probabilities? Let's introduce a symbol P which represents the probability of a logical statement. We'd like P to satisfy some mathematical coherence properties, like P (φ ∧ ψ) ≤ P (φ), and we'd also like P to be able to discuss itself. Now what happens when we consider the statement Y constructed such that Y ↔ (P (Y ) < 0.5)? Does this create a contradiction? It depends on what kind of statements P is allowed to make about itself! Namely, if P isn't allowed to make exact statements about its own values, but only arbitrarily precise approximations, then everything can work out consistently. Namely, in the place of the failed reflection principle T (T (φ) ↔ φ), we take the reflection principle ∀φ∀a, b ∈ Q (a < P (φ) < b) → (P (a < P (φ) < b) = 1) . (5.1) In the paper, they prove that with this reflection principle and the natural coherence conditions, there indeed is a probability valuation that works for the values of P . How does this avoid a collision with the statement Y above? It turns out that the true value of P (Y ) is 0.5, but P isn't able to know whether P (Y ) is exactly 0.5, or slightly above it, or slightly below it, and so P (P (Y ) < 0.5) is 0.5 as well, and so on. For any rational > 0, you can prove with certainty that P (0.5 − < P (Y ) < 0.5 + ) = 1, but you can't get from this anything sharp enough to produce a contradiction; in particular, you can't prove the generalization that ∀ > 0 P (0.5 − < P (Y ) < 0.5 + ) = 1. Remark. Nonstandard arithmetic isn't needed for this result, but it can shed some light on the fact that P (0.5 − < P (Y ) < 0.5 + ) = 1 for any rational . Since some models of N include nonstandard natural numbers, P can't rule out the possibility that reciprocals of nonstandard natural numbers (infinitesimals) exist, and so P (Y ) may as well be imagined as 0.5 plus or minus an infinitesimal...
 Remark. Since the result in the Definability of Truth paper is nonconstructive, it's worth noting that there's a followup paper by Christiano on computationally tractable approximations of probabilistic logic: Non-Omniscience, Probabilistic Inference, and Metamathematics  [6] .
 Crash Course in Gödel-Löb Modal Logic The final topics in this survey require some topics outside the usual first course in mathematical logic, namely the Gödel-Löb modal logic of provability. This modal logic captures exactly the parts of Peano Arithmetic that relate to self-reference, and by leaving out the rest, it has a pleasingly simple structure and plenty of powerful tools for analysis. Thus it's a great setting for model problems in decision theory, as we shall see.
 The modal logic of provability You may have heard of modal logic before, in the context of philosophy: Aristotle defined ways of arguing about which facts are necessary or possible rather than merely true or false. By the twentieth century, people had formalized various kinds of modal logic, where you take the usual language of propositional logic and add the symbols and , as well as new axioms and rules incorporating them. φ is usually interpreted as the statement "it is necessary that φ", and φ as "it is possible that φ". (We actually only need , since in all modal logics of interest to us, φ ↔ ¬ ¬φ.) Philosophers have tried out various sets of axioms, mostly in order to write intimidatingly technical-looking arguments for their preferred metaphysical conclusions 8 , but we're more interested in a particular modal logic that constitutes a perfectly rigorous-and quite useful-reflection of Löbian phenomena in Peano Arithmetic and other such systems. This is the Gödel-Löb modal logic (alternately, 'the modal logic of provability' or simply 'provability logic'), which we will denote as GL. We can construct GL in two different ways. First, we can let φ represent the formula of Peano Arithmetic that asserts that φ is provable in Peano Arithmetic, and then restrict ourselves to the formulas and proofs that use only , the logical operators, and Boolean variables (including the constants for true and ⊥ for false, but no numbers, arithmetical operations, or quantifiers). Or, equivalently, we can start with those elements of the language, and then add the following axioms and rules: • All tautologies of the propositional calculus are axioms of GL, including tautologies where an expression including has been substituted for a variable (for instance, p ∨ ¬ p is an axiom). • Modus Ponens Rule: if the expressions A and A → B are theorems, then so is B. • Distribution Axioms: for any expressions A and B in the language, we have the axiom (A → B) → ( A → B). • Generalization Rule: if the expression A is a theorem, then so is A. • Gödel-Löb Axioms: for any expression A, we have the axiom ( A → A) → A. It's a nontrivial theorem that these approaches give us the same modal logic! So GL really does capture the self-referential parts of Peano Arithmetic, while leaving aside the arithmetical parts; for instance, the statement that Peano Arithmetic is consistent is simply ¬ ⊥. Exercise. Show, using the rules of GL, that ⊥ ↔ p; recall that p = ¬ ¬p. Informally, that is, no statement can be proven consistent with Peano Arithmetic unless Peano Arithmetic is inconsistent. [Hint: First prove that ⊥ → p, then that p → ⊥. For the latter, it will help to rewrite ¬ ⊥ as ⊥ → ⊥, then consider the Löbian axiom ( ⊥ → ⊥) → ⊥.] While every specific deduction we would want to make in GL can be made via formal manipulations in the system, this is a computationally intractable way to handle deduction. (Deducing whether a given modal statement is a theorem of GL is NP-complete in general, just as it is in Peano Arithmetic.) However, there are some special cases where there are efficient algorithms for deducing provability in GL, and these happen to include cases of direct interest to decision theory...
 Fixed points of modal statements In Section 3.2, we figured out what happens when two different programs try to prove theorems about one another. In that case, the programs were simple enough that we could directly see where to apply Löb's Theorem, but in general we'd like to be able to handle more complicated phenomena. Fortunately, modal logic comes equipped with a powerful tool for doing precisely that: the theory of fixed points for modal sentences. In Peano Arithmetic, the Gödel statement G refers to itself via quining, in order to claim that it cannot be proved in PA; in GL, G corresponds to the formula p ↔ ¬ p. Similarly, we can have all sorts of formulas that refer to themselves and each other by quining, and these are represented by formulas p ↔ φ(p, q 1 , . . . , q k ) that are modalized in p: every occurrence of p in φ happens within the scope of some . (After all, quining can only refer to the provability of the statement's own Gödel number, not directly to the statement itself!) As it happens, whenever you have this setup in GL, with p equivalent to a formula that is modalized in p, then p is equivalent to some other formula which doesn't use p at all! For example, the Gödel statement is equivalent to the inconsistency of arithmetic: (p ↔ ¬p) ↔ (p ↔ ⊥). In the case of one-variable formulas p ↔ φ(p) with φ(p) modalized in p, there's a neat tool that helps us calculate these fixed points (where the result will involve no variables at all, just logical connectives, , , and ⊥). First, for many modal logics including GL, there's a corresponding class of sets and relations (called Kripke frames) such that a formula is provable in the modal logic if and only if a corresponding property holds for every Kripke frame in that class. And secondly, in the special case of sentences without variables in GL, we can reduce to checking a linear hierarchy corresponding to a world in which ⊥ holds, a world in which ⊥ ∧ ¬ ⊥ holds, and so on up the ladder of n+1 ⊥ ∧ ¬ n ⊥ for each n 9 . If you'll grant me that result (known as an World p p ¬p (p → ¬p) p → (p → ¬p) p p → ¬p ⊥ 1 1 1 1 1 1 1 ⊥ ∧ ¬ ⊥ 1 1 0 1 1 1 0 3 ⊥ ∧ ¬ 2 ⊥ ? ? ? ? ? ? ? Continuing with the second row, we find that p flips to false: World p p ¬p (p → ¬p) p → (p → ¬p) p p → ¬p ⊥ 1 1 1 1 1 1 1 ⊥ ∧ ¬ ⊥ 1 1 0 1 1 1 0 3 ⊥ ∧ ¬ 2 ⊥ 1 1 0 0 0 0 1 4 ⊥ ∧ ¬ 3 ⊥ ? ? ? ? ? ? ? And with the third row, when considering ¬p, it is important to remember that the lemma above requires A to be true in all previous rows, so even though ¬p is now true, ¬p remains false. We will fast-forward through the next few rows: World p p ¬p (p → ¬p) p → (p → ¬p) p p → ¬p ⊥ 1 1 1 1 1 1 1 ⊥ ∧ ¬ ⊥ 1 1 0 1 1 1 0 3 ⊥ ∧ ¬ 2 ⊥ 1 1 0 0 0 0 1 4 ⊥ ∧ ¬ 3 ⊥ 0 1 0 0 0 0 1 5 ⊥ ∧ ¬ 4 ⊥ 0 0 0 0 1 1 0 6 ⊥ ∧ ¬ 5 ⊥ 0 0 0 0 1 1 0 And here it stabilizes: every further row is identical  11  . And now that we've found the truth values of p, if we find a constant formula with the same truth values, then by the adequacy result I mentioned in passing, there must be a proof of equivalence in GL! We can get such a formula by taking Boolean combinations of the formulas that define the worlds (since the truth table stabilizes eventually, one only needs a finite combination of these); in the present case, that formula is ¬( 3 ⊥∧¬ 2 ⊥)∧¬( 4 ⊥∧¬ 3 ⊥), or equivalently 4 ⊥ → 2 ⊥. So in the end, we have the fixed point (p ↔ ( p → (p → ¬p))) ↔ (p ↔ ( 4 ⊥ → 2 ⊥)), and our algorithm for finding it is polynomial in the complexity of the statement (in fact, quadratic: we add columns for sub-expressions, and there's a linear bound for how many rows we need to calculate). Remark. We demonstrated the fixed-point algorithm only for one-variable expressions, but you can also take a statement of the form p ↔ φ(p, q 1 , . . . , q k ), where φ is modalized in p (not necessarily in any of the q i ), and find an equivalent expression p ↔ ψ(q 1 , . . . , q k ). The machinery for this is more difficult, as one needs to deal with more complicated Kripke frames (in the general case of GL, a Kripke frame is a set equipped with a transitive, irreflexive, well-founded relation), but it can be mechanized as well. I swept quite a lot under the rug in this section; if you want to get to know the topic on a much sounder basis, read The Logic of Provability  [4]  by Boolos. (This is an advanced textbook, so wait until after your first course in mathematical logic!) 7 Uses of Gödel-Löb Modal Logic in MIRI Research
 Modal Combat in the Prisoner's Dilemma We're now ready to handle the preprint Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic  [2] , which encompasses the results from Section 3.2, and studies "modal agents" represented by expressions in modal logic. Recall the idea of Löbian cooperation from Section 3.2. One embarrassing thing about FairBot is that it doesn't check whether its potential cooperation would actually make any difference. It happily cooperates with CooperateBot, the constant strategy that simply returns C for every opponent. That's a bit like cooperating against a rock because the rock "cooperated" with you. Let's consider the following alternative: first we define DefectBot as the constant strategy that simply returns D for every opponent; then we define PrudentBot as the algorithm that, given the source code of an opponent X, searches in Peano Arithmetic for a proof of the statement "X cooperates with PrudentBot, and if Peano Arithmetic is consistent  12  , then furthermore X defects against DefectBot", and cooperates if and only if it finds such a proof  13  . Now it's more difficult to figure out what happens when FairBot plays the open-sourcecode Prisoner's Dilemma against PrudentBot, if we view these as proof searches in Peano Arithmetic. But if we assume infinite computational power (i.e. the ability to consult a halting oracle about proof searches in Peano Arithmetic), then they can be written out as simple statements in modal logic. Let p = "FairBot cooperates with PrudentBot", q = "PrudentBot cooperates with FairBot", r = "FairBot cooperates with DefectBot", and  12  Why does PrudentBot include the consistency requirement? Because certain actions that depend on agents require the consistency of Peano Arithmetic; for instance, FairBot does defect against DefectBot, but knowing this requires knowing that Peano Arithmetic is consistent, since otherwise FairBot might find a proof that DefectBot cooperates, using the Principle of Explosion. 13 Also, why not directly check whether your cooperation causes the opponent's cooperation, by defining MagicBot which cooperates if and only if it finds a proof of the statement "X cooperates with MagicBot if and only if MagicBot cooperates with X"? Because this makes it susceptible to spurious counterfactuals as in Section 3.3, and in fact MagicBot is equivalent to FairBot when they are rendered as modal statements. s = "DefectBot cooperates with FairBot". Then we have p ↔ q q ↔ (p ∧ (¬ ⊥ → ¬r)) r ↔ s s ↔ ⊥. If we look for the fixed point of this modal system, we can use the truth table formalism of the fixed-point theorem (we do all of these simultaneously, since every variable is equivalent to a fully modalized expression) to show that p ↔ , q ↔ , r ↔ ⊥, and of course s ↔ ⊥.
 Exercise. Show this formally. So what actually happens when you run the bounded versions of these agents against one another (with sufficiently high bounds on how deep they search for proofs before giving up)? Again, the bounded analogue of Löb's Theorem helps any valid proof in GL to go through if the proof bounds are sufficiently large. And since we really don't expect there to be a contradiction in Peano Arithmetic or any of the systems built atop it by addition of the axioms ¬ n ⊥ 14 , in the bounded case we should expect that every search for n ⊥ comes up empty. Thus p and q are true, and r and s are false. (This is equivalent, in the infinite computation case, to specifying that we care about what actually holds in the standard model of the natural numbers.) Therefore we can use these tools to analyze interactions between "modal agents" represented by families of modal statements like FairBot and PrudentBot, and moreover, we can figure out what actually happens using an efficient algorithm rather than the huge bruteforce proof search! It's a little tricky to define these modal agents in one fell swoop, so let's start with the simplest case: agents that don't check their opponent's action against any third parties, but only against themselves. (We will call these "modal agents of rank 0".) A rank 0 modal agent is represented by a formula p ↔ φ(p, q), where φ is modalized in both p and q (we don't allow modal agents to run the other agent, only to prove things about them; this avoids the infinite regress of two agents simulating each other forever, waiting for the other to 'make the first move'). By the modal fixed-point theorem, this is equivalent in GL to p ↔ φ(q) for some φ modalized in q, so we will define rank 0 modal agents using those formulas. Then if we have two modal agents of rank 0, represented by p ↔ φ(q) and q ↔ ψ(p), the simultaneous fixed point of these formulas gives us constant sentences (combinations of n ⊥) from which we can deduce their actions. Now let's allow calls to third parties, like PrudentBot's call to the opponent's action against DefectBot. We'll insist that it bottom out in a finite number of statements: for this, it suffices that every modal agent can only predict its opponent's action against itself and against modal agents of strictly lower rank. That is, Definition. X = φ, Z 1 , . . . , Z k is a modal agent of rank n ≥ 0 if φ is a fully modalized formula of k + 1 variables p, r 1 , . . . , r k , and each Z i is a modal agent of rank < n. Given two modal agents X = φ X , Z 1 , . . . , Z k and Y = ψ Y , W 1 , . . . , W l , we then construct the family of formulas given by recursively applying the modal agents to each other and applying the third parties to the originals: p X ↔ φ X (q Y , r Z 1 , . . . , r Z k ) q y ↔ ψ Y (p X , s W 1 , . . . , s W l ) r Z i ↔ φ Z i (t (Y,Z i ) , u Z i 1 , . . . , u Z i m i ) t (Y,Z i ) ↔ ψ Y (r Z i , v (W 1 ,Z i ) , . . . , v (W l ,Z i ) ) and so on until it bottoms out. (Please excuse the proliferation of indices and the imperfect rigor, and take another look at the example above of PrudentBot versus FairBot, where FairBot= p , PrudentBot= (p ∧ (¬ ⊥ → ¬r)), DefectBot , and DefectBot= ⊥ .) Thus any two modal agents can be played against each other, and the result of that "modal combat  15  " computed by an effective algorithm. So there's an actual Haskell program, written by Mihaly Barasz and Marcello Herreshoff, that checks the result of the modal combat, and that program helped the authors to find many other patterns about modal agents (including the discovery of PrudentBot, which is never exploited, achieves mutual cooperation with itself and with FairBot, and which correctly defects against CooperateBot). Among the other results in that paper: • Third parties are necessary to get cooperation from FairBot while defecting against CooperateBot; no modal agent of rank 0 can achieve both. • No modal agent globally dominates another modal agent, since it is possible to write another modal agent which punishes or rewards agents for their other decisions. (For example, consider TrollBot, which cooperates with its opponent if and only if it proves that its opponent cooperates with DefectBot.) • In another obstacle to strong definitions of optimality, consider WaitFairBot N defined by ¬ N +1 ⊥∧ (¬ N ⊥ → p) . Then any modal agent that defects against DefectBot will fail to elicit mutual cooperation from WaitFairBot N for sufficiently large N . • Finally, there are some modal agents that can be exploited by non-modal agents but not by any modal agent. Consider MimicFairBot, which cooperates with X if and only if Peano Arithmetic proves that X cooperates with FairBot. (That is, MimicFairBot= r, FairBot .) It is possible for an algorithm to exploit MimicFairBot (consider the non-modal agent that cooperates if and only if the opponent's source code is identical to the source code of FairBot), but every modal agent treats MimicFairBot and FairBot identically. The field of modal combat is new and still wide-open, and I expect to see many more results soon! Furthermore, it inspired an unexpected development in decision theory for one-player games as well...
 Modal Decision Theory Recall the setup in Section 3.3: we have a universe U () which contains an agent A(). The universe runs some computation and returns some outcome; as a part of that computation, the agent runs some computation and returns some action. Everything is deterministic, but we can think of a decision theory as a condition on the allowable form of A(); if you have a certain decision theory, then you find yourself only in certain universes and you perform some specific algorithm in those universes. In order to define what we mean by a decision theory, then, we need to specify where in the universe the agent goes. So we start with some utility values over outcomes, and a universe template U (•) with a spot for an agent; then our decision theory specifies an agent A() that goes there, such that we now have the universe U A () = U (A) which computes a certain outcome; and we judge the decision theory based on how highly it values that outcome. Causal decision theory (CDT) and evidential decision theory (EDT) are two famous philosophical examples of decision theory. Both of these require the universe template to supply the agent with some additional information: in the case of CDT, a causal graph of the universe, and in the case of EDT, the actions and outcomes of other agents in the same universe template. Because CDT and EDT can be shown to make bad decisions on some problems even when given the right info, several people have proposed alternatives, including timeless decision theory  [17]  (TDT) and updateless decision theory (UDT). (For more on the philosophical side of things, see Toward Idealized Decision Theory  [13] .) While working out models of UDT, in particular, Vladimir Slepnev and Benja Fallenstein found a nice formalization of a simple decision theory using modal logic, one that they could even prove optimal on a certain class of modally defined problems! Consider a universe template U (•) with a finite set of possible actions A and a finite set of possible outcomes O. We have some preference ordering over O, and we pick some default action a 0 to return if we can't prove anything at all. We then define the decision theory which takes U (•) and returns the agent A() given by Algorithm 3. Note that this decision theory is immune to the spurious conterfactuals of Section 3.3: if it ever proves a counterfactual, it immediately makes the antecedent true by returning that action, so the consequent must be true as well. Moreover, if it proves any counterfactual, then it has already tried and failed to achieve every outcome higher in its preference order. Before we show an optimality result for Algorithm 3 in a special domain, we will have to show what can go wrong with it in other domains. Firstly, the universe template could be 'unfair', in the sense that it cares about aspects of the agent that aren't significantly related to its output. For instance, we could have a universe template that gives the best outcome if and only if A() is a tab-indented Python program, or if and only if A() can be proved to always select the first action of A in alphabetical order (regardless of the details of U (•). We will not worry about how to succeed on such pathological problems in general. A very strong fairness condition for a universe template is that U (•) should be a function of only the output of A, rather than depending on any other features  16  . We can define this in Peano Arithmetic by choosing an encoding for agents in a particular universe template, then writing a sentence that asserts (A() = B()) → (U A () = U B ()); the universe template is 'provably extensional' if that statement is a theorem of Peano Arithmetic. Everything thus far can be represented in modal logic. We will represent O and A by provably mutually exclusive and exhaustive (p.m.e.e.) families of modal statements. For example, {p∧q, p∧¬q, ¬p} is a p.m.e.e. family. Then if the universe template and the agent are modal expressions, the universe template is provably extensional if (a ↔ b) → (U (a) ↔ U (b)) is a theorem of GL. (You can check that the universe from Algorithm 1 corresponds to a provably extensional modal universe with 3 actions and 3 outcomes.) However, the modal version of Algorithm 3 can fail even on provably extensional modal universe templates; indeed, for any modally defined decision theory there is an "evil problem" which it fails on. (This can be shown by representing the modal decision theory itself in the universe template, and punishing any agent that acts identically to the action of the modal decision theory.) So our definition of optimality must allow some way around this. As it happens, although for any decision theory one can write a specific universe template that frustrates it, for any fixed universe template there is a modification of Algorithm 3 which succeeds on it, and this modification is simply a replacement of "proves in Peano Arithmetic" with "proves in Peano Arithmetic given ¬ n ⊥" for sufficiently large n. We can then show that this modification achieves the best outcome that is achievable by any modal decision theory. This result shows that we're on to something: in this restricted domain, we can do as well as it's possible to do, as long as we're bringing enough deductive capacity to bear! Note that here, the universe acts on the agent only by running it and using the output, while the agent interacts with the source code of the universe only by proving things about it. So in particular, it's less rich than modal combat, in which your agent is proving things about the universe, but another part of that universe (the opponent) is proving things about the agent as well. So before we count ourselves done with using modal logic in decision theory, we have many open questions about competition and bargaining and other phenomena. It's a beautiful Löbian universe, waiting for us to explore! In addition to the contributions of Vladimir Slepnev and Benja Fallenstein, the "evil decision problems" were introduced by Nick Bone. If you'd like to get into this topic, I recommend Benja's recent posts on evil decision problems in provability logic, optimality of the model of UDT  [8] , and its sequel.
 Acknowledgments I originally prepared about half of these notes for a talk at a MIRIx workshop in Los Altos, California in November 2014; the feedback from the participants was especially helpful. When writing up the notes, I got some great assistance from Benja Fallenstein, Marcello Herreshoff, Elliot Jin, and Nathaniel Thomas. product and sum of two natural numbers are also natural numbers) 4. ∀x ∈ N∀y ∈ N (Sx = Sy) → (x = y) (if the successors are equal, so are the originals) 5. ∀y ∈ N(y = O) ∨ (∃x ∈ N Sx = y) (every nonzero number has a predecessor) 6. ∀x ∈ N (x + O = x) ∧ (x • O = O) (how zero interacts with addition and multiplication) 7. ∀x ∈ N∀y ∈ N (x + Sy = S(x + y)) ∧ (x • Sy = (x • y) + x) (how successor interacts with addition and multiplication)
 Algorithm 3 : 3 A() (Slepnev-Fallenstein model of UDT) for x ∈ O (in descending order of preference) do for a ∈ A do if Peano Arithmetic proves "A() = a → U () = x" then return a end end end return a 0
			 Note the asymmetry: a proof of defection does not directly lead to defection, since FairBot acts based only on whether it finds a proof of cooperation, and it nowhere presumes that finding a proof of defection precludes finding another proof of cooperation. Again, the formal system can't correctly know that it is consistent!
			 Note, however, that it's impossible to prove a spurious counterfactual about the choice you actually take, and thus it could not be consistent to find a spurious proof that A() = 1 → U () = 2 or that A() = 2 → U () = 11
			 Of course, this example does take a leap of faith in accepting the existence of infinitely many natural numbers; some mathematicians reject that, call themselves strict finitists, and assert that we have no idea whether the axioms of arithmetic are consistent. But they do accept that any theory with a completely specified finite model must be consistent.
			 Actually, we need to get a bit more complicated to do all inductive proofs on the natural numbers; what we truly need is that if φ(x, z 1 , . . . , z k ) is a logical formula, then∀z 1 ∈ N . . . ∀z k ∈ N (φ(0, z 1 , . . . , z k ) ∧ (∀x ∈ Nφ(x, z 1 , . . . , z k ) → φ(Sx, z 1 , . . . , z k ))) → ∀y ∈ Nφ(y, z 1 , . . . , z k ).
			 Assuming that the standard ZFC axioms of set theory are consistent, as we do throughout this paper, since we use these to construct models of all the weird axiomatizations of Peano Arithmetic.
			 Gödel himself, toward the end of his life, circulated a modal logic version of Anselm's ontological proof of the existence of God. The proof is of course formally valid, but the axioms are a bit overpowered and underjustified.
			 Note that these all correspond to nonstandard models of Peano Arithmetic, but that in some sense they approach the standard model as n → ∞.
			 This part requires our assumption that the expression is modalized in p, since otherwise we couldn't fill in the value for it without first knowing the value for p.
			 Since every expression A either remains true forever or switches over to false and remains that way forever, we can see that every expression must eventually stabilize; and if we've decomposed it at each step involving a , once two rows are identical, all subsequent rows must be as well.
			 If this step worries you, you may be reassured to know that the standard axioms of set theory are strong enough to assert that this tower upon Peano Arithmetic is consistent all the way up. Of course, that simply means that maybe set theory is contradictory, but if so then the contradiction has been hiding pretty effectively from the mathematical community for the last few decades...
			 I came up with that pun. I'm proud of it.
			 Note that this condition is too strong to include modal combat, since there it also matters whether a given formal system can prove what the algorithm's output is, not only what the output is. Thus our optimality theorem will not cover modal combat, which is consistent, given that we mentioned in the last section that we lack a good optimality result for modal combat.


Abstract: The last few years have seen a proliferation of principles for AI ethics. There is substantial overlap between different sets of principles, with widespread agreement that AI should be used for the common good, should not be used to harm people or undermine their rights, and should respect widely held values such as fairness, privacy, and autonomy. While articulating and agreeing on principles is important, it is only a starting point. Drawing on comparisons with the field of bioethics, we highlight some of the limitations of principles: in particular, they are often too broad and high-level to guide ethics in practice. We suggest that an important next step for the field of AI ethics is to focus on exploring the tensions that inevitably arise as we try to implement principles in practice. By explicitly recognising these tensions we can begin to make decisions about how they should be resolved in specific cases, and develop frameworks and guidelines for AI ethics that are rigorous and practically relevant. We discuss some different specific ways that tensions arise in AI ethics, and what processes might be needed to resolve them.

Jess Whittlestone
Rune Nyrup
Anna Alexandrova
Stephen Cave

The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions

INTRODUCTION AI systems promise widespread benefits to society, while also posing substantial risks across almost all sectors. In the last few years, a number of different groups and initiatives have attempted to articulate and agree principles to guide the application of AI in society. In this paper we discuss the role and limitations of these principles, and argue that an important next step for AI ethics is to focus more on the tensions that arise as we try to implement them in practice. By 'AI ethics' we specifically mean the emerging field of practical AI ethics, which focuses on developing frameworks and guidelines to ensure the ethical use of AI in society (analogous to the field of biomedical ethics, which provides practical frameworks for ethical practice in medicine.) AI ethics therefore covers several different sectors and types of institutions, including the following: • Technology companies aiming to develop their own ethical guidelines (e.g. Google's 'AI Ethics principles'); • Professional bodies whose codes of ethics are aimed at guiding practitioners; • Standards-setting bodies that aim to set general standards for fields of research of industries (e.g. the IEEE or British Standards Institution); • Government bodies and legislators that aim to develop policy and regulation (e.g. the UK's new Centre for Data Ethics and Innovation); • Researchers across disciplines whose work aims to inform these ways that AI ethics is put into practice: exploring the technical, philosophical, or legal aspects of using algorithms in ethical ways  [20]  or synthesising and translating such research into practice  [27] . Agreeing on principles is valuable for the aims of all of these groups: for example, principles can provide a useful starting point from which to develop more formal standards and regulation, and can help to identify priority issues on which both research and policy should focus. However, we argue that current lists of principles for AI ethics are too high-level to be immediately useful for these groups' aims. When we look at specific cases, it becomes clear that principles will come into conflict with each other. This means their practical value is limited: without acknowledging these conflicts standards may be set unrealistically high, or regulation intended to protect one value might inadvertently compromise other important goals. In order to be practically useful, we suggest that the field of AI ethics should focus more on identifying and attempting to resolve the tensions that arise when we apply them to specific cases. We begin by reviewing how principles have evolved in AI ethics over the last two years. Drawing on comparisons with bioethics -a field with a robust and well-developed tradition in using principles to govern medical practice -we discuss some of the limitations of principles. We make the case that all areas of AI ethics would benefit from a more rigorous exploration of the tensions that arise when we try to apply principles to concrete cases. We outline some key tensions that already arise from the use of AI in society, and discuss what work might be needed to resolve them. To our knowledge, this is the first paper to explicitly examine the role and limits of principles in AI ethics, and the importance of focusing more on tensions as a next step.
 THE EMERGENCE OF PRINCIPLES IN AI ETHICS Though the field is in its infancy, there is widespread agreement on some of the core issues (such as bias) and values (such as fairness) that AI ethics should focus on. Over the last two years, these have begun to be codified in sets of 'principles' or 'tenets'. The Asilomar AI principles, developed in 2017 in conjunction with the Asilomar conference for Beneficial AI, outline guidelines on how research should be conducted, ethics and values that use of AI must respect, and important considerations for thinking about long-term issues  [12] . The principles were signed by several thousand AI researchers and others, including many academic ethicists and social scientists. Around the same time, the US Association for Computing Machinery (ACM) issued a statement and set of seven principles for Algorithmic Transparency and Accountability, addressing a narrower but closely related set of issues  [1] . Over the course of 2017, several other initiatives and organisations published additional sets of principles: including the Japanese Society for Artificial Intelligence's Ethical Guidelines in February 2017  [16] ; a set of draft principles from the Montreal Declaration on Responsible AI in November  [25] ; and the IEEE's General Principles of Ethical Autonomous and Intelligent Systems in December  [14] . This proliferation of principles has continued into 2018: with the Partnership on AI publishing a set of 'tenets' which its members agree to uphold  [18] ; the UK House of Lords suggesting five principles for a cross-sector AI code which could be adopted internationally  [23] , and Google publishing their 'AI ethics principles' in June  [19] . These different sets of principles have considerable overlap. There is widespread agreement that AI-based technologies should be used for the common good, should not be used to harm people or undermine their rights, and should respect widely-held values such as fairness, privacy, and autonomy.  [9]  suggest that many of the different existing sets can be synthesised into five key principles: the four that are already used in bioethics -autonomy, beneficence, non-maleficence, and justice  [4]  -plus the additional principle of explicability, which captures the challenges of intelligibility and accountability unique to AI systems. While this convergence is encouraging, it is unclear at this point whether this reflects a deep consensus about what is important, arrived at independently by numerous different actors, or merely a shallow consensus due to the fact that different groups have read similar papers and built on the work of one another. Principles can be a valuable part of applied ethics; agreeing on high-level principles is therefore an important step for ensuring that AI is developed and used for the benefit of society. Principles help condense complex ethical issues into a few central elements which can be clearly understood and agreed upon by people from diverse fields and sectors. They encourage widespread commitment to a shared set of values, and can give them a more prominent role in institutional decision-making processes. Principles can form a basis for more formal commitments in professional ethics, internationally agreed standards, and regulation. They can also help address public concerns, by clarifying the ethical commitments of researchers and industry. However, while principles are important, they are not in themselves enough to ensure society can reap the benefits and mitigate the risks of new technologies. In order to be useful in practice, principles need to be able to guide action -to help people navigate the competing demands and considerations of concrete situations. As we articulate in the next section, there are several obstacles to this.
 THE LIMITS OF PRINCIPLES The four principles of beneficence, non-maleficence, autonomy, and justice have played a prominent role in bioethics  [4] , a field with decades of experience in managing the challenges posed by new technologies. These principles aim to articulate general values on which everyone can agree, and to function as practical guidelines. But they have spurred substantial debate: some argue that we should put no weight on principles and focus entirely on the elements of specific cases  [10] , while others have advocated a more moderate view, whereby principles should be considered in close conjunction with analysis of 'paradigm' cases  [17] . However, even the strongest advocates of principlism in bioethics acknowledge that principles alone are not enough.  [4]  suggest that principles should be taken as guidelines, which need to be made specific for use in policy and clinical decision-making. They elaborate that in order to be action-guiding, principles need to be accompanied by an account of how they apply in specific situations, and how to balance them when they conflict. In this section, we review some of the main limitations of principles that have been highlighted in the bioethics literature and illustrate why these also apply to the principles proposed for AI ethics.
 Different Groups May Interpret Principles Differently The central terms used in principles are often ambiguous, masking conceptual complexity and differences in interpretation across populations. As  [8]  point out, the principle of 'justice' in bioethics does not say anything about what is just or unjust, leaving this to the agent to decide for themselves. Clouser and Gert argue that principles often mask important moral disagreements rather than presenting a well-developed unified theory as they propose to, and that it would be better if these disagreements were articulated and understood more explicitly. In particular, lists of broadly agreed-upon principles cannot recognise that important and legitimate differences in values exist across people and populations. While everyone might agree in principle that 'fairness' is important, there exist deep political disagreements about what exactly constitutes fairness  [5] . Groups may also vary in how much weight they put on one value relative to others in situations of conflict: more individualist cultures may put more weight on personal privacy than more collectivist cultures, for example. An important step in making principles more practical is to formalize them into standards and regulation  [27] . But this process is not a straightforward unpacking of the relevant principles, as different principles will come into conflict when applied to concrete cases. In the next section, we make the case that in order for principles to inform more practical aspects of AI ethics, including professional ethics, standards, and regulation, we need to begin by exploring in detail the different kinds of tensions that arise when principles are applied.
 Principles Are Highly General Relatedly, principles are by their nature highly general: their value is that they indicate important moral themes that apply across a wide range of scenarios. This means that they can be useful as a kind of checklist: as a set of important considerations that need to be taken into account in specific scenarios. However, the generality of most principles also limits their ability to guide practical action  [3] ). Many of the principles proposed in AI ethics are too broad to be action-guiding. For example, ensuring that AI is used for 'social good' or 'the benefit of humanity' is a common thread among all sets of principles. These are phrases on which a great majority can agree exactly because they carry with them few if any real commitments. A very wide range of differing ideological, political and philosophical standpoints could claim to be for the good, or for the benefit of humanity. Only principles that are narrower and more specific are likely to be useful in practice. Recent industry commitments to not develop technology for autonomous weapons are an example of a principle that is specific, action-guiding and can be used to hold people to account. But at the same time, exactly that specificity means its relevance is limited to one sector, and it has many dissenters.
 Principles Come into Conflict in Practice The gap between principles and practical judgement grows larger still when we consider that principles will inevitably conflict with each other. For example, the UK House of Lords AI Committee report states that, "it is not acceptable to deploy any artificial intelligence system which could have a substantial impact on an individual's life, unless it can generate a full and satisfactory explanation for the decisions it will take. " The intentions behind this principle are important, but it masks a crucial tension between using algorithms for social benefit ('beneficence') and ensuring those algorithms are fully intelligible to humans ('explicability'). For example, algorithms exist today that can diagnose medical conditions more accurately than doctors, potentially saving lives  [24] , but for which a full and satisfactory explanation cannot necessarily be provided (depending on how this is defined). In some situations, the benefit of using an algorithm may be high enough, and its accuracy reliable enough, that all users agree it is worth using even if a fully comprehensive explanation of its decisions cannot be given. There are complex and important trade-offs involved here  [15, 22] , and a principle that simply states that it is not acceptable to deploy AI systems without full explainability fails to recognise this. In conclusion, there is a risk that high-level principles give the impression of being the outcome of meaningful debate about how AI should be developed, but in reality they are simply postponing it.
 WHY THE FIELD SHOULD FOCUS ON TENSIONS We use the term 'tension' to refer to any conflict, whether apparent, contingent or fundamental, between important values or goals, where it appears necessary to give up one in order to realise the other. For example, the use of socially beneficial data-driven technologies might make it impossible for us to fully guarantee otherwise desirable levels of data privacy. If the potential gains of these technologies are significant enough -new and highly effective cancer treatments, say -we might decide that a higher risk of privacy breaches is a price worth paying. In some cases, a tension may reflect a strict moral tradeoff: a situation where two values or goals conflict and it is not possible to get more of one without sacrificing another. However, many tensions in AI are more contingent, and arise as a result of current technological or societal constraints. Using machine learning for social benefit may not be fundamentally in tension with privacy, transparency, or fairness, but many current methods employed for the former goal do conflict with these ideals. We do not yet know how far new technological or governance solutions could go to dissolve these tensions. Others have acknowledged the importance of recognising conflicts between values in AI ethics, but to our knowledge none have explored in detail why this would be beneficial or what it would look like in practice. For example,  [9]  say that, "Ensuring socially preferable outcomes of AI relies on resolving the tension between incorporating the benefits and mitigating the potential harms of AI, in short, simultaneously avoiding the misuse and underuse of these technologies", but do not discuss specific tensions in detail or how to resolve them. In this section, we discuss some of the benefits of focusing on tensions. We outline four reasons this is an important next step for AI ethics: (1) bridging the gap between principles and practice, (2) acknowledging differences in values, (3) highlighting areas where new solutions are needed and (4) identifying ambiguities and knowledge gaps.
 Bridging the Gap between Principles and Practice In general, we see focussing on tensions as an important way of bridging the gap between abstract ethical principles and specific cases, and therefore an important first step towards an ethics of AI that is practical and action-guiding. To identify tensions, we need to consider how different values and goals might come into practice in concrete cases. For example, when Google DeepMind collaborated with the Royal Free Hospital, they encountered a conflict between protecting the privacy of patient data, and their goal of using AI to improve early diagnosis of acute kidney injury  [20] . Since similar tensions will likely arise across a range of different cases, focusing on tensions means we are neither driven entirely by the specifics of an individual case, nor are we relying on abstract high-level values. If we can articulate important tensions by looking at a range of cases, and find ways to resolve them in specific scenarios, what we learn from this can then be used to develop standards and regulation that are more sensitive to how principles apply differently across scenarios.
 Acknowledging Differences in Values Focusing on tensions forces us to consider how different values might be interpreted and endorsed differently across groups. While some important tensions are due to conflicts between principles in practice, others arise because there are conflicting meanings or values within a single principle: broad terms like 'fairness' or 'justice' for example are subject to substantial moral and political disagreement  [5, 8] . It may never be possible to totally resolve all of these disagreements. But clearly articulating them is a crucial starting point for ensuring that all aspects of AI ethics are as inclusive as possible: for example, to ensure that international standards take full account of and accommodate cultural differences, and that agreement on such standards is meaningful.
 Highlighting Areas Where New Solutions Are Needed Noting a tension between two values does not necessarily mean we are forced to choose between them: often, we may be able to find some way to get more of both things we value. Recognising these tensions can therefore highlight high priority areas for both researchers and policymakers. For example, acknowledging that there is currently a tradeoff between performance and interpretability in state-of-the-art machine learning systems has motivated technical research that attempts to reduce or eliminate this trade-off  [2] . Acknowledging tensions will also help direct the development of AI in beneficial directions more generally. It is currently far from clear whether advances in AI will augment or degrade human capabilities and agency, but making this tension explicit focuses attention on the important question of which trajectories of development are most likely to lead to the former.
 Identifying Ambiguities and Knowledge Gaps Finally, a tension-focused approach helps to clearly highlight ambiguities and gaps in our understanding of how uses of AI are impacting society. Again, this can help identify new and important research directions. To think clearly about all tensions, we need to recognise and clarify ambiguities in terms: what do we really mean by things like 'fairness', 'justice', and 'autonomy', and how might these be interpreted differently across groups and contexts? To understand the nature of many tensions we need to understand what is currently technically possible: what are the best current methods for ensuring data privacy in machine learning, for example, and what are the costs of these methods? To understand how tensions arise in practice, we need better evidence on how AI is actually being applied in society today: what effect is automation already having on individual lives across different sectors? And to articulate and resolve conflicts between the interests of different groups, we need to really understand the needs and values of affected communities: how do the trade-offs people are willing to make differ based on demographic factors, for example? Focussing on tensions should help to drive this important work.
 WHICH TENSIONS? There are several different ways that applications of AI can introduce tensions between important goals and values. Some tensions arise due to the very nature of AI and machine learning: these techniques allow us to use and draw inferences from very large amounts of (often personal) data, and so challenge important notions of privacy. The most useful models also often quickly become very complex, introducing new issues around human interpretability  [11, 26] . This means we face tensions between using these technologies for socially beneficial goals: improving healthcare, justice, or security, for example, and other goals such as respecting privacy and maintaining trust and understanding in automated systems. Another possibility is that AI systems exacerbate already existing ethical or societal tensions: between different conflicting notions of fairness, for example. Often this is due to the fact that machine learning models are trained on historical data, and so inherit the biases or mistakes they contain  [13] . Here, applications of AI in society do not necessarily introduce new tensions, but increase the importance of already-existing ones such as how to make decision-making more accurate and efficient without inadvertently discriminating against minority groups. Other tensions arise because the harms and benefits of AI systems are unequally distributed in various ways. For example, the impacts of automation may be unequally distributed across populations and cultures: enhancing the agency of some groups by automating mundane tasks while wiping out the livelihood of others, thus threatening their basic needs. The risks and benefits of AI systems could also be unequally distributed over time, and uses of AI that present opportunities in the near term may compromise important long-term values. Increasing personalisation of messages and services may make our lives more convenient and enjoyable in the short run, but begin to undermine important aspects of autonomy, equality and solidarity over time  [21] . Finally, AI may have the potential to both enhance and threaten a given value. For example, depending on the precise direction in which technology develops, it could be used to either greatly enhance human capabilities -if we can develop sophisticated methods of intelligence augmentation  [7]  -or to degrade them: if our own capabilities atrophy as we outsource more and more tasks  [6] . As mentioned above, automation might enhance the agency of some groups while threatening the autonomy of others: whether we see AI as enhancing or degrading human agency could depend on how narrow or global a view we take of its impacts.
 FOUR KEY TENSIONS Given the wide range of tensions that may arise from applications of AI, now or in the future, there is unlikely to be an exhaustive list of all possible tensions. However, we believe that the following four tensions will be particularly central to thinking about the ethical issues arising from the applications of AI systems in society today. These capture a range of issues which are already salient or likely to grow in importance moving forward. Tension 1: Using data to improve the quality and efficiency of services vs. respecting privacy and autonomy of individuals. Machine learning and big data are already being used to improve various public services (including healthcare, education, and social care). These improvements could be hugely beneficial to citizens, but require large amounts of personal data, raising concerns about how to best protect privacy and ensure meaningful consent. Tension 2: Using algorithms to make decisions and predictions more accurate vs ensuring fair and equal treatment. This tension arises when public or private bodies base decisions on predictions about future behaviour of individuals (e.g. when probation officers estimate risk of reoffending) and when they employ machine learning algorithms to improve their predictions. These algorithms may improve accuracy overall, but discriminate against specific subgroups for whom representative data is not available. Tension 3: Reaping the benefits of increased personalisation in the digital sphere vs enhancing solidarity and citizenship. Companies and governments can use personal data to tailor the messages, offers, and services people see. This personalisation can make it easier for people to find the right products and services for them, but differentiating between people in such fine-grained ways may threaten societal ideals of citizenship and solidarity. Tension 4: Using automation to make people's lives more convenient and empowered vs promoting selfactualisation and dignity. Automated solutions may genuinely improve people's lives by saving them time on mundane tasks that could be better spent on more rewarding activities. But they also risk disrupting some of the practices that are an important part of what makes us human. With automation we may see the gifts of arts, languages and science become more accessible to those who were excluded in the pastbut we may also see widespread deskilling, atrophy, ossification of practices, homogenisation and cultural diversity.
 IDENTIFYING FURTHER TENSIONS The above tensions are important and represent areas where exploring tensions is likely to be fruitful for AI ethics. Going forward, further such areas can and should be identified. In order to do so, it is helpful to ask a range of questions, including: • Where AI is being used to serve a particular goal or value, or for 'social benefit' in general, what risks to other values are introduced? • Where might uses of AI that benefit one group, or the population as a whole, have negative consequences for a specific subgroup? How do we balance the interests of different groups? • Where might applications of AI that are beneficial in the near-term introduce risks in the long-term? How do we trade-off short and long-term impacts of society? • Where might future developments in AI either enhance or threaten important values, depending on the direction they take?
 RESOLVING TENSIONS The best approach to resolving a tension will depend on the nature of the tension in question. Where a strict trade-off between two values exists, a choice must be made to prioritise one set of values over another. For example, this may mean judging what risks to privacy it is acceptable to incur for the sake of better public health, or where to reject innovative automation technologies because the threats they pose to human skills and autonomy are too great. Making these trade-off judgements will be a complex political process. Weighing the costs and benefits of different solutions can be an important part of the process but alone is not enough, since it fails to recognise that values are vague and unquantifiable, and that numbers often hide complex value judgements. In addition, resolving trade-offs will require extensive public engagement, to give voice to a wide range of stakeholders and articulate their interests with rigour and respect. On the other hand, where tensions are more practical in nature, strict trade-offs may not be inevitable. It may be that we simply lack the knowledge or tools to advance conflicting values, and investing in further research could identify solutions that better serve all relevant values or goals. For example, it might be possible to use automation to improve people's lives without sacrificing self-actualisation and devaluing human skills, if a clear line can be drawn between the contexts where we do and do not want to pursue automation. In these situations we face a choice. Even if a tension between two goals is not a fundamentally irresolvable one, if we want to apply current technology in society, we will still need to make the kinds of trade-offs described above. On the other hand, if we can hold-off from implementing technologies that introduce tensionscertain kinds of automation, say -then we could instead invest in more research on how technological or governance solutions might reduce the need to navigate potentially difficult value trade-offs. Of course, this is not a binary choice: we can choose to strike a balance by making the trade-offs necessary to implement technology where doing so is relatively unproblematic, while still investing in research to explore how these tensions might be resolvable in future. This choice can be thought of as involving its own tension, between short-and long-term interests: to what extent should we postpone the benefits of new technologies in order to invest the time and resources necessary to resolve the tensions they introduce?
 CONCLUSION Over the last two years, many different sets of principles for the ethical use of AI have been developed. We argue that these highlevel principles, rather than representing the outcome of meaningful debate on how AI should be developed, risk simply postponing it. In order to make AI ethics practical, the field needs to now focus more on the tensions that arise when principles are applied to concrete cases. Though most of these tensions cannot be resolved straightforwardly, we believe articulating them more clearly and explicitly has several benefits. To be useful in practice, principles need to be formalized in standards, codes and ultimately regulation. To be effective, these in turn must acknowledge that there are tensions between the different high-level goals of AI ethics, and provide some guidance on how they should be resolved in different scenarios. They also need to acknowledge and accommodate different perspectives and values as far as possible, if they are to reflect genuine agreement. A focus on tensions can also help to direct research priorities in AI ethics. Articulating tensions can help to highlight important ambiguities and gaps in our understanding of how AI is currently being applied in society which need further research. More generally, much current research in AI ethics appears to be driven by questions of how we ensure that uses of AI respect important values, such as privacy, transparency, or fairness. Reframing research questions to be more focused on understanding and resolving tensions is an important step towards solving practical problems arising from the use of AI in society, since it directs attention to where new technological or governance solutions might help push the development of AI in robustly beneficial directions. 			 This work is licensed under a Creative Commons Attribution International 4.0 License.