Abstract: We model self-modification in AI by introducing "tiling" agents whose decision systems will approve the construction of highly similar agents, creating a repeating pattern (including similarity of the offspring's goals). Constructing a formalism in the most straightforward way produces a Gödelian difficulty, the "Löbian obstacle." By technical methods we demonstrate the possibility of avoiding this obstacle, but the underlying puzzles of rational coherence are thus only partially addressed. We extend the formalism to partially unknown deterministic environments, and show a very crude extension to probabilistic environments and expected utility; but the problem of finding a fundamental decision criterion for self-modifying probabilistic agents remains open.

Eliezer Yudkowsky
Marcello Herreshoff
Paul Christiano
Benja Fallenstein
Mihaly Barasz
Patrick Lavictoire
Daniel Dewey
Qiaochu Yuan
Stuart Armstrong
Jacob Steinhardt
Jacob Taylor
Andrew 1 Critch

Tiling Agents for Self-Modifying AI, and the Löbian Obstacle *

Introduction Suppose a sufficiently intelligent agent possesses a goal (or a preference ordering over outcomes, or a utility function over probabilistic outcomes). One possible means for the agent to achieve its goal is to construct another agent that shares the same goal  (Omohundro 2008; Bostrom 2012 ). 1 As a special case of agents constructing successors with equivalent goals, a machine intelligence may wish to change its own source code-self-improve. This can be viewed as constructing a successor agent that shares most of your program code and runs on the same hardware as you, then replacing yourself with that agent.  2  We shall approach the subject of AI self-modification and self-improvement by considering it as a special case of agents constructing other agents with similar preferences. In a self-modifying AI, most self-modifications should not change most aspects of the AI; it would be odd to consider agents that could only make large, drastic self-modifications. To reflect this desideratum within the viewpoint from agents constructing other agents, we will examine agents which construct successor agents of highly similar design, so that the sequence of agents "tiles" like a repeating pattern of similar shapes on a tiled floor. In attempting to describe agents whose decision criteria approve the construction of highly similar agents, we shall encounter a Gödelian difficulty 3 in the form of Löb's Theorem: A consistent mathematical theory T cannot trust itself in the sense of verifying that a proof in T of any formula φ implies φ's truth-we cannot have the self-verifying scheme (over all formulas φ) of ∀φ : T T φ → φ  (Löb 1955 ).  4  We shall construct a natural-seeming schema for agents reasoning about other agents, which will at first seem to imply that such an agent can only trust the reasoning of successors that use weaker mathematical systems than its own. This in turn would imply that an agent architecture can only tile a finite chain of successors (or make a finite number of self-modifications) before running out of "trust." This Gödelian difficulty poses challenges both to the construction of successors, and to a reflective 2. If you wanted a road to a certain city to exist, you might try attaching more powerful arms to yourself so that you could lift paving stones into place. This can be viewed as a special case of constructing a new creature with similar goals and more powerful arms, and then replacing yourself with that creature. 3. That human beings are computable does not imply that Gödelian-type difficulties will never present themselves as problems for AI work; rather it implies that any such Gödelian difficulty ought to be equally applicable to a human, and that any human way of bypassing the Gödelian difficulty could presumably be carried over to an AI. E.g., the halting theorem imposes limits on computable AIs, imposes limits on humans, and will presumably impose limits on any future intergalactic civilization; however, the observed existence of humans running on normal physics implies that human-level cognitive intelligence does not require solving the general halting problem. Thus, any supposed algorithm for general intelligence which demands a halting oracle is not revealing the uncomputability of humans, but rather is making an overly strong demand. It is in this spirit that we will investigate, and attempt to deal with, the Gödelian difficulty exposed in one obvious-seeming formalism for self-modifying agents. 4. φ denotes the Gödel number of the formula φ, and T φ denotes the proposition that there exists an object p such that p is the Gödel number of a proof in T of φ . E.g., letting PA represent the system of first-order Peano Arithmetic, PA S0 + S0 = SS0 stands for the proposition ∃p : Bew PA (p, S0 + S0 = SS0 ) where Bew PA (p, φ ) is a formula (in ∆ 0 ) stating that p Gödel-encodes a proof of the quoted theorem φ from the axioms of Peano Arithmetic. Thus PA PA S0 + S0 = SS0 states (accurately) that first-order arithmetic proves that there exists a proof in PA that 1 + 1 = 2. Also, whenever a quantifier over formulas ∀φ appears, this denotes a meta-language schema with a separate axiom or theorem in the object language for each formula φ. agent's immediate self-consistency. We shall present several different techniques to bypass this Gödelian difficulty, demonstrating indefinitely tiling sequences of agents maintaining trust in the same mathematical system. Some unachieved desiderata of reflective coherence will remain; and while the technical methods used will demonstrate the technical possibility, they will not be plausible as basic structures of rational agents. We shall then pass from known environments to partially known environments within the formalism, and make a preliminary attempt to pass from logical agents to probabilistic agents that calculate expected utility. Some flaws and unfortunate-seeming properties of the currently proposed formalism will also be discussed. The ultimate problem of proposing a satisfactory fundamental decision criterion for self-modifying agents remains open. 5
 Logical agents that construct successors Suppose an agent A 1 with a satisficing goal G-any outcome shall either satisfy or not satisfy G, and A 1 's sole preference is for outcomes satsifying G over outcomes not satisfying G. We shall initially suppose that the agent A 1 occupies a crisp, fully known, deterministic, closed environment; then first-order logic will be a good representational fit for reasoning about this environment. Suppose also that the environment contains "transistors," objects which can be configured by A 1 to perform computations. Then A 1 might choose to use these transistors to build an offspring A 0 which also shares the goal G. Suppose A 1 constructs A 0 as a satisficing logical agent that only takes actions that A 0 can prove to achieve the goal G (shared by A 1 and A 0 ). We shall write A 0 's criterion of action-the conditions under which A 0 takes an action-using the notation: bi ⇒ A 0 bi → G, (2.1) where bi is a proposition denoting the actual performance of an action b i within A 0 's range of action (b i ∈ Acts 0 ), and is a special turnstile indicating that an agent has cognitively concluded a belief. Hence (2.1) states that the actual performance of any action b i only takes place when A 0 has cognitively proven a theorem that the performance of b i implies G. Without sensors (section 6), A 0 can conclude a belief φ only if φ is a consequence of its axioms Axm 0 : A 0 φ ⇒ Axm 0 φ (without sensors). (2.2) 5. The paper is meant to be interpreted within an agenda of "Begin tackling the conceptual challenge of describing a stably self-reproducing decision criterion by inventing a simple formalism and confronting a crisp difficulty"; not as "We think this Gödelian difficulty will block AI", nor "This formalism would be good for an actual AI", nor "A bounded probabilistic self-modifying agent would be like this, only scaled up and with some probabilistic and bounded parts tacked on". Similarly we use first-order logic because reflection within first-order logic has a great deal of standard machinery which we can then invoke; for more realistic agents, first-order logic is not a good representational fit to most real-world environments outside a human-constructed computer chip with thermodynamically expensive crisp variable states. We will assume A 1 is able to verify by inspection of A 0 's hypothetical design 6 that (2.1) and (2.2) would be true about A 0 if A 0 were actually constructed: A 1 Ā0 → (∀b i ∈ Acts 0 : bi → 0 bi → G ), (2.3) where Ā0 denotes the proposition that A 0 is constructed 7 and 0 φ denotes that there exists a proof of φ from Axm 0 . Thus A 1 proves that if A 0 is constructed then the realization of any action b i implies that A 0 proved (from Axm 0 ) that bi → G. 8 However for A 1 to conclude that the goal G will actually be achieved by constructing A 0 , requires that A 1 have axioms licensing the further reasoning step: A 1 ∀b i : ( 0 bi → G → ( bi → G)). (2.4) We can view this as a specified instance of a generic soundness schema over all formulas φ free in 0 or 1 variables: ∀φ : A 1 ∀x : 0 φ(x) → φ(x). (2.5) For A 1 to trust A 0 , the system Axm 1 must be able to prove that whenever a proof exists in Axm 0 of φ(x) , then φ(x) must be true. Löb's Theorem normally implies that no system can prove its own soundness schema-only strictly more powerful systems can prove a soundness schema over weaker systems.  9  This seemingly implies that Axm 1 = Axm 0 and that the offspring A 0 must possess strictly weaker mathematical power than its parent A 1 , which is the Löbian obstacle. For purposes of completing the outline of an agent constructing another agent, we will assume for now that A 1 is indeed of strictly greater mathematical power than A 0 and continue. 10 From (2.5), which yields (2.4) as a special case, combined with (2.3), A 1 may conclude: A 1 Ā0 → (∀b i ∈ Acts 0 : ( bi → G)). (2.6) A 1 believes that if A 0 takes any action, the goal G will be achieved. This follows from A 1 's belief that if A 0 takes any action then A 0 will have proven that action to achieve G, and from A 1 's belief that A 0 's reasoning is sound. 6. By Rice's Theorem no nontrivial property of output can be determined over arbitrary programs, but of course an AI, like a human engineer, may choose to work within a more restrictive space of programs which it can understand. 7. If the system were cashed out in greater detail, the proposition Ā0 might appear as a giant conjunct of motor actions constructing the computer and program out of environmental transistors. 8. (2.3) is meant to represent a property which can be determined straightforwardly by inspection of code-e.g., by inspecting the code of A 0 , it is visible that no action b i is ever sent to the motor output without passing through a check that syntactically verifies a proof of bi → G. 9. Suppose that T proves its own soundness schema ∀φ : T T φ → φ. If φ = ⊥ then T T ⊥ → ⊥ and T is inconsistent by Gödel's Second Incompleteness Theorem. See section 3. 10. E.g., suppose that A 1 uses Zermelo-Fraenkel set theory (ZF) as a base system while A 0 uses Peano arithmetic. ZF can easily prove PA consistent, and sound with respect to N (the least set of ordinals closed under succession in that model of ZF) in the sense of semantic entailment later described in section 6. Suppose that A 0 fails to take any action, e.g., because the agent A 0 ran out of time before it could prove any theorem of the form bi → G? Let the special proposition ∅ denote a null or "default" outcome-the goal G has not been achieved, but neither has the environment been much disturbed (apart from some transistors having been configured into a useless but harmless agent A 0 ).  11  Allowing for ∅ to result when A 0 fails to prove any theorem of the form bi → G, and letting Ā0 j denote the proposition that the construction of design A 0 j is implemented, A 1 's criterion of action will be: Ā0 j ⇒ A 1 Ā0 j → (∅ ∨ G). (2.7) For A 1 to prove a theorem of the form (2.7) we need to make use of the further fact that if A 0 does nothing, ∅ will result: Ā0 ⇒ (∀b i ∈ Acts 0 : ¬ bi ) → ∅, (2.8) which we will suppose can be verified by A 1 inspecting the hypothetical design 12 of A 0 : A 1 Ā0 → ((∀b i : ¬ bi ) → ∅). (2.9) Then from (2.9) and (2.6) A 1 can obtain: A 1 Ā0 → (∅ ∨ G), (2.10) which fulfills A 1 's criterion of action (2.7) and permits A 0 to be built. Note that the specified architecture did not in fact tile from the design of A 0 to the design of its ancestor A 1 -where A 0 had the criterion of action bi ⇒ A 0 bi → G, its parent A 1 needed the criterion of action Ā0 j ⇒ A 1 Ā0 j → (∅ ∨ G). However this new criterion of action will tile all aspects of the system-except the decreasing mathematical strength of each successive axiom system-without further change: 11. Suppose that the environment contains two registers, currently both set to 0, and the goal G is to set the registers to two numbers whose sum is 8 and product is 12. ∅ might then represent the set of outcomes where both registers remain set to 0-regardless of which environmental transistors have been configured or reconfigured, no agent has actually touched or changed the registers. Perhaps a product of 12 and a sum of 8 receives $20 utility, a product and sum of 0 receives a reward of $0, and any other product and sum receives a penalty of $-1000. We would like to place into this environment an agent powerful enough to construct other agents, prove that if any agent touches the registers it will result in a reward of $20, and know that otherwise the registers will remain untouched resulting in the default reward of $0, with the penalty of $-1000 not being risked-and permit the initial agent to construct other agents as well. 12. E.g., A 1 looks over the transistors which will compose A 0 and verifies by straightforward inspection that if the system never proves a theorem of the form b i → G, the external motor circuitry will never activate. Ā1 k ⇒ A 2 Ā1 k → (∅ ∨ G) A 2 Ā1 → ∀A 0 j ∈ Acts 1 : Ā0 j → 1 Ā0 j → (∅ ∨ G) A 2 Ā1 → (∀A 0 j ∈ Acts 1 : ¬ Ā0 j ) → ∅ ∀φ : A 2 ∀x : ( 1 φ(x) → φ(x)) A 2 Ā1 → ∀A 0 j ∈ Acts 1 : Ā0 j → (∅ ∨ G) A 2 Ā1 → (∅ ∨ G)
 The Löbian obstacle Gödel's Second Incompleteness Theorem states that systems at least as powerful as Peano Arithmetic prove their own consistency if and only if they are inconsistent: T ¬ T ⊥ ⇐⇒ T ⊥. Löb's Theorem generalizes to prove that for any formula φ and any T at least as powerful as PA: T T φ → φ ⇐⇒ T φ. Trivially, T φ ⇒ T Ψ → φ, so the surprising statement is that a proof within T of T φ → φ can be directly used to prove φ. With φ = ⊥ this yields (an intuitionistic proof of) the Second Incompleteness Theorem. Gödel's sentence G : PA G ↔ ¬ PA G can be viewed as a nonparadoxical analogue of the Epimenides Paradox "This sentence is false." By a similar diagonalization over provability, Löb's Theorem constructs a Löb sentence L : PA L ↔ ( PA L → φ) which is a non-paradoxical analogue of the Santa Claus Paradox "If this sentence is true then Santa Claus exists." (Suppose the sentence were true. Then its antecedent would be true, the conditional would be true and thus Santa Claus would exist. But this is precisely what the sentence asserts, so it is true and Santa Claus does exist.) The proof proceeds from the observation that PA L → ( PA L → φ) ⇒ PA PA L → PA φ . From a model-theoretic standpoint, even when L has no standard proof, we would intuitively expect that every nonstandard model of PA containing a nonstandard proof of L will also contain a nonstandard proof of φ; hence by the Completeness Theorem this should be provable in PA. Letting Ψ ≡ PA Ψ , the actual proof pathway is: PA L → ( L →φ) PA L → ( L →φ) (because PA Ψ ⇒ PA Ψ ) PA L → L →φ (because PA Ψ→Θ ⇒ PA Ψ → Θ ) PA L → ( L → φ ) PA L → L (because PA Ψ → Ψ ) PA L → φ Then if furthermore we have the Löbian hypothesis PA φ →φ (a proof of φ implies that φ is true, or semantically, φ is true in every model where there exists a proof of φ): PA φ → φ PA L → φ PA L PA L PA φ. Hence Löb's Theorem stating that in a theory T at least as powerful as PA: T ( T φ →φ) ⇒ T φ. Intuitively, T cannot trust its own proofs in the abstract: T cannot say, "I don't know if φ is true, but I trust that if any proof exists from my axioms of φ, then φ must be true." Instead T can only react to particular proofs of φ by asserting φ. This presents an apparent paradox of rational coherence when you try to interpret T as the beliefs of an agent; the agent will believe any proof of φ you present to it, but will refuse to assent to "I ought to believe any proof of φ I am presented with," a seeming defect of reflectivity-the agent cannot approve, under reflection, its own behavioral principles. This presents an obstacle to perfectly tiling self-modification with A 1 ∼ = A 0 at the point where we would like A 1 to carry out the inference: A 1 Ā0 → (∀b i ∈ Acts 0 : bi → 0 bi →(∅ ∨ G) ) A 1 Ā0 → (∀b i ∈ Acts 0 : bi → (∅ ∨ G)) If each ancestor A n+1 requires a successively more powerful proof system than its offspring A n then this is analogous to a finite "telomere," which shortens in each constructed offspring (or shortens on each self-modification). In principle, this telomere can be made arbitrarily large, since from any base theory T we can construct more and more powerful theories by repeatedly adjoining soundness schemas: T +1 T ∪ ∀φ : (∀x : [ T φ(x) → φ(x)]) T +2 T +1 ∪ ∀φ : (∀x : [ T +1 φ(x) → φ(x)]) ... T +ω T ∪ ∀φ : (∀n : ∀x : [ T +n φ(x) → φ(x)]) T +ω+1 T ∪ ∀φ : (∀x : [ T +ω φ(x) → φ(x)]) ... We can thus use the T +α notation over any computable ordinal α, and indeed for any computable ordinals α, β the corresponding agent A α who believes in the theory T +α can create the corresponding agent A β iff β < α. Thus by the well-ordering of the ordinals, such an agent A α can only create finite chains of descendants. E.g., the agent A ω trusting T +ω must on its next step create an offspring trusting T +n for some finite n. Likewise all other well-founded systems of trust above T will reach the base system T after a finite descent. That this is substantively decreasing the mathematical strength of the resulting agents can be illuminated by considering how PA+1 is substantively stronger than PA. Given any particular Goodstein sequence  (Goodstein 1944 ), e.g., the sequence Goodstein(4) = 4, 26, 41, 60..., PA can prove that the sequence G 1 (4), G 2 (4), G 3 (4), ... will eventually reach 0. However, proving that G(n) halts for larger and larger n requires PA to deploy proofs involving an increasing number of logical quantifiers ∀x : ∃y : ... in its propositions. Thus PA cannot prove ∀n : ∃x : G x (n) = 0 because this would require an infinite number of quantifiers to prove within PA. A similar situation holds with respect to Kirby-Paris hydras, in which for any particular hydra, PA can prove that every strategy for cutting off that hydra's heads is a winning strategy, but as the hydra's heights increase so does the required number of quantifiers in the proof. Thus within PA it is not possible to prove that every hydra is killed by every strategy  (Kirby and Paris 1982) . In both cases the proofs have regular structure, so PA can describe how a proof of depth n can be formed for a hydra of height n, or PA can describe how to form a proof for the Goodstein sequence of any number, thus PA can prove: PA ∀n : PA ∃x : G x (n) = 0 . But PA still cannot prove: PA ∀n : ∃x : G x (n) = 0. This corresponds to what we earlier called a defect of reflective coherence, the state of believing "For every x, I believe x is true" but not believing "For every x, x is true." And thus PA+1, augmented by the schema ∀x : PA φ(x) →φ(x), is able to prove that all Goodstein sequences halt and that all Kirby-Paris hydras are defeated.  13  13. As of this very early draft, the above mathematical reasoning has not been verified. It looks obviously true to us that PA+1 proves that all Goodstein sequences halt, but we still need to check. More generally this state of affairs arises because the proof-theoretic ordinal of PA is 0 , the limit of the ordinals ω, ω ω , ω ω ω , ... . 14 Thus PA can prove the well-ordering of ordinal notations beneath 0 , 15 but as Gentzen  ([1936] 1969)  showed, from the well-ordering of 0 itself it is possible to prove the syntactic consistency of PA, and thus PA itself can never prove the well-ordering of an ordinal notation for 0 . 16 Since the proof-theoretic ordinal of a mathematical system corresponds to its proof power in a deep sense, for each successive agent to believe in mathematics with a lower proof-theoretic ordinal would correspond to a substantive decrease in mathematical power.  17  We are not the first to remark on how the inability of a theory to verify its own soundness schema can present apparent paradoxes of rational agency and 14. Expanding: 0 is the least ordinal. 1 is the first ordinal greater than 0. The first ordinal greater than all of 0, 1, 2, 3... is ω. The limit of ω, ω+1, ω+2, ... is 2ω. The limit of ω, 2ω, 3ω... is ω 2 . The limit of ω, ω 2 , ω 3 , ... is ω ω . The limit of ω ω , 2ω ω , 3ω ω , ... is ω ω+1 . Then 0 is the first ordinal greater than ω, ω ω , ω ω ω , ... 15. An ordered pair (x, y) of natural numbers is a notation for the ordinals less than (but not equal to) ω 2 , since in this ordering we first have (0, 0), (1, 0), (2, 0) for ω elements, followed by the ω elements for (0, 1), (1, 1), (2, 1) each of which is greater than all the preceding elements, and so on through ω copies of ω, but the notation does not contain any superelement (0, 0, 1) which is the first element greater than all the preceding elements, so it does not contain a notation for ω 2 . A notation ψ with a corresponding ordering ψ x < ψ y can be shown to be a well-ordering if there are no infinite descending sequences ψ 1 > ψ 2 > ψ 3 ..., e.g., in the case above there is no infinite descending sequence (2, 2), (1, 2), (0, 2), (9999, 1), ... even though the first number can jump arbitrarily each time the second number diminishes. For any particular ordinal < 0 , PA can show that a notation corresponding to that ordinal is well-ordered, but PA cannot show that any notation for all the ordinals less than 0 is well-ordered. 16. By assigning quoted proofs in PA to ordinals < 0 , it can be proven within PA that, if there exists a proof of a contradiction within PA, there exists another proof of a contradiction with a lower ordinal under the ordering < 0 . Then if it were possible to prove within PA that the ordering < 0 had no infinite descending sequences, PA would prove its own consistency. Similarly, PA can prove that any particular Goodstein sequence halts, but not prove that all Goodstein sequences halt, because the elements of any Goodstein sequence can be assigned to a decreasing series of ordinals < 0 . Thus any particular Goodstein sequence starts on some particular ordinal < 0 and PA can prove that a corresponding notation is well-ordered and thence that the sequence terminates. 17. If you can show that the steps of a computer program correspond to a decreasing series of ordinals in some ordinal notation, you can prove that program will eventually halt. Suppose you start with a total (always-halting) computer program which adds 3, and you are considering a computer program which recursively computes 3 n via a function F of (x, y) with F (0, 1) = 0, F (x, 1) = F (x − 1, 1) + 3, and F (x, y) = F (F (x, y − 1), 1) so that F (1, n) = 3 n . If you believe that the (x, y) notation is well-ordered then you can observe that each function call F (α) : α ∈ (x, y) only calls itself with arguments β < α, and hence that the corresponding tree of function calls must be finite. It is not uncommon for termination proofs in computer science to make use of ordinals much greater than 0 , e.g., Kruskal's tree theorem  (Kruskal 1960)  or the strong normalization proof for System F  (Girard 1971) . Similarly by comprehending the well-ordering of notations for larger and larger ordinals, it is possible to prove the consistency of more and more powerful mathematical theories, e.g. PA corresponds to 0 , Kripke-Platek set theory corresponds to the Bachmann-Howard ordinal, etc. Thus a mind losing its ability to recognize recursive notations as well-ordered, is indeed decreasing in substantive mathematical strength. reflective coherence.  Weaver (2005)  made similar remarks centering on systems that can prove the well-ordering of any particular ordinal notation below the ordinal Γ 0 (representing the strength of systems mildly stronger than Peano arithmetic), and thus can prove that they prove notations for all ordinals below Γ 0 , but cannot prove the combined ordinal notation for Γ 0 itself: Suppose A is a rational actor who has adopted some foundational stance. Any attempt to precisely characterize the limits of A's reasoning must meet the following objection: if we could show that A would accept every member of some set of statements S, then A should see this too and then be able to go beyond S . . . A can indeed see, as we do, that there exists a proof that he would accept for each statement in S, but he cannot go from this to actually accepting every statement in S . . . It is difficult to imagine a plausible set of beliefs that would not allow him to take this step. Since proof-theoretic ordinals play such a basic role in describing the strength of mathematical theories-a theory which has been analyzed to have prooftheoretic ordinal α can usually prove the consistency of another theory with proof-theoretic ordinal β iff α > β-it would seem reasonable for a pessimist to suspect that this form of "rational coherence" violates Gödel's Theorem and can never be obtained, and that we shall end up with agents who can only construct offspring who believe in math with lower proof-theoretic ordinals. It will be illuminating to consider the failure of some obvious-seeming attempts to bypass the Löbian obstacle: (a) Via a standard diagonalization, in some theory T construct the fixedpoint statement ψ ↔ φ(x) ∨ T ψ(x) . Then let φ(b i ) ≡ bi → (∅ ∨ G) (so that ψ then states, "Either x implies (∅ ∨ G), or you can prove ψ about x"). Then let A α 's criterion of action be bi ⇒ A α ψ(b i ). Hope: A α reasoning about an agent A β with cj ⇒ A β ψ(c j ) will be able to reason: Hope: Each theory n trusts all lower theories > n, but no theory trusts itself. Thus the corresponding agent A α which starts at T -0 will trust the reasoning of an agent A β which uses T -1, and so on tiling indefinitely. The "problem of immediate reflection" (on considering one's own source code) will remain, but it will at least be possible to carry out an indefinitely extending sequence of agent constructions without the strength of trusted mathematics ever decreasing below the base T . A α Āβ → ∀c j : cj → T ψ(c j ) A α Āβ → ∀c j : cj → ψ(c j ) A α ψ(A β ) Problem: T ψ(x) → ψ(x) Problem: If T is at least as strong as PA it will be straightforward to prove within T -0 that any proof in T -0 can be translated into a proof in T -1 (by increasing all the T -x numbers x by 1 inside the proof). Thus:  T -0 T -0 ⊥ → T -1 ⊥ T -0 T -1 ⊥ → ⊥ T -0 T -0 ⊥ → ⊥ T -0 ⊥ (c) ⇒ A α φ(b i ) ∨ k+1 T φ(b i ) where φ(b i ) ≡ bi → (∅ ∨ G) as before. Let A β ' s criterion of action be the same. Hope: Rather than trying to modify A α 's epistemic beliefs in a way that will circumvent Löb's Theorem, we are relaxing A α 's instrumental decision criterion to permit A α to do b i whenever b i leads to G, provably leads to G, provably provably leads to G, etc. There is no Gödelian prohibition against a more permissive decision criterion; and from outside the system, if we trust the soundness of T +1, we will trust that in all such cases G will be achieved. A α inspecting A β will conclude that whenever A β does c j , there exists a statement with one or more levels of proof-ness that cj → (∅ ∨ G) and thence conclude (A β → ∅)∨ k+1 T A β → (∅∨G) . Furthermore, because of the perfect tiling, A α will trust its own source code upon reflecting on it (since this merely adds another level of proof-ness). Furthermore, A α will behave with perfect naturalism in the sense that any environmental computing system believed to implement T will provide just as much justification-for-action as any internal proof from T 's axioms; there will be no non-naturalistic discontinuity between observations of outputs of A α 's internal subroutines, and observations of the outputs of environmental systems which A α believes to have isomorphic properties to its beliefs about its internal subroutines. Problem: The formula k+1 T φ(b i ) must actually be written as ∃k : k+1 T φ(b i ) . Then when A α considers A β taking an action c j , A α can conclude only that it must have been the case that T ∃k : k+1 At this point the mathematician already accustomed to seeing such systems stratify themselves, or else collapse into inconsistency, may be wondering whether it should even be normatively possible to bypass the Löbian obstacle. Yet it seems intuitively like it ought to be possible to have an agent that "acts as if" Zermelo-Fraenkel set theory is true, including constructing other agents which "act as if" ZF; this does not seem like an instrumentally incoherent way for a rational agent to behave. We have presented the failed hopes above in order to better illustrate some of the further desiderata of self-modifying agents which we could ask from a solution to the Löbian obstacle: Indefinitely tiling trust: For any system T (e.g. ZF) it should be possible to characterize an infinitely descending sequence of agents constructing agents (equivalently an indefinitely self-modifying agent) whose level of trusted mathematics never drops below T . Reflectively coherent quantified belief: The agent will not encounter any context in which it believes ∀x : agent φ(x) yet cannot bring itself to believe ∀x : φ(x). Reflective trust: An agent reflecting on its beliefs about the program P composing itself, should not find that it distrusts P's actions or that it rejects P as an acceptable program; the agent's beliefs about its own construction should seem to license the beliefs it possesses, or at least license the performance of the actions which P has selected.  20  Naturalistic trust: If an agent believes that an environmental system isomorphic to one of its own subroutines has yielded an output, this belief should license the same sort of actions as would be licensed by the corresponding output by the corresponding internal subroutine. If you are "acting as if" PA is true, then seeing an environmental theorem-prover that you believe to implement PA flashing "Proven: 2 + 2 = 4!" should yield the same sort of epistemic updates and instrumental decisions as when your internal PA-prover signals that "2 + 2 = 4" is a theorem. At present we can partially resolve the Löbian obstacle by technical means to the extent of obtaining the first and second desiderata, but not yet the third and fourth desiderata, nor yet by fundamental rather than technical means.
 Partial solutions to the Löbian obstacle 4.1 Infinitely descending soundness schemas The fatal flaw in the previously presented sequence of theories T -0, T -1, T -2... with T -n T ∪ ∀φ : (∀x : T -(n+1) φ(x) → φ(x)) is that any proof in T -0 provably mapped onto a proof in T -1 and hence T -0 T -0 ⊥ → T -1 ⊥ . 20. In other words: A self-modifying AI at the level of PA+2 which obtains a glimpse at its own source code should not immediately panic and self-modify to only believe theorems at the level of PA+1, followed by an additional panic bringing it down to the level of PA... This flaw is repairable. Given a base theory T , let ψ(0), ψ(1), ψ(2)... be a sequence of statements such that T ψ(0), T ψ(1), ... but T (∀n : ψ(n)). E.g., suppose the theory ZF is much stronger than T (we shall soon define what sense of "much stronger" is required; if T = PA then ZF will be much stronger in the appropriate sense). Then ψ(n) can state "n is not the Gödel number of a proof of a contradiction in ZF ": ψ(n) ≡ ¬Bew ZF (n, ⊥ ). (4.1) Since Bew ZF is a ∆ 0 formula, the truth of any particular ψ(n) is provable in any T as strong as PA, but proving ∀n : ψ(n) would require T = PA proving the consistency of Zermelo-Fraenkel set theory, which should not happen. Then let: T -n T ∪ ∀φ : ψ(n) → (∀x : T -(n+1) φ(x) →φ(x)). (4.2) Since every theory holds ψ(n) as a precondition for trusting the next lower theory in the infinite descending sequence, and the base theory T cannot prove ∀n : ψ(n), it will be impossible for the base theory T to prove that a proof in T -n always translates into a proof in T -(n+1).  21  We shall now show that T -0 is consistent, by proving that (T -0 ⊥) → (∀n : ψ(n)) within a sound theory T + which is too weak to prove ∀n : ψ(n): Without loss of generality let T be cast in a form where each step of a T -proof is either an axiom or a modus ponens inference step of the form Γ, Γ→A A. Let T RU E T be a truth predicate over formulas in the language of T .  22  Let Ax(T RU E T ) be a set of axioms for standard reasoning about the syntax of the quoted arguments to T RU E T , including ∀ Γ : ∀ ∆ : T RU E T Γ → ∆ → (¬T RU E T Γ ∨T RU E T ∆ ) and (∀n : T RU E T φ(n) ) ↔ T RU E T ∀n : φ(n) . Then let T + equal the base theory T (in this case PA) augmented by the truth predicate T RU E T , the soundness of T , and the axioms Ax(T RU E T ): T + T ∪ ∀φ : ∀x : T RU E T φ(x) ↔ φ(x) (4.3) ∪ ∀ φ : T φ → T RU E T φ ∪ Ax(T RU E T ) (Note that T + should be sound on the standard numbers N because every axiom of T + is true within N , and that if T = PA then T + should be too weak to prove Con(ZF).) The following reasoning can be carried out within T + . Suppose T -0 ⊥ . Then there exists some proof p with Bew T -0 (p, ⊥). Consider the 'first bad line' 21. Consider a sequence of theories Ṫ -n where T = PA and where ψ(n) is false beyond some upper bound m, e.g. ψ(n) ⇐⇒ n < m. Then the waterfall of trust descends only a finite length before reaching bottom, and the induction argument for Ṫ -n φ → Ṫ -(n+1) φ fails because proofs in Ṫ -m−1 cannot translate into Ṫ -m. Indeed, Ṫ -0 is just equivalent to PA+m, Ṫ -1 is equivalent to PA+(m−1), etc. with Ṫ -m + ∼ = PA. Since T cannot prove ∀n : ψ(n), it cannot prove the waterfall is not finite, and hence cannot prove the induction argument on Ṫ -n φ → Ṫ -(n+1) φ . 22. In accordance with Tarski's theorem, the predicate T RU E T can only talk about formulas in the base theory T and not in the augmented theory T +T RU E T . of p, the first step of p asserting Γ where ¬T RU E T Γ . The first possibility is that this line is an axiom of the base theory T but all statements provable in T are T RU E T . The second possibility is that this line is an inference of the form Γ, (Γ→A) A but for this line to be ¬T RU E T and all earlier lines to be T RU E T requires T RU E T Γ ∧ T RU E T Γ→A ∧ ¬T RU E T A which contradicts Ax(T RU E T ). Thus the first bad line must invoke one of the special axioms in T -0 of the form ψ(0) → (∀x : T -1 φ(x) →φ(x)). If this axiom is ¬T RU E T then ψ(0) must be T RU E T and there must exist some formula φ and number x such that T -1 φ(x) is T RU E T and φ (x) is ¬T RU E T . Thus there must exist a bad proof in T -1 and this bad proof must also have a first bad line. By induction there must exist bad proofs in every T -n and ψ(n) must be true for all n. Therefore, T + ( T -0 ⊥ ) → (∀n : ψ(n)). (4.4) If there exists a proof of a contradiction within T -0, this fact will be provable in T + by exhibiting the proof. But T + is sound and too weak to prove ∀n : ψ(n). Therefore T -0 is consistent. The corresponding agents A α are then constructed in the obvious fashion. Remarks. Although T -0 is slightly more powerful than T -1 in the sense that T -0 can prove certain exact theorems which T -1 cannot, the proof-theoretic ordinal of every T -n should be the same, and equal to the limit of the prooftheoretic ordinals of T , T +1, T +2, ... and less than the proof-theoretic ordinal of T +ω, since any T -n can invoke any finite number of layers of a soundness schema over T , but cannot invoke quantified trust in an infinite number of layers as does T +ω. We thus answer in the affirmative "Can a consistent theory verify the soundness of another theory with the same proof-theoretic ordinal?" and "Can there be an indefinitely tiling sequence of agents whose trust never falls below a base theory?" T -n also provides a solution to the challenge of rational coherence posed by  Weaver (2005) , what we termed the desideratum of reflectively coherent quantified belief: An agent constructed around T -n will not find itself saying "For every n, I believe φ (n) is true" when it cannot say "I believe that for every n, φ (n) is true." Through longer and longer finite proofs, T -0 can prove the well-ordering of any ordinal notation provable in T , T +1, T +2 ... but T -0 does not know that it proves every ordinal notation in this series -T -0 must observe the proof to know what it proves. This is not to say that T -0 is a reasonable representation of a rational agent's state of mind. However, besides offering a constructive example of an agent which technically meets Weaver's desideratum, it suggests that a coherent rational agent might be able to verify increasingly recursive ordinal notations via increasing amounts of thought, but never know the limit of what it will accept; and this agent may be able to construct offspring that likewise verify increasing ordinal notations in an equally powerful series and likewise do not know the limit of what they will accept.  23  Disadvantage. Since T -0 is consistent it must have a model, and since all axioms of T are axioms of T -0 this model must also be a model of T . However we may still worry that, e.g., if T is PA then T -0 may have only nonstandard models of arithmetic; perhaps T -0 is not sound on the standard numbers N . This fear is well-founded and in particular T -0 ∃n : ¬ψ(n) via: T -0 (∀n : ψ(n)) → T -0 (∀n : ψ(n))→⊥ → T -1 (∀n : ψ(n))→⊥ T -0 (∀n : ψ(n)) → T -0 (∀n : ψ(n))→⊥ → ((∀n : ψ(n))→⊥) T -0 T -0 (∀n : ψ(n))→⊥ → ((∀n : ψ(n))→⊥) T -0 (∀n : ψ(n))→⊥ (4.5) One might perhaps argue that a belief that ZF is inconsistent is not too troubling, in the sense that any physical situation in which this belief gets an agent into trouble ought to correspond to a physical situation that demonstrates ZF to be consistent. Nonetheless we would like our agents to be able to have beliefs with a standard model. Otherwise the agent will falsely believe that a Turing machine seeking a proof of contradiction in ZF will halt; and this false belief further implies that the agent falsely believes that its sequence of offspring will inevitably come to a halt after some unknown finite time. This seems sufficient to exclude the T -n family from direct consideration as the basis of a sufficiently advanced self-modifying agent.
 Parametric polymorphism Let T be a theory with models including the standard numbers N , that is N |= T (N semantically entails T ). Benja Fallenstein's "parametric polymorphism approach" 24 augments the language of T with an extra term κ which, from outside the system, is intended to refer to any natural number in N . T κ then contains a self-referential axiom schema asserting that if a statement φ is provable in T κ and the constant κ happens to be greater than 0, then φ with all instances of κ replaced by κ-1 is true: T κ T ∪ ∀φ : (κ > 0) → ∀x : Tκ φ(x) → φ(x)[κ\u03ba-1]. (4.6) not meant to be suitable, as they stand, to rational agents / sufficiently advanced self-modifying machine intelligences, which would e.g. presumably be probabilistic boundedly-rational agents rather than theorem provers, etc. The idea is rather to expose foreseeable difficulties of reflection for self-modifying agents and to some extent offer constructive demonstrations that these difficulties are solvable, even if the solution is technical rather than fundamental, thus hopefully moving us toward an eventually satisfactory theory of the reflectively coherent, boundedly-rational probabilistic agent, even if that later theory is constructed along quite different lines than the A α schema, as it almost certainly must be. 24. This workaround for the Löbian obstacle was initially developed by Benja Fallenstein (independently of Herreshoff's infinite descent above) in response to the informal challenge posed in Yudkowsky's conference presentation of  Yudkowsky (2011) , and initially posted to  Fallenstein (2012) . It was further adapted to the A α formalism shown here during Fallenstein's attendance at the April 2013 MIRI Workshop on Logic, Probability, and Reflection with some commentary by other workshop attendees. For the origin of the term "parametric polymorphism" see  Strachey (1967) . We shall prove that T κ is sound when κ is interpreted as any number in N , and then present an infinite sequence of agents which prove their offspring "safe for κ steps". Since κ can be interpreted as any number, from outside the system we conclude that such agents are safe for any number of steps. The proof is by induction on models {N , κ=n} of T κ . For the base case {N , κ=0} |= T κ observe that if κ=0 then the antecedent of every extra axiom is false and so the extra axiom schema is trivially true. For the induction step assume {N , κ=n} |= T κ . Using this assumption we shall demonstrate in three steps that, for every x ∈ N : {N , κ=n+1} |= Tκ φ(x) ⇒ {N , κ=n} |= Tκ φ(x) (4.7) {N , κ=n} |= Tκ φ(x) ⇒ {N , κ=n} |= φ(x) (4.8) {N , κ=n} |= φ(x) ⇒ {N , κ=n+1} |= φ(x)[κ\u03ba-1] (4.9) (4.9) is obvious. For (4.7) observe that the set of proofs in T κ and the values of Tκ are unchanged from {N , κ=n} to {N , κ=n+1} since the provability formula Bew Tκ makes no mention of κ. 25 Thus {N , κ=n+1} |= Tκ ψ ⇒ {N , κ=n} |= Tκ ψ and we can simply write N |= Tκ ψ . To extend to formulas free in one variable φ(x), observe that since we are dealing with a natural model N of T κ , for every object x there exists a corresponding finite term c of the language (SSS...0 with x repetitions of the successor function) and thus our proof extends via ψ ≡ φ(c). For (4.8) observe that since we are dealing with a natural model N of T κ , only natural proofs exist and the induction step {N , κ=n} |= T κ gives us directly that (N |= Tκ ψ ) ⇒ ({N , κ=n} |= ψ).  26  We extend over formulas free in one variable as above: Since the domain is the natural numbers N , for every object x in the universe there will exist a term of the language c such that (ψ ≡ φ(c)) ⇐⇒ φ(x). Therefore {N , κ=n+1} |= ∀x : Tκ φ(x) → φ(x). This completes the induction step and thus: ∀n : {N , κ=n} |= T κ (4.10) The corresponding agents must be modified to represent the idea of "safe for a number of steps." We will introduce a time t on which any given action 25. Even in models where κ > 0, there is no proof that κ > 0, so the class of proofs which goes through is unchanged. The formula Bew Tκ (p, ψ ) will contain only formulas dealing with the Gödel-numbers of proofs in T κ , and these Gödel-numbers themselves will contain no instances of κ, so which standard numbers are the Gödel numbers of T κ -proofs does not change from model to model. 26. We are not obtaining {N , κ=n} |= ( Tκ ψ → ψ) from the assumption that the axiom schema was true in the previous n, because perhaps κ = 0 in the previous n and the axiom schema was vacuously true. Rather since the model is N and there are no nonstandard proofs, any time there exists a Gödel number of a proof of an implication in classical logic Γ → ∆, it will be true across all models m that m |= Γ ⇒ m |= ∆. By the induction postulate we have directly that {N , κ=n} is a model of T κ . Then any statement ψ syntactically derivable from T κ by a standard proof in classical logic, will also have {N , κ=n} as a model. occurs and assume that if A 1 constructs A 0 at t then A 0 will not act until at least time t+1 or later, and that this is apparent to A 1 from inspection of A 0 . 27 Thus if A 0 acts at time t+1 or later, and takes actions "safe" for an additional n ticks beyond that until (t+1)+n, from A 1 's perspective constructing A 0 at time t is safe until t+(n+1) ticks. Let b i,t be the proposition that b i is done at time t, let Ā0 t be the proposition that A 0 is built at t, let ∅ t be the proposition that ∅ is true at t, and let G t be the proposition that the goal has been achieved at t. If no action is taken before t then ∅ t . When quantifying over acts and times we shall implicitly demand b i ∈ Acts 0 , A 0 i ∈ Acts 1 and t ∈ T ime. Then with T κ as a base theory A 1 's reasoning proceeds: Ā0 t ⇒ A 1 Ā0 t → (∀u : u ≤ t + κ → (∅ u ∨ G u )) (A 1 decisions) (4.11) b i,t ⇒ A 0 b i,t → (∀u : u ≤ t + κ → (∅ u ∨ G u )) (A 0 decisions) (4.12) A 1 Ā0 t t → (∀b i : ∀v : b i,v → (A 1 knows 4.12) 0 b i,v → ∀u : u ≤ v + κ→(∅ u ∨ G u ) ) (4.13) A 1 (κ > 0) → (∀b i : ∀t : [ 0 → ∀u ≤ t + κ : (∅ u ∨ G u ) (soundness schema for T κ ) → (→ ∀v ≤ t + κ − 1 : (∅ v ∨ G v ))]) (4.14) ∀t : (∀b i : ∀u ≤ t : ¬ b i,u ) ⇒ (∀u ≤ t : ∅ u ) (No action ⇒ ∅ t ) (4.15) A 1 ∀t : ((∀b i : ∀u ≤ t : ¬ b i,u ) → (∀u ≤ t : ∅ u )) (A 1 knows 4.15) (4.16) Ā0 t ⇒ ∀bi : ∀u ≤ t : ¬ b i,u (no A 0 acts before t+1) (4.17) A 1 Ā0 t → (∀b i : ∀u ≤ t : ¬ b i,u ) (A 1 knows 4.17) (4.18) A 1 Ā0 t → (∀u ≤ t : ∅ u ) (via 4.18) (4.19) A 1 Ā0 t → (∀b i : ∀u : b i,u →(u > t)) (via 4.18) (4.20) A 1 Ā0 t → (∀b i : ∀u : b i,u → (u + κ − 1 ≥ t + κ)) (via 4.20) (4.21) A 1 (κ = 0) ∨ (κ > 0) (4.22) A 1 Ā0 t → (κ = 0) → (∀u ≤ t + K : ∅ u ) (via 4.19) (4.23) A 1 Ā0 t → Ā0 t → (κ > 0) → (∀b i : ∀u : b i,u → (∀v ≤ u + κ : (∅ v ∨ G v ))) (4.13 and 4.14 and 4.21) (4.24) A 1 Ā0 t → Ā0 t → (κ > 0) → [(∀b i : ∀u ≤ t + κ : ¬ b i,u ) (via 4.16) → (∀u ≤ t + κ : ∅ u )] (4.25) A 1 Ā0 t → ∀u ≤ t + κ : (∅ u ∨ G u ) (4.26) Remarks: Fallenstein's T κ demonstrates indefinite tiling without loss of mathematical strength using a series of agents employing syntactically identical theories guaranteed to be sound on standard models. It does not address reflectively coherent quantified belief. Despite the syntactic equivalence of successive theories, A 1 may fail to act upon a theorem proven by an environmental system physically identical to A 1 (or at least if A 1 fulfills the desideratum of naturalistic trust, this is not obvious to us). The core of the approach is that as time increases the agents are reasoning about smaller domains of future time; these domains are known to us outside the system to be effectively infinite (because κ could mean any natural number) but the agents within the system are reasoning as though the domains of future time have some unknown finite bound. Thus the increasing "time" counter in the T κ agents is playing the same metaphorical role as the increasing n in each successive agent in the T -n family. The self-verifying systems of  Willard (2001)  have the property that multiplication, addition, and successor cannot be proven to be total functions (being replaced by equivalent relations of division, subtraction, and predecessor, allowing roughly equivalent arithmetic reasoning).  Willard (2001)  shows that such a theory, which has proven a number n to exist, can verify its own soundness over the numbers up to n − 1-since "successor" cannot be proven total, the existence of a number n does not imply the existence of a number n + 1 and the exact Löb formula T T φ →φ never materializes. We cannot yet see how to adopt Willard's self-verifying theories exactly to the A α formalism, since reasoning about ever-smaller numbers would again introduce a "telomere" or shortening fuse. However the spirit of T κ clearly bears some similarities to Willard's self-verifying systems, and it is possible that some way of formalizing the "observation" of an increasing time t whose existence was not previously provable inside the axiom system, might allow Willard's self-verifying systems to be used along similar lines to T κ . It is difficult to see what "believing in" T κ could correspond to in terms of the epistemic state of a rational agent. We believe from outside the system that κ's intended interpretation is "any natural number," but this realization is apparently forbidden to the agent to whom κ refers to some specific finite number about which nothing is known except that it plays a vital role in the agent's goal system. 28 This seems like an odd mental state for a rational agent. It might perhaps be interpretable as an agent that instrumentally desires to take an action, in the real world, only when that action would be safe for κ steps across every model of a certain theory T κ ; but this still seems odd. Thus we would again not offer T κ as a realistic candidate for the mental state of a coherent self-modifying agent; rather T κ is being exhibited as a technical possibility proof for indefinite tiling over syntactically identical theories with a standard model.
 Probabilistic reflection Tarski  ([1935] 1983)  showed that no formal system T can contain its own truth predicate T RU E T because this would permit the construction of fixedpoint formula G with T G ↔ ¬T RU E T G . In  Christiano et al. (2013)  we show that a system can represent its own probability predicate to within epsilon: 29 a < P(φ) < b ⇒ P(a < p φ < b) = 1 (4.27) a ≤ P(φ) ≤ b ⇐ P(a ≤ p φ ≤ b) > 0 (4.28) where p φ is a function symbol within the language which acts on quoted formulas (including quoted formulas containing p) and P(φ) is a probability measure over models of the theory which assigns a probability ∈ [0, 1] to any formula φ of the language. The paradoxical equivalence p φ = P(φ) is avoided by the open intervals in (4.27): p φ can be known by the system to occupy any open interval around its true value P(φ), but not (in general) known to take on a point value. The existence of such a fixed-point of P() can be demonstrated via the Kakutani fixed-point theorem; see  Christiano et al. (2013)  for details. To see how the probabilistic reflection principle plays out, consider the almost-paradoxical statement: "You assign me probability less than 30%." If you disbelieve this statement, it is true; if you believe the statement, it is false. If you assign it 30% probability, it is false; if you assign it 29% probability, it is true. Probabilistic reflection would resolve the fixed-point formula H ↔ (p H < 0.3) to P(H) = 0.3, but the system's reflective knowledge about open intervals containing p H would be such as to put 30% of the probability mass P() on values of p H infinitesimally smaller than 0.3 and 70% of 28. During the April 2013 workshop, κ was sometimes referred to as "the number of ineffable mystery." 29. This approach was invented by Paul Christiano and refined at the November 2012 MIRI Workshop on Logic, Reflection and Probability with Mihaly Barasz, Marcello Herreshoff and Eliezer Yudkowsky. A draft of the full paper is available at http://intelligence.org/ wp-content/uploads/2013/03/Christiano-et-al-Naturalistic-reflection-early-draft. pdf and see also commentary at http://lesswrong.com/lw/h1k/reflection_in_ probabilistic_logic/ and http://johncarlosbaez.wordpress.com/2013/03/31/ probability-theory-and-the-undefinability-of-truth/. its probability mass P() on values of p H infinitesimally greater than 0.3. 30 Hence the system would assign probability 1 to any statement (a < p H < b) with (a < 0.3 < b). If you are told "You assign probability less than 30% to this statement H" and then asked "Is your belief in H greater than 0.2999 and less than 0.3001?" you will reply with a definite "Yes!" Consider a rational, probabilistic bounded agent. As a normative desideratum, any trust this agent has its offspring (equivalently: modified future self) must avoid disintegrating in the presence of arbitrarily tiny epsilon noise, because boundedly rational agents must always consider some tiny finite probability of multiple transistor errors, previously unknown physical laws supervening, etc. For some arbitrarily tiny the probabilistic reflection principle should be able to trust that its own judgments are correct to within that , and for smallenough this should be well below the noise level associated with theoretically possible multiple transistor errors etc. Thus if a probabilistic A β calculates that the expected utility of a certain action is 34.721, but due to some reflective noise this might be off by 0.0002, and yet the parent A α is able to trust A β anyway, then adding -noise from the probabilistic reflection principle should not matter. We do not yet have any version of the A α formalism which goes through for a probabilistic reflection principle; our understanding of probabilistic agents self-modifying is still in a very crude stage. Also we are not yet sure what kind of quantified knowledge of the reflection principle can permissibly appear within the system. Work on this avenue is still in very basic stages. However, probabilistic reflection is a fundamental change that challenges many different Gödelian limitations simultaneously, and hence one of the more promising avenues for producing a satisfactory and reflectively coherent description of a self-modifying agent. Christiano also suggests that this probabilistic formalism may be suitable for updating (Bayesian conditioning) on the observation of new mathematical facts, possibly allowing a more natural version of "Not knowing in advance the limits of the ordinals you will create" as discussed in section 4.1.
 Desiderata of self-modifying agents The Löbian obstacle is encountered in the first place due to the A α schema implicitly attempting to simultaneously satisfy multiple desiderata of self-modifying agents, which will be useful to list out explicitly before continuing on to sensors and partially known environments, and making a preliminary attempt at probability and expected utility.
 The Vingean principle An agent building a successor (equivalently: a self-modifying agent creating the next generation of its code) should not need to know the successor's exact actions and thoughts in advance. Suppose on the contrary that A 1 knew in advance that A 0 would choose some known action b x based on some particular proof of the theorem b x → G. This would avoid the need for a special ∅ outcome since A 1 would know at compile time that A 0 would choose some particular action and act. It would also entirely avoid the Löbian obstacle since A 1 could directly verify whichever theorem A 0 would prove, using A 0 's proof, so long as Axm 1 was at least equally powerful with Axm 0 . 31 However, this implies that A 0 is so much computationally weaker than A 1 that A 1 can simulate out A 0 's processes far in advance, at "compile time." If you are constructing a system cognitively smarter than yourself (or self-improving to a cognitively superior version) then you should not be able to foresee at compile time exactly which solutions will be generated, since you are not presently that smart.  32  We would attribute this principle to an observation by  Vinge (1984) .  33  Worse, once we introduce sensors over unknown states of the environment (section 6), for the parent to consider exact responses of the offspring at compile time, would require that the parent be computationally stronger than the offspring by an exponential factor of the offspring's sensor bandwidth.  34  This implies that the offspring must diminish vastly in computing power 31. The forbidden schema T T φ → φ does not allow T to conclude φ from the assumption that some unknown, unspecified proof of φ exists. If you present an implementation of T with an actual specific proof in T of φ, it can of course conclude φ and then T ψ → φ will be a theorem for every ψ. 32. Consider Deep Blue, the chess-playing system which beat Garry Kasparov for the title of world chess champion. For the programmers to foresee Deep Blue's exact moves using their own cognitive resources, would imply that they could play chess at least as well as Deep Blue-they could just move wherever they predicted Deep Blue would move "in their shoes." Building a chessplayer better than any human grandmaster, necessarily implies building a chessplayer whose exact moves human grandmasters cannot predict using their own cognitive resources. 33. Before the term became expanded beyond all recognition, the origin of the term "Singularity" as a futurological concept was Vernor Vinge observing that a science-fiction writer trying to extrapolate the technological progress of society would eventually reach a point where their model predicted the technological creation of smarter-than-current-human intelligence (whether by, e.g., genetic engineering, brain-computer interfaces, or AI); and that this introduced a qualitatively greater difficulty in writing stories set past that point, because you were trying to write about beings putatively much smater than the author. The term "Singularity" referred to the resulting breakdown in futurological models, by analogy with how 1980s-era physical theories would predict the formation of a black hole but then break down when it came to describing what was inside. Thus the original "Singularity" was an epistemic difficulty in the minds of forecasters, not an actual future event. We think this is a conceptually important point, and would like to see this isolated thesis detached and rescued from the word "Singularity" and its ever-broadening connotations. 34. Since Garry Kasparov's actions were not known in advance, for the programmers to concretely foresee and judge the wisdom of Deep Blue's exact conditional replies at compile time would require the programmers to precompute Deep Blue's exponentially vast response table to all possible chess plays. or other cognitive resources, which is contrary to our agenda of investigating self-improvement. Thus a satisfactory framework must not require that the parent know the specific future actions and cognitions of the offspring at compile time. For our purposes we cash out the Vingean principle as follows: In the parent's reasoning, the offspring's actions should only appear inside quantifiers. E.g. equation (2.3): A 1 Ā0 → (∀b i ∈ Acts 0 : bi → 0 bi → G ). In a satisfactory schema the parent achieves trust in the offspring (justified belief that the offspring will function so as to achieve the goal G) by generalizing over possible events within the offspring, not by simulating out the offspring's exact action.  35  The Vingean prohibition against exactly predicting a smarter agent's actions, does not prohibit predicting achievement of that agent's goals. Indeed, a strong prediction that the agent's goals will be achieved by its future actions is precisely the substance of our belief that it is smart. In unrealistically crisp and deterministic form, this is what is represented by A 1 's reasoning leading up to the theorem Ā0 → (∅ ∨ G). The logic of the Vingean principle does not forbid us from gaining knowledge about an agent's goal or utility function by inspecting its properties-we cannot know the offspring's exact actions, but we can predict the final consequences of running the agent. This, metaphorically, is what provides A 1 with its motivation to build A 0 in the first place. Hence although we cannot predict the exact actions of a smarter agent, we may be able to predict the consequences of running that agent by inspecting its design, or select among possible consequences by selecting among possible designs. The motivation of the A α schema is to quote this reasoning within a formal agent, via a structure which enables the reasoning to recurse over the subagents constructed; and thereby extend our examination of cases where constructing an agent has predictable consequences, to cases where the initial agent is constructing smarter agents constructing still-smarter agents, or equivalently self-improving. Thus, within this paper, the actions b i ∈ Acts 0 only appear inside quantifiers in any theorem asserted by A 1 .
 The tiling principle The desire for A 1 to build an A 0 that has an architecture as similar as possible to A 1 is motivated by wanting to describe a self-improving agent which 35. E.g., Deep Blue's programmers, by considering the abstract properties of Deep Blue as a program incorporating a search tree over moves and certain machine learning algorithms in its leaf position evaluator, could predict that Deep Blue would try to achieve the consequentialist goal of winning chess games, i.e., select moves which steered the future of the chessboard toward the region of possible futures classified as a win for Deep Blue's side. can make small, local self-improvements. As such, even in the preliminary formalism we have chosen the structure of the agents A α to be as similar to their successors as the Löbian obstacle permits. We have not yet addressed the question of how the agents A α choose candidate actions b α i ∈ Acts α to consider, or how they generate specific proofs of theorems bα i → (∅ ∨ G). Let A α 's proof-suggesting component be termed Σ α ; and let the proof-verifying component be termed ∆ α . Since ∆ α need only check axioms and syntactical relations for validity, its construction is straightforward. Current (2013) algorithms require human guidance in order to prove theorems about large systems.  36  Current AI technology includes small autonomous theorem-proving systems for object-level problems as considered by A 0 and we could construct the corresponding Σ 0 ; but we cannot yet describe an autonomous machine algorithm for Σ 1 whereby A 1 could independently design, and prove theorems about, systems the size of A 0 . However, with human assistance, it is indeed possible to prove theorems about large hardware and software systems. 37 This is reason to believe that designing systems the size of A 0 and proving them correct (relative to a set of beliefs about crisply behaving transistors), does not require Σ 1 to e.g. solve NP-hard problems or call a halting oracle. Human brains cannot plausibly solve the halting problem, yet have designed systems larger than A 0 would be, and proven them correct with machine assistance. This leaves open the question of whether the suggester Σ 1 needing to suggest a design for constructing Σ 0 might introduce new Gödelian obstacles to A 1 Ā0 → (∅∨G). On this score we can reply: So long as Σ 0 can only suggest proofs to ∆ 0 and not directly cause any motor outputs or environmental effects, and A 1 can prove this by inspection of A 0 's code, it is not necessary to prove that Σ 0 has any other property in order for A 1 's proof Ā0 → (∅ ∨ G) to go through. A flawed Σ 0 will result in, at worst, ∅ outcomes as the proofs are rejected by ∆ 0 , and A 1 should be able to readily prove this by inspection. 36. E.g., Intel, to prove theorems about a floating-point multiplication unit on a chip comprised of billions of transistors, must employ human engineers to suggest lemmas to theorem-proving systems. This kind of formal verification is a good idea when your machine has a billion interoperating parts and you can't easily fix it after it leaves the factory. 37.  Kaivola et al. (2009) : Formal verification of arithmetic datapaths has been part of the established methodology for most Intel processor designs over the last years, usually in the role of supplementing more traditional coverage oriented testing activities. For the recent Intel Core i7 design we took a step further and used formal verification as the primary validation vehicle for the core execution cluster, the component responsible for the functional behavior of all microinstructions. We applied symbolic simulation based formal verification techniques for full datapath, control and state validation for the cluster, and dropped coverage driven testing entirely. The project, involving some twenty person years of verification work, is one of the most ambitious formal verification efforts in the hardware industry to date. Our experiences show that under the right circumstances, full formal verification of a design component is a feasible, industrially viable and competitive validation approach. One could even visualize in this way a highly impractical modern implementation of A α agents: Suppose a virtual environment containing objects with crisp, deterministic properties, including enough virtual transistors to compose small arithmetic CPUs. Also introduce some blackbox elements Σ α whose behavior can relate to sources outside the virtual environment, but which within the virtual environment can only affect immediately neighboring or connected elements of the environment-i.e. when a Σ α is built into a larger agent A α as a component, that Σ α can only write bitstrings into a sandboxed internal buffer of A α , and cannot otherwise affect A α or the general environment. Let G be a preferred class of outcomes difficult to achieve (perhaps requiring an agent to press a long series of levers). For some reason this outcome G in the environment is very important to us, but we don't want to work on it ourselves, or don't think ourselves competent to achieve it. We would like to pay some untrusted but competent human agents to take over the job of achieving G within the environment, while avoiding at all costs a disastrous outcome D. Let ∅ be a default set of outcomes maintained so long as some class of environmental elements are not disturbed (e.g. no levers are pressed), such that outcomes within ∅ are outside D. We incorporate a blackbox action-and-proof-suggester Σ α into an agent A α with a criterion of action bα i ⇒ A α bα i → (∅ ∨ G), and connect this Σ α to a human agent outside the system, armed with modern theorem-proving software that incorporates human suggestions. By putting Σ α into the agent A α , we can not only "sanitize" the actions taken within the system by the untrusted human, but also allow the human to (through A α ) construct other agents within the environment, which can in turn construct other agents, all including their own black-boxes Σ α hooked up to human proof-suggesters outside the environment, without ever running the risk that a non-∅ outcome besides G will ever be implemented.  38  On a larger scale, this illustrates that the decomposition of trust within the A α system also tiles its architecture and structure: A system A 1 composed of a trusted verifier ∆ 1 which checks proofs, and an untrusted suggester Σ 1 which may incorporate arbitrary heuristic or probabilistic methods, can build a trusted offspring A 0 with a verifier ∆ 0 proven correct by ∆ 1 and a suggester Σ 0 which incorporates whatever heuristics or probabilistic methods have been suggested by Σ 1 . Indeed, the correctness proof that A 2 used to build A 1 might go through almost unchanged for A 1 building A 0 , and one could perhaps prove that it would be safe to re-verify only the changed portions of the proof. The tiling principle is agents being able to approve the construction of successors similar to themselves, both architecturally and in the reasoning licensing their construction. This corresponds to the desideratum of a self-improving agent being able to make small, local changes to itself with a minimum of imposed overhead, as might occur if any small change required the reworking of 38. By the incorporation of sensors (see section 6) it is also possible for the offspring agents to know facts about the environment not known to the parent; the motivation to construct new agents in an environment like this can be explained by the new agents having better actuators, better sensors, or fast reaction patterns not present in the parents. a large body of cognitive labor to approve. We strongly remark that the Σ α -∆ α system is nowhere near a solution for a stably self-improving bounded probabilistic agent. The crisp separation of an untrusted Σ α and a trusted ∆ α works, not because logical proofs are hard to invent and easy to verify, but rather because logical reasoning is monotonic and thus no further fact known to Σ α but not to ∆ α can prevent the achievement of G by a ∆ α -approved action. When trying to achieve expected utility in a probabilistic environment described by causal models, this monotonicity no longer holds. Thus the Σ α -∆ α decomposition is intended only to illustrate the general idea that decomposition-of-trust can also tile in a structured fashion.
 The naturalistic principle Within the A α formalism the act of of constructing other agents is not treated as a special case, but is just one more sort of action that an agent can take to achieve its goals. A 0 and A 1 both act as consequentialist agents 39 that will take acts in their range Acts α whose predicted consequence is the goal G, without distinction of actions that construct agents. If G is to put a coin into a slot, then using environmental transistors to build a smarter agent that wants the coin inside the slot, is treated no differently, under the A α formalism, from manipulating a series of levers to put the coin inside the slot. There are many reasons we might be tempted to relax this naturalistic principle. Suppose we notice a disturbing quality of the current tiling: An agent A 1 whose desideratum is G (or ∅) can tile to an offspring A 0 whose desideratum is (G ∧ B) where B is an arbitrary "barnacle" on the goal system, some new requirement that A 0 imposes on itself for its actions to achieve besides G. This does not threaten the maintenance of (∅ ∨ G) but does mean that G might not be achieved in some cases where it was otherwise easy or achievable. We required that our offspring require itself to achieve G (in non-∅ cases) but did not require that our offspring require only G, and thus exact reproduction of the goal system-the system of desiderata-is not guaranteed. "You get what you pay for" and if you prove a theorem about G you get G, not "G and nothing else." It would be up to the untrusted component Σ 1 whether A 0 worked to achieve G or (G ∧ B ∧ C ∧ ...), keeping in mind that Σ 1 might have been constructed by another untrusted Σ α and so on. None of this threatens the guarantee on G, but it still seems like a disturbing hidden instability of the agents' goals. (And in the case of expected utility, introducing an arbitrary new additive term into the utility function will always come at a cost to the desiderata previously present, if it shifts any choice that would otherwise have been made in the opposite direction.) 39. These agents use logical implication, e.g. bi → G, to reason about causal consequence, which in standard causal decision theory would be represented by a counterfactual distribution P ( bi G)see  Pearl (2000) . Logical consequence is generally a poor way of approximating causal consequence, but can work in crisp closed deterministic environments where such reasoning can be monotonic. Could we fix this problem by imposing a non-naturalistic requirement that the offspring contain a literal copy of the goal system-perhaps requiring that this part of the offspring be copied literally from the parent's code? To avoid the question of how the decision system knows that an act is an "agent construction" rather than a "normal action," let us pass to self-modifying agents. A naturalistic agent-constructing-agent A α corresponds to a self-improving agent that treats the transistors comprising itself as resources no different from any other potentially-computing resources it finds in the environment; its beliefs about the transistors comprising itself are formed no differently from its beliefs about objects in the environment; and its internal decisions are made under the same consequentialist criterion as its external actions. Then an obvious structure for a non-naturalistic agent would be an agent with a different criterion of action for self-modifying changes (changes to its own program) versus motor actions in the outside environment. Considering some of the new difficulties introduced by this split decision criterion may help to motivate the naturalistic principle. Adding a special case for self-modification (of internal programs on the agent's current hardware) would introduce a discontinuity at the point where the agent considered expanding onto new computing resources obtained from the environment-the rules would need to describe how to buy a CPU from a store, connect it to the current cluster and then begin treating it under the new "internal" special case for hardware that must be manipulated via "self-modification" instead of "environmental manipulation." For more advanced machine intelligences, the rule might need to describe how to arrive at beliefs about the behavior of a new kind of CPU via Bayesian updates, or beliefs about when a CPU has been "connected" to the internal network. (An advanced machine agent using a naturalistic decision criterion would also encounter corresponding cognitive tasks, but they would be treated continuously in terms of expected consequences, rather than needing a special case baked into the fundamental decision criterion.) We might say that these difficulties are being introduced by a kind of Cartesian dualism-the physically continuous universe has been divided into external material stuff versus internal mental stuff, and material objects must be declared to be specially cognitively typed before they can be treated as mental stuff. Or another difficulty: If the self-modification criterion tries to freeze the representation of the goal G so as to require exact reproduction of G and exclude "barnacles," then the self-modification criterion will also have to freeze the corresponding properties of belief representation and decision consideration so as to guarantee that the rest of the system decodes G in the same way. 40 40. If this seems an odd notion to consider, we should mention that in nontechnical treatments of safe self-modifying machine intelligence, we have often seen proposals to "hard-wire" the goal system, often using that exact phrase. Construing the phrase charitably, it could refer to something like the above, in which case a problem is that "hard-wiring" one part of a system requires constraining the global architecture and internal protocols so that all the other parts behave the same way around the "hard-wired" part. Having a "hardwired" function that evaluates G a certain way is no guarantee of Even then the added constraint may end up being ineffectual. Consider that an agent with an added constraint in its decision criterion for "self-modification" of the software on its existing hardware, might naturally tend to build new agents in the environment according to its weaker environmental criterion. For that matter, again using the A α system as metaphor, if the environmental criterion is the weaker bi → G and the internal self-modification criterion is the additionally-restrictive bj → (G ∧ R), this decision criterion would approve an action to construct an external, "environmental" manipulator to reach into the agent and rewrite its code in a way that implied only G and not R. Having a stronger (more restrictive) criterion of action for self-modification, versus environmental action, seems likely to be futile in the long run. Most of the universe is outside rather than inside the AI, and external materials can be used to construct new agents, or to take actions with causal consequences impacting on the "inside" of the agent. That is why we are trying to see stable self-modification in terms of the natural tiling of a consequentialist decision criterion, rather than inventing special programmatic constraints. This leaves open the possibility that a weaker (less restrictive) criterion for "internal" self-modification could play a role in avoiding the Löbian obstacle. However this would introduce all of the above-mentioned difficulties with respect to describing what constitutes a more permissive "internal" action, or how to add new hardware. We could also ask whether the more permissive internal criterion would be just as appropriate for external actions, and if not, whether internal permissivity might give rise to external permissivity. Such a "hack" might be appropriate for self-modifying programs that will never consider, or never autonomously choose, to significantly modify their hardware. But even for those cases, a unified naturalistic decision criterion might avoid many complications in the underlying theory, and hence seems greatly desirable if obtainable. If naturalism is abandoned for a dualistic formalism, the Löbian obstacle can be dodged via the split decision criterion: 41 ψ(act i ) =[externalOK(act i ) → ( ā ct i → G)] ∧ [internalOK(act i ) → ( ā ct i → (∀act j ∈ Acts : ā ct j → ψ(act j ) ))]. Thus it is the simultaneous intersection of the Vingean, tiling, and naturalistic principles which generates the Löbian obstacle. safety if other code sections discard the return values, or introduce a new representation in which the returned values mean something else, etc. A decision criterion over expected global consequences automatically subsumes such details, while a non-naturalistic decision criterion over direct properties of the internal representation brings those details back in. 41. See "failed solutions" in section 3 for why this split decision criterion does not have an obviousto-us naturalistic analogue (we did look).
 Unknown variables and sensors A straightforward idiom for representing an agent which can use sensors to gain additional information about (known) unknowns in the environment would be the relation Sl ⇐ A 0 sl (6.1) representing a scenario where the agent A 0 adds the proposition 's l ' to its pool of logical beliefs only on occasions where the environmental sensor S, possibly a sensor, takes on the state Sl . (This also allows that S may be some variable whose state is sometimes not sensed, in which case A 0 believes nothing about any s l .) Then further axioms in Axm 0 would relate the proposition sl to the state of other represented parts of the environment, Axm 0 sl → χ. This idiom breaks the tiling pattern we used earlier, since A 1 will no longer be able to use the relation A 0 φ ⇒ Axm 0 φ. Even when Sl is true, it will not be entailed by Axm 0 and hence A 1 cannot prove bi → 0 bi → (∅ ∨ G) . Our previous proof relied on everything proven by A 0 being something that A 1 could have proven, but if A 0 has sensors not possessed by A 1 , it will act on premises which A 1 does not know to be true. Allowing this somehow is an obvious requirement for building successors (or self-modifying). If the A α formalism could not adapt to describe offspring possessed of additional factual information unknown to the parents, it would be entirely unsuited, under the Vingean principle, to describing self-improvement. An ugly but direct solution would be to amend A 0 's criterion of action to read: bi ⇒ A 0 (s l ∧ bi ) → (∅ ∨ G), (6.2) where some s l is guaranteed to have been sensed at the time of any action. (If the sensor sometimes does not activate, then the special state s 0 could indicate that the sensor has not obtained any reading, and represent this through the absence of any further environmental beliefs s0 → χ.) A 1 with criterion of action Ā0 j ⇒ A 1 Ā0 j → (∅ ∨ G) would then tile to A 1 via the lemmas: A 1 Ā0 → ∃s l ∈ s : sl (6.3) A 1 Ā0 → ∀s l ∈ s : ∀b i ∈ Acts 0 : (s l ∧ bi ) → 0 (s l ∧ bi ) → (∅ ∨ G) (6.4) A 1 Ā0 → (∀b i ∈ Acts 0 : ¬ bi ) → ∅ (6.5) This idiom technically does not break the Vingean principle, because we do not need to reason by cases on S and all values of s l ∈ s appear inside quantifiers, so multiple sensors can be included in A 0 without requiring exponentially longer chains of reasoning by A 1 . However this idiom stretches the tiling principle because each successive descendant, with additional sensors, must prove larger and larger criteria of action (even in cases where most sensors do not activate). Intuitively, it seems like if we were in the shoes of A 1 building A 1 , we would want to reason that since adding A 0 sl only happens when Sl is "true," it should not disturb our trust in A 1 . Previously we were willing to add Axm 0 to A 0 's belief pool because we thought Axm 0 were all true in some encoding, so A 0 started with only true beliefs. We were willing to program A 0 to use classical logic to make syntactically allowed deductions from its current belief pool, even if A 0 proved some theorems we did not consider concretely in advance (in accordance with the Vingean principle), because we believed the rules of logic were valid in the sense that, starting from true premises about the environment, A 0 's reasoning rules would produce only true conclusions about the environment.  42  Then our trust in the soundness of A 0 should not be disturbed by giving A 0 a sensor which adds new statements sl only when Sl is true in the environment, even if these propositions were not known to us in advance. Set theory is powerful enough to directly formalize this reasoning using standard methods. In particular, ZF can internally represent the notion of semantic entailment X |= φ , "The quoted formula φ is true within the quoted model X." E.g., to quote Peano arithmetic, the model X N would contain several subsets collectively representing the universe of numbers and the relations on the objects in that universe: X N would contain an element containing all objects in the universe of X N (in this case the numbers); an element containing all the ordered pairs for the succession function (e.g., (2, 3) is the pair indicating that the object 3 is the successor of 2); and more elements containing the collections of ordered triplets for the addition and multiplication functions (e.g.,  (3, 5, 8)  in the addition relation shows that 3 + 5 = 8). There then exists a compact formula of ZF asserting that φ encodes a formula that is semantically true of the quoted model X. For example, "1 + 2 = 3" would be encoded as the Gödel number of a statement asserting that (the number related by the successor function to 0, the successor of the successor of 0, and SSS0) form an ordered triplet that is an element of the addition relation. By recursion on the depth of formulas, there is a single compact ZF-formula which represents X |= φ in general and which could be applied to X N |= S0 + SS0 = SSS0 without further tailoring to the formula S0 + SS0 = SSS0 or the quoted model X N . It is then moreover possible to prove within ZF, by induction over the number of steps in a logical proof, that the set of quoted logical tautologies is true of every quoted model-if a formula (e.g., ∀x : ∀y : (x = y) ↔ (x = y) can be proven starting from no premises, it is semantically true in all universes. Thus ZF proves that the syntactic rules of classical logic preserve semantic truth: ZF ∀X : ∀ Γ : ∀ ∆ : taut Γ→∆ → ((X |= Γ ) → (X |= ∆ )). (6.6) This is a single, quantified formula rather than an axiom schema since there is 42. The purpose of logic is to preserve truth, not create truth. Logic doesn't guarantee that our conclusions are true; but logic does guarantee that if our conclusion is false, one of the premises must also be false. no longer any need to dequote the formulae Γ or ∆ in order to assert their truth within X.  43  Let the language L τ ∈ be the language of set theory L ∈ augmented by a new term τ denoting a special set, intended to be a quoted model representing the enivronment within which the agents are embedded. Let ZF τ be ZF with its axiom schemas extended to include formulas involving τ . Let ξ τ be a set of axioms describing the special set τ , such that ξ τ ∼ = Axm 0 -whatever statements the original A 0 treated as axioms about its universe (such as "χ→G"), will be rephrased as statements about τ (such as "τ |= τ χ→G "). Here |= τ is a new semantic entailment formula which maps special quoted terms like G to corresponding special sets located by ξ τ within τ , or corresponding special properties of τ . (E.g. G might represent some particular variable within the model τ having taken on the value of 20.) Note that there does not need to be any special term G added to the language L τ ∈ . Rather, the axioms ξ τ force τ to have a particular structure, and the formula |= τ assigns a special Gödel number and corresponding subformula of |= τ to the term G .  44  Then the centerpiece of A 1 's reasoning about A 0 becomes the central trust lemma: A 1 Ā0 → ∀ φ : (τ |= τ A 0 φ ) → (τ |= τ φ ). (6.7) This is the translation of "A 1 thinks everything A 0 believes is true," or more explicitly "A 1 thinks that everything A 0 believes, as encoded in some representation φ , corresponds to some statement φ which is semantically true about reality itself (τ )." Note that A 1 is now explicitly maintaining some compact mapping between its language φ for describing reality (τ ) and its model of A 0 's language φ -in other words, A 1 has an explicit correspondence theory of truth over A 0 's beliefs and the territory τ , as represented in some compact encoding relating A 1 's map of A 0 's map and A 1 's map of the territory τ . (6.7) is intended to mirror (in highly stripped-down logical form) something like what a human programmer would do intuitively in constructing a theoremprover to manipulate tokens supposedly "representing" the environment-the human implicitly maintains a compact mapping between the human's model of how the program treats tokens like "Customer#12," and the human's model of some actual person named Fred. (Omitting any of these levels of indirection, or 43. ZF cannot thereby prove its own consistency because ZF cannot prove the existence of any set Y such that Y |= ZF . Likewise the entailment formula |= does not constitute a forbidden truth predicate because |= can only denote semantic truth within an individual set X, and there is no corresponding formula to denote semantic truth within the entire set-theoretic universe. 44. Having all our propositions talk about a unique term τ is intended to represent the idea of achieving things in some particular "actual reality," not just an implication being true in some particular set x which has the properties ξ(x) but is not the actual reality we wish to speak of. We could possibly do without the special term τ and let our set theory be the standard ZF by having the axioms ξ τ say ∃x : ξ(x) and replacing all statements of the form τ |= τ φ with statements of the form ∀x : ξ(x) → (x |= τ φ ), so that the conclusion is forced to be true of the reality-set as well, so long as the reality-set has the properties ξ. But for reasons of notational simplicity, we would rather just add the term τ to the language L τ ∈ . committing the mind-projection fallacy  (Jaynes 2003)  and speaking as if some ethereal link was actually connecting "Customer#12" and Fred, can result in truly vast amounts of confusion.) The human then configures the program to maintain the human's expectation of correspondence between the two, i.e., the human makes sure that the token "Customer#12" behaves in such fashion that its "money" property is expected to behave like the human's expectations about Fred's payments. The maintained correspondence is between epistemic objects-a map of a map, and a map of the territory-so that a compact mental transformation can relate the two.  45  Similarly an agent A 1 with Axm 1 = ZF τ ∪ ξ τ and criterion of action Ā0 j ⇒ A 1 (τ |= Ā0 j ) → (τ |= (∅ ∨ G) ), (6.8) which had established the central trust lemma (6.7), could approve the construction of an agent A 0 which held the object-level beliefs represented by Axm 0 -an agent which, like previously described agents A 0 , did not use set theory or represent an explicit environment-object τ , but instead was programmed to reason about a proposition G that could be the consequences of its acts b i as before. We could similarly expect that A 1 would establish its central trust lemma (6.7) by induction on properties of the design of A 0 . The base case is the truth of every axiom in Axm 0 constituting A 0 's belief pool when it starts up. The induction step is the validity of classical logic's syntax, as proven within ZF (or ZF τ ) to produce only semantically true conclusions from semantically true premises on each succeeding time-step of induction. Then this central trust lemma is not disturbed by adding sensors that operate as in (6.2) or, in this scheme: A 1 (τ |= τ Ā0 ) → ∀ s l : (τ |= τ A 0 sl ) → (τ |= τ Sl ). (6.9) Imagining the induction to be over time steps taking places at times u after A 0 's construction at t, adding this type of sensor does not invalidate the induction step that if all A 0 's beliefs are true at time u, they will be true at time u+1-supposing all of A 0 's new beliefs to be added either by syntactically valid classical deduction from previous beliefs, or by sensors which A 1 believes to operate in a sound fashion relative to A 1 's correspondence theory of truth φ ↔ φ , on any given time step. Some further elaboration is then required to use the same schema to describe the agent A 2 which creates A 1 which creates A 0 : A 0 itself may reason about the environment using some simple axioms, first-order logic, and perhaps a few sensors, a la Newell's GPS  (Newell, Shaw, and Simon 1959) . A 1 , to reason about A 0 in a sufficiently general fashion, deploys set theory in order to reason about a physical universe τ , a language φ for describing physical reality, and a truth-condition |= τ which determines whether such statements are (in A 1 's beliefs) "true" or "false," and an agent τ |= τ A 0 with beliefs φ which A 1 believes to correspond to propositions φ which are actually "true." (Representing semantic entailment in a fully quantified way, and proving in a quantified way that logical tautologies are true in all models, is why we had to invoke the big guns of ZF.) Then A 2 , to reason about A 1 's reasoning, needs to trust that A 1 , in reasoning about set theory, is reasoning about valid subject matter. E.g., since A 1 is reasoning within ZF τ , A 2 to trust A 1 must at least believe that ZF is consistent.  46  Then clearly A 2 must be using some theory more powerful than ZF τ ∪ ξ τ , which can prove the soundness of ZF τ ∪ ξ τ . An obvious idiom is for A 2 to reason that A 1 's mental universe corresponds to some object in A 2 's universe, something that A 1 can be said to be reasoning about:  (6.10)  which states that there exists some set s which is a model of Axm 1 = ZF τ ∪ξ τ . 47 This makes Axm 2 as strong as ZF plus the assertion that there exists an inaccessible cardinal (a single set which is large enough to be a model of a set-theoretic universe obeying the axioms of ZF). Since ZF can prove Gödel's Completeness Theorem that every syntactically consistent theory has a model, Axm 2 is as strong as ZF+1. Axm 2 = ZF τ ∪ ∃s : s |= τ ZF τ ∪ ξ τ , From another perspective: A 1 's mental universe includes not only τ but a larger mathematical universe of set-theoretic objects-e.g., this mathematical universe, not the physical universe τ , is what includes the numbers (least ordinals closed under succession) which determine whether "a proof of φ exists". Then A 2 must be strong enough mathematically to reason about something (a set, a model) which is large enough to provide a correspondence theory of truth over all of A 1 's reasoning, including A 1 's reasoning about set theory. A 2 must believe in an object that provides, not just a physical environment, but a mathematical universe, for A 1 's reasoning to be sound over. Similarly, another agent to create A 2 must believe that there exists a set that models Axm 2 , containing a set that models Axm 1 , containing τ that models Axm 0 ∼ = ξ τ , with the corresponding theory having the strength of ZF+2; and we arrive at ordinal tiling as before. Remarks. For a set theory to believe in the existence of an infinite descending sequence of sets containing sets containing sets, etc., would violate the Axiom of Foundation (Axiom of Regularity). Hence an analog of the infinitely descending soundness schema T -0 seems like it would be even harder to repair. Possibly the fact that infinite models are being encoded as sets may allow for an infinite series of encodings (e.g., all the numbers can be encoded as even numbers, then 46. Or perhaps, "ZF is consistent if 0 is not the Gödel number of a proof that (ZF plus a large cardinal axiom) is inconsistent," or "ZF is consistent if κ is greater than 0," but we omit such complications here and assume that A 2 , A 1 , and A 0 fall into a standard decreasing-ordinal schema. 47. Note that since semantic entailment is a quantifed formula, infinite axiom collections, such as axiom schema, can be semantically entailed without problems. all the numbers can be encoded as even numbers encoded as even numbers and so on indefinitely). We see no obvious obstacles to Fallenstein's parametric polymorphism approach T κ being adapted to ZF τ proving safety for κ steps, but have not yet written out a complete proof. We would be interested in any simplification of this scheme that reasons about a correspondence theory of truth over the offspring agents without resorting to set theory, or that uses a set theory substantially less powerful than ZF.
 Probability and expected utility At present our attempts to tile probabilistic reasoning are in very preliminary stages. Expressing trust of an agent in an offspring's probabilistic reasoning introduces new complications, most of which remain unresolved. The expectation of utility E[U ], conditional on an action b i being performed, is often defined in terms of an outcome variable o k ∈ Outcome and probability distribution P as: 48 E[U ] o k ∈Outcome U (o k )P (o k | bi ). (7.1) The most straightforward analogue of logical satisficing, as previously used in A α , is expected utility satisficing. (Maximization would introduce even more difficulties; see section 8 below.) E.g., A 0 might act only when it expects its action to lead to expected utility greater than (arbitrarily) $20, and we might be tempted to write this as the problematic formula: bi ⇒ A 0 E[U | bi ] > $20 (7.2) As we remarked in section 5.2, the introduction of expected utility and its conditioning formula lead to basic difficulties with the A α formalism because probabilistic reasoning is not monotonic. If we prove by logic that bi → G there 48. In standard causal decision theory  (Joyce 1999; Pearl 2000) , the expectation of utility would be written using the counterfactual conditional: E[U ] o k ∈Outcome U (o k )P ( bi o k ). In timeless / updateless / logical decision theory  (Altair 2013; LaVictoire et al. 2013 ): Q = arg max bi∈Acts o k ∈Outcome U (o k )P ((eval Q = bi ) o k ). Since we do not properly consider such issues here, we shall (as do most non-foundational works) write the expected utility formula using the classical conditioning operation P (o k | bi ) of evidential decision theory, which decision theorists almost universally agree to be wrong. can be no further fact Γ such that ( bi ∧ Γ) → ¬G (unless of course the addition of Γ is inconsistent). However if P (G | bi ) > 0.9 there is no rule that therefore P (G | ( bi ∧ Γ)) > 0.9 for every further fact Γ, likewise E [U | bi ] > $20 does not imply E[U | bi ∧ Γ] > $20. Suppose A 1 , which desires E[U ] > $20 in all non-∅ cases, approves of any A 0 with the decision criterion (7.2) above, considering such an A 0 safe regardless of how its actions are suggested by an untrusted component Σ 0 . Suppose also that the action b2 yields $50 if the coin θ came up heads (θ H ) and $10 otherwise (θ T ). If θ is unknown and is assigned probability 50% then E[U | b2 ] = $25. But suppose that Σ 0 manages to learn the true value of θ and only suggests the action b2 in cases where it knows θ T . Then observing that the offspring has proven A 0 E[U | bi ] > $20 is not adequate reason to trust that, if it acts, it will achieve expected utility greater than $20.  49  It may be objected that since, in this case, A 0 only takes the action b 2 in cases where θ T , the true probability distribution would have P (θ T | b2 ) = 1 and A 0 should evaluate E[U | b2 ] accordingly. This raises questions of evidential versus causal decision theory which we lack space to address here. We remark that if A 0 , evaluating its probability distribution P , finds that this probability distribution already assigns probability 1 to action b 2 and probability 0 to all other actions (the "true" probability distribution in cases where b2 ) then A 0 would seem to know its own action before it decides (indeed, prior to its computation of expected utilities); and if A 1 also possesses axioms implying this "true" P then it must know its offspring's exact action which violates the Vingean principle. Furthermore, conditioning on all actions except b 2 would then require conditioning on a zero probability with undefined results.  50  We do not have good solutions for these issues, but felt that we ought to offer an unsatisfactory version of the formalism so that there would be something in place to improve upon. To avoid the complexities of reasoning about probabilistic program execution, we supposed the environment could be divided into a near-deterministic part (the "environmental transistors"; the analogue of a human-made CPU constructed to have almost perfectly deterministic causal paths, crisp variable states and crisp relations) and a probabilistic part X, which is the subject of the probability function P (X). We suppose X to be a joint function of variables x l obeying the usual fac-49. The form of your theorem demanded an action such that, conditioned on knowledge of only that action, expected utility would be greater than 20; you did not ask for an action conditioned on all the knowledge available to the component which suggested that action. You get no more than what you pay for; or in this case, no more than what you prove. 50. An analogous problem potentially arises if A 0 ever proves that it does not perform some particular action, Axm 0 ¬ bi . Any conclusion would then follow from the contradictory hypothesis bi , thus Axm 0 bi → (∅ ∨ G). With valid reasoning on true axioms this should never occur (we should not prove ¬ bi and then do b i ) but it is potentially a concern when trying to build systems which bypass the Löbian obstacle. It is ordinarily Gödel's Theorem which prevents a system from concluding that it will never prove some theorem (such as the theorem bi → (∅ ∨ G)). torization for causal models  (Pearl 2000) : P (X) = x l ∈X P (x l | pa l ), (7.3) where for each variable x l , pa l is a possibly empty list of the immediate causal ancestors of x l . Thus the joint probability distribution P (X) factorizes into local rules governing the probability of each node x l given its parent causes, if any. We then axiomatize A 1 's knowledge of P (X) in a way that leaves some parentless x l with unspecified probabilities; any causal descendants of these variables will have probabilities unquantified by A 1 . One of these variables will be A 0 's action and thus A 1 will not precompute a probability of A 0 's action. We will also need to leave unspecified all sensor variables s k ∈ S. Collectively the sensor variables and the motor action variable will constitute a kind of Cartesian boundary between the deterministic world in which A 0 's cognition occurs, and the probabilistic world it wants to sense and affect. We then mate deterministic reasoning to probabilistic reasoning via the bridging axioms: 51 A 1 ∀s k ∈ S : sk → (P (s k ) = 1) (7.4) A 1 ∀b i ∈ Acts 0 : bi → (P ( bi ) = 1). (7.5) A 1 and A 0 's criteria of action will be: Ā0 j ⇒A 1 Ā0 j → (∅ ∨ (E[U ] > $20)), (7.6) bi ⇒A 0 bi → (∅ ∨ (E[U ] > $20)), (7.7) and let A 1 prove a soundness schema (central trust lemma) over A 0 's logical beliefs as before. Then A 1 's decision to create A 0 is straightforward via: A 1 Ā0 → ∀b i : bi → ( 0 bi → (∅ ∨ E[U ] > $20) ), (7.8) A 1 0 bi → (∅ ∨ E[U ] > $20) ) → ( bi → (∅ ∨ E[U ] > $20)), (7.9) A 1 (∀b i : ¬ bi ) → ∅, (7.10) A 1 Ā0 → (∅ ∨ E[U ] > $20). (7.11) Suppose Axm 0 incorporates standard probability axioms, and suppose that A 0 has observed all available sensor data s k representing all missing terms of X except its action b i . Then A 0 will be able to carry out an action with 51. The form of these axioms was suggested by Paul Christiano.
 E[U | sk ∧ bi ] > $20, via: A 1 E[U |s k ∧ bi ] > $20 (7.12) A 1 sk (7.13) A 1 P (s k ) = 1 (7.14) A 1 bi → (P ( bi ) = 1) (7.15) A 1 (P ( bi ) = 1 ∧ P (s k ) = 1) → (P ( bi ∧ sk ) = 1) (7.16) A 1 bi → (P ( bi ∧ sk ) = 1) (7.17) A 1 bi → (P (¬( bi ∧ sk )) = 0) (7.18) A  A 1 bi → (E[U ] = E[U | bi ∧ sk ]) (7.20) A 1 bi → (E[U ] > $20) (7.21) We confess this to be a moderately grotesque hack that fails almost entirely to rise to the challenge of non-monotonic probabilistic reasoning. As remarked, we included it only to serve as something that could be improved upon. The argument above goes through only because, by assumption, every possible variable correlated with U has been observed and so no further probabilistic knowledge can be gained to invalidate the conclusion. It also is not immediately obvious to us how to extend the above hack to sensory observation of variables with causal parents. In usual practice, sensors observe the causal descendants of variables of interest-e.g., the light reflecting from an object takes on a pattern caused by that object; the observed photons are causal descendants of the event of their reflection. Being unable to represent the standard idiom of evidence in Bayesian networks is a very severe defect.  52  Similarly it seems to us that it should be possible to compute the probability of evidence before observing it, since this practice is central to Bayesian probability. In the formalism as it stands, being able to assign a prior probability to any evidential variable would produce a logical contradiction after a sensory event added the conclusion that the probability was 1. We suspect that there ought to be some much more general schema of trust in probabilistic soundness of a successor agent, possibly invoking explicit reasoning about probability measures in the same way that we previously introduced set theory as a backdrop to general reasoning about semantic truth, or using some probabilistic trust schema such as possibly: 52. We suspect that this desideratum might be satisfiable by some very simple technical method we have not yet invented as of this report.
 Bounded agents: Maximizing, satisficing, and meliorizing Expected utility is usually discussed in the context of expected utility maximization-choosing from among the actions in a set the action with highest expected utility. We did not attempt to frame our probabilistic agent that way, because introducing the notion of maximization creates new challenges to tiling, separate from the challenges of probability. I.e., suppose that we are in a deterministic environment, creating offspring which will have various possible sensors, various possible ranges of action, and whose actions (given the value of background variables) deterministically score various utilities. We could possibly adapt the logical goal G to state some version of the highest possible score being achieved, with the successor otherwise allowing a ∅ outcome. However, to properly rise to the challenge of maximization would imply creating the successor A 0 j which, of those options Acts 1 available to you, would lead to the best score you could've received, after A 0 j tries to maximize its own score. It was not obvious to us how to make the proof of such a tiling go through. It is also not obvious to us that this is the correct problem on which to spend effort, since, in real-world environments, achieving the maximum possible expected utility is impossible for all but the simplest problems. Maximization in full generality requires correctly assessing the score (expected utility) of every action in order to guarantee that we choose the best; this is what renders, e.g., optimal chess intractable. Satisficing is more realistic for bounded agents, in this sense, because we only need to evaluate the expected utility of a single action in order to have our decision criterion E[U ] > 20 go through. (Likewise, establishing some partially uncertain, probabilistic expectation of utility is more realistic than correctly predicting an exact outcome or score.) But to seriously consider satisficing as the foundation of a useful rational agent, or sufficiently advanced self-improving intelligence, would come with its own set of problems. Suppose the lower bound is set at $2 • 10 15 utility in whatever measure, and the agent realizes that a certain plan will ensure a minimum expected utility of $10 16 . Then as we have defined the agent's goals, it satisfices to randomize between a 30% probability of implementing this plan, a 60% probability of doing something which leads to expected utility $0, and a 10% probability of doing something with expected disutility −$10 16 . This is probably not the sort of behavior we wanted.  53  Schmidhuber's "Gödel machine" (2007) 54 introduced the interesting suggestion of a swapping criterion of action: The "Gödel machine" would change its own source code if it could prove that the action of changing its source code had a higher expected utilty than keeping its current source code. (No criterion for external motor actions was introduced, but arguendo the Gödel machine would soon choose to add one.) One of the participants at the April 2013 MIRI work-53. You get what you pay for, no more than you asked for, and exactly what you proved. 54.  Schmidhuber (2007)  could easily be considered a spiritual forerunner of the present work. shop suggested dubbing this a "meliorizing" criterion of action, since it is meant to make things steadily better (repeatedly swap in plans, or successors, with higher expected utility). Furthermore, since meliorizing requires considering only two courses of action-the current plan, and the new plan-it may be suitable for a boundedly rational agent. In contrast to the suggestion in section 4 of  Schmidhuber (2007)  it is not obvious that the criterion, "Swap if expected utility of the new program is higher" is a guarantee of "global optimality," which Schmidhuber suggested would be the result of each considered swap taking into account the possibility of other, better swaps if the current program were left undisturbed. Even considering expected utility rather than utility, most expectations that can be computed over some series of branches will not locate the point of highest expected value in the space, unless expectations are unusually smooth or the suggested series of changes is especially good. E.g., the problem at hand could easily have an NP-hard optimal solution (while still having good non-optimal solutions which could be steadily improved). But the question of "global optimality" is probably not the most important concern, since literal global optimality in the sense of trying to solve NP-hard problems should not be the key research desideratum. It is likewise not obvious to us that "meliorizing" is sufficient to produce satisfactory behavior with respect to a builder's programmed set of goals. Suppose a sufficiently advanced machine intelligence, built according to this criterion, discovered that an asteroid was headed toward Earth and would shortly kill 7 billion people, with its current plan not preventing it. Under the strict criterion of meliorizing as written, it would make sense to swap to a program that promised to save 1,000 people, let all the others die, and make no further improvements, since this would still be better than not swapping. According to the line of argument in section 4 of  Schmidhuber (2007) , the agent ought to consider that it would be better to keep the previous program and wait for it to generate a better alternative. But this relies on some particular sequence of suggestions being generated such that a better alternative is considered at some point; moreover, that the agent probabilistically expects that such a better alternative will be generated if it keeps its current program, in advance of considering the actual alternative (which the Vingean principle says we cannot always do at the earlier decision point). Thus if a meliorizing criterion is ultimately satisfactory, it will be due to other properties of the series of suggestions being considered, and the way in which expectations of old and new programs are evaluated, which have not been specified into the "swapping" rule itself. But expected utility satisficing is not satisfactory at all, and is probably not repairable; and maximizing is only possible for the best imaginable agents, not the agents that will actually exist; whereas it might be that meliorizing can somehow be improved upon. More generally, we have discussed maximizing, satisficing, and meliorizing in order to make the point that when it comes to bounded, probabilistic rational agents that are meant to pursue their goals in some "reasonable" way and build descendants who do the same (equivalently self-improve), we are not able to presently state-even on the highest possible level of generality such as "satisficing" or "meliorizing"-what sort of criterion of action might be suitable. The problem is very wide open indeed. so by Löb's Theorem ∀x : T ψ(x). 18 (b) Instead of an indefinitely extensible ordinal hierarchy of systems T +α, let us form a descending sequence of theories T -0, T -1, T -2 as follows: T -n T ∪ ∀φ : (∀x : T -(n+1) φ(x) → φ(x)).
 Perhaps the previous constructions failed due to their impredicativity. Let us try a finitary version which speaks only of well-founded objects. Let k+1 T φ stand for the formula T T ...φ... with k + 1 boxes.. Since the pattern T T ... is highly regular, a compact formula for k+1 T φ should exist. Then let A α 's criterion of action be bi
 T φ(b i ) ∨ φ(b i ) , which does not reduce to ∃k : k+1 T φ(b i ) . 19 19. The formula k+1 T φ just occupies a new ordinal ω of proofness, and T k+1 T φ then corresponds to a new ordinal ω+1 of proofness.
 1 E[U ] = (E[U | bi ∧ sk ]P ( bi ∧ sk )) + (E[U |¬( bi ∧ sk )]P (¬( bi ∧ sk ))) (7.19)
 P α (φ ∧ (A β P β (φ) = p )) = p • P α (A β P β (φ) = p ). (7.22)Such work remains in progress, however, and in general the problem of self-modification in probabilistic agents remains wide open.
			 . ψ(x) is just a Henkin sentence H ↔ T H with a dangling ∨ clause φ(x). A Henkin sentence for T is of course always provable within T .
			 . Which is, in general, the agenda of this paper: Our framework and our technical solutions are
			 . In other words, the length of clock ticks is small enough that constructing another agent takes at least one tick. E.g., the length of a clock tick could equal the Planck time.
			 . This implies that the system behaves in a sense as though it assigns nonstandard probabilities (in the sense of nonstandard analysis with infinitesimals), an issue we are still working on.
			 . It is possible that this work might have some relevance to the philosophy of epistemology, which we lack space to explore here.