Abstract: A Bayesian agent acting in a multi-agent environment learns to predict the other agents' policies if its prior assigns positive probability to them (in other words, its prior contains a grain of truth). Finding a reasonably large class of policies that contains the Bayes-optimal policies with respect to this class is known as the grain of truth problem. Only small classes are known to have a grain of truth and the literature contains several related impossibility results. In this paper we present a formal and general solution to the full grain of truth problem: we construct a class of policies that contains all computable policies as well as Bayes-optimal policies for every lower semicomputable prior over the class. When the environment is unknown, Bayes-optimal agents may fail to act optimally even asymptotically. However, agents based on Thompson sampling converge to play ε-Nash equilibria in arbitrary unknown computable multi-agent environments. While these results are purely theoretical, we show that they can be computationally approximated arbitrarily closely.

Jan Leike
Jessica Taylor
Benya Fallenstein

A Formal Solution to the Grain of Truth Problem

INTRODUCTION Consider the general setup of multiple reinforcement learning agents interacting sequentially in a known environment with the goal to maximize discounted reward.  1  Each agent knows how the environment behaves, but does not know the other agents' behavior. The natural (Bayesian) approach would be to define a class of possible policies that the other  1  We mostly use the terminology of reinforcement learning. For readers from game theory we provide a dictionary in Table  1 .
 Reinforcement learning Game theory stochastic policy mixed strategy deterministic policy pure strategy agent player multi-agent environment infinite extensive-form game reward payoff/utility (finite) history history infinite history path of play Table  1 : Terminology dictionary between reinforcement learning and game theory. agents could adopt and take a prior over this class. During the interaction, this prior gets updated to the posterior as our agent learns the others' behavior. Our agent then acts optimally with respect to this posterior belief. A famous result for infinitely repeated games states that as long as each agent assigns positive prior probability to the other agents' policies (a grain of truth) and each agent acts Bayes-optimal, then the agents converge to playing an ε-Nash equilibrium  [KL93] . As an example, consider an infinitely repeated prisoners dilemma between two agents. In every time step the payoff matrix is as follows, where C means cooperate and D means defect. C D C 3/4, 3/4 0, 1 D 1, 0 1/4, 1/4 Define the set of policies Π := {π ∞ , π 0 , π 1 , . . .} where policy π t cooperates until time step t or the opponent defects (whatever happens first) and defects thereafter. The Bayes-optimal behavior is to cooperate until the posterior belief that the other agent defects in the time step after the next is greater than some constant (depending on the discount function) and then defect afterwards. Therefore Bayes-optimal behavior leads to a policy from the set Π (regardless of the prior). If both agents are Bayes-optimal with respect to some prior, they both have a grain of truth and therefore they converge to a Nash equilibrium: either they both cooperate forever or after some finite time they both defect forever. Alternating strategies like TitForTat (cooperate first, then play the opponent's last action) are not part of the policy class Π, and adding them to the class breaks the grain of truth property: the Bayes-optimal behavior is no longer in the class. This is rather typical; a Bayesian agent usually needs to be more powerful than its environment  [LH15b] . Until now, classes that admit a grain of truth were known only for small toy examples such as the iterated prisoner's dilemma above  [SLB09, Ch. 7.3 ]. The quest to find a large class admitting a grain of truth is known as the grain of truth problem  [Hut09, Q. 5j ]. The literature contains several impossibility results on the grain of truth problem  [FY01, Nac97, Nac05]  that identify properties that cannot be simultaneously satisfied for classes that allow a grain of truth. In this paper we present a formal solution to multi-agent reinforcement learning and the grain of truth problem in the general setting (Section 3). We assume that our multiagent environment is computable, but it does not need to be stationary/Markov, ergodic, or finite-state  [Hut05] . Our class of policies is large enough to contain all computable (stochastic) policies, as well as all relevant Bayes-optimal policies. At the same time, our class is small enough to be limit computable. This is important because it allows our result to be computationally approximated. In Section 4 we consider the setting where the multi-agent environment is unknown to the agents and has to be learned in addition to the other agents' behavior. A Bayes-optimal agent may not learn to act optimally in unknown multiagent environments even though it has a grain of truth. This effect occurs in non-recoverable environments where taking one wrong action can mean a permanent loss of future value. In this case, a Bayes-optimal agent avoids taking these dangerous actions and therefore will not explore enough to wash out the prior's bias  [LH15a] . Therefore, Bayesian agents are not asymptotically optimal, i.e., they do not always learn to act optimally  [Ors13] . However, asymptotic optimality is achieved by Thompson sampling because the inherent randomness of Thompson sampling leads to enough exploration to learn the entire environment class  [LLOH16] . This leads to our main result: if all agents use Thompson sampling over our class of multi-agent environments, then for every ε > 0 they converge to an ε-Nash equilibrium asymptotically. The central idea to our construction is based on reflective oracles  [FST15, FTC15b] . Reflective oracles are probabilistic oracles similar to halting oracles that answer whether the probability that a given probabilistic Turing machine T outputs 1 is higher than a given rational number p. The oracles are reflective in the sense that the machine T may itself query the oracle, so the oracle has to answer queries about itself. This invites issues caused by self-referential liar paradoxes of the form "if the oracle says that I return 1 with probability > 1/2, then return 0, else return 1." Reflective oracles avoid these issues by being allowed to randomize if the machines do not halt or the rational number is exactly the probability to output 1. We introduce reflective oracles formally in Section 2 and prove that there is a limit computable reflective oracle.
 REFLECTIVE ORACLES
 PRELIMINARIES Let X denote a finite set called alphabet. The set X * := ∞ n=0 X n is the set of all finite strings over the alphabet X , the set X ∞ is the set of all infinite strings over the alphabet X , and the set X ♯ := X * ∪ X ∞ is their union. The empty string is denoted by ǫ, not to be confused with the small positive real number ε. Given a string x ∈ X ♯ , we denote its length by |x|. For a (finite or infinite) string x of length ≥ k, we denote with x 1:k the first k characters of x, and with x <k the first k − 1 characters of x. The notation x 1:∞ stresses that x is an infinite string.
 A function f : X * → R is lower semicomputable iff the set {(x, p) ∈ X * × Q | f (x) > p} is recursively enu- merable. The function f is computable iff both f and −f are lower semicomputable. Finally, the function f is limit computable iff there is a computable function φ such that lim k→∞ φ(x, k) = f (x). The program φ that limit computes f can be thought of as an anytime algorithm for f : we can stop φ at any time k and get a preliminary answer. If the program φ ran long enough (which we do not know), this preliminary answer will be close to the correct one. We use ∆Y to denote the set of probability distributions over Y. A list of notation can be found in Appendix A.
 DEFINITION A semimeasure over the alphabet X is a function ν : X * → [0, 1] such that (i) ν(ǫ) ≤ 1, and (ii) ν(x) ≥ a∈X ν(xa) for all x ∈ X * . In the terminology of measure theory, semimeasures are probability measures on the probability space X ♯ = X * ∪ X ∞ whose σ-algebra is generated by the cylinder sets Γ x := {xz | z ∈ X ♯ } [LV08, Ch. 4.2]. We call a semimeasure (probability) a measure iff equalities hold in (i) and (ii) for all x ∈ X * . Next, we connect semimeasures to Turing machines. The literature uses monotone Turing machines, which naturally correspond to lower semicomputable semimeasures [LV08, Sec. 4.5.2] that describe the distribution that arises when piping fair coin flips into the monotone machine. Here we take a different route. A probabilistic Turing machine is a Turing machine that has access to an unlimited number of uniformly random coin flips. Let T denote the set of all probabilistic Turing machines that take some input in X * and may query an oracle (formally defined below). We take a Turing machine T ∈ T to correspond to a semimeasure λ T where λ T (a | x) is the probability that T outputs a ∈ X when given x ∈ X * as input. The value of λ T (x) is then given by the chain rule λ T (x) := |x| k=1 λ T (x k | x <k ). (1) Thus T gives rise to the set of semimeasures M where the conditionals λ(a | x) are lower semicomputable. In contrast, the literature typically considers semimeasures whose joint probability (1) is lower semicomputable. This set M contains all computable measures. However, M is a proper subset of the set of all lower semicomputable semimeasures because the product (1) is lower semicomputale, but there are some lower semicomputable semimeasures whose conditional is not lower semicomputable [LH15c, Thm. 6]. In the following we assume that our alphabet is binary, i.e., X := {0, 1}. Definition 1 (Oracle). An oracle is a function O : T × {0, 1} * × Q → ∆{0, 1}. Oracles are understood to be probabilistic: they randomly return 0 or 1. Let T O denote the machine T ∈ T when run with the oracle O, and let λ O T denote the semimeasure induced by T O . This means that drawing from λ O T involves two sources of randomness: one from the distribution induced by the probabilistic Turing machine T and one from the oracle's answers. The intended semantics of an oracle are that it takes a query (T, x, p) and returns 1 if the machine T O outputs 1 on input x with probability greater than p when run with the oracle O, i.e., when λ O T (1 | x) > p. Furthermore, the oracle returns 0 if the machine T O outputs 1 on input x with probability less than p when run with the oracle O, i.e., when λ O T (1 | x) < p. To fulfill this, the oracle O has to make statements about itself, since the machine T from the query may again query O. Therefore we call oracles of this kind reflective oracles. This has to be defined very carefully to avoid the obvious diagonalization issues that are caused by programs that ask the oracle about themselves. We impose the following self-consistency constraint. Definition 2 (Reflective Oracle). An oracle O is reflective iff for all queries (T, x, p) ∈ T × {0, 1} * × Q,  (i) λ O T (1 | x) > p implies O(T, x, p) = 1, and 0 1 λ O T (0 | x) λ O T (1 | x) O returns 1 O may randomize O returns 0 (ii) λ O T (0 | x) > 1 − p implies O(T, x, p) = 0. If p under-or overshoots the true probability of λ O T ( • | x), then the oracle must reveal this information. However, in the critical case when p = λ O T (1 | x), the oracle is allowed to return anything and may randomize its result. Furthermore, since T might not output any symbol, it is possible that λ O T (0 | x) + λ O T (1 | x) < 1. In this case the oracle can reassign the non-halting probability mass to 0, 1, or randomize; see Figure  1 . Example 3 (Reflective Oracles and Diagonalization). Let T ∈ T be a probabilistic Turing machine that outputs 1 − O(T, ǫ, 1/2) (T can know its own source code by quining [Kle52, Thm. 27]). In other words, T queries the oracle about whether it is more likely to output 1 or 0, and then does whichever the oracle says is less likely. In this case we can use an oracle O(T, ǫ, 1/2) := 1/2 (answer 0 or 1 with equal probability), which implies λ O T (1 | ǫ) = λ O T (0 | ǫ) = 1/2, so the conditions of Definition 2 are satisfied. In fact, for this machine T we must have O(T, ǫ, 1/2) = 1/2 for all reflective oracles O. ♦ The following theorem establishes that reflective oracles exist.
 Theorem 4 ([FTC15a, App. B]). There is a reflective oracle. Definition 5 (Reflective-Oracle-Computable). A semimeasure is called reflective-oracle-computable iff it is computable on a probabilistic Turing machine with access to a reflective oracle. For any probabilistic Turing machine T ∈ T we can complete the semimeasure λ O T ( • | x) into a reflective- oracle-computable measure λ O T ( • | x): Using the oracle O and a binary search on the parameter p we search for the crossover point p where O(T, x, p) goes from returning 1 to returning 0. The limit point p * ∈ R of the binary search is random since the oracle's answers may be random. But the main point is that the expectation of p * exists, so λ O T (1 | x) = E[p * ] = 1 − λ O T (0 | x) for all x ∈ X * . Hence λ O T is a measure. Moreover, if the oracle is reflective, then λ O T (x) ≥ λ O T (x) for all x ∈ X * . In this sense the oracle O can be viewed as a way of 'completing' all semimeasures λ O T to measures by arbitrarily assigning the non-halting probability mass. If the oracle O is reflective this is consistent in the sense that Turing machines who run other Turing machines will be completed in the same way. This is especially important for a universal machine that runs all other Turing machines to induce a Solomonoffstyle distribution.
 A LIMIT COMPUTABLE REFLECTIVE ORACLE The proof of Theorem 4 given in [FTC15a, App. B] is nonconstructive and uses the axiom of choice. In Section 2.4 we give a constructive proof for the existence of reflective oracles and show that there is one that is limit computable.
 Theorem 6 (A Limit Computable Reflective Oracle). There is a reflective oracle that is limit computable. This theorem has the immediate consequence that reflective oracles cannot be used as halting oracles. At first, this result may seem surprising: according to the definition of reflective oracles, they make concrete statements about the output of probabilistic Turing machines. However, the fact that the oracles may randomize some of the time actually removes enough information such that halting can no longer be decided from the oracle output.
 Corollary 7 (Reflective Oracles are not Halting Oracles). There is no probabilistic Turing machine T such that for every prefix program p and every reflective oracle O, we have that λ O T (1 | p) > 1/2 if p halts and λ O T (1 | p) < 1/2 otherwise. Proof. Assume there was such a machine T and let O be the limit computable oracle from Theorem 6. Since O is reflective we can turn T into a deterministic halting oracle by calling O(T, p, 1/2) which deterministically returns 1 if p halts and 0 otherwise. Since O is limit computable, we can finitely compute the output of O on any query to arbitrary finite precision using our deterministic halting oracle. We construct a probabilistic Turing machine T ′ that uses our halting oracle to compute (rather than query) the oracle O on (T ′ , ǫ, 1/2) to a precision of 1/3 in finite time. If O(T ′ , ǫ, 1/2) ± 1/3 > 1/2, the machine T ′ outputs 0, otherwise T ′ outputs 1. Since our halting oracle is entirely deterministic, the output of T ′ is entirely deterministic as well (and T ′ always halts), so λ O T ′ (0 | ǫ) = 1 or λ O T ′ (1 | ǫ) = 1. Therefore O(T ′ , ǫ, 1/2) = 1 or O(T ′ , ǫ, 1/2) = 0 because O is reflective. A precision of 1/3 is enough to tell them apart, hence T ′ returns 0 if O(T ′ , ǫ, 1/2) = 1 and T ′ re- turns 1 if O(T ′ , ǫ, 1/2) = 0. This is a contradiction. A similar argument can also be used to show that reflective oracles are not computable.
 PROOF OF THEOREM 6 The idea for the proof of Theorem 6 is to construct an algorithm that outputs an infinite series of partial oracles converging to a reflective oracle in the limit. The set of queries is countable, so we can assume that we have some computable enumeration of it: T × {0, 1} * × Q =: {q 1 , q 2 , . . .} Definition 8 (k-Partial Oracle). A k-partial oracle Õ is function from the first k queries to the multiples of 2 −k in [0, 1]: Õ : {q 1 , q 2 , . . . , q k } → {n2 −k | 0 ≤ n ≤ 2 k } Definition 9 (Approximating an Oracle). A k-partial or- acle Õ approximates an oracle O iff |O(q i ) − Õ(q i )| ≤ 2 −k−1 for all i ≤ k. Let k ∈ N, let Õ be a k-partial oracle, and let T ∈ T be an oracle machine. The machine T Õ that we get when we run T with the k-partial oracle Õ is defined as follows (this is with slight abuse of notation since k is taken to be understood implicitly). 1. Run T for at most k steps. 2. If T calls the oracle on q i for i ≤ k, (a) return 1 with probability Õ(q i ) − 2 −k−1 , (b) return 0 with probability 1 − Õ(q i ) − 2 −k−1 , and (c) halt otherwise. 3. If T calls the oracle on q j for j > k, halt. Furthermore, we define λ Õ T analogously to λ O T as the distribution generated by the machine T Õ. Lemma 10. If a k-partial oracle Õ approximates a re- flective oracle O, then λ O T (1 | x) ≥ λ Õ T (1 | x) and λ O T (0 | x) ≥ λ Õ T (0 | x) for all x ∈ {0, 1} * and all T ∈ T . Proof. This follows from the definition of T Õ : when running T with Õ instead of O, we can only lose probability mass. If T makes calls whose index is > k or runs for more than k steps, then the execution is aborted and no further output is generated. If T makes calls whose index i ≤ k, then Õ(q i ) − 2 −k−1 ≤ O(q i ) since Õ approximates O. Therefore the return of the call q i is underestimated as well. Definition 11 (k-Partially Reflective). A k-partial oracle Õ is k-partially reflective iff for the first k queries (T, x, p) • λ Õ T (1 | x) > p implies Õ(T, x, p) = 1, and • λ Õ T (0 | x) > 1 − p implies Õ(T, x, p) = 0. It is important to note that we can check whether a k-partial oracle is k-partially reflective in finite time by running all machines T from the first k queries for k steps and tallying up the probabilities to compute λ Õ T . Lemma 12. If O is a reflective oracle and Õ is a k-partial oracle that approximates O, then Õ is k-partially reflective. Lemma 12 only holds because we use semimeasures whose conditionals are lower semicomputable. Proof. Assuming λ Õ T (1 | x) > p we get from Lemma 10 that λ O T (1 | x) ≥ λ Õ T (1 | x) > p. Thus O(T, x, p) = 1 because O is reflective. Since Õ approximates O, we get 1 = O(T, x, p) ≤ Õ(T, x, p)+ 2 −k−1 , and since Õ assigns values in a 2 −k -grid, it follows that Õ(T, x, p) = 1. The second implication is proved analogously.
 Definition 13 (Extending Partial Oracles ). A k + 1-partial oracle Õ′ extends a k-partial oracle Õ iff | Õ(q i )− Õ′ (q i )| ≤ 2 −k−1 for all i ≤ k. Lemma 14. There is an infinite sequence of partial oracles ( Õk ) k∈N such that for each k, Õk is a k-partially reflective k-partial oracle and Õk+1 extends Õk . Proof. By Theorem 4 there is a reflective oracle O. For every k, there is a canonical k-partial oracle Õk that approximates O: restrict O to the first k queries and for any such query q pick the value in the 2 −k -grid which is closest to O(q). By construction, Õk+1 extends Õk and by Lemma 12, each Õk is k-partially reflective. Lemma 15. If the k + 1-partial oracle Õk+1 extends the k-partial oracle Õk , then λ Õk+1 T (1 | x) ≥ λ Õk T (1 | x) and λ Õk+1 T (0 | x) ≥ λ Õk T (0 | x) for all x ∈ {0, 1} * and all T ∈ T . Proof. T Õk+1 runs for one more step than T Õk , can answer one more query and has increased oracle precision. Moreover, since Õk+1 extends Õk , we have | Õk+1 (q i ) − Õk (q i )| ≤ 2 −k−1 , and thus Õk+1 (q i )−2 −k−1 ≥ Õk (q i )− 2 −k . Therefore the success to answers to the oracle calls (case 2(a) and 2(b)) will not decrease in probability. Now everything is in place to state the algorithm that constructs a reflective oracle in the limit. It recursively traverses a tree of partial oracles. The tree's nodes are the partial oracles; level k of the tree contains all k-partial oracles. There is an edge in the tree from the k-partial oracle Õk to the i-partial oracle Õi if and only if i = k + 1 and Õi extends Õk . For every k, there are only finitely many k-partial oracles, since they are functions from finite sets to finite sets. In particular, there are exactly two 1-partial oracles (so the search tree has two roots). Pick one of them to start with, and proceed recursively as follows. Given a k-partial oracle Õk , there are finitely many (k + 1)-partial oracles that extend Õk (finite branching of the tree). Pick one that is (k + 1)partially reflective (which can be checked in finite time). If there is no (k + 1)-partially reflective extension, backtrack. By Lemma 14 our search tree is infinitely deep and thus the tree search does not terminate. Moreover, it can backtrack to each level only a finite number of times because at each level there is only a finite number of possible extensions. Therefore the algorithm will produce an infinite sequence of partial oracles, each extending the previous. Because of finite backtracking, the output eventually stabilizes on a sequence of partial oracles Õ1 , Õ2 , . . .. By the following lemma, this sequence converges to a reflective oracle, which concludes the proof of Theorem 6. Lemma 16. Let Õ1 , Õ2 , . . . be a sequence where Õk is a k-partially reflective k-partial oracle and Õk+1 extends Õk for all k ∈ N. Let O := lim k→∞ Õk be the pointwise limit. Then Proof. First note that the pointwise limit must exists because | Õk (q i ) − Õk+1 (q i )| ≤ 2 −k−1 by Definition 13.  (a) λ Õk T (1 | x) → λ O T (1 | x) and λ Õk T (0 | x) → λ O T (0 | x) as k → ∞ for all x ∈ {0,
 A GRAIN OF TRUTH
 NOTATION In reinforcement learning, an agent interacts with an environment in cycles: at time step t the agent chooses an action a t ∈ A and receives a percept e t = (o t , r t ) ∈ E consisting of an observation o t ∈ O and a real-valued reward r t ∈ R; the cycle then repeats for t + 1. A history is an element of (A × E) * . In this section, we use ae ∈ A × E to denote one interaction cycle, and ae <t to denote a history of length t − 1. We fix a discount function γ : N → R with γ t ≥ 0 and ∞ t=1 γ t < ∞. The goal in reinforcement learning is to maximize discounted rewards ∞ t=1 γ t r t . The discount normalization factor is defined as Γ t := ∞ k=t γ k . The effective horizon H t (ε) is a horizon that is long enough to encompass all but an ε of the discount function's mass: H t (ε) := min{k | Γ t+k /Γ t ≤ ε} (2) A policy is a function π : (A × E) * → ∆A that maps a history ae <t to a distribution over actions taken after seeing this history. The probability of taking action a after history ae <t is denoted with π(a | ae <t ). An environment is a function ν : (A × E) * × A → ∆E where ν(e | ae <t a t ) denotes the probability of receiving the percept e when taking the action a t after the history ae <t . Together, a policy π and an environment ν give rise to a distribution ν π over histories. Throughout this paper, we make the following assumptions. Assumption 17. (a) Rewards are bounded between 0 and 1. (b) The set of actions A and the set of percepts E are both finite. (c) The discount function γ and the discount normalization factor Γ are computable. Definition 18 (Value Function). The value of a policy π in an environment ν given history ae <t is defined recursively as V π ν (ae <t ) := a∈A π(a | ae <t )V π ν (ae <t a) and V π ν (ae <t a t ) := 1 Γ t et∈E ν(e t | ae <t a t ) γ t r t + Γ t+1 V π ν (ae 1:t ) if Γ t > 0 and V π ν (ae <t a t ) := 0 if Γ t = 0. The optimal value is defined as V * ν (ae <t ) := sup π V π ν (ae <t ). Definition 19 (Optimal Policy). A policy π is optimal in environment ν (ν-optimal) iff for all histories ae <t ∈ (A × E) * the policy π attains the optimal value: V π ν (ae <t ) = V * ν (ae <t ). We assumed that the discount function is summable, rewards are bounded (Assumption 17a), and actions and percepts spaces are both finite (Assumption 17b). Therefore an optimal deterministic policy exists for every environment [LH14, Thm. 10].
 REFLECTIVE BAYESIAN AGENTS Fix O to be a reflective oracle. From now on, we assume that the action space A := {α, β} is binary. We can treat computable measures over binary strings as environments: the environment ν corresponding to a probabilistic Turing machine T ∈ T is defined by ν(e t | ae <t a t ) := λ O T (y | x) = k i=1 λ O T (y i | xy 1 . . . y i−1 ) where y 1:k is a binary encoding of e t and x is a binary encoding of ae <t a t . The actions a 1:∞ are only contextual, and not part of the environment distribution. We define ν(e <t | a <t ) := t−1 k=1 ν(e k | ae <k ). Let T 1 , T 2 , . . . be an enumeration of all probabilistic Turing machines in T . We define the class of reflective environments M O refl := λ O T1 , λ O T2 , . . . . This is the class of all environments computable on a probabilistic Turing machine with reflective oracle O, that have been completed from semimeasures to measures using O. T . This is the distribution that is used to compute the posterior. There are no cyclic dependencies since ξ is called on the shorter history ae <t . We arrive at the following statement.
 Analogously to AIXI
 Proposition 20 (Bayes is in the Class ). ξ ∈ M O refl . Moreover  (µ) > 0, V π µ (ae <t ) − V π ξ (ae <t ) → 0 µ π -almost surely as t → ∞.
 REFLECTIVE-ORACLE-COMPUTABLE POLICIES This subsection is dedicated to the following result that was previously stated but not proved in [FST15, Alg. 6]. It contrasts results on arbitrary semicomputable environments where optimal policies are not limit computable [LH15b, Sec. 4]. Theorem 22 (Optimal Policies are Oracle Computable). For every ν ∈ M O refl , there is a ν-optimal (stochastic) policy π * ν that is reflective-oracle-computable. Note that even though deterministic optimal policies always exist, those policies are typically not reflectiveoracle-computable. To prove Theorem 22 we need the following lemma. Lemma 23 (Reflective-Oracle-Computable Optimal Value Function). For every environment ν ∈ M O refl the optimal value function V * ν is reflective-oracle-computable. Proof. This proof follows the proof of  [LH15b, Cor. 13 ]. We write the optimal value explicitly as V * ν (ae <t ) = 1 Γ t lim m→∞ max aet:m m k=t γ k r k k i=t ν(e i | ae <i ), (5) where max denotes the expectimax operator: For a fixed m, all involved quantities are reflective-oraclecomputable. Moreover, this quantity is monotone increasing in m and the tail sum from m + 1 to ∞ is bounded by Γ m+1 which is computable according to Assumption 17c and converges to 0 as m → ∞. Therefore we can enumerate all rationals above and below V * ν . Proof of Theorem 22. According to Lemma 23 the optimal value function V * ν is reflective-oracle-computable. Hence there is a probabilistic Turing machine T such that λ O T (1 | ae <t ) = V * ν (ae <t α) − V * ν (ae <t β) + 1 /2. We define a policy π that takes action α if O(T, ae <t , 1/2) = 1 and action β if O(T, ae <t , 1/2) = 0. (This policy is stochastic because the answer of the oracle O is stochastic.) It remains to show that π is a ν-optimal policy. If V * ν (ae <t α) > V * ν (ae <t β), then λ O T (1 | ae <t ) > 1/2, thus O(T, ae <t , 1/2) = 1 since O is reflective, and hence π takes action α. Conversely, if V * ν (ae <t α) < V * ν (ae <t β), then λ O T (1 | ae <t ) < 1/2, thus O(T, ae <t , 1/2) = 0 since O is reflective, and hence π takes action β. Lastly, if V * ν (ae <t α) = V * ν (ae <t β) , then both actions are optimal and thus it does not matter which action is returned by policy π. (This is the case where the oracle may randomize.)
 SOLUTION TO THE GRAIN OF TRUTH PROBLEM Together, Proposition 20 and Theorem 22 provide the necessary ingredients to solve the grain of truth problem. Corollary 24 (Solution to the Grain of Truth Problem). For every lower semicomputable prior w ∈ ∆M O refl the Bayesoptimal policy π * ξ is reflective-oracle-computable where ξ is the Bayes-mixture corresponding to w defined in (3). Proof. From Proposition 20 and Theorem 22.
 Hence the environment class M O refl contains any reflectiveoracle-computable modification of the Bayes-optimal policy π * ξ . In particular, this includes computable multi-agent environments that contain other Bayesian agents over the class M O refl . So any Bayesian agent over the class M O refl has a grain of truth even though the environment may contain other Bayesian agents of equal power. We proceed to sketch the implications for multi-agent environments in the next section.
 MULTI-AGENT ENVIRONMENTS This section summarizes our results for multi-agent systems. The proofs can be found in  [Lei16] .
 SETUP In a multi-agent environment there are n agents each taking sequential actions from the finite action space A. and ae <t = a 1 e 1 . . . a t−1 e t−1 . We define a multi-agent environment as a function σ : (A n × E n ) * × A n → ∆(E n ). The agents are given by n policies π 1 , . . . , π n where π i : (A × E) * → ∆A. Together they specify the history distribution σ π1:n (ǫ) : = 1 σ π1:n (ae 1:t ) : = σ π1:n (ae <t a t )σ(e t | ae <t a t ) σ π1:n (ae <t a t ) : = σ π1:n (ae <t ) n i=1 π i (a i t | ae i <t ). Each agent i acts in a subjective environment σ i given by joining the multi-agent environment σ with the policies π 1 , . . . , π i−1 , π i+1 , . . . , π n by marginalizing over the histories that π i does not see. Together with policy π i , the environment σ i yields a distribution over the histories of agent i σ πi i (ae i <t ) := ae j <t ,j =i σ π1:n (ae <t ). We get the definition of the subjective environment σ i with the identity σ i (e i t | ae i <t a i t ) := σ πi i (e i t | ae i <t a i t ). It is crucial to note that the subjective environment σ i and the policy π i are ordinary environments and policies, so we can use the formalism from Section 3. Our definition of a multi-agent environment is very general and encompasses most of game theory. It allows for cooperative, competitive, and mixed games; infinitely repeated games or any (infinite-length) extensive form games with finitely many players. The policy π i is an ε-best response after history ae i <t iff V * σi (ae i <t ) − V πi σi (ae i <t ) < ε. If at some time step t, all agents' policies are ε-best responses, we have an ε-Nash equilibrium. The property of multi-agent systems that is analogous to asymptotic optimality is convergence to an ε-Nash equilibrium.
 INFORMED REFLECTIVE AGENTS Let σ be a multi-agent environment and let π * σ1 , . . . π * σn be such that for each i the policy π * σi is an optimal policy in agent i's subjective environment σ i . At first glance this seems ill-defined: The subjective environment σ i depends on each other policy π * σj for j = i, which depends on the subjective environment σ j , which in turn depends on the policy π * σi . However, this circular definition actually has a well-defined solution. Theorem 25 (Optimal Multi-Agent Policies). For any reflective-oracle-computable multi-agent environment σ, the optimal policies π * σ1 , . . . , π * σn exist and are reflectiveoracle-computable. Note the strength of Theorem 25: each of the policies π * σi is acting optimally given the knowledge of everyone else's policies. Hence optimal policies play 0-best responses by definition, so if every agent is playing an optimal policy, we have a Nash equilibrium. Moreover, this Nash equilibrium is also a subgame perfect Nash equilibrium, because each agent also acts optimally on the counterfactual histories that do not end up being played. In other words, Theorem 25 states the existence and reflectiveoracle-computability of a subgame perfect Nash equilibrium in any reflective-oracle-computable multi-agent environment. From Theorem 6 we then get that these subgame perfect Nash equilibria are limit computable. Corollary 26 (Solution to Computable Multi-Agent Environments). For any computable multi-agent environment σ, the optimal policies π * σ1 , . . . , π * σn exist and are limit computable.
 LEARNING REFLECTIVE AGENTS Since our class M O refl solves the grain of truth problem, the result by Kalai and Lehrer  [KL93]  immediately implies that for any Bayesian agents π 1 , . . . , π n interacting in an infinitely repeated game and for all ε > 0 and all i ∈ {1, . . . , n} there is almost surely a t 0 ∈ N such that for all t ≥ t 0 the policy π i is an ε-best response. However, this hinges on the important fact that every agent has to know the game and also that all other agents are Bayesian agents. Otherwise the convergence to an ε-Nash equilibrium may fail, as illustrated by the following example. At the core of the following construction is a dogmatic prior [LH15a, Sec. 3.2]. A dogmatic prior assigns very high probability to going to hell (reward 0 forever) if the agent deviates from a given computable policy π. For a Bayesian agent it is thus only worth deviating from the policy π if the agent thinks that the prospects of following π are very poor already. This implies that for general multi-agent environments and without additional assumptions on the prior, we cannot prove any meaningful convergence result about Bayesian agents acting in an unknown multi-agent environment. Example 27 (Reflective Bayesians Playing Matching Pennies). In the game of matching pennies there are two agents (n = 2), and two actions A = {α, β} representing the two sides of a penny. In each time step agent 1 wins if the two actions are identical and agent 2 wins if the two actions are different. The payoff matrix is as follows. α β α 1,0 0,1 β 0,1 1,0 We use E = {0, 1} to be the set of rewards (observations are vacuous) and define the multi-agent environment σ to give reward 1 to agent 1 iff a 1 t = a 2 t (0 otherwise) and reward 1 to agent 2 iff a 1 t = a 2 t (0 otherwise). Note that neither agent knows a priori that they are playing matching pennies, nor that they are playing an infinite repeated game with one other player. Let π 1 be the policy that takes the action sequence (ααβ) ∞ and let π 2 := π α be the policy that always takes action α. The average reward of policy π 1 is 2/3 and the average reward of policy π 2 is 1/3. Let ξ be a universal mixture (3). By Lemma 21, V π1 ξ → c 1 ≈ 2/3 and V π2 ξ → c 2 ≈ 1/3 almost surely when following policies (π 1 , π 2 ). Therefore there is an ε > 0 such that V π1 ξ > ε and V π2 ξ > ε for all time steps. Now we can apply [LH15a, Thm. 7] to conclude that there are (dogmatic) mixtures ξ ′ 1 and ξ ′ 2 such that π * ξ ′ 1 always follows policy π 1 and π * ξ ′ 2 always follows policy π 2 . This does not converge to a (ε-)Nash equilibrium. ♦ A policy π is asymptotically optimal in mean in an environment class M iff for all µ ∈ M E π µ V * µ (ae <t ) − V π µ (ae <t ) → 0 as t → ∞ (6) where E π µ denotes the expectation with respect to the probability distribution µ π over histories generated by policy π acting in environment µ. Asymptotic optimality stands out because it is currently the only known nontrivial objective notion of optimality in general reinforcement learning  [LH15a] . The following theorem is the main convergence result. It states that for asymptotically optimal agents we get convergence to ε-Nash equilibria in any reflective-oraclecomputable multi-agent environment. Theorem 28 (Convergence to Equilibrium). Let σ be an reflective-oracle-computable multi-agent environment and let π 1 , . . . , π n be reflective-oracle-computable policies that are asymptotically optimal in mean in the class M O refl . Then for all ε > 0 and all i ∈ {1, . . . , n} the σ π1:n -probability that the policy π i is an ε-best response converges to 1 as t → ∞. In contrast to Theorem 25 which yields policies that play a subgame perfect equilibrium, this is not the case for Theorem 28: the agents typically do not learn to predict off-policy and thus will generally not play ε-best responses in the counterfactual histories that they never see. This weaker form of equilibrium is unavoidable if the agents do not know the environment because it is impossible to learn the parts that they do not interact with. Together with Theorem 6 and the asymptotic optimality of the Thompson sampling policy [LLOH16, Thm. 4] that is reflective-oracle computable we get the following corollary. Corollary 29 (Convergence to Equilibrium). There are limit computable policies π 1 , . . . , π n such that for any computable multi-agent environment σ and for all ε > 0 and all i ∈ {1, . . . , n} the σ π1:n -probability that the policy π i is an ε-best response converges to 1 as t → ∞.
 DISCUSSION This paper introduced the class of all reflective-oraclecomputable environments M O refl . This class solves the grain of truth problem because it contains (any computable modification of) Bayesian agents defined over M O refl : the optimal agents and Bayes-optimal agents over the class are all reflective-oracle-computable (Theorem 22 and Corollary 24). If the environment is unknown, then a Bayesian agent may end up playing suboptimally (Example 27). However, if each agent uses a policy that is asymptotically optimal in mean (such as the Thompson sampling policy  [LLOH16] ) then for every ε > 0 the agents converge to an ε-Nash equilibrium (Theorem 28 and Corollary 29). Our solution to the grain of truth problem is purely theoretical. However, Theorem 6 shows that our class M O refl allows for computable approximations. This suggests that practical approaches can be derived from this result, and reflective oracles have already seen applications in one-shot games  [FTC15b] . Figure 1 : 1 Figure 1: Answer options of a reflective oracle O for the query (T, x, p); the rational p ∈ [0, 1] falls into one of the three regions above. The values of λ O T (0 | x) and λ O T (1 | x) are depicted as the length of the line segment under which they are written.
 1} * and all T ∈ T , and (b) O is a reflective oracle.
 (a) Since Õk+1 extends Õk , each Õk approximates O. Let x ∈ {0, 1} * and T ∈ T and consider the sequence a k := λ Õk T (1 | x) for k ∈ N. By Lemma 15, a k ≤ a k+1 , so the sequence is monotone increasing. By Lemma 10, a k ≤ λ O T (1 | x), so the sequence is bounded. Therefore it must converge. But it cannot converge to anything strictly below λ O T (1 | x) by the definition of T O . (b) By definition, O is an oracle; it remains to show that O is reflective. Let q i = (T, x, p) be some query. If p < λ O T (1 | x), then by (a) there is a k large enough such that p < λ Õt T (1 | x) for all t ≥ k. For any t ≥ max{k, i}, we have Õt (T, x, p) = 1 since Õt is t-partially reflective. Therefore 1 = lim k→∞ Õk (T, x, p) = O(T, x, p). The case 1 − p < λ O T (0 | x) is analogous.
 [Hut05], we define a Bayesian mixture over the class M O refl . Let w ∈ ∆M O refl be a lower semicomputable prior probability distribution on M O refl . Possible choices for the prior include the Solomonoff prior w λ O T := 2 −K(T ) , where K(T ) denotes the length of the shortest input to some universal Turing machine that encodes T [Sol78]. 2 We define the corresponding Bayesian mixture ξ(e t | ae <t a t ) := ν∈M O refl w(ν | ae <t )ν(e t | ae <t a t ) (3) where w(ν | ae <t ) is the (renormalized) posterior, w(ν | ae <t ) := w(ν) ν(e <t | a <t ) ξ(e <t | a <t ) .(4)The mixture ξ is lower semicomputable on an oracle Turing machine because the posterior w( • | ae <t ) is lower semicomputable. Hence there is an oracle machine T such that ξ = λ O T . We define its completion ξ := λ O T as the completion of λ O
 , since O is reflective, we have that ξ dominates all environments ν ∈ M O refl : ξ(e 1:t | a 1:t ) = ξ(e t | ae <t a t )ξ(e <t | a <t ) ≥ ξ(e t | ae <t a t )ξ(e <t | a <t ) = ξ(e <t | a <t ) | ae <t )ν(e t | ae <t a t ) = ξ(e <t | a <t ) ν∈M O refl refl w(ν ν∈M O w(ν) ν(e <t | a <t ) ξ(e <t | a <t ) ν(e t | ae <t a t ) = w(ν)ν(e 1:t | a 1:t ) ν∈M O refl ≥ w(ν)ν(e 1:t | a 1:t ) This property is crucial for on-policy value convergence. Lemma 21 (On-Policy Value Convergence [Hut05, Thm. 5.36]). For any policy π and any environment µ ∈ M O refl with w
 In each time step t = 1, 2, . . ., the environment receives action a i a 1 t agent π 1 e 1 t a 2 t agent π 2 e 2 t multi-agent environment σ . . . a n t agent π n e n t Figure 2: Agents π 1 , . . . , π n interacting in a multi-agent environment. each agent. Each percept e i t = (o i t , r i t ) contains an obser-vation o i t and a reward r i t ∈ [0, 1]. Importantly, agent i only sees its own action a i t and its own percept e i t (see Figure 2). We use the shorthand notation a t := (a 1 t , . . . , a n t ) and e t := (e 1 t , . . . , e n t ) and denote ae i <t = a i 1 e i 1 . . . a i t−1 e i t−1 t from agent i and outputs n percepts e 1 t , . . . , e n t ∈ E, one for
			 Technically, the lower semicomputable prior 2 −K(T ) is only a semidistribution because it does not sum to 1. This turns out to be unimportant.


Abstract: This paper looks at the convergence property of off-policy Monte Carlo agents with variable behaviour policies. It presents results about convergence and lack of convergence. Even if the agent generates every possible episode history infinitely often, the algorithm can fail to converge on the correct Q-values. On the other hand, it can converge on the correct Q-values under certain conditions. For instance, if, during the n-th episode, the agent has an independent probability of 1/ log(n) of following the original policy at any given state, then it will converge on the right Q-values for that policy.

Off-policy Monte Carlo agents with variable behaviour policies

Off-policy Monte Carlo The Monte Carlo agent is a model-free reinforcement learning agent  [3] . These operate when the environment is a Markov decision process (MDP). In an MDP, the next observation depends only on the current observation -the state -and the current action. The full set of state action pairs is designated by S × A. In each state, an agent chooses its actions according to a (possibly stochastic) policy π, that is assumed to be Markov. Monte Carlo agents operate by computing the Q-values of a policy for each state-action pair (s, a), which is the expected return if the agent choose a in s and subsequently follows π. It is episodic, exploring the same MDP repeatedly to compute Q-values. Episodic MDP's have initial state (where the agent starts an episode) and terminal states (where the agent ends an episode). This paper focuses on computing Q-values, not on updating policy choice in consequence. Assumption 1 In the following, these are always assumed, required for convergence results: (a) The MDP is finite, (b) Whatever policy the agent uses, its expected time until reaching a terminal state is finite. (c) The rewards after each step have finite expectation (consequently the total reward has finite expectation). If an agent is following one policy (e.g., ρ) but wishes to compute the Qvalues of another (e.g., π), it can use the off-policy Monte Carlo algorithm  [3] . In this case ρ is the behaviour policy, and π is the evaluation policy. The algorithm requires that for all state-action pairs (s, a): π(a|s) > 0 ⇒ ρ(a|s) > 0. ( ) 1 Because of this requirement, π(a|s) is zero whenever ρ(a|s) is, and the ratio π(a|s)/ρ(a|s) can be defined as 0 in these cases. The full algorithm is then given in box 1 -there are two variants, ordinary importance sampling using N (s, a) (the number of episodes in which the agent has encountered the state-action pair (s, a)) as the denominator, and weighted importance sampling using W (s, a) (the sum of the weights) instead  [4, 1] . 1. For all s ∈ S and a ∈ As, initialise N (s, a), W (s, a) and W R(s, a) to zero. 2. For all n ≥ 1: (a) Generate an episode history hn by following policy ρ. (b) The episode data will consist of the state, action chosen in the state, the immediate reward experienced by the agent. (c) For each state-action pair (s, a) appearing in the episode history: i. Let t be the first appearance of (s, a) in the history. ii. Let Rn(s, a) be the total subsequent reward. iii. Define  Note that W (s, a) and N (s, a) have the same expectation: let H (s,a) be the set of possible histories in the MDP subsequent to (s, a). For h ∈ H (s,a) , relabel the indexes so that h starts in step zero at (s, a). Use ρ and π to denote the probabilities of certain events conditional on the agent following those policies, and E ρ and E π similarly. Then if I n (s, a) denotes the event that (s, a) appears in the n-the episode, wn(s, a) = k>t π(a k |s k ) ρ(a k |s k ) . (2) E ρ (w(s, a)|I n (s, a)) = h∈H (s,a) ρ(h|I n (s, a)) k>0 π(h a k |h s k ) ρ(h a k |h s k ) = h∈H (s,a) ρ(h|I n (s, a)) π(h|I n (s, a)) ρ(h|I n (s, a)) = h∈H (s,a) π(h|I n (s, a)) = π(H (s,a) |I n (s, a)) = 1. ( ) 3 where h a k denotes the k-th action of history h, and h s k denotes the k-th state of history h. Then simply note that the value of N n (s, a) is simply the sum The same argument as in equation  (3)  shows that E ρ (w n (s, a)R n (s, a)|I n (s, a)) = h∈H (s,a) R h (s, a)ρ(h|I n (s, a)) k>0 π(h a k |h s k ) ρ(h a k |h s k ) = E π (R(s, a)|I n (s, a)), (4) where R h (s, a) is the reward along the history h subsequent to the first (s, a). Thus the expected weighted reward from following policy ρ, is the expected reward from following policy π. Assume that agents following ρ would almost surely explore every stateaction pair infinitely often (equivalently, that N n (s, a) → ∞ almost surely). Then the convergence of ordinary importance sampling is simply a consequence of the law of large numbers applied to every episode that visits (s, a). Convergence of weighted importance sampling is a consequence of Corollary 1 below.
 Varying the behaviour policy The previous proof, however, assumes that policy ρ is fixed. But what if it varies from episode to episode? If π and π are two policies, any p ∈ [0, 1] defines the mixed policy: (1 − p)π + pπ . This is the policy that, in each state, independently chooses whether to follow π, with probability 1 − p, and π , with probability p. Because the decisions are independent, the mixed policy is Markov if π and π are. A general form for all behaviour policies is then: Proposition 1. If ρ and π obey the restriction in equation (  1 ), then there exists a θ ∈ [0, 1) and a policy π such that ρ = (1 − θ)π + θπ . Proof. Let σ be the maximum across S × A of the ratio π(a|s)/ρ(a|s), in all cases where ρ(a|s) = 0. Then define (1 − θ) = 1/σ. Since π(a|s)/ρ(a|s) ≤ σ, (1 − θ) = 1/σ ≤ ρ(a|s)/π(a|s) and hence ρ(a|s) ≥ (1 − θ)π(a|s). Then define π as π (a|s) = 1 θ ρ(a|s) − (1 − θ)π(a|s) . To check that this quantity is less than 1, note that ρ(a|s ) = 1 − b =a ρ(b|s) ≤ 1 − b =a (1 − θ)π(b|s) hence that ρ(a|s) − (1 − θ)π(a|s) ≤ 1 − b (1 − θ)π(b|s) ≤ 1 − (1 − θ) ≤ θ. Then, by construction, ρ(a|s) = (1 − θ)π(a|s) + (θ)π (a|s). For n being the episode number, define ρ n by ρ n = (1 − θ n )π + θ n π n , (5) for some π n and θ n ∈ [0, 1). For convenience, further define the set S ⊂ S, which is the set of states s such that ρ n (s) = π(s). Then neither off-policy Monte Carlo algorithm need converge: Theorem 1. Neither off-policy Monte Carlo algorithm following policy ρ n need converge to the correct Q-values for π, even if the agent generates every possible episode history infinitely often. The proof is given in Section 3. But that convergence failure need not happen for all such ρ n . A sufficient condition for convergence is: Theorem 2. Let σ n = 1/(1−θ n ). Assume that σ n is eventually non-decreasing, and that there exists a δ > 0 such that, for large enough n, (σ n ) √ log(n) < n 1−δ . Assume that π visits each state-action pair with non-zero probability. Then both off-policy Monte Carlo algorithms following policy ρ n will almost surely converge on the correct Q-values for π. This will be proved in Section 4. Note that the requirement for visiting all (s, a) pairs is for the evaluation policy π -the behaviour policy will also do so, as a consequence of the conditions above. One immediate corollary of theorem 2 is: Corollary 1. If θ n is bounded above by κ < 1, both off-policy Monte Carlo algorithms using ρ n will converge on the correct Q-values for π, if π visits each state-action pair with non-zero probability. Proof. This kind of result has been proved before  [5] . The σ n are bounded above by 1/(1 − κ), and thus, for any δ, are less than n 1−δ for sufficiently large n. Then Theorem 2 implies the result. A second corollary is: Corollary 2. If θ n = 1 − 1 log(n) , both off-policy Monte Carlo algorithms using ρ n will converge on the correct Q-values for π, if π visits each state-action pair with non-zero probability. Proof. Set δ = 1/2, note that σ n = log(n), and that log (σ n ) √ log(n) = log(n) log(log(n)) < log(n) 2 = log( √ n) for large enough n. Hence, for large enough n, (σ n ) √ log(n) < √ n = n 1−1/2 . Then Theorem 2 implies the result. And a third corollary is: Corollary 3. Assume π n (a|s) > r (s,a) whenever π(a|s) > 0, for constants r (s,a) > 0. Then both off-policy Monte Carlo algorithms using ρ n will converge on the correct Q-values for π, if π visits each state-action pair with non-zero probability. Proof. Let r be the minimum value of r (s,a) /π(a|s), across all (s, a) where π(a|s) > 0. Then π n can be rewritten as: π n = rπ + (1 − r)π n , for some π n . Similarly ρ n can be rewritten as ρ n = (1 − (1 − r)θ n )π + (θ n )(1 − r)π n . Then note that (θ n )(1 − r) is bounded above by 1 − r, and the result follows by corollary 1. The rest of this paper will be dedicated to proving theorems 1 and 2.
 Failure to converge This Section aims to prove Theorem 1, by constructing a counter-example using the MDP of figure  1 . Set π(a|s 1 ) = 1, π(a|s 2 ) = π(b|s 2 ) = 1/2, S = {s 2 }, π n (a|s 2 ) = 0, π n (b|s 2 ) = 1, and define θ n as The probability of the n-th episode history being h b = (s 1 , a, 0, s 2 , b, −3, s 4 ) is greater or equal to 1/2: this history therefore almost surely gets generated infinitely often. Now consider the episode history h a = (s 1 , a, 0, s 2 , a, −1, s 4 ). It will get generated during episode n with probability θ n = 1 − 1/(n log(n)). 1 2 • (1 − θ n ) = 1 2n log(n) . Each episode is independent and n 1/(2n log(n)) = ∞, so by the converse Borel-Cantelli lemma, the episode is generated infinitely often. Then note that if the history h a is generated during the n-th episode, it is generated with weight 1/(1 − θ n ) = n log(n), while the history h b is generated with weight 1/(1 + θ n ) = 1/(2 − 1/(n log(n))). Split the weight total W n (s 1 , a) as W a n (s 1 , a) + W b n (s 1 , a) (the weight totals due to the histories h a and h b respectively). Then ordinary importance sampling will give Q n as: Q n (s 1 , a) = −W a n (s 1 , a) − 3W b n (s 1 , a) N n (s 1 , a) , (6) while weighted importance sampling gives: Q n (s 1 , a) = −W a n (s 1 , a) − 3W b n (s 1 , a) W a n (s 1 , a) + W b n (s 1 , a) . (7) In state s 1 , both actions are equiprobable and independent, so the law of large numbers implies N n (s 1 , a)/n → 1 almost surely. The W b n (s 1 , a) is the sum of weights less than 1, so W b n (s 1 , a) ≤ N n (s 1 , a). The probability of h b increases to 1, and the weights for that history are larger than 1/2 for n ≥ 1. Thus, almost surely, for sufficiently large n, n/3 and 2n are lower and upper bounds for both N n (s 1 , a) and W b n (s 1 , a). Now pick an n where episode history h a is generated, which must happen infinitely often. The weight of this history is n log(n), so W a n (s 1 , a) ≥ n. In the limit, this must dominate the n-bounded contributions of N n (s 1 , a) and W b n (s 1 , a). Thus, for a large enough n where h a is generated, equation (6) then gives the upper bound Q n (s 1 , a) ≤ −C log(n), for some constant C. Conversely, equation (7) gives an upper bound: Q n (s 1 , a) ≥ −1 − C/ log(n), for some C. The correct Q-value for Q(s 1 , a) under π is clearly (−3−1)/2 = −2, so neither algorithm can converge to the correct values. Ordinary importance sampling clearly cannot find the optimal policy (of choosing a in state s 2 ) either. If b had a reward of 0 instead of a reward of −2, then weighted importance sampling would fail to find the optimal policy on that MDP, so both algorithms can fail to find optimal policies. Remark 1. The lack of convergence can also be proved for θ n = 1 − 1/n, but the proofs are more involved.
 Proof of convergence This Section aims to prove 3 Theorem 2.
 Infinite variance The reason that such mathematical machinery is needed is because, in many cases, the variance of the reward become infinite. Consider the MDP of figure  2 . Two actions are available in s 2 : action a which, with p probability will return the agent to state 1 with a reward of 1, and otherwise send them to state s 3 with no reward. And b, which goes straight to s 3 with no reward. Set π(a|s 1 ) = 1, π n (b|s 2 ) = 1, and θ n = 1 − q. The weights are powers of 1/q, ρ n (a|s 2 ) = q, so the expected weighted reward is the correct ∞ l=0 (l/q l )(pq) l , which is finite. Since for any random variable X, Var(X) = E(X 2 ) − E(X) 2 , the variance of the weighted reward is finite iff the expected squared weighted reward is finite. Then the expected squared weighted reward is: ∞ l=0 (l/q l ) 2 (pq) l = ∞ l=0 l 2 (p/q) l . And this sum diverges for q ≤ p.
 Proving convergence This proof will use ordinary importance sampling, then generalise to weighted importance sampling. Fix any pair (s, a). Since π must visit that pair with finite probability, there is an m < |S × A| and a probability τ > 0 such that an agent following π would  s 3 s 2 s 1 a, 0 a, 0, 1 − q b, 0 a, 1, q (σ n ) m < (σ n ) √ log(n) < n 1−δ < n. Hence, for large enough n, τ (1 − θ n ) m = τ (1/σ n ) > τ /n, so ∞ τ (1 − θ n ) m = ∞. Then since each episode is independent, the converse Borel-Cantelli lemma implies that an agent following ρ will almost surely visit (s, a) infinitely often. How regular will these visits be? The expected number of visits during the episodes up to the n-th is simply n j=1 τ (1 − θ j ) m . For large enough n, σ n < n 1−δ √ log(n) and so eventually τ (1 − θ n ) m > τ n −m 1−δ √ log(n) > n −δ , for any δ > 0. Therefore, for large enough n, the number of visits to (s, a) among the n episodes must be almost surely greater than n j=1 j −δ = O(n 1−δ ). Thus, for large enough n, N n (s, a) > n 1−δ almost surely. Let Q * be the true Q-value of the MDP under π, and let H be the set of possible episode histories, ignoring rewards. Let H l be the set of histories of length l. Write π(h n ) for h ∈ H to designate the probability that episode n has history h if the agent follows the policy π. Note that since π is fixed, π(h n ) is independent of n (unlike ρ n (h n )). Let η be the policy designed to maximise the expected time the agent spends in the MDP. This means that η is a Markov policy, as the policy that maximises the time spent if the agent is in state (s , a ) does not depend on the agent's prior history. Combined with the MDP, η describes a Markov chain, with absorbing final states. Its transition matrix is of the form: P R 0 Id , for P a transition matrix on the non-absorbing state-action pairs, and Id the identity matrix on the absorbing ones. Since any episode must terminate with probability 1, the matrix P has a single maximal real eigenvalue µ < 1  [2] . For large n, the probability that the matrix will not have terminated by the n-th episode is bounded by (µ ) n . Since η is the policy that maximises the expected time spent in the MDP, an agent following π cannot expect to stay in MDP longer than that, so there exists a C such that π(H l ) ≤ C (µ ) l . Fix any µ < µ < 1, then because l 2 (µ ) l must eventually be less than µ l , there exists a C such that lπ(H l ) ≤ l 2 π(H l ) ≤ Cµ l . Let E be the maximal expected reward the agent can generate from a single state-action pair. Let S be the maximal expected squared reward the agent can generate from a single state-action pair. Then if R h is the random variable denoting the reward generated along history h ∈ H l , E (R h |h) ≤ lE Var (R h |h) ≤ E R 2 h |h ≤ l 2 S. Let W R l n denote the random variable that returns 0 if the length of the n-th episode is not l, and the (weighted) reward otherwise, under the assumption that the agent visits (s, a) during episode n. Therefore: W R l n = h∈H l w(h n )R h ρ(h n ). Note that w(h n ) = π(h n )/ρ(h n ) ≤ (σ n ) i h , where i h ≤ l is the number of times that the agent goes through a state in S along h. Then E ρn W R l n = h∈H l w(h n )E (R h |h) ρ n (h n ) = h∈H l E (R h |h) π(h n ), which is the expected reward from episode histories of length l from an agent following π. Therefore ∞ l=1 E ρn W R l n = h E (R h |h) π(h n ) = Q * . The expectation and variance of W R l n can be bounded as: E ρn W R l n = h∈H l E (R h |h) π(h n ) ≤ lEπ(H l ) ≤ ECµ l Var ρn W R l n ≤ E ρn W R l n 2 ≤ h∈H l E (w(h n )R h ) 2 |h ρ(h n ) ≤ h∈H l (σ n ) i h E (R h ) 2 |h π(h n ) ≤ π(H l )(σ n ) l l 2 S ≤ SC(µσ n ) l . (8) Redefine C as max(EC, SC) so that these bounds are Cµ l and C(µσ n ) l respectively. Then define W R <l n = j<l W R j n and W R ≥l n = j≥l W R j n . Equation (8) then implies that E ρn W R ≥l n ≤ C ∞ j=l µ j ≤ Cµ l 1 − µ Var ρn W R <l n ≤ l−1 j=1 Var ρn W R <l n ≤ l−1 j=1 C(µσ n ) j ≤ C(σ n µ) l − 1 σ n µ − 1 ≤ C(σ n ) l σ n µ − 1 Redefine C as C/(1 − µ) so that the first bound is Cµ l . For large enough n, σ n µ − 1 > 1; so, redefining C if needed to cover the finitely many smaller values of n, the second bound is C(σ n ) l : E ρn W R ≥l n ≤ Cµ l Var ρn W R <l n ≤ C(σ n ) l Let I n be the indexing variable that denotes that the agent visited (s, a) during episode n. By assumption, N n (s, a) = n j=1 I n (s, a), and this section has shown that N n (s, a) > n 1−δ for any δ > 0 and large enough n. Define Q ≥l n = 1 N n (s, a) n j=1 I j W R ≥l j Q <l n = 1 N n (s, a) n j=1 I j W R <l j . Then, since σ n are eventually non-decreasing, for any δ > 0 and for large enough n: E ρn Q ≥l n ≤ 1 N n (s, a) n j=1 I j Cµ l ≤ n j=1 I j N n (s, a) Cµ l ≤ Cµ l Var ρn Q <l n ≤ 1 N n (s, a) 2 n j=1 CI j (σ j ) l ≤ C N n (s, a) (σ n ) l ≤ C n 1−δ (σ n ) l . The following is a key Lemma: Lemma 1. Let 1 ≤ m 1 ≤ m 2 . . . be a sequence that is lacunary in that there exists a c > 1 such that m j+1 /m j > c for sufficiently large j. Then Q mj converges to Q * almost surely as j → ∞. Proof. Fix any > 0, and consider the subsequence n j = e √ j . Then since {m j } is eventually an exponentially growing sequence, m j > n j for sufficiently large j. Set l j = 4 √ j and consider Q mj = Q ≥lj mj + Q <lj mj . Then note that if k = 4 √ j, E Q ≥lj mj ≤ Cµ lj , and j 2 2 Cµ lj ≤ k 8 2 Cµ k−1 → 0, as k → ∞. This implies that for large enough j, E Q ≥lj mj ≤ j 2 . Any random variable X with finite expectation has the first moment bound: P (|X| ≥ λ) ≤ E|X| λ . Setting λ = , this implies that P (|Q ≥lj mj | ≥ ) ≤ 1 j 2 for large enough j. Since ∞ j=1 1 j 2 < ∞, the Borel-Cantelli lemma implies that, almost surely, |Q ≥lj mj | ≥ only finitely often. Since Q * = E Q mj = E Q ≥lj mj +E Q <lj mj , this also implies |E Q <lj mj − Q * | ≥ only finitely often. Consider now Var Q <lj mj ≤ C (mj ) 1−δ (σ mj ) lj . Note that l j = 4 √ j ≤ √ j ≤ log(n j ) ≤ log(m j ) . So, by the assumption of Theorem 2, for large enough j, (σ mj ) lj < m 1−δ j for some δ > 0. Fix δ = δ/2. Hence for large enough j, C (mj ) 1−δ/2 (σ mj ) lj < 1 (mj ) δ/2 . Since m δ/2 j > n δ/2 j ≥ e (δ/2) √ j > Cj 2 / 2 for large enough j, eventually Var Q <lj mj ≤ 2 j 2 . Any random variable X with finite variance has the second moment bound: P (|X − E(X)| ≥ λ) ≤ Var(X) λ 2 . Setting λ = , this implies that for large enough j, P (|Q <lj mj − E Q <lj mj | ≥ ) ≤ 1 j 2 . Using the Borel-Cantelli lemma a second time, |Q <lj mj − E Q <lj mj | < except finitely often. Putting together the three bounds completes the proof by showing that, almost surely, except for finitely many j, |Q mj − Q * | < 3 . Since was arbitrary, this completes the proof. The next step is to generalise from lacunary sequences to all sequences. Since all rewards have been assumed positive, nQ n ≤ mQ m for n < m. Then for any 1 > > 0, define the lacunary sequence m j = (1 + ) j . For m j < n < m j+1 , (1 + ) j Q mj ≤ nQ n ≤ Q mj+1 (1 + ) j+1 Q mj+1 . This implies that for large enough j, Q n ≥ Q mj (1/(1 + ) − 2 ) ≥ Q mj (1 − ) and Q n ≤ Q mj ((1 + ) + 2 ) ≤ Q mj (1 + 2 ) , where the 2 term comes from the fact that (1 + ) j need not be an integer. By the Lemma above, the sequence Q mj converge almost surely to Q * , thus, for large enough j and n > (1 + ) j , Q n must be within 3 (Q * + 1) of Q * . Since was arbitrary, this proves that Q n converges to Q * almost surely as n → ∞. This completes the proof for all MDP's with positive rewards. Note that the same proof works for MDP's with negative rewards. Then the general proof is established for by dividing the rewards into positive and negative parts, noting their separate convergence, and noting that the Q-values update process is linear in rewards. Since this proves the convergence of Q n (s, a) to Q * (s, a) for any (s, a), and there are finitely many (s, a) pairs, this proves the almost sure convergence of Q n to Q * in general. It is now necessary to extend the result to weighted importance sampling. To do that, it will suffice to show that, under the conditions above, W n (s, a)/N n (s, a) → 1 almost surely. To see this, change the MDP by setting all rewards to 0, except for the final reward when the agent reaches a terminal state, where they will get 1. This means that the reward along each history is 1, and the weighted reward is just the weight. Thus the new Q n for ordinary importance sampling is Q n (s, a) = W n (s, a)/N n (s, a). By the result we've just proved, these must converge almost surely to the correct Q-values for the modified MDP, i.e., to 1. This proves that the ratios converge almost surely to 1, as required.
 A Note on the independence assumption The policy ρ n , as defined in equation (  5 ), assumes that the agent choose independently between π and π n for each s ∈ S. If this independence is dropped, the situation can get even worse for convergence -the agent may fail to converge to the right values 4 even for fixed θ < 1. Consider the MDP in figure  3 . Define the policy π as choosing randomly amongst the two actions at s 1 , and choosing a otherwise. Define S = {s 2 , s 3 } and π n as choosing b from all states in S . For ρ, the probabilities of choosing π and π n in S are equal (and the same from episode to episode), θ 2 = θ 3 = 0.5, but they are strictly anti-correlated within a given episode. Notice that the episode history (s 1 , a, 0, s 2 , a, 1, s 3 , a, 1, s 4 ) never appears. This is because the uses of π n at s 2 and s 3 are anti-correlated, so the agent cannot avoid both of them. Therefore the agent never experiences a total reward of 2; moreover, the only episodes with rewards are the (equiprobable) iv. Nn(s, a) = Nn−1(s, a) + 1. v. Wn(s, a) = Wn−1(s, a) + wn(s, a). vi. W Rn(s, a) = W Rn−1(s, a) + wn(s, a)R(s, a). vii. Either Qn(s, a) = W Rn(s,a) Nn(s,a) , or Qn(s, a) = W Rn(s,a) Wn(s,a) . (d) Discard the episode data.
 n j=1 I j (s,a) , to see that N n (s, a) and W n (s, a) = n j=1 I j (s,a) w n (s, a) have same expectation. If ρ is fixed, then by sampling only those histories including (s, a), the strong law of large numbers implies that the ratio of N n (s, a) and W n (s, a) converges to 1 almost surely.
 3 a, − 1 Fig. 1 . 311 Fig. 1. An MDP where the off-policy Monte Carlo algorithms can fail to compute the correct Q-values.
 Fig. 2 . 2 Fig. 2. An MDP where the variance of the reward can become infinite within a single episode.
 1 Fig. 3 . 13 Fig. 3. An MDP on which the agent has non-Markov policy choices.
 Table 1 . 1 Off-policy Monte Carlo algorithm
			 The proof will closely mirror the standard proofs of the strong law of large numbers, see for instance https://terrytao.wordpress.com/2008/06/18/the-strong-lawof-large-numbers/
			 For non-Markov policies like this one, the off-policy algorithm has to be adjusted to consider ratios π(h)/ρ(h) for entire histories h (rather the product of state-action pair probabilities), but this is not a large change.