SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We can now present the main Theorem of the paper, this theorem states that as Self-AIXI interacts with the true environment  $\mu$  Self-AIXI will learn to act like the optimal Bayesian agent AIXI.
Theorem 18 (Main Theorem: Self-AIXI converges to AIXI in  $\mu$ -expectation). For all  $\mu \in \mathcal{M}$  we have
$$
\mathbb {E} _ {\mu} ^ {\pi_ {S}} \left[ V _ {\xi} ^ {*} (h _ {<   t}) - V _ {\xi} ^ {\pi_ {S}} (h _ {<   t}) \right] \to 0 a s t \to \infty .
$$
Proof. Theorem 16 and Lemma 17.
We can rewrite the above theorem in the form of the Universal Intelligence measure  $\Upsilon$ ,
Corollary 19 (LH Form of Theorem 18). For all  $\mu \in \mathcal{M}$
$$
\mathbb {E} _ {\mu} ^ {\pi_ {S}} \left[ \max  _ {\pi} \Upsilon (\pi | h _ {<   t}) - \Upsilon (\pi_ {S} | h _ {<   t}) \right]\rightarrow 0 a s t \rightarrow \infty
$$
Finally we can generalise our previous result to work for the value function of any  $\mu$ . This is important because we are most interested in the performance of our agent on the true environment. We first need to show that convergence of the value function with  $\xi$  implies convergence of the value function with  $\mu$  (in  $\mu$ -expectation).
Lemma 20 ( $V_{\xi}^{\pi'} \to V_{\xi}^{\pi}$  implies  $V_{\mu}^{\pi'} \to V_{\mu}^{\pi}$  in  $\mu$ -expectation). If  $\pi$  is such that for all  $\mu \in \mathcal{M}$
$$
\mathbb {E} _ {\mu} ^ {\pi} \left[ V _ {\xi} ^ {\pi^ {\prime}} \left(h _ {<   t}\right) - V _ {\xi} ^ {\pi} \left(h _ {<   t}\right)\right]\rightarrow 0 a s t \rightarrow \infty .
$$
and  $D_{\infty}(\mu^{\pi '},\xi^{\pi '}|h_{< t})\to 0$ $\mu^{\pi}$  -almost surely then we have
$$
\mathbb {E} _ {\mu} ^ {\pi} \left[ V _ {\mu} ^ {\pi^ {\prime}} \left(h _ {<   t}\right) - V _ {\mu} ^ {\pi} \left(h _ {<   t}\right)\right]\rightarrow 0 a s t \rightarrow \infty .
$$
Now we can present a convergence in value functions with the true environment  $\mu$ .
Theorem 21 (Self-AIXI converges to the  $\xi$ -optimal agent in  $\mu$ -expectation). For all  $\mu \in \mathcal{M}$  if  $D_{\infty}(\mu^{\pi_{\xi}^{*}}, \xi^{\pi_{\xi}^{*}} | h_{<t}) \to 0$ $\mu^{\pi_S}$ -almost surely then we have
$$
\mathbb {E} _ {\mu} ^ {\pi_ {S}} \left[ | V _ {\mu} ^ {\pi_ {\xi} ^ {*}} (h _ {<   t}) - V _ {\mu} ^ {\pi_ {S}} (h _ {<   t}) | \right]\rightarrow 0 a s t \rightarrow \infty .
$$
Proof. Immediate result from Theorem 18 and Lemma 20
This result states that as Self-AIXI interacts with the true environment  $\mu$ , the expected future performance difference between Self-AIXI and AIXI in  $\mu$  will go to zero, that is to say, they will (asymptotically) have equal performance in  $\mu$ .
4.2 Self-optimization
We have previously shown the performance of Self-AIXI using convergence of value functions  $\mu$ -expectation. We can also demonstrate the performance of Self-AIXI in a more direct way, using self-optimization. That is, showing that if it is possible for a policy to learn to be optimal in all environments in a given class, then Self-AIXI will learn to be optimal in all environments in this class.
Self-AIXI can self-optimize in the same way as AIXI, with an additional assumption on the self model. The additional assumption is that for all  $t$ ,  $h_{< t}$  we have  $V_{\xi}^{\zeta}(h_{< t}) \geq V_{\xi}^{\overline{\pi_t}}(h_{< t}) - \epsilon_t$ .
Theorem 22 (Self-AIXI is Self-optimizing). Let  $\mu$  be some environment. If there is a policy  $\pi$  and a sequence of policies  $\overline{\pi_1},\overline{\pi_2}\ldots$  all contained within  $\mathcal{P}$  such that for all  $t,h_{< t}$  we have  $V_{\xi}^{\zeta}(h_{< t})\geq V_{\xi}^{\overline{\pi_t}}(h_{< t}) - \epsilon_t$  with  $\epsilon_t\to 0$ , and for all  $\nu \in \mathcal{M}$
$$
V _ {\nu} ^ {*} \left(h _ {<   t}\right) - V _ {\nu} ^ {\overline {{\pi^ {*}}}} \left(h _ {<   t}\right)\rightarrow 0 \quad a s \quad t \rightarrow \infty \quad \mu^ {\pi} - a l m o s t s u r e l y \tag {4}
$$
$$
V _ {\nu} ^ {*} (h _ {<   t}) - V _ {\nu} ^ {\pi s} (h _ {<   t}) \rightarrow 0 \quad a s \quad t \rightarrow \infty \quad \mu^ {\pi} - a l m o s t s u r e l y
$$
If  $\pi = \pi_S$  and Equation 4 holds for all  $\mu \in \mathcal{M}$ , then  $\pi_S$  is strongly asymptotically optimal in the class  $\mathcal{M}$ .
There are known important cases of model classes which admit self-optimising policies such as Ergodic MDPs [33] and Ergodic k-MDPs [34]. However, we have not yet shown there exist (interesting) classes  $\mathcal{M}$  and  $\mathcal{P}$  where the assumption that for all  $t,h_{< t}$  we have  $V_{\xi}^{\zeta}(h_{< t})\geq V_{\xi}^{\overline{\pi}_{t}}(h_{< t}) - \epsilon_{t}$  with  $\epsilon_t\to 0$  and the other self-optimizing assumptions hold. We leave this to future work.
Policy Learning (or distillation)	Planning	Environment Model	Bayes Optimal	Practical
AIXI	No	Yes	Yes	Yes	No
Self-AIXI	Yes	No	Yes	Yes	No
MC-AIXI-CTW	No	Yes	Yes	Yes	Yes
μ-Zero	Yes	Yes	Yes	No	Yes
DQN	Yes	No	No	No	Yes
CnC	No	No	No	Yes	Yes
PhiMDP	No	No	No	No	No
Table 1: Comparison between different methods
All models in the table learn an (action) value function.
5.1 Universal Artificial Intelligence
There has been much work studying the Bayesian optimal agent AIXI and extending it in various ways [14, 21, 23]. Some of these extensions include: Knowledge seeking agent (KSA), an extension of AIXI which seeks information leading to a form of maximal exploration [35, 36]. BayesExp, which combines the exploiting of AIXI and exploring of KSA in a way that allows the agent to be weakly asymptotically optimal [37, 38]. An AIXI variant that uses Thompson sampling [39] for additional exploration is asymptotically optimal in expectation [28]. Inquisitive Agent (Inq) uses an information gaining strategy based on expeditions to achieve strong asymptotic optimality [40].
AIXI achieves (by definition) Bayesian optimality, and although there are problems with this notion of optimality, it is widely agreed to be the (current) best choice for an optimality criterion. Every notion of optimality possesses inherit difficulties; these difficulties have been extensively studied Leike and Hutter [29], Lattimore and Hutter [30], Cohen et al. [31].
5.2 Comparison with other methods in the literature
To overcome the computational burden of the AIXI's planning phase, approximation techniques usually relax the planning problem into an estimation problem [41, 42]. The main approach for off-the-shelf estimation methods such as Monte Carlo Tree Search (MCTS) is an (optimistic) estimation of the Q-values. Though seemingly different, planning and learning, in the general case, can be seen as following very similar computational processes as shown by [43]. Other practical methods which reduce or remove the planning problem are distributional reinforcement learning [44], Compress and Control (CnC) [45] and feature reinforcement learning (FRL) [46, 47]. In distributional reinforcement learning and CnC powerful sequence prediction methods are utilized to estimate the distribution of returns directly. While it does not remove the planning entirely, FRL adaptively compresses the general RL problem to a more simple (often MDP) problem and solves that problem by traditional methods. The planning required in the simple problem is often far less than in the general problem.
In Table 1 we compare some exemplary agents where each column denotes whether they use some form of learning (or policy distillation), whether they use explicit search/planning, whether they learn an explicit model of the environment, if they are a Bayes-optimal agent, and if there are practical in the sense that there is an actual implementation of the agent. As can be seen, Self-AIXI is the only Bayes-optimal agent that is universal and exploits learning instead of planning.
5.3 Agents modeling themselves
Self-prediction can be thought as an agent modeling itself. Within the universal AI literature, there are other works that go further and drop the dualism of traditional reinforcement learning for the (more realistic) physicalism where agents are contained within their own environment. In this case many complications arise which previous work tries to address. For example, in [48] the behavior of several agents able to modify their own source code is analyzed. In [49], Orseau et. al. presents a space-time embedded definition of the value function and intelligence, and in [50] Leike et. al. aims to solve the grain of truth (a self-referential) problem of Bayesian agents in GRL, showing that there
is a non-trivial class of environments for which the optimal Bayesian agent over that class is within the class.
5.4 Self Prediction in Traditional RL
Recently traditional reinforcement learning has begun to embrace self-prediction. In [51] Predictions of Bootstrapped Latents (PBL) is presented as a method which can use predictive representations on multitask environments. [52] used Self-Predictive Representations (SPR) to train agents to anticipate future latent states, which showed improvements on Atari games benchmarks. In [53], exploration in visually complex environments was streamlined with BYOL-Explore using a bootstrapped prediction. Additionally, [54] underscores the challenges of representation collapse in self-predictive learning and introduces bidirectional learning to address this. Together, these studies underscore the evolving landscape of self-predictive strategies in traditional RL.
6 Discussion, Future Work and Limitations
Practical considerations for computing the mixture component in each step. Computing the Self-AIXI policy requires estimating Q-values using the Bayesian mixture policy  $\zeta (a_t|h_{< t})$ . Naively, this means evaluating each policy in the mixture and computing its respective posterior weight. However, more efficient schemes can be derived, e.g., similar to how the CTW algorithm computes the Bayesian mixture without explicitly evaluating each hypothesis and computing the (marginalizing) sum over hypotheses in each step.
Leveraging large sequence models. Powerful sequence predictors like large language models (LLMs) are the cornerstone of modern general AI due to their impressive zero- and few-shot performance [55-57]. An open question in AGI research is how to leverage the strength of sequence prediction not only for environment modeling, but also to derive goal-directed policies; how to build agents from predictors. We have shown how to shift computational effort for planning (in AIXI) can be transferred into effort devoted to prediction in Self-AIXI, making self-prediction theoretically very well suited to exploit strong predictors such as LLMs. In contrast to similar ad-hoc approaches [58], Self-AIXI is derived from sound theoretical design principles and comes with theoretical guarantees.
Safety implications. Self-AIXI constitutes a theoretical blueprint for building powerful general AI agents that may be very suitable to leverage strong predictors, such as LLMs, in practice. The implications of such agents have recently been discussed in the philosophical and technical literature, and concerns regarding their safety have been raised [59, 60]. Self-AIXI comes with some interesting favorable safety properties. First and foremost, Self-AIXI is derived from sound reasoning and decision-making principles: Bayesian posterior inference and rational optimization of rewards. Self-AIXI comes with a well-understood theoretical model to study abstract properties, guarantees and bounds. An advantage over AIXI, by design, is the self-model of the agent, which can be analyzed on its own. The explicit separation of beliefs over the environment and posterior over the own policy leads to an agent that is more interpretable and can be verified more easily compared to a monolithic agent where components are mixed. If the self-predictor used is not a Bayesian one, but instead some black box predictor, then both the guarantees of Self-AIXI and the guarantees on interpretability are no longer maintained. We strongly encourage an open and differentiated wider discussion on the safety of (universal) RL agents, and believe that theoretical models, like Self-AIXI, can facilitate this discussion by providing a formal and mathematically concrete formulation.
Limitations. Our theoretical results require that the Self-AIXI policy is part of the policy class under consideration, i.e.,  $\pi_S \in \mathcal{P}$  (meaning that the mixture policy is itself part of the policy class). Generally, if there is a policy in the policy class that is "close" to the optimal policy (in terms of minimal KL divergence), Self-AIXI will still perform well. This result can be stated formally using known formulations and bounds for Bayesian mixtures. We leave a detailed mathematical analysis for future work, since the aim of this work is to introduce the theory of Self-AIXI. Another issue is that Self-AIXI, like the original AIXI, is incomputable for environment classes required for (strict) universality. To tackle this, many practical approximations of AIXI have been proposed in the literature, which provide straightforward starting points for similar approximations for Self-AIXI.
Designing and studying non-trivial practical approximations of Self-AIXI is beyond the scope of this paper.
Future work. On the theoretical side there are several promising directions of future work: exploring which non-trivial policy classes  $\mathcal{P}$  satisfy  $\pi_S \in \mathcal{P}$  (or replacing this assumption); extending this work with a sampling-first approach (e.g. Thompson Sampling). On the practical side the first step is to empirically demonstrate the performance of the self-prediction approach proposed in this paper. We believe that especially when coupled with powerful modern predictors, e.g., transformer-based sequence predictors, high-fidelity approximations of  $\xi$  and  $\zeta$  are possible, which can lead to strong, general agents.
In this paper we presented Self-AIXI, a theoretical framework for universal AI that extends AIXI by self-prediction (the agent performs inference over its own policy). The result is an agent that matches AIXIs performance guarantees, while shifting the computational focus on prediction instead of planning. This makes Self-AIXI interesting for building general agents by leveraging recent breakthroughs in sequence predictors, and combining them with RL in a sound fashion. Current approaches to do this, such as RL from human/artificial feedback (RLHF/RLAF), are quite limited (in terms of generality of the RL formulation) and do not come with optimality or universality guarantees. We believe our results to be an important cornerstone for building agents that go beyond zero- or few-shot transfer, towards rapid and data efficient RL.
We thank Tor Lattimore, Laurent Orseau and Anian Ruoss for their helpful feedback and insightful discussions.
[1] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
[2] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning, pages 5-32, 1992.
[3] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv.org/abs/1506.02438.
[4] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.
[5] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[6] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[7] Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1-43, 2012.
[8] Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: Theory and practice—a survey. Automatica, 25(3):335-348, 1989.
[59] Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., USA, 1st edition, 2014. ISBN 0199678111.
[60] Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019.
[61] Cameron B. Browne, Edward Powley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games, 4(1):1-43, 2012. doi: 10.1109/TCIAIG.2012.2186810.
[62] F.M.J. Willems, Y.M. Shtarkov, and T.J. Tjalkens. The context-tree weighting method: basic properties. IEEE Transactions on Information Theory, 41(3):653-664, 1995. doi: 10.1109/18.382012.
$$
I f \quad \mathbb {E} _ {\xi} ^ {\pi} \left| \max  _ {a} Q _ {\xi} ^ {\pi} (h, a) - V _ {\xi} ^ {\pi} (h) \right| <   \beta \quad \forall t \geq t _ {0}
$$
$$
t h e n \mathbb {E} _ {\xi} ^ {\pi} \Delta (h) <   \frac {\beta}{1 - \gamma (1 + \alpha)} \forall t \geq t _ {0} p r o v i d e d 1 + \alpha <   1 / \gamma
$$
Proof. Let  $\delta \coloneqq \sup_{t\geq t_0}\mathbb{E}\Delta (h)$ , where  $h\in (\mathcal{A}\times \mathcal{E})^t$  and  $\mathbb{E}$  is short for  $\mathbb{E}_{\xi}^{\pi}$ .
$$
\begin{array}{l} \mathbb {E} \Delta (h) = | \max  _ {a} Q _ {\xi} ^ {*} (h, a) - V _ {\xi} ^ {\pi} (h) | \\ = \mathbb {E} \left| \max  _ {a} Q _ {\xi} ^ {*} (h, a) - \max  _ {a} Q _ {\xi} ^ {\pi} (h, a) + \max  _ {a} Q _ {\xi} ^ {\pi} (h, a) - V _ {\xi} ^ {\pi} (h) \right| \\ \leq \mathbb {E} \left| \max  _ {a} Q _ {\xi} ^ {\pi} (h, a) - V _ {\xi} ^ {\pi} (h) \right| + \mathbb {E} \left| \max  _ {a} Q _ {\xi} ^ {*} (h, a) - \max  _ {a} Q _ {\xi} ^ {\pi} (h, a) \right| \\ \stackrel {(1)} {<  } \beta + \mathbb {E} \left| \max  _ {a} \sum_ {e} \xi (e | h a) \left(r + \gamma V _ {\xi} ^ {*} (h a e)\right) - \max  _ {a} \sum_ {e} \xi (e | h a) \left(r + \gamma V _ {\xi} ^ {\pi} (h a e)\right) \right| \\ \leq \beta + \gamma \mathbb {E} \max  _ {a} \sum_ {e} \xi (e | h a) | V _ {\xi} ^ {*} (h a e) - V _ {\xi} ^ {\pi} (h a e) | \\ \leq \beta + \gamma (1 + \alpha) \mathbb {E} \Delta (h a e) \\ \end{array}
$$
Taking  $\sup_{t\geq t_0}$  on both sides implies  $\delta <  \beta +\gamma (1 + \alpha)\delta$  implies  $\delta <  \beta /(1 - \gamma (1 + \alpha))$
Lemma 17  $(\mathbb{E}_{\xi}^{\pi}\to 0$  implies  $\mathbb{E}_{\mu}^{\pi}\to 0)$  . If  $\pi$  is such that
$$
\mathbb {E} _ {\mu} ^ {\pi} \left[ V _ {\xi} ^ {*} (h _ {<   t}) - V _ {\xi} ^ {\pi} (h _ {<   t}) \right] \leq \frac {1}{w (\mu)} \mathbb {E} _ {\xi} ^ {\pi} \left[ V _ {\xi} ^ {*} (h _ {<   t}) - V _ {\xi} ^ {\pi} (h _ {<   t}) \right]\rightarrow 0
$$
by the dominance of  $\xi (\cdot)\geq w(\mu)\mu (\cdot)$
$$
\mathbb {E} _ {\mu} ^ {\pi} \left[ V _ {\xi} ^ {\pi^ {\prime}} (h _ {<   t}) - V _ {\xi} ^ {\pi} (h _ {<   t}) \right] \to 0 a s t \to \infty .
$$
$$
\begin{array}{l} \mathbb {E} _ {\mu} ^ {\pi} \left[ | V _ {\mu} ^ {\pi^ {\prime}} (h _ {<   t}) - V _ {\mu} ^ {\pi} (h _ {<   t}) | \right] \\ = \mathbb {E} _ {\mu} ^ {\pi} \left[ \left| V _ {\mu} ^ {\pi^ {\prime}} (h _ {<   t}) - V _ {\xi} ^ {\pi^ {\prime}} (h _ {<   t}) + V _ {\xi} ^ {\pi^ {\prime}} (h _ {<   t}) - V _ {\xi} ^ {\pi} (h _ {<   t}) + V _ {\xi} ^ {\pi} (h _ {<   t}) - V _ {\mu} ^ {\pi} (h _ {<   t}) \right| \right] \\ \leq \mathbb {E} _ {\mu} ^ {\pi} \left[ | V _ {\mu} ^ {\pi^ {\prime}} (h _ {<   t}) - V _ {\xi} ^ {\pi^ {\prime}} (h _ {<   t}) | \right] + \mathbb {E} _ {\mu} ^ {\pi} \left[ | V _ {\xi} ^ {\pi^ {\prime}} (h _ {<   t}) - V _ {\xi} ^ {\pi} (h _ {<   t}) | \right] + \mathbb {E} _ {\mu} ^ {\pi} \left[ | V _ {\xi} ^ {\pi} (h _ {<   t}) - V _ {\mu} ^ {\pi} (h _ {<   t}) | \right] \\ \end{array}
$$
The second and third term go to 0 as  $t\to \infty$  by the assumptions and Lemma 3 with Lemma 13. The first term goes to 0 as  $D_{\infty}(\mu^{\pi '},\xi^{\pi '}|h_{< t})\to 0\mu^{\pi}$  -almost surely implies  $\mathbb{E}_{\mu}^{\pi}\left[D_{\infty}(\mu^{\pi '},\xi^{\pi '}|h_{< t})\right]\to 0$  and we have  $\mathbb{E}_{\mu}^{\pi}\left[|V_{\mu}^{\pi '}(h_{< t}) - V_{\xi}^{\pi '}(h_{< t})|\right]\leq \mathbb{E}_{\mu}^{\pi}\left[D_{\infty}(\mu^{\pi '},\xi^{\pi '}|h_{< t})\right].$
$$
V _ {\nu} ^ {*} \left(h _ {<   t}\right) - V _ {\nu} ^ {\overline {{\pi_ {t}}}} \left(h _ {<   t}\right)\rightarrow 0 \quad a s \quad t \rightarrow \infty \quad \mu^ {\pi} - a l m o s t s u r e l y \tag {4}
$$
$$
\begin{array}{l} 0 \leq w (\mu | h _ {<   t}) \left(V _ {\mu} ^ {*} (h _ {<   t}) - V _ {\mu} ^ {\pi_ {S}} (h _ {<   t})\right) (5) \\ \leq \sum_ {\nu \in \mathcal {M}} w (\nu | h _ {<   t}) \left(V _ {\nu} ^ {*} (h _ {<   t}) - V _ {\nu} ^ {\pi_ {S}} (h _ {<   t})\right) (6) \\ = \sum_ {\nu \in \mathcal {M}} w (\nu | h _ {<   t}) V _ {\nu} ^ {*} (h _ {<   t}) - V _ {\xi} ^ {\pi s} (h _ {<   t}) (7) \\ \leq \sum_ {\nu \in \mathcal {M}} w (\nu | h _ {<   t}) V _ {\nu} ^ {*} (h _ {<   t}) - V _ {\xi} ^ {\zeta} (h _ {<   t}) (8) \\ \leq \sum_ {\nu \in \mathcal {M}} w (\nu | h _ {<   t}) V _ {\nu} ^ {*} (h _ {<   t}) - V _ {\xi} ^ {\overline {{\pi_ {t}}}} (h _ {<   t}) + \epsilon_ {t} (9) \\ = \sum_ {\nu \in \mathcal {M}} w (\nu | h _ {<   t}) \left(V _ {\nu} ^ {*} (h _ {<   t}) - V _ {\nu} ^ {\overline {{\pi_ {t}}}} (h _ {<   t})\right) + \epsilon_ {t} (10) \\ \rightarrow 0 (11) \\ \end{array}
$$
Equation 6 comes from adding positive terms. Equations 7 and 10 comes from the linearity of the value function. Equation 8 comes from  $\pi_S$  being one step optimal then following  $\zeta$  and . Equation 9 comes from the assumptions. Lastly, 11 comes from Equation 4 and [14, Lem.5.28ii].
$w(\mu |h_{< t})\nrightarrow 0$  as  $h_{< t}$  is generated from  $\mu^{\pi}$  (for more details see Self-Optimizing proof in [14]). Therefore  $V_{\mu}^{*}(h_{< t}) - V_{\mu}^{\pi s}(h_{< t})\to 0\mu^{\pi}$  -almost surely.
B Experimental Validation
In this section we heuristically evaluate the effect of self-modelling using the AIXI approximation provided by Veness et al. This AIXI approximation uses Monte-Carlo Tree Search (MCTS) [61] and the Context Tree Weighting [62] technique respectively to approximate the expectimax planning and Bayesian environment learning components of the AIXI optimal agent. At each timestep the approximation performs a MCTS to estimate the action-value function with respect to its CTW-based environment model. We consider two agent configurations: MC-AIXI, which chooses each action uniformly at random in the rollout stage of MCTS, i.e.  $\hat{\pi}_{\hat{\xi},u}^{*}(a_{t} \mid h_{<t}) := 1 / |\mathcal{A}|$ , and matches identically the original configuration in [20]; and MC-Self-AIXI, which uses Context Tree Weighting, i.e.  $\hat{\pi}_{\hat{\xi},\hat{\zeta}}^{*}(a_{t} \mid h_{<t}) := \mathrm{CTW}_{d}(a_{t} \mid h_{<t})$ , to predict the outcome of each MCTS decision.