SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Elliot Catt, Jordi Grau Moya, Marcus Hutter, Matthew Aitchison  
Tim Genewein, Gregoire Deletang, Kevin Li Wenliang, Joel Veness
ecatt@google.com
Reinforcement Learning (RL) algorithms typically utilize learning and/or planning techniques to derive effective policies. Integrating both approaches has proven to be highly successful in addressing complex sequential decision-making challenges, as evidenced by algorithms such as AlphaZero and MuZero, which consolidate the planning process into a parametric search-policy. AIXI, the universal Bayes-optimal agent, leverages planning through comprehensive search as its primary means to find an optimal policy. Here we define an alternative universal Bayesian agent, which we call Self-AIXI, that on the contrary to AIXI, maximally exploits learning to obtain good policies. It does so by self-predicting its own stream of action data, which is generated, similarly to other TD(0) agents, by taking an action maximization step over the current on-policy (universal mixture-policy) Q-value estimates. We prove that Self-AIXI converges to AIXI, and inherits a series of properties like maximal Legg-Hutter intelligence and the self-optimizing property.
Reinforcement Learning (RL) [1] algorithms exploit learning, planning  $^{1}$ , or their combination, to obtain good policies from experience. Pure learning consists of using real experience for improving a policy via a (parametric) model, possibly representing an explicit policy-model and/or the Q-values [2-6]. In a sense, learning stores the computational effort of policy-improvement into the parameters, which makes it a computationally efficient approach when needing to reuse the policy later on. In contrast, pure planning finds good policies via simulated experience using an environment model and a randomized (or exhaustive) search policy [1, 7, 8]. In the case of unknown or stochastic environments, one must re-plan after receiving a new observation, thus wasting all computational-effort from the previous step. This makes pure planning a wasteful approach. Using both, planning and learning, is a good way to improve performance and efficiency as demonstrated by modern high-performant RL algorithms such as MuZero [9-12]. These algorithms distill the planning effort back into the parametric search-policy by training it to predict the good actions obtained from planning. In a way, these agents are self-predicting their own policy-improvements. Although empirically successful and widely used, this distillation [13] or self-prediction  $^{2}$  process is motivated in a purely heuristic way without much theoretical understanding on its optimality condition.
The AIXI agent [14, 15] is a theoretical universal Bayes-optimal agent obtained through pure planning without relying on distilling the search effort as described above. AIXI learns an environment model via a Solomonoff predictor [16, 17] and uses it for exhaustive (computationally intractable) planning. Thus, although it uses learning for the environment model, we say AIXI adopts a pure planning approach in the context of policy generation. Two desirable properties of Solomonoff prediction are universality—obtained by considering a huge hypothesis class containing all computable
<sup>1</sup>We use the terms planning and search interchangeably.
Policy distillation usually refers to the process of amortizing one or several policies into another policy model. We view self-prediction as a type of distillation where a search-policy is consolidated into another model.
environments—and fast convergence to the true environment statistics [18, 19]. Unfortunately, universality makes Solomonoff prediction incomputable, a property that AIXI also inherits. Nevertheless, the AIXI agent is considered as the most powerful agent and the gold-standard in decision-making under unknown general environments. An existing gap in the literature, however, is the lack of an alternative universal agent that maximally exploits learning and distillation, instead of the more wasteful planning approach used by AIXI. Understanding this distillation process from a theoretical standpoint can inspire taking new directions in approximating the AIXI agent as done in [20], which is based on a pure planning approach.
The central aim of this paper is to define an agent, which we name Self-AIXI, that maximally exploits self-prediction (or distillation) instead of planning, and to prove that this agent is optimal in the sense that converges to the gold-standard AIXI agent. We define self-prediction as the process of predicting the stream of action-data generated by the agent itself. Self-AIXI generates action-data in a similar fashion as other Temporal Difference TD(0) [1] algorithms in which there is only a single "planning" step when choosing an action i.e.  $\arg \max_{a_t} \hat{Q}(h_t, a_t)$  for history  $h_t$ , action  $a_t$  and current Q-value estimates  $\hat{Q}$ . The difference lies in that Self-AIXI does exact Bayesian inference on the policy space by holding a universal mixture  $\zeta$  over policies (in addition to  $\xi$ , the universal mixture over environments), and consumes action-data generated by maximizing the Self-AIXI policy i.e.  $\arg \max_{a} Q_{\xi}^{\zeta}(h_t, a)$ . In summary our contributions are:
- We define Self-AIXI, a novel universal agent based on self-prediction.
- We prove that Self-AIXI's Q-values converge to AIXI's Q-values asymptotically.
- We show how Self-AIXI leads to maximum Legg-Hutter [21] intelligence and inherits the self-optimization [22] property.
These results provide compelling evidence that self-prediction can effectively serve as a robust alternative to traditional planning methods.
2.1 General Reinforcement Learning
Let  $\mathcal{A},\mathcal{O},\mathcal{R}$  denote the (finite) action, observation and reward set respectively. Let the set of percepts be defined as  $\mathcal{E} := \mathcal{O} \times \mathcal{R}$ .  $\Delta \mathcal{X}$  denotes a distribution over  $\mathcal{X}$ . Define the set of histories  $\mathcal{H} := (\mathcal{A} \times \mathcal{E})^*$ . A policy  $\pi : \mathcal{H} \to \Delta \mathcal{A}$  is a distribution over actions given a history. An environment  $\mu : \mathcal{H} \times \mathcal{A} \to \Delta \mathcal{E}$  is a distribution over percepts given history and action. We use  $h_{<t} := a_1e_1 \ldots a_{t-1}e_{t-1} = a_1o_1r_1 \ldots a_{t-1}o_{t-1}r_{t-1}$  to denote the history of interactions up to time  $t-1$ . Given an environment  $\mu$  and policy  $\pi$  we use  $\mu^{\pi}$  to denote the induced probability measure on histories, i.e.  $h_{<t}$  is assumed to be sampled from  $\mu^{\pi}$ . We are interested in how well agents (policies) perform. We measure this performance by the future expected sum of discounted rewards, called value function.
Definition 1 (Value function and optimal policy). The value  $V_{\nu}^{\pi} : \mathcal{H} \to [0,1]$  of a policy  $\pi$  in an environment  $\nu$ , and discount factor  $\gamma$  given a history  $h_{<t} \in \mathcal{H}$  is defined as
$$
V _ {\nu} ^ {\pi} \left(h _ {<   t}\right) := \frac {1}{\Gamma_ {t}} \mathbb {E} _ {\nu} ^ {\pi} \left[ \sum_ {k = t} ^ {\infty} \gamma^ {k - t} r _ {k} \Bigg | h _ {<   t} \right]
$$
Where  $\Gamma_{t} = \sum_{k = t}^{\infty}\gamma^{k - t} = \frac{1}{1 - \gamma}$  is the normalization factor of the discount. The optimal value is defined as  $V_{\nu}^{*}(h_{<  t})\coloneqq \sup_{\pi}V_{\nu}^{\pi}(h_{<  t})$  , the set of optimal policies with respect to that value is defined as  $\arg \max_{\pi}V_{\nu}^{\pi}(h_{<  t})$  , and an optimal policy with respect to the value is defined as  $\pi_{\nu}^{*}(\cdot |h_{<  t})\in \arg \max_{\pi}V_{\nu}^{\pi}(h_{<  t})$
We can similarly define the action value or Q-value function as  $Q_{\nu}^{\pi}(h_{< t},a_{t}):= V_{\nu}^{\pi}(h_{< t}a_{t})$ , which allows to write the pseudo-recursive Bellman forms
$$
V _ {\nu} ^ {\pi} \left(h _ {<   t}\right) = \sum_ {a _ {t} \in \mathcal {A}} \pi \left(a _ {t} \mid h _ {<   t}\right) Q _ {\nu} ^ {\pi} \left(h _ {<   t}, a _ {t}\right), Q _ {\nu} ^ {\pi} \left(h _ {<   t}, a _ {t}\right) = \sum_ {e _ {t} \in \mathcal {E}} \nu \left(e _ {t} \mid h _ {<   t} a _ {t}\right) \left(r _ {t} + \gamma V _ {\nu} ^ {\pi} \left(h _ {1: t}\right)\right) (1)
$$
We use throughout the paper the Total-Variation distance as a convenient way to compare two policies.
Definition 2 (TV distance). We define the Total Variation (TV) distance of two measures  $\nu_{1}^{\pi_{1}},\nu_{2}^{\pi_{2}}$  between timesteps  $t$  and  $t\leq m\leq \infty$  as
$$
D _ {m} \left(\nu_ {1} ^ {\pi_ {1}}, \nu_ {2} ^ {\pi_ {2}} | h _ {<   t}\right) := \sup  _ {H \subseteq (\mathcal {A} \times \mathcal {E}) ^ {m}} \left| \nu_ {1} ^ {\pi_ {1}} (H | h _ {<   t}) - \nu_ {2} ^ {\pi_ {2}} (H | h _ {<   t}) \right| \leq 1
$$
The following lemma, useful for our later proofs, shows that TV distance is an upper bound for value function difference in general reinforcement learning.
Lemma 3 ([23, Lemma 4.17]). For any two policies  $\pi_1, \pi_2$  and two environments  $\nu_1, \nu_2$
$$
\left| V _ {\nu_ {1}} ^ {\pi_ {1}} (h _ {<   t}) - V _ {\nu_ {2}} ^ {\pi_ {2}} (h _ {<   t}) \right| \leq D _ {\infty} (\nu_ {1} ^ {\pi_ {1}}, \nu_ {2} ^ {\pi_ {2}} | h _ {<   t})
$$
2.2 Universal Artificial Intelligence
We achieve universality by considering a Bayesian mixture over a class of potential environments  $\mathcal{M}$  which is large enough so as to include the true environment the agent is interacting with. A typical choice is the class of all computable functions.
Definition 4 (Bayesian mixture environment). The Bayesian mixture over the class of environments  $\mathcal{M}$  is defined as
$$
\xi \left(e _ {t} \mid h _ {<   t} a _ {t}\right) := \sum_ {\nu \in \mathcal {M}} w (\nu \mid h _ {<   t}) \nu \left(e _ {t} \mid h _ {<   t} a _ {t}\right) \quad \text {w i t h p o s t e r i o r} \quad w (\nu \mid h _ {1: t}) := w (\nu \mid h _ {<   t}) \frac {\nu \left(e _ {t} \mid h _ {<   t} a _ {t}\right)}{\xi \left(e _ {t} \mid h _ {<   t} a _ {t}\right)}
$$
and  $w(\nu |\epsilon) \coloneqq w(\nu)$  is the prior probability of  $\nu$ .
Given this environment mixture  $\xi$ , we define  $\mathrm{AI}\xi$  (AIXI) as acting optimally with respect to  $\xi$ .
Definition 5 (AI\xi). AIXI, the optimal Bayesian agent is defined as
$$
\pi_{\xi}^{*}(h_{<  t}):= \operatorname *{arg  max}_{a}Q_{\xi}^{*}(h_{<  t},a)
$$
The following results prove that the AIXI agent achieves maximal intelligence in the Legg-Hutter sense and is self-optimizing. These are important to have in mind throughout the paper since we will prove the same type of results for our Self-AIXI agent.
Legg-Hutter Intelligence measures the performance of an agent  $\pi$  in a wide range of environments  $\mathcal{M}$ , weighted by their p(oste)rior plausibility  $w(\cdot)$ :
Definition 6 (LH Intelligence [21]). The Legg-Hutter Intelligence measure  $\Upsilon$  of a policy  $\pi$  given a history  $h_{<t}$  is defined as
$$
\Upsilon (\pi | h _ {<   t}) := \sum_ {\nu \in \mathcal {M}} w (\nu | h _ {<   t}) V _ {\nu} ^ {\pi} (h _ {<   t}) = V _ {\xi} ^ {\pi} (h _ {<   t})
$$
It has been shown that if  $\mathcal{M}$  is such that there exists a policy (sequence) which is able to achieve strong asymptotic optimality then AIXI can achieve this as well. This property is known as self-optimizing.
Theorem 7 (AIXI is Self-optimizing [14]). Let  $\mu$  be some environment. If there is a policy  $\pi$  and a sequence of policies  $\overline{\pi_1},\overline{\pi_2}\ldots$  such that for all  $\nu \in \mathcal{M}$
$$
V _ {\nu} ^ {*} \left(h _ {<   t}\right) - V _ {\nu} ^ {\overline {{\pi_ {t}}}} \left(h _ {<   t}\right)\rightarrow 0 \quad a s \quad t \rightarrow \infty \quad \mu^ {\pi} - a l m o s t s u r e l y \tag {2}
$$
$$
V _ {\nu} ^ {*} (h _ {<   t}) - V _ {\nu} ^ {\pi_ {\xi} ^ {*}} (h _ {<   t}) \rightarrow 0 a s t \rightarrow \infty \mu^ {\pi} - a l m o s t s u r e l y
$$
If  $\pi = \pi_{\xi}^{*}$  and Equation 2 holds for all  $\mu \in \mathcal{M}$ , then  $\pi_{\xi}^{*}$  is strongly asymptotically optimal in the class  $\mathcal{M}$ .
For a taxonomy of classes that do (not) allow for self-optimizing policies, see [21, Figure 3.1]. A challenge that is difficult to overcome is that for geometric discounting, there may not yet be widely-recognized classes of environments that allow self-optimizing policies.
3 Self-prediction in General Reinforcement Learning
We are ready to define our Self-AIXI agent that is based on self-prediction instead of pure planning. We start with our definition of the mixture policy  $\zeta$  and discuss the choices of model class and prior.
Definition 8 (Bayesian mixture policy). The Bayesian mixture over the class of policies  $\mathcal{P}$  is defined as
$$
\zeta (a _ {t} | h _ {<   t}) := \sum_ {\pi \in \mathcal {P}} \omega (\pi | h _ {<   t}) \pi (a _ {t} | h _ {<   t}) \quad \text {w i t h p o s t e r i o r} \quad \omega (\pi | h _ {1: t}) := \omega (\pi | h _ {<   t}) \frac {\pi (a _ {t} | h _ {<   t})}{\zeta (a _ {t} | h _ {<   t})}
$$
and  $\omega(\pi|\epsilon) \coloneqq \omega(\pi)$  is the prior probability of  $\pi$ . Note  $\omega(\cdot)$  is a prior for the mixture policy, while  $w(\cdot)$  was a prior for the mixture environment.
Lemma 9 (Linearity of Q-values). For all  $\mu, \pi, h_{<t}$  and  $a_t$  we have
$$
Q _ {\xi} ^ {\pi} \left(h _ {<   t}, a _ {t}\right) = \sum_ {\nu \in \mathcal {M}} w (\nu | h _ {<   t}) Q _ {\nu} ^ {\pi} \left(h _ {<   t}, a _ {t}\right), \quad Q _ {\mu} ^ {\zeta} \left(h _ {<   t}, a _ {t}\right) = \sum_ {\pi \in \mathcal {P}} \omega (\pi | h _ {<   t}) Q _ {\mu} ^ {\pi} \left(h _ {<   t}, a _ {t}\right)
$$
From the linearity of  $Q$ -values we can define the following  $Q$ -value over the Bayesian mixture environment and Bayesian mixture policy.
$$
Q _ {\xi} ^ {\zeta} \left(h _ {<   t}, a _ {t}\right) := \sum_ {\pi \in \mathcal {P}} \omega (\pi | h _ {<   t}) \sum_ {\nu \in \mathcal {M}} w (\nu | h _ {<   t}) Q _ {\nu} ^ {\pi} \left(h _ {<   t}, a _ {t}\right) \tag {3}
$$
Definition 10 (Self-AIXI). The agent Self-AIXI is defined as taking the (one step) optimal action with respect to  $Q_{\xi}^{\zeta}$ . Formally,
$$
\pi_{S}(h_{<  t}):= \operatorname *{arg  }\max_{a_{t}}Q_{\xi}^{\zeta}(h_{<  t},a_{t})
$$
Remark 11 (Self-AIXI does not optimize the future.). Importantly, Self-AIXI maximizes the  $Q$ -values from the mixture policy instead of the optimal policy, which means that there is no need to optimize the future but it needs to know the  $Q$ -values of the current  $\zeta$ -mixture. Pragmatically, these  $Q$ -values are on-policy, typically easier to estimate than the optimal off-policy values.
Note how action selection and history interact with the policy-mixture. The action selected by the Self-AIXI agent necessarily improves, by definition, over the current value estimates  $\max_{a_t} Q_\xi(h_{<t}, a_t) \geq V_\xi^\zeta(h_{<t})$ . Then, this action is added to the next history  $h_{<t+1}$  which is consumed by the policy-mixture at the next time step i.e.,  $\zeta(a_{t+1}|h_{<t+1})$ . This leads to the following remark.
Remark 12 (Self-predicting minimal improvements.). The policy-mixture does Bayesian inference over the incoming self-generated action-data and makes better predictions over time. Thus, Self-AIXI is self-predicting its own small improvements made by the arg max operation in Definition 10. In the case of using the largest class of policies (all computable functions), the mixture is a Solomonoff predictor with enough power to represent, within itself, the policy evaluation and improvement operation.
Showing that the self-prediction process employed by Self-AIXI converges to AIXI is the aim of our theoretical results in Section 4.
3.1 Choice of class and prior
When choosing an environment class  $\mathcal{M}$  we need to pick one large enough so as to contain the true environment  $\mu$ ; the larger the class the weaker the assumption that  $\mu \in \mathcal{M}$ . However, the larger the class we choose the harder it will be to compute the Bayesian mixture of that class, to the point that  $\xi$  may become incomputable (or at least more incomputable than any  $\nu \in \mathcal{M}$ ). With all of this in mind there are two directions that this can be taken: The first is to choose the largest class such that computing  $\xi$  is still tractable and the second is to choose the largest possible class such that it contains all possible environments our universe could contain (or be). In the first case we can choose something like the class of variable-order MDPs with efficient Context Tree Weighting
(CTW) algorithm [24] for  $\xi$ , as is done in [20]. In the second case we can pick the set of all (cumulatively lower-) semicomputable semimeasures [25]. Without going too much into the choice of prior, a simplicity-based prior is an ideal choice. There is an interesting poly-time computable class of all measures of logarithmic complexity [26] for which  $\xi \in \mathcal{M}$ $((\alpha, \gamma)$ -simple measures). One easy way of achieving  $\xi \in \mathcal{M}$  is to add  $\xi$  to  $\mathcal{M}$  with some weight  $w_0' > 0$  and renormalize  $w_i \sim w_i' = (1 - w_0')w_i$ , then  $\xi' \equiv \xi \in \mathcal{M}' := \{\xi\} \cup \mathcal{M}$ . We can apply the same logic to the policy class  $\mathcal{P}$  and policy prior  $\omega(\pi)$  as we just did for  $\mathcal{M}$  and  $w(\mu)$ , similarly we need to have that  $\pi_S \in \mathcal{P}$ . We leave the full investigation of policy classes to future work. Full discussion on the choice of class  $\mathcal{M}$  and prior  $w(\nu)$  are beyond the scope of this work [14, 27].
While our work is mainly theoretical, we also conducted experiments (see Appendix B) comparing self-prediction, using a Self-AIXI approximation, against the pure planning approach, using an AIXI approximation, using Context Tree Weighting as predictor and Monte-Carlo Tree Search for the Q-value estimates. In short, the Self-AIXI approximation outperforms the AIXI approximation in three environments and performs equally in the remaining two.
In this section we will demonstrate the intelligence of Self-AIXI by showing that it asymptotically converges to AIXI in expectation. We specifically chose this criterion over other types of optimality, since Cesaro and Almost-Sure asymptotic optimality (converging to the optimal policy) are too strong [28-30], and are not satisfied by AIXI. In fact, agents which satisfy these also die with certainty [31].
The goal is to show that Self-AIXI eventually performs as well as the most intelligent agent, and AIXI is the most intelligent agent [21]. We will do this by proving an asymptotic convergence result to the Bayes optimal policy in all environments.
To summarize the results of this section:
- Theorem 18:  $\forall \mu$ .  $V_{\xi}^{\pi_S} \to V_{\xi}^*$  in  $\mu^{\pi_S}$ -expectation.
Theorem 21:  $\forall \mu .V_{\mu}^{\pi_S}\to V_\mu^{\pi_\xi^*}$  in  $\mu^{\pi_S}$  -expectation.
- Theorem 22:  $V_{\mu}^{\pi s} \to V_{\mu}^{*} \mu^{\pi s}$ -almost surely, under similar conditions to the AIXI Self-optimizing result from Theorem 7.
The initial three results successively broaden in scope, with the second being a more general form of the first, and the third further generalizing the second, holding for all  $\mu$  instead of just  $\xi$ . All proofs not found in this section can be found in the appendix.
4.1 Expected AIXI-like behavior
To start with, we will present a useful Lemma about the environment mixture  $\xi$ :
Lemma 13 (Convergence of  $\xi$  to  $\mu$  in TV). For any environment  $\mu \in \mathcal{M}$  and policy  $\pi$  we have
$$
D _ {\infty} \left(\xi^ {\pi}, \mu^ {\pi} | h _ {<   t}\right)\rightarrow 0 \quad a s \quad t \rightarrow \infty \quad \mu^ {\pi} - a l m o s t s u r e l y
$$
Proof. Dominance  $\xi^{\pi}(\cdot) \geq w(\mu)\mu^{\pi}(\cdot)$  implies that  $\mu^{\pi}$  is absolutely continuous with respect to  $\xi^{\pi}$  and therefore  $\xi^{\pi}$  merges strongly with  $\mu^{\pi}$  [32]. For full details and definitions of strong merging and absolute continuity see [23].
We can derive a dual to Lemma 13 and restate it in terms of asymptotic convergence in expectation:
Lemma 14 (Convergence of  $\zeta$  to  $\pi$  in expectation). If  $\pi \in \mathcal{P}$  then for all  $\nu$  we have
$$
\mathbb {E} _ {\nu} ^ {\pi} \left[ D _ {\infty} \left(\nu^ {\pi}, \nu^ {\zeta} \mid h _ {<   t}\right)\right]\rightarrow 0 a s t \rightarrow \infty
$$
Proof. The proof is dual to the proof of Lemma 13 with policies and environments exchanged. Finally, almost sure convergence implies convergence in expectation for bounded random variables.  $\square$
Now we can show that if a policy is close to one-step optimal, then it is also close to optimal (this result is independent of  $\xi$  and will work for any environment, but we are only interested in the  $\xi$  case).
Lemma 15 (A one-step good policy is close to optimal). Let  $\Delta(h) \coloneqq |V_{\xi}^{*}(h) - V_{\xi}^{\pi}(h)|$  with  $h \in (\mathcal{A} \times \mathcal{E})^t$  for  $t \geq t_0 \in \mathbb{N}$ .
$$
I f \mathbb {E} _ {\xi} ^ {\pi} | \max _ {a} Q _ {\xi} ^ {\pi} (h, a) - V _ {\xi} ^ {\pi} (h) | <   \beta \forall t \geq t _ {0}
$$
$$
a n d \mathbb {E} _ {\xi} ^ {\pi} \left[ \max  _ {a} \sum_ {e} \xi (e | h a) \Delta (h a e) \right] \leq (1 + \alpha) \mathbb {E} _ {\xi} ^ {\pi} \Delta (h a e) \quad \forall t \geq t _ {0}
$$
$$
\text {t h e n} \mathbb {E} _ {\xi} ^ {\pi} \Delta (h) <   \frac {\beta}{1 - \gamma (1 + \alpha)} \forall t \geq t _ {0} p r o v i d e d 1 + \alpha <   1 / \gamma
$$
Using the previous Lemma we can show our main result about the behavior of Self-AIXI. This result shows that the performance of Self-AIXI converges to that of AIXI on (expected) histories generated by Self-AIXI. We call a policy that satisfies the second condition for some  $1 + \alpha < 1 / \gamma$  for some (large)  $t_0$ , sensible off-policy. We conjecture this plausible condition holds for  $\pi_S$ , bar some exotic counter-example classes  $\mathcal{M}$ .
Theorem 16 (Self-AIXI converges to AIXI in  $\xi$ -expectation). Assuming  $\pi_S$  is sensible off-policy,
$$
\mathbb {E} _ {\xi} ^ {\pi_ {S}} \left[ V _ {\xi} ^ {*} (h _ {<   t}) - V _ {\xi} ^ {\pi_ {S}} (h _ {<   t}) \right] \to 0 a s t \to \infty .
$$
Proof.  $0 \stackrel{(a)}{\leq} \max_{a} Q_{\xi}^{\zeta}(h, a) - V_{\xi}^{\zeta}(h) \stackrel{(b)}{=} V_{\xi}^{\pi_{S}}(h) - V_{\xi}^{\zeta}(h) \stackrel{(c)}{\leq} D_{\infty}(\xi^{\pi_{S}}, \xi^{\zeta}|h)$ .
where (a) and (b) follow from
$$
V _ {\xi} ^ {\pi_ {S}} (h) \equiv \max  _ {a} Q _ {\xi} ^ {\zeta} (h, a) \geq \sum_ {a} \zeta (a | h) Q _ {\xi} ^ {\zeta} (h, a) = V _ {\xi} ^ {\zeta} (h)
$$
and (c) follows from Lemma 3. Let  $\beta >0$  and  $h\in (\mathcal{A}\times \mathcal{E})^t$ . Now Lemma 14 and the above implies that there exists a  $t_0$  such that for all  $t\geq t_0$
$$
0 \leq \mathbb {E} _ {\xi} ^ {\pi_ {S}} \left[ \max  _ {a} Q _ {\xi} ^ {\zeta} (h, a) - V _ {\xi} ^ {\zeta} (h) \right] \leq \mathbb {E} _ {\xi} ^ {\pi_ {S}} D _ {\infty} \left(\xi^ {\pi_ {S}}, \xi^ {\zeta} | h\right) <   \beta
$$
Taking the expectation of  $0 \leq V_{\xi}^{*} - V_{\xi}^{\pi_S} \leq V_{\xi}^{*} - V_{\xi}^{\zeta}$  (which follows from  $V_{\xi}^{*} \geq V_{\xi}^{\pi_S} \geq V_{\xi}^{\zeta}$ ), Lemma 15 implies
$$
0 \leq \mathbb {E} _ {\xi} ^ {\pi_ {S}} \left[ V _ {\xi} ^ {*} (h) - V _ {\xi} ^ {\pi_ {S}} (h) \right] \leq \mathbb {E} _ {\xi} ^ {\pi_ {S}} \left[ V _ {\xi} ^ {*} - V _ {\xi} ^ {\zeta} \right] \leq \beta / (1 - \gamma (1 + \alpha)).
$$
Since  $\beta > 0$  was arbitrary, we get the  $\mathbb{E}_{\xi}^{\pi_S}[V_{\xi}^*(h) - V_{\xi}^{\pi_S}(h)] \to 0$ .
The previous theorem also holds in expectation over the true environment  $\mu$ , instead of in expectation w.r.t.  $\xi$ , but first we need a small lemma:
Lemma 17  $(\mathbb{E}_{\xi}^{\pi}\to 0$  implies  $\mathbb{E}_\mu^\pi \to 0)$  . If  $\pi$  is such that
$$
\mathbb {E} _ {\xi} ^ {\pi} \left[ V _ {\xi} ^ {*} (h _ {<   t}) - V _ {\xi} ^ {\pi} (h _ {<   t}) \right]\rightarrow 0 a s t \rightarrow \infty .
$$
then for all  $\mu \in \mathcal{M}$  we have
$$
\mathbb {E} _ {\mu} ^ {\pi} \left[ V _ {\xi} ^ {*} (h _ {<   t}) - V _ {\xi} ^ {\pi} (h _ {<   t}) \right]\rightarrow 0 a s t \rightarrow \infty .
$$