SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 7 (Regret of planning optimistically for sub-Gaussian noise). Let  $\pi_n^*$ ,  $\pi_n$  denote the solution to Equation (6) and Equation (7) respectively, and  $z_{n,t}^{*}, z_{n,t}$  the corresponding state-action pairs visited during their respective trajectories. Furthermore, let Assumption 1 and 2 hold, and relax Assumption 3 to  $\sigma$ -sub Gaussian noise. Then, the following is true for all  $n \geq 0$  with probability at least  $1 - \delta$
$$
J _ {n} \left(\boldsymbol {\pi} _ {n} ^ {*}\right) - J _ {n} \left(\boldsymbol {\pi} _ {n}\right) \leq \mathcal {O} \left(L _ {\boldsymbol {\sigma}} ^ {T - 1} \beta_ {n - 1} ^ {T} (\delta) T \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \| \boldsymbol {\sigma} _ {n - 1, j} \left(\boldsymbol {z} _ {n, t}\right) \| \right]\right)
$$
Proof. Curi et al. (2020, Lemma 5) bound the regret with the sum of epistemic uncertainties for Lipschitz continuous reward functions, under Assumption 1 and 2 for sub-Gaussian noise (c.f., Rothfuss et al. (2023, Theorem 3.5) for a more rigorous derivation). For the active exploration setting, the reward in episode  $n + 1$  is
$$
r (\boldsymbol {z}) = \frac {1}{2} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} (\boldsymbol {z})}{\sigma^ {2}}\right).
$$
We show in Lemma 4 that our choice of exploration reward is Lipschitz continuous. Thus, can use the regret bound from Curi et al. (2020).
Compared to the Gaussian case,  $\sigma$ -sub Gaussian noise has the additional exponential dependence on the horizon  $T$ , i.e., the  $\beta_{n}^{T}$  term. This follows from the analysis through Lipschitz continuity. Moreover, as we show in Lemma 2, the regret of planning optimistically is proportional to the change in value under the same optimistic dynamics and policy, but different initial states. The Lipschitz continuity property of our objective allows us to relate the difference in values to the discrepancy in the trajectories. Even for linear systems, trajectories under the same dynamics and policy but different initial states can deviate exponentially in the horizon.
A.3 Proof for general Bayesian models
In this section, we analyze the information gain for general Bayesian models and prove Theorem 1.
Theorem 3 (Entropy of a RV with finite second moment is upper bounded by Gaussian entropy (Theorem 8.6.5 Cover and Thomas (2006))). Let the random vector  $\pmb{x} \in \mathbb{R}^n$  have covariance  $\pmb{K} = \mathbb{E}\left[\pmb{x}\pmb{x}^\top\right]$  (i.e.,  $\pmb{K}_{ij} = E[\pmb{x}_i\pmb{x}_j], 1 \leq i, j \leq n$ ). Then
$$
H (X) \leq \frac {1}{2} \log ((2 \pi e) ^ {n} | \boldsymbol {K} |)
$$
with equality if and only if  $\pmb{x} \sim \mathcal{N}(\pmb{\mu}, \pmb{K})$  for  $\pmb{\mu} = E[\pmb{x}]$ .
Lemma 8 (Monotonocity of information gain). Let  $\pmb{\tau}^{\pi}$  denote the trajectory induced by the policy  $\pmb{\pi}$ . Then, the following is true for all  $n\geq 0$ , policies  $\pmb{\pi}$
$$
\mathbb {E} _ {\mathcal {D} _ {1: n}} \left[ I \left(\boldsymbol {f} _ {\boldsymbol {\tau} ^ {\pi}} ^ {*}; \boldsymbol {y} _ {\boldsymbol {\tau} ^ {\pi}} \mid \mathcal {D} _ {1: n}\right) \right] \leq \mathbb {E} _ {\mathcal {D} _ {1: n - 1}} \left[ I \left(\boldsymbol {f} _ {\boldsymbol {\tau} ^ {\pi}} ^ {*}; \boldsymbol {y} _ {\boldsymbol {\tau} ^ {\pi}} \mid \mathcal {D} _ {1: n - 1}\right) \right].
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {1: n}} \left[ I \left(\boldsymbol {f} _ {\boldsymbol {\tau} ^ {\pi}} ^ {*}; \boldsymbol {y} _ {\boldsymbol {\tau} ^ {\pi}} \mid \mathcal {D} _ {1: n - 1}\right) - I \left(\boldsymbol {f} _ {\boldsymbol {\tau} ^ {\pi}} ^ {*}; \boldsymbol {y} _ {\boldsymbol {\tau} ^ {\pi}} \mid \mathcal {D} _ {1: n}\right) \right] \\ = \mathbb {E} _ {\mathcal {D} _ {1: n}} \left[ H \left(\boldsymbol {y} _ {\tau^ {\pi}} \mid \mathcal {D} _ {1: n - 1}\right) - H \left(\boldsymbol {y} _ {\tau^ {\pi}} \mid \boldsymbol {f} _ {\tau^ {\pi}} ^ {*}, \mathcal {D} _ {1: n - 1}\right) \right. \\ \left. - \left(H \left(\boldsymbol {y} _ {\tau^ {\pi}} \mid \mathcal {D} _ {1: n}\right) - H \left(\boldsymbol {y} _ {\tau^ {\pi}} \mid \boldsymbol {f} _ {\tau^ {\pi}} ^ {*}, \mathcal {D} _ {1: n}\right)\right) \right] \\ = \mathbb {E} _ {\mathcal {D} _ {1: n}} \left[ H \left(\boldsymbol {y} _ {\tau^ {\pi}} \mid \mathcal {D} _ {1: n - 1}\right) - H \left(\boldsymbol {y} _ {\tau^ {\pi}} \mid \mathcal {D} _ {1: n}\right) \right] \\ + \mathbb {E} _ {\mathcal {D} _ {1: n}} \left[ H \left(\boldsymbol {y} _ {\tau^ {\pi}} \mid \boldsymbol {f} _ {\tau^ {\pi}} ^ {*}\right) - H \left(\boldsymbol {y} _ {\tau^ {\pi}} \mid \boldsymbol {f} _ {\tau^ {\pi}} ^ {*}\right) \right] \\ \geq 0 \quad (\text {i n f o r m a t i o n n e v e r h u r t s}) \\ \end{array}
$$
A direct consequence of Lemma 8 is the following corollary.
Corollary 4 (Information gain at  $N$  is less than the average gain till  $N$ ). Let  $\tau^{\pi}$  denote the trajectory induced by the policy  $\pi$ . Then, the following is true for all  $N \geq 1$ , policies  $\pi$ ,
$$
\mathbb {E} _ {\mathcal {D} _ {1: N - 1}} \left[ I \left(\boldsymbol {f} _ {\boldsymbol {\tau} ^ {\pi}} ^ {*}; \boldsymbol {y} _ {\boldsymbol {\tau} ^ {\pi}} \mid \mathcal {D} _ {1: N - 1}\right) \right] \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}} \left[ I \left(\boldsymbol {f} _ {\boldsymbol {\tau} ^ {\pi}} ^ {*}; \boldsymbol {y} _ {\boldsymbol {\tau} ^ {\pi}} \mid \mathcal {D} _ {1: n - 1}\right) \right].
$$
Next, we prove Lemma 1, which is central to our proposed active exploration objective in Equation (6).
Proof of Lemma 1. Let  $\pmb{y}_{\tau^{\pi}} = \{\pmb{y}_t\}_{t=0}^{T-1} = \{\pmb{f}_t^* + \pmb{w}_t\}_{t=0}^{T-1}$ , where  $\pmb{f}_t^* = \pmb{f}^*(\pmb{z}_t)$ . Furthermore, denote with  $\Sigma_n(\pmb{f}_{0:T-1}^*)$  the covariance of  $\pmb{f}_{0:T-1}^*$ .
$$
\begin{array}{l} I \left(\boldsymbol {f} _ {\tau^ {\pi}} ^ {*}; \boldsymbol {y} _ {\tau^ {\pi}} \mid \mathcal {D} _ {1: n}\right) = I \left(\boldsymbol {f} _ {0: T - 1} ^ {*}; \boldsymbol {y} _ {0: T - 1} \mid \mathcal {D} _ {1: n}\right) \\ = H \left(\boldsymbol {y} _ {0: T - 1} \mid \mathcal {D} _ {1: n}\right) - H \left(\boldsymbol {y} _ {0: T - 1} \mid \boldsymbol {f} _ {0: T - 1} ^ {*}, \mathcal {D} _ {1: n}\right) \\ \leq \frac {1}{2} \log \left(\left| \sigma^ {2} \mathbb {I} + \Sigma_ {n} \left(\boldsymbol {f} _ {0: T - 1} ^ {*}\right) \right|\right) - \frac {1}{2} \log \left(\left| \sigma^ {2} \mathbb {I} \right|\right) \tag {Theorem3} \\ \leq \frac {1}{2} \log \left(\left| \operatorname {d i a g} \left(\mathbb {I} + \sigma^ {- 2} \Sigma_ {n} \left(\boldsymbol {f} _ {0: T - 1} ^ {*}\right)\right) \right|\right) \quad \text {(H a d a m a r d ' s i n e q u a l i t y)} \\ = \frac {1}{2} \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n , j} ^ {2} (\boldsymbol {z} _ {t})}{\sigma^ {2}}\right). \\ \end{array}
$$
We can leverage the result from Lemma 1 to bound the average mutual information with the sum of epistemic uncertainties.
Lemma 9 (Average information gain is less than sum of average epistemic uncertainties). Let Assumption 3 hold and denote with  $\bar{\pi}_N$  be the solution of Equation (4). Then, for all  $N\geq 1$  and dataset  $\mathcal{D}_{1:N}$  the following is true
$$
\begin{array}{l} \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}, \boldsymbol {\tau} ^ {\bar {\pi} _ {n}}} \left[ I \left(\boldsymbol {f} _ {\boldsymbol {\tau} ^ {\bar {\pi} _ {N}}}; \boldsymbol {y} _ {\boldsymbol {\tau} ^ {\bar {\pi} _ {N}}} \mid \mathcal {D} _ {1: n - 1}\right) \right] \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}, \boldsymbol {\tau} ^ {\pi_ {n} ^ {*}}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \left(\frac {1}{2} \log \left(1 + \frac {\boldsymbol {\sigma} _ {n , j} ^ {2} (\boldsymbol {z} _ {n , t} ^ {*})}{\sigma^ {2}}\right)\right) \right], \\ \end{array}
$$
where  $z_{n,t}^{*}$  are the state-action tuples visited by the solution of Equation (6), i.e.,  $\pi_n^*$ .
$$
\begin{array}{l} \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}, \boldsymbol {\tau} ^ {\bar {\pi} _ {N}}} \left[ I \left(\boldsymbol {f} _ {\boldsymbol {\tau} ^ {\bar {\pi}} _ {N}} ^ {*}; \boldsymbol {y} _ {\boldsymbol {\tau} ^ {\bar {\pi}} _ {N}} \mid \mathcal {D} _ {1: n - 1}\right) \right] \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}, \boldsymbol {\tau} ^ {\bar {\pi} _ {N}}} \left[ \left(\sum_ {\boldsymbol {z} _ {t} \in \boldsymbol {\tau} ^ {\bar {\pi} _ {N}}} \frac {1}{2} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n , j} ^ {2} (\boldsymbol {z} _ {t})}{\sigma^ {2}}\right)\right) \right] (Lemma 1) \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}} \left[ \max  _ {\pi \in \Pi} \mathbb {E} _ {\tau^ {\pi}} \left[ \sum_ {z _ {t} \in \tau_ {\pi}} \sum_ {j = 1} ^ {d _ {x}} \left(\frac {1}{2} \log \left(1 + \frac {\sigma_ {n , j} ^ {2} (z _ {t})}{\sigma^ {2}}\right)\right) \right] \right] (1) \\ = \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}, \boldsymbol {\tau} ^ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \left(\frac {1}{2} \log \left(1 + \frac {\sigma_ {n , j} ^ {2} \left(\boldsymbol {z} _ {n , t} ^ {*}\right)}{\sigma^ {2}}\right)\right) \right]. \\ \end{array}
$$
Here (1) follows from the tower property. Note that the second expectation in (1) is wrt  $\pmb{\tau}^{\pi}$  conditioned on a realization of  $\mathcal{D}_{1:n - 1}$ , where the conditioning is captured in the epistemic uncertainty  $\sigma_{n}(\cdot)$ .
We use the results from above, to prove Theorem 1.
Proof of theorem 1. Let  $\bar{\pi}_n$  denote the solution to Equation (4) at iteration  $n\geq 1$ . We first relate the information gain from OPAX to the information gain of  $\bar{\pi}_n$ .
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {1: N - 1}, \boldsymbol {\tau} ^ {\pi_ {N}}} \left[ I \left(\boldsymbol {f} _ {\boldsymbol {\tau} ^ {\pi_ {N}}} ^ {*}; \boldsymbol {y} _ {\boldsymbol {\tau}} ^ {\pi_ {N}} \mid \mathcal {D} _ {1: N - 1}\right) \right] \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}, \boldsymbol {\tau} ^ {\pi_ {n}}} \left[ I \left(\boldsymbol {f} _ {\boldsymbol {\tau} ^ {\pi_ {n}}}; \boldsymbol {y} _ {\boldsymbol {\tau} ^ {\pi_ {n}}} \mid \mathcal {D} _ {1: n - 1}\right) \right] (Corollary 4) \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}, \boldsymbol {\tau} ^ {\pi_ {n}}} \left[ \left(\sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} \left(\boldsymbol {z} _ {n , t} ^ {*}\right)}{\sigma^ {2}}\right)\right) \right] (Lemma9) \\ = \frac {1}{N} \sum_ {n = 1} ^ {N} \left(\mathbb {E} _ {\mathcal {D} _ {1: n - 1}, \boldsymbol {\pi} _ {n}} \left[ \sum_ {t = 0} ^ {T - 1} \frac {1}{2} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} (\boldsymbol {z} _ {n , t})}{\sigma^ {2}}\right) \right] + J _ {n} (\boldsymbol {\pi} _ {n} ^ {*}) - J _ {n} (\boldsymbol {\pi} _ {n})\right) \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}} \left[ \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \frac {1}{2} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\boldsymbol {\sigma} _ {n - 1 , j} ^ {2} (\boldsymbol {z} _ {n , t})}{\sigma^ {2}}\right) \right] \right. \\ \left. + \mathcal {O} \left(\beta_ {n - 1} (\delta) T \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, t}\right) \| _ {2} \right]\right)\right)\left. \right] (Corollary3) \\ \end{array}
$$
In summary, the maximum expected mutual information at episode  $N$  is less than the mutual information of OPAX and the sum of model epistemic uncertainties. Crucial to the proof is the regret
bound for optimistic planning from Corollary 3.
$$
\begin{array}{l} \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}} \left[ \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \frac {1}{2} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\boldsymbol {\sigma} _ {n - 1 , j} ^ {2} (\boldsymbol {z} _ {n , t})}{\sigma^ {2}}\right) \right] \right. \\ \left. + \mathcal {O} \left(T \beta_ {n - 1} (\delta) \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n, t}) \| _ {2} \right]\right)\right) \\ = \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}} \left[ \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \log \left(\sqrt {1 + \frac {\boldsymbol {\sigma} _ {n - 1 , j} ^ {2} (\boldsymbol {z} _ {n , t})}{\sigma^ {2}}}\right) \right] \right. \\ \left. + \mathcal {O} \left(T \beta_ {n - 1} (\delta) \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n, t}) \| _ {2} \right]\right)\right) \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}} \left[ \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\boldsymbol {\sigma} _ {n - 1 , j} (\boldsymbol {z} _ {n , t})}{\sigma}\right) \right] \right. \\ \left. + \mathcal {O} \left(T \beta_ {n - 1} (\delta) \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n, t}) \| _ {2} \right]\right)\right) \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}} \left[ \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {\sigma_ {n - 1 , j} (\boldsymbol {z} _ {n , t})}{\sigma} \right] \right. \\ \left. + \mathcal {O} \left(T \beta_ {n - 1} (\delta) \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n, t}) \| _ {2} \right]\right)\right) \quad (\log (1 + x) \leq x \text {f o r} x \geq 0.) \\ \leq \mathcal {O} \left(\frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}} \left[ T \beta_ {n - 1} (\delta) \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n, t}) \| _ {2} \right] \right]\right) \\ \end{array}
$$
Above, we show that the maximum expected mutual information can be upper bounded with the sum of epistemic uncertainties for the states OPAX visits during learning. Finally, we further upper bound this with the model complexity measure.
$$
\begin{array}{l} \mathcal {O} \left(\frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\mathcal {D} _ {1: n - 1}} \left[ \beta_ {n - 1} (\delta) T \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n, t}) \| _ {2} \right] \right]\right) \\ = \mathcal {O} \left(\frac {1}{N} \sqrt {\left(\mathbb {E} _ {\mathcal {D} _ {1 : N}} \left[ \sum_ {n = 1} ^ {N} (T \beta_ {n - 1} (\delta)) \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n , t}) \| _ {2} \right]\right]\right)\right) ^ {2}\left. \right) \\ \leq \mathcal {O} \left(\frac {1}{N} T \beta_ {N} (\delta)) \sqrt {T N \mathbb {E} _ {\mathcal {D} _ {1 : N}} \left[ \sum_ {n = 1} ^ {N} \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} | | \boldsymbol {\sigma} _ {n} (\boldsymbol {z} _ {n , t}) | | _ {2} ^ {2} \right] \right]}\right) \\ \leq \mathcal {O} \left(\beta_ {N} (\delta) T ^ {3 / 2} \sqrt {\frac {\mathcal {M C} _ {N} \left(\boldsymbol {f} ^ {*}\right)}{N}}\right) \\ \end{array}
$$
Theorem 1 gives a bound on the maximum expected mutual information w.r.t. the model complexity. We can use concentration inequalities such as Markov, to give a high probability bound on the
information gain. In particular, we have for all  $\epsilon > 0$
$$
\begin{array}{l} \Pr \left(I \left(\boldsymbol {f} _ {\tau^ {\bar {\pi}} N}; \boldsymbol {y} _ {\tau^ {\bar {\pi}} N} \mid \mathcal {D} _ {1: N - 1}\right) \geq \epsilon\right) \leq \frac {\mathbb {E} _ {\mathcal {D} _ {1 : N - 1} , \tau^ {\bar {\pi}} N} \left[ I \left(\boldsymbol {f} _ {\tau^ {\bar {\pi}} N} ^ {*} ; \boldsymbol {y} _ {\tau^ {\bar {\pi}} N} \mid \mathcal {D} _ {1 : N - 1}\right) \right]}{\epsilon} \\ \leq \mathcal {O} \left(T ^ {3 / 2} \beta_ {N} (\delta) \sqrt {\frac {\mathcal {M C} _ {N} \left(\boldsymbol {f} ^ {*}\right)}{N \epsilon^ {2}}}\right). \\ \end{array}
$$
A.4 Proof of GP results
This section presents our results for the frequentist setting where the dynamics are modeled using GPs. Since the information gain has no meaning in the frequentist setting, we study the epistemic uncertainty of the GP models.
Corollary 5 (Monotonicity of the variance). For all  $n \geq 0$ , and policies  $\pi$  the following is true.
$$
\sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \log \left(1 + \frac {\sigma_ {N - 1 , j} ^ {2} (\boldsymbol {z} _ {t})}{\sigma^ {2}}\right) \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} (\boldsymbol {z} _ {t})}{\sigma^ {2}}\right)
$$
Proof. Follows directly due to the monotonicity of GP posterior variance.
Next, we prove that the trajectory of Equation (6) at iteration  $n$  is upper-bounded with the maximum information gain.
Lemma 10. Let Assumption 2 - 4 hold Then, for all  $N \geq 1$ , with probability at least  $1 - \delta$ , we have
$$
\max  _ {\pi \in \Pi} \mathbb {E} _ {\tau_ {\pi}} \left[ \left(\sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \log \left(1 + \frac {\sigma_ {N , j} ^ {2} \left(\boldsymbol {z} _ {t}\right)}{\sigma^ {2}}\right)\right) \right] \leq \mathcal {O} \left(\beta_ {N} (\delta) T ^ {3 / 2} \sqrt {\frac {\gamma_ {N}}{N}}\right).
$$
Moreover, relax noise Assumption 3 to  $\sigma$ -sub Gaussian. Then, for all  $N \geq 1$ , with probability at least  $1 - \delta$ , we have
$$
\operatorname * {m a x} _ {\boldsymbol {\pi} \in \Pi} \mathbb {E} _ {\boldsymbol {\tau} _ {\boldsymbol {\pi}}} \left[ \left(\sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \log \left(1 + \frac {\sigma_ {N , j} ^ {2} (\boldsymbol {z} _ {t})}{\sigma^ {2}}\right)\right) \right] \leq \mathcal {O} \left(L _ {\boldsymbol {\sigma}} ^ {T} \beta_ {N} ^ {T} (\delta) T ^ {3 / 2} \sqrt {\frac {\gamma_ {N}}{N}}\right)
$$
Proof. Gaussian noise case: Let  $z_{n,t}^{*}$  denote the state-action pair at time  $t$  for the trajectory of Equation (6) at iteration  $n \geq 1$  and  $\pi_n^*$  the corresponding policy.
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n} ^ {*}}} \left[ \left(\sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \log \left(1 + \frac {\sigma_ {N , j} ^ {2} (\boldsymbol {z} _ {N , t} ^ {*})}{\sigma^ {2}}\right)\right) \right] \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\tau_ {\pi_ {n} ^ {*}}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \log \left(1 + \frac {\sigma_ {n , j} ^ {2} \left(z _ {N , t} ^ {*}\right)}{\sigma^ {2}}\right) \right] (Corollary 5) \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\boldsymbol {\tau} _ {\pi_ {n} ^ {*}}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \log \left(1 + \frac {\sigma_ {n , j} ^ {2} \left(\boldsymbol {z} _ {n , t} ^ {*}\right)}{\sigma^ {2}}\right) \right] \quad (\text {B y}) \\ \leq \mathcal {O} \left(\beta_ {N} (\delta) T ^ {3 / 2} \sqrt {\frac {\mathcal {M C} _ {N} \left(\boldsymbol {f} ^ {*}\right)}{N}}\right) \quad \text {(S e e p r o o f T h e o r e m 1)} \\ \leq \mathcal {O} \left(\beta_ {N} (\delta) T ^ {3 / 2} \sqrt {\frac {\gamma_ {N}}{N}}\right) (Curi et al., 2020, Lemma 17) \\ \end{array}
$$
Sub-Gaussian noise case: The only difference between the Gaussian and sub-Gaussian case is the regret term (c.f., Corollary 3 and Lemma 7). In particular, the regret for the sub-Gaussian case leverages the Lipschitz continuity properties of the system (Assumption 2). This results in an exponential dependence on the horizon for our bound. We refer the reader to Curi et al. (2020); Rothfuss et al. (2023) for a more detailed derivation.
Lemma 10 gives a sample complexity bound that holds for a richer class of kernels. Moreover, for GP models,  $\beta_{N} \propto \sqrt{\gamma_{N}}$  (Chowdhury and Gopalan, 2017). Therefore, for kernels, where  $\lim_{N \to \infty} \gamma_{N}^{2} / N \to 0$ , we can show convergence (for the Gaussian case). We summarize bounds on  $\gamma_{N}$  from Vakili et al. (2021) in Table 1.
From hereon, we focus on deriving the results for the case of Gaussian noise case. All our results can be easily extended for the sub-Gaussian setting by considering its corresponding bound.
Lemma 11. The following is true for all  $N \geq 0$  and policies  $\pi \in \Pi$ ,
$$
\mathbb {E} _ {\boldsymbol {\tau} _ {\pi}} \left[ \max _ {\boldsymbol {z} \in \boldsymbol {\tau} ^ {\pi}} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \sigma_ {N, j} ^ {2} (\boldsymbol {z}) \right] \leq C _ {\boldsymbol {\sigma}} \mathbb {E} _ {\boldsymbol {\tau} _ {\pi}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \log \left(1 + \frac {\sigma_ {N , j} ^ {2} (\boldsymbol {z} _ {t})}{\sigma^ {2}}\right) \right],
$$
with  $C_{\sigma} = \frac{\sigma_{\mathrm{max}}}{\log(1 + \sigma^{-2}\sigma_{\mathrm{max}})}$
$$
\begin{array}{l} C _ {\sigma} \mathbb {E} _ {\tau_ {\pi}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \log \left(1 + \frac {\sigma_ {N , j} ^ {2} \left(\boldsymbol {z} _ {t}\right)}{\sigma^ {2}}\right) \right] \\ \geq \mathbb {E} _ {\tau_ {\pi}} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \sigma_ {N, j} ^ {2} \left(\boldsymbol {z} _ {t}\right) \right], \quad (\text {C u r i e t a l .}, 2 0 2 0, \text {L e m m a}. 1 5) \\ \geq \mathbb {E} _ {\boldsymbol {\tau} _ {\pi}} \left[ \max  _ {\boldsymbol {z} \in \boldsymbol {\tau} ^ {\pi}} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \sigma_ {N, j} ^ {2} (\boldsymbol {z}) \right]. \\ \end{array}
$$
Corollary 6. Let Assumption 2 and 4 hold, and relax noise Assumption 3 to  $\sigma$ -sub Gaussian. Then, for all  $N \geq 1$ , with probability at least  $1 - \delta$ , we have
$$
\max _ {\boldsymbol {\pi} \in \Pi} \mathbb {E} _ {\boldsymbol {\tau} _ {\boldsymbol {\pi}}} \left[ \max _ {\boldsymbol {z} \in \boldsymbol {\tau} ^ {\pi}} \sum_ {j = 1} ^ {d _ {x}} \frac {1}{2} \sigma_ {N, j} ^ {2} (\boldsymbol {z}) \right] \leq \mathcal {O} \left(\beta_ {N} (\delta) T ^ {3 / 2} \sqrt {\frac {\gamma N}{N}}\right).
$$
Lemma 12. Let Assumption 2 and 4 hold, and relax noise Assumption 3 to  $\sigma$ -sub Gaussian. Furthermore, assume  $\lim_{N\to \infty}\beta_N^2 (\delta)\gamma_N(k) / N\to 0$ . Then for all  $N\geq 1$ ,  $z\in \mathcal{R}$ , and  $1\le j\le d_x$ , with probability at least  $1 - \delta$ , we have
$$
\sigma_ {n, j} (z) \xrightarrow {\text {a . s .}} 0 f o r n \to \infty .
$$
Proof. We first show that the expected epistemic uncertainty along a trajectory converges to zero almost surely. Then we leverage this result to show almost sure convergence of all trajectories induced
by  $\pmb{\pi} \in \Pi$ . To this end, let  $S_{n} = \mathbb{E}_{\pmb{\tau}_{\pi_{n}^{*}}}\left[\left(\sum_{t=0}^{T-1} \sum_{j=1}^{d_{x}} \frac{1}{2} \log \left(1 + \frac{\sigma_{n,j}^{2}(\pmb{z}_{n,t}^{*})}{\sigma^{2}}\right)\right)\right]$  for all  $n \geq 0$ . So far we have,
$$
\Pr \left(S _ {n} \leq \mathcal {O} \left(\beta_ {N} (\delta) T ^ {3 / 2} \sqrt {\frac {\gamma_ {n}}{n}}\right)\right) \geq 1 - \delta
$$
Consider a sequence  $\{\delta_n\}_{n\geq 0}$  such that  $\lim_{n\to \infty}\delta_n = 0$ , and  $\lim_{n\to \infty}\beta_n(\delta_n)T^{3 / 2}\sqrt{\frac{\gamma_n}{n}}\to 0$ . Note, for GP models with  $\lim_{N\to \infty}\beta_N^2 (\delta)\gamma_N(k) / N\to 0$ , such a sequence of  $\delta_{n}$  exists (Chowdhury and Gopalan, 2017). Consider any  $\epsilon >0$  and let  $N^{*}(\epsilon)$  be the smallest integer such that