SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rothfuss, J., Sukhija, B., Birchler, T., Kassraie, P., and Krause, A. (2023). Hallucinated adversarial control for conservative offline policy evaluation. UAI.
Sancaktar, C., Blaes, S., and Martius, G. (2022). Curious exploration via structured world models yields zero-shot object manipulation. In NeurIPS 35 (NeurIPS 2022).
Schmidhuber, J. (1991). A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animals, pages 222-227.
Schön, T. B., Wills, A., and Ninness, B. (2011). System identification of nonlinear state-space models. Automatica, pages 39-49.
Schoukens, J. and Ljung, L. (2019). Nonlinear system identification: A user-oriented road map. IEEE Control Systems Magazine, 39(6):28-99.
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. (2020). Planning to explore via self-supervised world models. In ICML, pages 8583-8592. PMLR.
Settles, B. (2009). Active learning literature survey.
Shyam, P., Jaskowski, W., and Gomez, F. (2019). Model-based active exploration. In ICML, pages 5779-5788. PMLR.
Simchowitz, M. and Foster, D. (2020). Naive exploration is optimal for online LQR. In ICML, Proceedings of Machine Learning Research, pages 8937-8948. PMLR.
Simchowitz, M., Mania, H., Tu, S., Jordan, M. I., and Recht, B. (2018). Learning without mixing: Towards a sharp analysis of linear system identification. In  $COLT$ , pages 439-473. PMLR.
Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. W. (2012). Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory.
Stadie, B. C., Levine, S., and Abbeel, P. (2015). Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814.
Sussex, S., Makarova, A., and Krause, A. (2023). Model-based causal bayesian optimization. In ICLR.
Tarbouriech, J., Shekhar, S., Pirotta, M., Ghavamzadeh, M., and Lazaric, A. (2020). Active model estimation in markov decision processes. In UAI, pages 1019-1028. PMLR.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018). Deepmind control suite. arXiv preprint arXiv:1801.00690.
Vakili, S., Khezeli, K., and Picheny, V. (2021). On information gain and regret bounds in gaussian process bandits. In AISTATS.
Wagenmaker, A. and Jamieson, K. (2020). Active learning for identification of linear dynamical systems. In  $COLT$ , pages 3487-3582. PMLR.
Wagenmaker, A., Shi, G., and Jamieson, K. (2023). Optimal exploration for model-based rl in nonlinear systems. arXiv preprint arXiv:2306.09210.
Wagenmaker, A. J., Chen, Y., Simchowitz, M., Du, S., and Jamieson, K. (2022). Reward-free rl is no harder than reward-aware rl in linear markov decision processes. In ICML, pages 22430-22456. PMLR.
Wang, H. and Yeung, D.-Y. (2020). A survey on bayesian deep learning. ACM computing surveys (csur), pages 1-37.
Weiss, K., Khoshgoftaar, T. M., and Wang, D. (2016). A survey of transfer learning. Journal of Big data, pages 1-40.
Zhang, Y. and Yang, Q. (2021). A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, pages 5586-5609.
Ziemann, I. and Tu, S. (2022). Learning with little mixing. NeurIPS, 35:4626-4637.
Ziemann, I. M., Sandberg, H., and Matni, N. (2022). Single trajectory nonparametric learning of nonlinear dynamics. In  $COLT$ , pages 3333-3364. PMLR.
A Proofs for section 4 16
A.1 Proof of Lemma 2 17
A.2 Analyzing regret of optimistic planning 18
A.3 Proof for general Bayesian models 20
A.4 Proof of GP results 23
A.5 Zero-shot guarantees 26
BExperiment Details 27
B.1 Environment Details 27
B.2 OPAX in the High-dimensional Fetch Pick & Place Environment 29
C Study of exploration intrinsic rewards 32
A Proofs for section 4
We first prove some key properties of our active exploration objective in Equation (6). Then, we prove Theorem 1 which holds for general Bayesian models, and finally we prove Theorem 2, which guarantees convergence for the frequentist setting where the dynamics are modeled using a GP.
Lemma 4 (Properties of OPAX's objective). Let Assumption 1 and 2 hold, then the following is true for all  $n \geq 0$ ,
$$
\log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} \left(\boldsymbol {x} _ {t} , \boldsymbol {\pi} \left(\boldsymbol {x} _ {t}\right)\right)}{\sigma^ {2}}\right) \geq 0 \tag {1}
$$
$$
\begin{array}{l} \frac {1}{2} \sup  _ {\boldsymbol {\pi} \in \Pi , \boldsymbol {\eta} \in \Xi} \mathbb {E} \left[ \left(\sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} (\boldsymbol {x} _ {t} , \boldsymbol {\pi} (\boldsymbol {x} _ {t}))}{\sigma^ {2}}\right)\right) ^ {2} \right] \leq \frac {1}{2} T ^ {2} d _ {x} ^ {2} \log^ {2} \left(1 + \frac {\sigma_ {\max } ^ {2}}{\sigma^ {2}}\right), (2) \\ \left| \frac {1}{2} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n , j} ^ {2} (\boldsymbol {z})}{\sigma^ {2}}\right) - \log \left(1 + \frac {\sigma_ {n , j} ^ {2} \left(\boldsymbol {z} ^ {\prime}\right)}{\sigma^ {2}}\right) \right| \leq \frac {d _ {x} \sigma_ {\max } L _ {\boldsymbol {\sigma}}}{\sigma^ {2}} \| \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \|. (3) \\ \end{array}
$$
where  $\sigma_{\mathrm{max}} = \sup_{\mathbf{z}\in \mathcal{Z};i\geq 0;1\leq j\leq d_x}\sigma_{i,j}(\mathbf{z})$
Proof. The positivity of the reward follows from the positive definiteness of the epistemic uncertainty  $\sigma_{n - 1,j}$ . For (2), the following holds
$$
\begin{array}{l} \mathbb {E} \left[ \right.\left.\left(\sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} \left(\boldsymbol {x} _ {t} , \boldsymbol {\pi} \left(\boldsymbol {x} _ {t}\right)\right)}{\sigma^ {2}}\right)\right) ^ {2} \right] \\ \leq \mathbb {E} \left[ \sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} T d _ {x} \log^ {2} \left(1 + \frac {\sigma_ {n - 1 , j} ^ {2} \left(\boldsymbol {x} _ {t} , \boldsymbol {\pi} \left(\boldsymbol {x} _ {t}\right)\right)}{\sigma^ {2}}\right) \right] \\ \leq \mathbb {E} \left[ \left(\sum_ {t = 0} ^ {T - 1} \sum_ {j = 1} ^ {d _ {x}} T d _ {x} \log^ {2} \left(1 + \frac {\sigma_ {\max } ^ {2}}{\sigma^ {2}}\right)\right) \right] \\ \leq T ^ {2} d _ {x} ^ {2} \log^ {2} \left(1 + \frac {\sigma_ {\operatorname* {m a x}} ^ {2}}{\sigma^ {2}}\right) \\ \end{array}
$$
From hereon, let  $J_{\mathrm{max}} = \frac{1}{2} T^2 d_x^2 \log^2 \left(1 + \sigma^{-2} \sigma_{\mathrm{max}}^2 \right)$ .
Finally, we show that this reward is Lipschitz continuous.
$$
\begin{array}{l} \left| \sigma_ {n, j} ^ {2} (\boldsymbol {z}) - \sigma_ {n, j} ^ {2} \left(\boldsymbol {z} ^ {\prime}\right) \right| = \left| \sigma_ {n, j} (\boldsymbol {z}) \sigma_ {n, j} (\boldsymbol {z}) - \sigma_ {n, j} (\boldsymbol {z}) \sigma_ {n, j} \left(\boldsymbol {z} ^ {\prime}\right) + \sigma_ {n, j} (\boldsymbol {z}) \sigma_ {n, j} \left(\boldsymbol {z} ^ {\prime}\right) - \sigma_ {n, j} \left(\boldsymbol {z} ^ {\prime}\right) \sigma_ {n, j} \left(\boldsymbol {z} ^ {\prime}\right) \right| \\ \leq L _ {\sigma} \| \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \| \sigma_ {n, j} (\boldsymbol {z}) + L _ {\sigma} \| \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \| \sigma_ {n, j} \left(\boldsymbol {z} ^ {\prime}\right) \\ \leq 2 \sigma_ {\max } L _ {\sigma} \| z - z ^ {\prime} \|. \\ \end{array}
$$
$$
\begin{array}{l} \left| \frac {1}{2} \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\sigma_ {n , j} ^ {2} (z)}{\sigma^ {2}}\right) - \log \left(1 + \frac {\sigma_ {n , j} ^ {2} \left(z ^ {\prime}\right)}{\sigma^ {2}}\right) \right| = \frac {1}{2} \left| \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\frac {\sigma_ {n , j} ^ {2} (z) - \sigma_ {n , j} ^ {2} \left(z ^ {\prime}\right)}{\sigma^ {2}}}{1 + \frac {\sigma_ {n , j} ^ {2} \left(z ^ {\prime}\right)}{\sigma^ {2}}}\right) \right| \\ \leq \frac {1}{2} \left| \sum_ {j = 1} ^ {d _ {x}} \log \left(1 + \frac {\mid \sigma_ {n , j} ^ {2} (z) - \sigma_ {n , j} ^ {2} \left(z ^ {\prime}\right) \mid}{\sigma^ {2}}\right) \right| \\ \leq \frac {1}{2 \sigma^ {2}} \sum_ {j = 1} ^ {d _ {x}} \left| \sigma_ {n, j} ^ {2} (\boldsymbol {z}) - \sigma_ {n, j} ^ {2} \left(\boldsymbol {z} ^ {\prime}\right) \right| \tag {*} \\ \leq \frac {d _ {x} \sigma_ {\operatorname* {m a x}} L _ {\boldsymbol {\sigma}}}{\sigma^ {2}} \left\| \boldsymbol {z} - \boldsymbol {z} ^ {\prime} \right\|. \\ \end{array}
$$
Where  $(^{*})$  is true because for all  $x\geq 0,\log (1 + x)\leq x$
Corollary 1 (OPAX gives an optimistic estimate on Equation (6)). Let Assumption 1 hold and  $\pi_n^*$  denote the solution to Equation (6) and  $J_{n}(\pi_{n}^{*})$  the resulting objective. Similarly, let  $\pi_{n}$  and  $\pmb{\eta}_{n}$  be the solution to Equation (7) and  $J_{n}(\pi_{n},\pmb{\eta}_{n})$  the corresponding value of the objective. Then with probability at least  $1 - \delta$  we have for every episode  $n\in \{1,\dots ,N\}$ :
$$
J _ {n} \left(\boldsymbol {\pi} _ {n} ^ {*}\right) \leq J _ {n} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}\right).
$$
Proof. Follows directly from Assumption 1.
Lemma 5 (Difference in Policy performance). Let  $J_{r,k}(\pmb{\pi}, \pmb{x}_k) = \mathbb{E}_{\pmb{\tau}^\pi}\left[\sum_{t=k}^{T-1} r(\pmb{x}_t, \pmb{\pi}(\pmb{x}_t))\right]$  and  $A_{r,k}(\pmb{\pi}, \pmb{x}, \pmb{a}) = \mathbb{E}_{\pmb{\tau}^\pi}\left[r(\pmb{x}, \pmb{a}) + J_{r,k+1}(\pmb{\pi}, \pmb{x}') - J_{r,k}(\pmb{\pi}, \pmb{x})\right]$  with  $\pmb{x}' = \pmb{f}^*(\pmb{x}, \pmb{a}) + \pmb{w}$ . For simplicity we refer to  $J_{r,0}(\pmb{\pi}, \pmb{x}_0) = J_r(\pmb{\pi}, \pmb{x}_0)$ . The following holds for all  $\pmb{x}_0 \in \mathcal{X}$ :
$$
J _ {r} (\boldsymbol {\pi} ^ {\prime}, \boldsymbol {x} _ {0}) - J _ {r} (\boldsymbol {\pi}, \boldsymbol {x} _ {0}) = \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi^ {\prime}}} \left[ \sum_ {t = 0} ^ {T - 1} A _ {r, t} (\boldsymbol {\pi}, \boldsymbol {x} _ {t} ^ {\prime}, \boldsymbol {\pi} ^ {\prime} (\boldsymbol {x} _ {t} ^ {\prime})) \right]
$$
$$
\begin{array}{l} J _ {r} \left(\boldsymbol {\pi} ^ {\prime}, \boldsymbol {x} _ {0}\right) = \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi^ {\prime}}} \left[ \sum_ {t = 0} ^ {T - 1} r \left(\boldsymbol {x} _ {t} ^ {\prime}, \boldsymbol {\pi} ^ {\prime} \left(\boldsymbol {x} _ {t} ^ {\prime}\right)\right) \right] = \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi^ {\prime}}} \left[ r \left(\boldsymbol {x} _ {0}, \boldsymbol {\pi} ^ {\prime} \left(\boldsymbol {x} _ {0}\right)\right) + J _ {r, 1} \left(\boldsymbol {\pi} ^ {\prime}, \boldsymbol {x} _ {1} ^ {\prime}\right) \right] \\ = \mathbb {E} _ {\tau^ {\pi^ {\prime}}} \left[ r \left(\boldsymbol {x} _ {0}, \boldsymbol {\pi} ^ {\prime} \left(\boldsymbol {x} _ {0}\right)\right) + J _ {r, 1} \left(\boldsymbol {\pi}, \boldsymbol {x} _ {1} ^ {\prime}\right) + J _ {r, 1} \left(\boldsymbol {\pi} ^ {\prime}, \boldsymbol {x} _ {1} ^ {\prime}\right) - J _ {r, 1} \left(\boldsymbol {\pi}, \boldsymbol {x} _ {1} ^ {\prime}\right) \right] \\ = \mathbb {E} _ {\tau^ {\pi^ {\prime}}} \left[ r \left(\boldsymbol {x} _ {0}, \boldsymbol {\pi} ^ {\prime} \left(\boldsymbol {x} _ {0}\right)\right) + J _ {r, 1} \left(\boldsymbol {\pi}, \boldsymbol {x} _ {1} ^ {\prime}\right) - J _ {r} \left(\boldsymbol {\pi}, \boldsymbol {x} _ {0}\right) \right] \\ + J _ {r} (\boldsymbol {\pi}, \boldsymbol {x} _ {0}) + \mathbb {E} _ {\tau \pi^ {\prime}} \left[ J _ {r, 1} \left(\boldsymbol {\pi} ^ {\prime}, \boldsymbol {x} _ {1} ^ {\prime}\right) - J _ {r, 1} \left(\boldsymbol {\pi}, \boldsymbol {x} _ {1} ^ {\prime}\right) \right] \\ = \mathbb {E} _ {\tau^ {\pi^ {\prime}}} \left[ A _ {r, 0} \left(\boldsymbol {\pi}, \boldsymbol {x} _ {0}, \boldsymbol {\pi} ^ {\prime} \left(\boldsymbol {x} _ {0}\right)\right) \right] + J _ {r} \left(\boldsymbol {\pi}, \boldsymbol {x} _ {0}\right) + \mathbb {E} _ {\tau^ {\pi^ {\prime}}} \left[ J _ {r, 1} \left(\boldsymbol {\pi} ^ {\prime}, \boldsymbol {x} _ {1}\right) - J _ {r, 1} \left(\boldsymbol {\pi}, \boldsymbol {x} _ {1}\right) \right] \\ \end{array}
$$
Therefore we obtain
$$
J _ {r} \left(\boldsymbol {\pi} ^ {\prime}, \boldsymbol {x} _ {0}\right) - J _ {r} (\boldsymbol {\pi}, \boldsymbol {x} _ {0}) = \mathbb {E} _ {\tau^ {\pi^ {\prime}}} \left[ A _ {0} \left(\boldsymbol {\pi}, \boldsymbol {x} _ {0}, \boldsymbol {\pi} ^ {\prime} \left(\boldsymbol {x} _ {0}\right)\right) \right] + \mathbb {E} _ {\tau^ {\pi^ {\prime}}} \left[ J _ {r, 1} \left(\boldsymbol {\pi} ^ {\prime}, \boldsymbol {x} _ {1} ^ {\prime}\right) - J _ {r, 1} \left(\boldsymbol {\pi}, \boldsymbol {x} _ {1} ^ {\prime}\right) \right].
$$
Using the same argument for  $J_{r,1}, J_{r,2}, \ldots, J_{r,T-1}$  and that  $J_{r,T}(\pmb{\pi}, \pmb{x}) = 0$  for all  $\pmb{\pi} \in \Pi$  and  $\pmb{x} \in \mathcal{X}$  completes the proof.
Assume a policy  $\pi$  is fixed and dynamics are of the form:
$$
\boldsymbol {x} ^ {\prime} = \boldsymbol {\mu} _ {n} (\boldsymbol {x}, \boldsymbol {\pi} (\boldsymbol {x})) + \beta_ {n} (\delta) \boldsymbol {\sigma} (\boldsymbol {x}, \boldsymbol {\pi} (\boldsymbol {x})) \boldsymbol {u} + \boldsymbol {w}. \tag {14}
$$
Here  $\pmb{u} \in [-1, 1]^{d_x}$ . Furthermore, assume that the associated running rewards do not depend on  $\pmb{u}$ , that is,  $r(\pmb{x}_t)$ , and let  $\pmb{\eta} \in \Xi$  denote the policy, i.e.,  $\pmb{\eta}: \mathcal{X} \to [-1, 1]^{d_x}$ .
Corollary 2. The following holds for all  $\pmb{x}_0 \in \mathcal{X}$  and policy  $\pi$ :
$$
J _ {r} \left(\boldsymbol {\pi}, \boldsymbol {\eta} ^ {\prime}, \boldsymbol {x} _ {0}\right) - J _ {r} \left(\boldsymbol {\pi}, \boldsymbol {\eta}, \boldsymbol {x} _ {0}\right) = \mathbb {E} _ {\boldsymbol {\tau} \eta^ {\prime}} \left[ \sum_ {t = 0} ^ {T - 1} J _ {r, t + 1} \left(\boldsymbol {\pi}, \boldsymbol {\eta}, \boldsymbol {x} _ {t + 1} ^ {\prime}\right) - J _ {r, t + 1} \left(\boldsymbol {\pi}, \boldsymbol {\eta}, \boldsymbol {x} _ {t + 1}\right) \right],
$$
with  $\pmb{x}_{t+1} = \pmb{\mu}_n(\pmb{x}_t', \pmb{\pi}(\pmb{x}_t')) + \beta_n(\delta) \pmb{\sigma}(\pmb{x}_t', \pmb{\pi}(\pmb{x}_t')) \pmb{\eta}(\pmb{x}_t') + \pmb{w}_t,$  and  $\pmb{x}_{t+1}' = \pmb{\mu}_n(\pmb{x}_t', \pmb{\pi}(\pmb{x}_t')) + \beta_n(\delta) \pmb{\sigma}(\pmb{x}_t', \pmb{\pi}(\pmb{x}_t')) \pmb{\eta}'(\pmb{x}_t') + \pmb{w}_t$ .
Proof. From Lemma 5 we have
$$
J _ {r} \left(\boldsymbol {\pi}, \boldsymbol {\eta} ^ {\prime}, \boldsymbol {x} _ {0}\right) - J _ {r} \left(\boldsymbol {\pi}, \boldsymbol {\eta}, \boldsymbol {x} _ {0}\right) = \mathbb {E} _ {\boldsymbol {\tau} \boldsymbol {\eta} ^ {\prime}} \left[ \sum_ {t = 0} ^ {T - 1} A _ {r, t} \left(\boldsymbol {\eta}, \boldsymbol {x} _ {t} ^ {\prime}, \boldsymbol {\eta} ^ {\prime} \left(\boldsymbol {x} _ {t} ^ {\prime}\right)\right) \right].
$$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\tau} ^ {\eta^ {\prime}}} \left[ A _ {r, t} \left(\boldsymbol {\eta}, \boldsymbol {x} _ {t} ^ {\prime}, \boldsymbol {\eta} ^ {\prime} \left(\boldsymbol {x} _ {t} ^ {\prime}\right)\right) \right] = \mathbb {E} _ {\boldsymbol {\tau} ^ {\eta^ {\prime}}} \left[ r \left(\boldsymbol {x} _ {t} ^ {\prime}\right) + J _ {r, t + 1} \left(\boldsymbol {\pi}, \boldsymbol {\eta}, \boldsymbol {x} _ {t + 1} ^ {\prime}\right) - J _ {r, t} \left(\boldsymbol {\pi}, \boldsymbol {\eta}, \boldsymbol {x} _ {t} ^ {\prime}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {\tau} ^ {\eta^ {\prime}}} \left[ r \left(\boldsymbol {x} _ {t} ^ {\prime}\right) + J _ {r, t + 1} \left(\boldsymbol {\pi}, \boldsymbol {\eta}, \boldsymbol {x} _ {t + 1} ^ {\prime}\right) - r \left(\boldsymbol {x} _ {t} ^ {\prime}\right) - J _ {r, t + 1} \left(\boldsymbol {\pi}, \boldsymbol {\eta}, \boldsymbol {x} _ {t + 1}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {\tau} ^ {\eta^ {\prime}}} \left[ J _ {r, t + 1} (\boldsymbol {\pi}, \boldsymbol {\eta}, \boldsymbol {x} _ {t + 1} ^ {\prime}) - J _ {r, t + 1} (\boldsymbol {\pi}, \boldsymbol {\eta}, \boldsymbol {x} _ {t + 1}) \right]. \\ \end{array}
$$
Proof of Lemma 2. From Assumption 1 we know that with probability at least  $1 - \delta$  there exists a  $\bar{\eta}$  such that  $\pmb{f}^{*}(\pmb {z}) = \pmb{\mu}_{n}(\pmb {z}) + \beta_{n}(\delta)\pmb {\sigma}(\pmb {z})\bar{\eta} (\pmb {x})$  for all  $\pmb {z}\in \mathcal{Z}$
$$
\begin{array}{l} J _ {n} \left(\boldsymbol {\pi} _ {n} ^ {*}\right) - J _ {n} \left(\boldsymbol {\pi} _ {n}\right) \leq J _ {n} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}\right) - J _ {n} \left(\boldsymbol {\pi} _ {n}\right) \tag {Corollary 1} \\ = J _ {n} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}\right) - J _ {n} \left(\boldsymbol {\pi} _ {n}, \bar {\boldsymbol {\eta}}\right) \\ = \mathbb {E} _ {\boldsymbol {\tau} ^ {\bar {\eta}}} \left[ \sum_ {t = 0} ^ {T - 1} J _ {n, t + 1} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x} _ {t + 1} ^ {\prime}\right) - J _ {n, t + 1} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x} _ {t + 1}\right) \right] \quad (\text {C o r o l l a r y} 2) \\ = \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} J _ {n, t + 1} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x} _ {t + 1} ^ {\prime}\right) - J _ {n, t + 1} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x} _ {t + 1}\right) \right], \\ \end{array}
$$
(Expectation wrt  $\pi_{n}$  under true dynamics  $f^{*}$ )
$$
\text {w i t h} \boldsymbol {x} _ {t + 1} = \boldsymbol {f} ^ {*} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) + \boldsymbol {w} _ {t},
$$
$$
\text {a n d} \boldsymbol {x} _ {t + 1} ^ {\prime} = \boldsymbol {\mu} _ {n - 1} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) + \beta_ {n - 1} (\delta) \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) \boldsymbol {\eta} _ {n} (\boldsymbol {x} _ {t}) + \boldsymbol {w} _ {t}.
$$
A.2 Analyzing regret of optimistic planning
In the following, we analyze the regret of optimistic planning for both  $\sigma$ -Gaussian noise and  $\sigma$ -sub Gaussian noise case. We start with the Gaussian case.
Lemma 6 (Absolute expectation Difference Under Two Gaussians (Lemma C.2. Kakade et al. (2020))). For Gaussian distribution  $\mathcal{N}(\mu_1, \sigma^2\mathbb{I})$  and  $\mathcal{N}(\mu_2, \sigma^2\mathbb{I})$ , and for any (appropriately measurable) positive function  $g$ , it holds that:
$$
\left| \mathbb {E} _ {z \sim \mathcal {N} _ {1}} [ g (z) ] - \mathbb {E} _ {z \sim \mathcal {N} _ {2}} [ g (z) ] \right| \leq \min  \left\{\frac {\left\| \mu_ {1} - \mu_ {2} \right\|}{\sigma^ {2}}, 1 \right\} \sqrt {\mathbb {E} _ {z \sim \mathcal {N} _ {1}} [ g ^ {2} (z) ]}
$$
(Lemma C.2. Kakade et al. (2020))
$$
\begin{array}{l} \left| \mathbb {E} _ {z \sim \mathcal {N} _ {1}} [ g (z) ] - \mathbb {E} _ {z \sim \mathcal {N} _ {2}} [ g (z) ] \right| = \left| \mathbb {E} _ {z \sim \mathcal {N} _ {1}} \left[ g (z) \left(1 - \frac {\mathcal {N} _ {2}}{\mathcal {N} _ {1}}\right) \right] \right| \\ \leq \left| \sqrt {\mathbb {E} _ {z \sim \mathcal {N} _ {1}} \left[ g ^ {2} (z) \right]} \sqrt {\mathbb {E} _ {z \sim \mathcal {N} _ {1}} \left[ \left(1 - \frac {\mathcal {N} _ {2}}{\mathcal {N} _ {1}}\right) ^ {2} \right]} \right| \\ = \sqrt {\mathbb {E} _ {z \sim \mathcal {N} _ {1}} \left[ g ^ {2} (z) \right]} \sqrt {\mathbb {E} _ {z \sim \mathcal {N} _ {1}} \left[ \left(1 - \frac {\mathcal {N} _ {2}}{\mathcal {N} _ {1}}\right) ^ {2} \right]} \\ \leq \sqrt {\mathbb {E} _ {z \sim \mathcal {N} _ {1}} [ g ^ {2} (z) ]} \min  \left\{\frac {\| \mu_ {1} - \mu_ {2} \|}{\sigma^ {2}}, 1 \right\} \\ \end{array}
$$
Corollary 3 (Regret of optimistic planning for Gaussian noise). Let  $\pi_n^*$ ,  $\pi_n$  denote the solution to Equation (6) and Equation (7) respectively, and  $z_{n,t}^{*}$ ,  $z_{n,t}$  the corresponding state-action pairs visited during their respective trajectories. Furthermore, let Assumption 1 - 3 hold. Then, the following is true for all  $n\geq 0$ ,  $t\in [0,T - 1]$ , with probability at least  $1 - \delta$
$$
J _ {n} \left(\boldsymbol {\pi} _ {n} ^ {*}\right) - J _ {n} \left(\boldsymbol {\pi} _ {n}\right) \leq \mathcal {O} \left(T \mathbb {E} _ {\boldsymbol {\tau} ^ {\boldsymbol {\pi} _ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \frac {(1 + \sqrt {d _ {x}}) \beta_ {n - 1} (\delta) \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n , t}) \|}{\sigma^ {2}} \right]\right)
$$
Proof. For simplicity, define  $g_{n}(\pmb{x}) = J_{n,t + 1}(\pmb{\pi}_{n},\pmb{\eta}_{n},\pmb{x})$ . Note since  $\pmb{w}_t \sim \mathcal{N}(0,\sigma^2\mathbb{I})$  (Assumption 3), we have that  $\pmb{x}_{t + 1}^{\prime}$  and  $\pmb{x}_{t + 1}$  are also Gaussians. Therefore, we can leverage Lemma 6.
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi_ {n}}} \left[ J _ {n, t + 1} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x} _ {t + 1} ^ {\prime}\right) - J _ {n, t + 1} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x} _ {t + 1}\right) \right] = \mathbb {E} \left[ g _ {n} \left(\boldsymbol {x} _ {t + 1} ^ {\prime}\right) - g _ {n} \left(\boldsymbol {x} _ {t + 1}\right) \right] \\ \leq \sqrt {\mathbb {E} \left[ g _ {n} ^ {2} \left(\boldsymbol {x} _ {t + 1}\right) \right]} \min  \left\{\frac {\left\| \boldsymbol {x} _ {t + 1} ^ {\prime} - \boldsymbol {x} _ {t + 1} \right\|}{\sigma^ {2}}, 1 \right\} (Lemma6) \\ \leq \sqrt {J _ {\max }} \min  \left\{\frac {\left\| \boldsymbol {x} _ {t + 1} ^ {\prime} - \boldsymbol {x} _ {t + 1} \right\|}{\sigma^ {2}}, 1 \right\}. (Lemma4) \\ \end{array}
$$
$$
\begin{array}{l} \left\| \boldsymbol {x} _ {t + 1} ^ {\prime} - \boldsymbol {x} _ {t + 1} \right\| = \| \boldsymbol {\mu} _ {n - 1} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) + \beta_ {n - 1} (\delta) \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) \boldsymbol {\eta} _ {n} (\boldsymbol {x} _ {t}) - \boldsymbol {f} ^ {*} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) \| \\ \leq \left\| \boldsymbol {\mu} _ {n - 1} \left(\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} \left(\boldsymbol {x} _ {t}\right)\right) - \boldsymbol {f} ^ {*} \left(\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} \left(\boldsymbol {x} _ {t}\right)\right) \right\| \\ + \beta_ {n - 1} (\delta) \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) \| \| \boldsymbol {\eta} _ {n} (\boldsymbol {x} _ {t}) \| \\ \leq \left(1 + \sqrt {d _ {x}}\right) \beta_ {n - 1} (\delta) \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {x} _ {t}, \boldsymbol {\pi} _ {n} \left(\boldsymbol {x} _ {t}\right)\right) \|. \tag {Assumption1} \\ \end{array}
$$
Next, we use Lemma 2
$$
\begin{array}{l} J _ {n} \left(\boldsymbol {\pi} _ {n} ^ {*}\right) - J _ {n} \left(\boldsymbol {\pi} _ {n}\right) \leq \mathbb {E} _ {\boldsymbol {\tau} ^ {\boldsymbol {\pi} _ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} J _ {n, t + 1} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x} _ {t + 1} ^ {\prime}\right) - J _ {n, t + 1} \left(\boldsymbol {\pi} _ {n}, \boldsymbol {\eta} _ {n}, \boldsymbol {x} _ {t + 1}\right) \right], \\ \leq \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \sqrt {J _ {\max }} \min  \left\{\frac {(1 + \sqrt {d _ {x}}) \beta_ {n - 1} (\delta) \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {x} _ {t} , \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) \|}{\sigma^ {2}}, 1 \right\} \right], \\ \leq \sqrt {J _ {\max }} \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \frac {(1 + \sqrt {d _ {x}}) \beta_ {n - 1} (\delta) \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {x} _ {t} , \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) \|}{\sigma^ {2}} \right], \\ = \mathcal {O} \left(T \mathbb {E} _ {\boldsymbol {\tau} ^ {\pi_ {n}}} \left[ \sum_ {t = 0} ^ {T - 1} \frac {(1 + \sqrt {d _ {x}}) \beta_ {n - 1} (\delta) \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {x} _ {t} , \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {t})) \|}{\sigma^ {2}} \right]\right). \\ \end{array}
$$