SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {x}} _ {n} (t) - \boldsymbol {x} _ {n} (t) \right\| = \left\| \int_ {0} ^ {t} \boldsymbol {f} _ {n - 1} (\widehat {\boldsymbol {z}} _ {n} (s)) - \boldsymbol {f} ^ {*} (\boldsymbol {z} _ {n} (s)) d s \right\| \\ \leq \int_ {0} ^ {t} \| \boldsymbol {f} _ {n - 1} (\widehat {\boldsymbol {z}} _ {n} (s)) - \boldsymbol {f} ^ {*} (\boldsymbol {z} _ {n} (s)) \| d s \\ \leq \int_ {0} ^ {t} \| \boldsymbol {f} _ {n - 1} (\widehat {\boldsymbol {z}} _ {n} (s)) - \boldsymbol {f} _ {n - 1} (\boldsymbol {z} _ {n} (s)) \| + \| \boldsymbol {f} _ {n - 1} (\boldsymbol {z} _ {n} (s)) - \boldsymbol {f} ^ {*} (\boldsymbol {z} _ {n} (s)) \| d s \\ \leq L _ {\boldsymbol {f}} \left(1 + L _ {\boldsymbol {\pi}}\right) \int_ {0} ^ {t} \| \boldsymbol {x} _ {n} (s) - \widehat {\boldsymbol {x}} _ {n} (s) \| d s + 2 \beta_ {n} \int_ {0} ^ {t} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (s)\right) \| d s \\ \end{array}
$$
where the final inequality follows from Lemma 15 and Definition 2 (with high probability). We obtain Equation (8) by applying Gronwall's inequality (Fact 14).
Equation (9) is obtained analogously by expanding with  $\pmb{f}^{*}(\widehat{\pmb{z}}_{n}(s))$  rather than  $\pmb{f}_{n - 1}(\pmb{z}_n(s))$ .
Lemma 5. For any MSS  $S$ , the regret of any episode  $n$  is bounded with high probability by
$$
r _ {n} (S) \leq 2 \beta_ {n} L _ {c} \left(1 + L _ {\pi}\right) T e ^ {L _ {f} \left(1 + L _ {\pi}\right) T} \int_ {0} ^ {T} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (t)\right) \| d t. \tag {10}
$$
Proof. By definition of  $\pi_n$  in OCoRL we have with high probability,  $C(\pmb{f}^*,\pmb{\pi}^*)\geq C(\pmb{f}_{n - 1},\pmb{\pi}_n)$ . Therefore, with high probability,
$$
\begin{array}{l} r _ {n} (S) = C \left(\boldsymbol {f} ^ {*}, \boldsymbol {\pi} _ {n}\right) - C \left(\boldsymbol {f} ^ {*}, \boldsymbol {\pi} ^ {*}\right) \\ \leq C \left(\boldsymbol {f} ^ {*}, \boldsymbol {\pi} _ {n}\right) - C \left(\boldsymbol {f} _ {n - 1}, \boldsymbol {\pi} _ {n}\right) \\ = \int_ {0} ^ {T} c (\boldsymbol {z} _ {n} (t)) - c (\widehat {\boldsymbol {z}} _ {n} (t)) d t. \\ \end{array}
$$
By Lemma 15, we further have that
$$
r _ {n} (S) \leq L _ {c} \left(1 + L _ {\pi}\right) \int_ {0} ^ {T} \| \boldsymbol {x} _ {n} (t) - \widehat {\boldsymbol {x}} _ {n} (t) \| d t.
$$
By Lemma 4 (with high probability), we further have that
$$
\begin{array}{l} r _ {n} (S) \leq 2 \beta_ {n} L _ {c} \left(1 + L _ {\pi}\right) e ^ {L _ {f} \left(1 + L _ {\pi}\right) T} \int_ {0} ^ {T} \int_ {0} ^ {t} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (s)\right) \| d s d t \\ \leq 2 \beta_ {n} L _ {c} \left(1 + L _ {\pi}\right) e ^ {L _ {f} (1 + L _ {\pi}) T} \int_ {0} ^ {T} \int_ {0} ^ {T} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (s)\right) \| d s d t \\ = 2 \beta_ {n} L _ {c} \left(1 + L _ {\pi}\right) T e ^ {L f (1 + L _ {\pi}) T} \int_ {0} ^ {T} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (t)\right) \| d t. \\ \end{array}
$$
Proof of Proposition 1. Let us first bound  $R_N^2(S)$ . By the Cauchy-Schwarz inequality,
$$
R _ {N} ^ {2} (S) \leq N \sum_ {n = 1} ^ {N} r _ {n} ^ {2} (S).
$$
By Lemma 5 (with high probability), we have that
$$
\begin{array}{l} R _ {N} ^ {2} (S) \leq N 4 L _ {c} ^ {2} \left(1 + L _ {\pi}\right) ^ {2} T ^ {2} e ^ {2 L _ {f} (1 + L _ {\pi}) T} \sum_ {n = 1} ^ {N} \beta_ {n} ^ {2} \left(\int_ {0} ^ {T} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n} (t)) \| d t\right) ^ {2} \\ \leq N 4 \beta_ {N} ^ {2} L _ {c} ^ {2} (1 + L _ {\pi}) ^ {2} T ^ {3} e ^ {2 L _ {f} (1 + L _ {\pi}) T} \sum_ {n = 1} ^ {N} \int_ {0} ^ {T} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n} (t)) \| ^ {2} d t. \\ \end{array}
$$
Taking the square root, we obtain
$$
R _ {N} (S) \leq 2 \beta_ {N} L _ {c} (1 + L _ {\pi}) T ^ {\frac {3}{2}} e ^ {L _ {f} (1 + L _ {\pi}) T} \sqrt {N \sum_ {n = 1} ^ {N} \int_ {0} ^ {T} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (t)\right) \right\| ^ {2} d t}.
$$
The result follows by noting that
$$
\sum_ {n = 1} ^ {N} \int_ {0} ^ {T} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n} (t)) \| ^ {2} d t \leq \mathcal {I} _ {N} (\boldsymbol {f} ^ {*}, S).
$$
A.2 Bounding Model Complexities
In this section, we derive the model complexity bounds for the presented MSSs. Our bounds are based on the approximation of integrals using an upper Darboux sum.
Fact 6 (Upper Darboux approximation). Given any  $a \leq b$ , a function  $f: \mathbb{R} \to \mathbb{R}$ , and a partition  $P = (a, \ldots, b)$  of  $[a, b]$  into  $k$  sub-intervals,
$$
\int_ {a} ^ {b} f (x) d x \leq \sum_ {i = 1} ^ {k} \Delta_ {i} \max  _ {x \in P [ i ]} f (x) \tag {11}
$$
where  $P[i]$  denotes the  $i$ -th sub-interval and  $\Delta_i$  denotes the length of  $P[i]$ .
We are now ready to bound the model complexities.
Lemma 7 (Oracle model complexity). For any  $N \geq 1$ ,
$$
\mathcal {I} _ {N} \left(\boldsymbol {f} ^ {*}, S ^ {\text {O R A}}\right) \leq \max  _ {\substack {\boldsymbol {\pi} _ {1}, \dots , \boldsymbol {\pi} _ {N} \\ \boldsymbol {\pi} _ {n} \in \Pi}} T \sum_ {n = 1} ^ {N} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, 1}\right)\right) \right\| ^ {2}. \tag{12}
$$
$$
\mathcal {I} _ {N} (\pmb {f} ^ {*}, S ^ {\mathrm {O R A}}) = \max _ {\substack {\pmb {\pi} _ {1}, \dots , \pmb {\pi} _ {N} \\ \pmb {\pi} _ {n} \in \Pi}} \sum_ {n = 1} ^ {N} \int_ {0} ^ {T} \| \pmb {\sigma} _ {n - 1} (\pmb {z} _ {n} (t)) \| ^ {2} d t.
$$
For each episode  $n$ , use an upper Darboux approximation (Fact 6) with  $k = 1$  to obtain
$$
\begin{array}{l} \leq \max_{\substack{\boldsymbol{\pi}_{1},\ldots ,\boldsymbol{\pi}_{N}\\ \boldsymbol{\pi}_{n}\in \Pi}}T\sum_{n = 1}^{N}\max_{0\leq t\leq T}\| \boldsymbol{\sigma}_{n - 1}\left(\boldsymbol{z}_{n}(t)\right)\|^{2} \\ = \max_{\substack{\boldsymbol{\pi}_{1},\ldots ,\boldsymbol{\pi}_{N}\\ \boldsymbol{\pi}_{n}\in \Pi}}T\sum_{n = 1}^{N}\| \boldsymbol{\sigma}_{n - 1}\left(\boldsymbol{z}_{n}(t_{n,1})\right)\|^{2}. \\ \end{array}
$$
Lemma 8 (Equidistant model complexity). For any  $N \geq 1$ ,
$$
\mathcal {I} _ {N} \left(\boldsymbol {f} ^ {*}, S ^ {E Q I}\right) \leq \max  _ {\substack {\boldsymbol {\pi} _ {1}, \dots , \boldsymbol {\pi} _ {N} \\ \boldsymbol {\pi} _ {n} \in \Pi}} \sum_ {n = 1} ^ {N} \frac {T}{m _ {n}} \sum_ {i = 1} ^ {m _ {n}} \left(\left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right)\right) \right\| ^ {2} + \frac {T L _ {\sigma^ {2}}}{m _ {n}}\right). \tag{13}
$$
$$
\mathcal{I}_{N}(\boldsymbol{f}^{*},S^{\mathrm{EQI}}) = \max_{\substack{\boldsymbol{\pi}_{1},\ldots ,\boldsymbol{\pi}_{N}\\ \boldsymbol{\pi}_{n}\in \Pi}}\sum_{n = 1}^{N}\int_{0}^{T}\| \boldsymbol{\sigma}_{n - 1}\left(\boldsymbol {z}_{n}(t)\right)\|^{2}  dt.
$$
For each episode  $n$ , let  $P_{n} \stackrel{\mathrm{def}}{=} (0, \Delta_{n}, 2\Delta_{n}, \ldots, T)$  with  $\Delta_{n} \stackrel{\mathrm{def}}{=} \frac{T}{m_{n}}$  be a partition of  $[0, T]$  into  $m_{n}$  sub-intervals, each of length  $\Delta_{n}$ . Using the upper Darboux approximations (Fact 6) of  $\| \sigma_{n-1}(\cdot) \|^{2}$  with respect to  $P_{n}$ , we have
$$
\leq \max_{\substack{\boldsymbol{\pi}_{1},\ldots ,\boldsymbol{\pi}_{N}\\ \boldsymbol{\pi}_{n}\in \Pi}}\sum_{n = 1}^{N}\Delta_{n}\sum_{i = 1}^{m_{n}}\max_{t\in [(i - 1)\Delta_{n},i\Delta_{n}]}\| \boldsymbol{\sigma}_{n - 1}\left(\boldsymbol{z}_{n}(t)\right)\|^{2}.
$$
Observe that  $t_{n,i}\in [(i - 1)\Delta_n,i\Delta_n]$ , and hence, using that  $\| \pmb{\sigma}_{n - 1}(\cdot)\|^2$  is  $L_{\sigma^2}$ -Lipschitz continuous,
$$
\leq \max_{\substack{\boldsymbol{\pi}_{1},\ldots ,\boldsymbol{\pi}_{N}\\ \boldsymbol{\pi}_{n}\in \Pi}}\sum_{n = 1}^{N}\Delta_{n}\sum_{i = 1}^{m_{n}}\Bigl(\| \boldsymbol{\sigma}_{n - 1}\left(\boldsymbol{z}_{n}(t_{n,i})\right)\|^{2} + L_{\sigma^{2}}\Delta_{n}\Bigr).
$$
In the remainder of this section, we study the model complexity of the adaptive MSS. In each episode  $n$ , we partition  $[0,T]$  into equally sized sub-intervals (called "buckets") of length  $\Delta_{n}$ , where  $\Delta_{n}$  is the solution to
$$
\Delta_ {n} = \frac {1}{2 \Gamma_ {n} (\Delta_ {n})} \tag {14}
$$
where  $\Gamma_{n}(\Delta_{n})\stackrel {\mathrm{def}}{=}2\beta_{n}L_{\sigma}(1 + L_{\pi})e^{L_{f}(1 + L_{\pi})\Delta_{n}}$ . As  $\Gamma_{n}$  is a monotonically increasing function of  $\Delta_{n}$  it is clear that a suitable  $\Delta_{n}$  exists. Moreover, it follows from  $\Delta_{n}\leq T$  that  $\Gamma_{n}\in \mathcal{O}(\beta_{n})$ , and hence that  $m_{n} = T / \Delta_{n}\in \mathcal{O}(\beta_{n})$ . Without loss of generality, we assume  $m_{n}\in \mathbb{N}$ .
The definition of the adaptive MSS can be interpreted as follows: In every bucket, we take two measurements. One measurement is saved and used later for updating the statistical model. The other measurement is taken at the beginning of every bucket, and used to inform OCoRL of the current state position. For the purposes of our theoretical analysis, these initial measurements are not saved and not used for the construction of the statistical model.
Lemma 9. For any episode  $n$ , bucket  $i \in \{1, \dots, m_n\}$ , and  $t \in [(i - 1)\Delta_n, i\Delta_n]$  where the real and hallucinated trajectories are "synced" initially, i.e.,  $\hat{\mathbf{z}}_n((i - 1)\Delta_n) = \mathbf{z}_n((i - 1)\Delta_n)$ , we have with high probability that
$$
\left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} (t)\right) - \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (t)\right) \right\| \leq \Gamma_ {n} \int_ {(i - 1) \Delta_ {n}} ^ {t} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (s)\right) \| d s \quad a n d \tag {15}
$$
$$
\left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} (t)\right) - \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (t)\right) \right\| \leq \Gamma_ {n} \int_ {(i - 1) \Delta_ {n}} ^ {t} \| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} (s)\right) \| d s \tag {16}
$$
Proof. Using that  $\sigma_{n-1}$  is  $L_{\sigma}$ -Lipschitz continuous and applying Lemma 15,
$$
\left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} (t)\right) - \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (t)\right) \right\| \leq L _ {\boldsymbol {\sigma}} \left(1 + L _ {\boldsymbol {\pi}}\right) \left\| \widehat {\boldsymbol {x}} _ {n} (t) - \boldsymbol {x} _ {n} (t) \right\|.
$$
The result then follows from Lemma 4, where the lower integral bounds can be tightened to  $(i - 1)\Delta_{n}$  as the real and hallucinated trajectories are "synced" in the beginning of the bucket.
Lemma 10. For any episode  $n$  and  $i \in \{1, \dots, m_n\}$ , if  $t_{n,i}$  is selected according to the adaptive MSS then with high probability,
$$
\max  _ {t \in [ (i - 1) \Delta_ {n}, i \Delta_ {n} ]} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n} (t)) \| ^ {2} \leq 9 \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n} (t _ {n, i})) \| ^ {2}. \tag {17}
$$
Proof. By applying Lemma 9 we have (with high probability) for every  $t \in [(i - 1)\Delta_n, \Delta_n]$ ,
$$
\left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (t)\right) \right\| \leq \left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} (t)\right) \right\| + \Gamma_ {n} \int_ {(i - 1) \Delta_ {n}} ^ {t} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} (s)\right) \right\| d s.
$$
Note that  $\| \pmb{\sigma}_{n - 1}(\widehat{\pmb{z}}_n(t))\| \leq \| \pmb{\sigma}_{n - 1}(\widehat{\pmb{z}}_n(t_{n,i}))\|$  by the definition of  $t_{n,i}$ , and hence,
$$
\begin{array}{l} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (t)\right) \right\| \leq \left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} \left(t _ {n, i}\right)\right) \right\| + \Gamma_ {n} \Delta_ {n} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} \left(t _ {n, i}\right)\right) \right\| \\ = \left(1 + \frac {1}{2}\right) \| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} \left(t _ {n, i}\right)\right) \|. \tag {18} \\ \end{array}
$$
Similarly, by applying Lemma 9, we have (with high probability) that
$$
\begin{array}{l} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right)\right) \right\| \geq \left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} \left(t _ {n, i}\right)\right) \right\| - \Gamma_ {n} \int_ {(i - 1) \Delta_ {n}} ^ {t _ {n, i}} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} (s)\right) \right\| d s \\ \geq \left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} \left(t _ {n, i}\right)\right) \right\| - \Gamma_ {n} \Delta_ {n} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} \left(t _ {n, i}\right)\right) \right\| \\ = \left(1 - \frac {1}{2}\right) \| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} \left(t _ {n, i}\right)\right) \|. \tag {19} \\ \end{array}
$$
Combining Equations (18) and (19), we obtain
$$
\left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (t)\right) \right\| ^ {2} \leq \left(\frac {1 + \frac {1}{2}}{1 - \frac {1}{2}}\right) ^ {2} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right)\right) \right\| ^ {2} = 9 \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right)\right) \right\| ^ {2}.
$$
Corollary 11 (Adaptive model complexity). For any  $N \geq 1$ ,
$$
\mathcal {I} _ {N} \left(\boldsymbol {f} ^ {*}, S ^ {A D P}\right) \leq \max  _ {\substack {\boldsymbol {\pi} _ {1}, \dots , \boldsymbol {\pi} _ {N} \\ \boldsymbol {\pi} _ {n} \in \Pi}} 9 \sum_ {n = 1} ^ {N} \frac {T}{m _ {n}} \sum_ {i = 1} ^ {m _ {n}} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right)\right) \right\| ^ {2}. \tag{20}
$$
Proof. The result follows analogously to the proof of Lemma 8, using Lemma 10 in the last step.
A.3 Bounding Measurement Uncertainty of GPs
Our goal in this section is to prove Theorem 3.
The informativeness of a set of sampling points  $A \subset \mathcal{Z}$  about a function  $f \sim \mathcal{GP}(0, k)$  with the noisy observations  $\pmb{y}_A = \pmb{f}_A + \pmb{\epsilon}_A$  can be measured by the information gain (c.f., Cover (1999); Srinivas et al. (2009)),
$$
\mathrm {I} (\boldsymbol {y} _ {A}; f) \stackrel {\text {d e f}} {=} \mathrm {I} (\boldsymbol {y} _ {A}; \boldsymbol {f} _ {A}) = \mathrm {H} (\boldsymbol {y} _ {A}) - \mathrm {H} (\boldsymbol {y} _ {A} \mid \boldsymbol {f} _ {A}), \tag {21}
$$
which quantifies the reduction in entropy of  $\pmb{f}_A$  when observing  $\pmb{y}_A$ . Here, we write  $\pmb{f}_A = [f(\pmb{x})]_{\pmb{x} \in A}$  and  $\epsilon_A \sim \mathcal{N}(\mathbf{0}, \sigma^2 \pmb{I})$ . For a Gaussian,  $\mathrm{H}(\mathcal{N}(\pmb{\mu}, \pmb{\Sigma})) = \frac{1}{2} \log \det(2\pi e \pmb{\Sigma})$ , so that when  $f$  is a Gaussian process,  $\mathrm{I}(\pmb{y}_A; f) = \frac{1}{2} \log \det(\pmb{I} + \sigma^{-2} \pmb{K}_{AA})$  where  $\pmb{K}_{AA} = [k(\pmb{x}, \pmb{x}')]_{\pmb{x}, \pmb{x}' \in A}$ . We denote the maximal information gain by observing  $n$  points by
$$
\gamma_ {n} \stackrel {\text {d e f}} {=} \max  _ { \begin{array}{c} A \subset \mathcal {Z} \\ | A | = n \end{array} } I (\boldsymbol {y} _ {A}; f). \tag {22}
$$
Clearly,  $\gamma_{n}$  depends on the kernel  $k$ . In Table 2, we state the magnitude of  $\gamma_{n}$  for different kernels. We take the magnitudes from Theorem 5 of Srinivas et al. (2009) and Remark 2 of Vakili et al. (2021).
Table 2: Here we present different magnitudes of  $\gamma_{n}$ . The magnitudes hold under the assumption that  $\mathcal{Z}$  is compact. Here,  $B_{\nu}$  is the modified Bessel function.
Kernel	k(x,x')	γn
Linear	x^T x'	O(d log(n))
RBF	e^-||x-x'|^2/2l^2	O(logd+1(n))
Matérn	1/Γ(ν)2ν-1(√2ν||x-x'|/l)νBν(√2ν||x-x'|/l)	O(n^d/2ν+d log2ν/2ν+d (n))
The maximum information gain can be interpreted analogously in the setting where  $f \in \mathcal{H}_{k,B}$ , see Srinivas et al. (2009) and section 5.2 of Kirschner and Krause (2018). The following result relates mutual information and epistemic uncertainty.
Lemma 12 (Lemma 5.3 from Srinivas et al. (2009)). For any  $A \subset \mathcal{Z}$ , if  $f \sim \mathcal{GP}(0, k)$  or  $f \in \mathcal{H}_{k,B}$  and if the observation noise is i.i.d.  $\mathcal{N}(0; \sigma^2)$  then
$$
\operatorname {I} \left(\boldsymbol {y} _ {A}; f\right) = \frac {1}{2} \sum_ {i = 1} ^ {n} \log \left(1 + \frac {\operatorname {V a r} \left[ f \left(\boldsymbol {x} _ {i}\right) \mid \boldsymbol {y} \left(\boldsymbol {x} _ {1 : i - 1}\right) \right]}{\sigma^ {2}}\right) \tag {23}
$$
where  $\{\pmb{x}_1,\dots ,\pmb{x}_n\} \stackrel {def}{=}A.$
In our setting, denote by  $A_N \stackrel{\mathrm{def}}{=} (S_1, \ldots, S_N)$  the set of times of observations  $\mathcal{D}_{1:N}$  until episode  $N$ . We define for every episode  $n$ ,
$$
i _ {n} ^ {*} \stackrel {\text {d e f}} {=} \operatorname * {a r g m a x} _ {i \in \{1, \dots , m _ {n} \}} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right)\right) \| _ {2} ^ {2}. \tag {24}
$$
We write  $t_{n}^{*} \stackrel{\mathrm{def}}{=} t_{n,i_{n}^{*}}$  and  $\tilde{A}_{N} \stackrel{\mathrm{def}}{=} (\{t_{1}^{*}\}, \ldots, \{t_{N}^{*}\})$ . Note that  $\tilde{A}_{N}[n] \subseteq A_{N}[n]$  ( $\forall n$ ) and  $\tilde{A}_{N}$  comprises exactly  $N$  observations.
Lemma 13. If  $f_{j} \sim \mathcal{GP}(0,k)$  for all  $j \in \{1, \dots, d_x\}$  or if  $\pmb{f} \in \mathcal{H}_{k,B}^{d_x}$ , and if the observation noise is i.i.d.  $\mathcal{N}(\mathbf{0}; \sigma^2 \mathbf{I})$  then
$$
\sum_ {n = 1} ^ {N} \frac {T}{m _ {n}} \sum_ {i = 1} ^ {m _ {n}} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right)\right) \| _ {2} ^ {2} \leq 2 \bar {\sigma} T d _ {x} \gamma_ {N} \tag {25}
$$
where  $\bar{\sigma} = \frac{\sigma_{\mathrm{max}}^2}{\log(1 + \sigma^{-2}\sigma_{\mathrm{max}}^2)}$ ,  $\sigma_{\mathrm{max}}^2 = \max_{\boldsymbol {z}\in \mathcal{Z},j\in \{1,\dots,d_x\}}\sigma_{0,j}^2 (\boldsymbol {z})$ , and  $\gamma_N$  is with respect to kernel  $k$ .
$$
\sum_ {n = 1} ^ {N} \frac {T}{m _ {n}} \sum_ {i = 1} ^ {m _ {n}} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n} (t _ {n, i})) \| _ {2} ^ {2} \leq T \sum_ {n = 1} ^ {N} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n} (t _ {n} ^ {*})) \| _ {2} ^ {2}.
$$
This inequality is tight when  $m_{n} = 1 (\forall n)$ . When more than one measurement is obtained per episode then the regret may be smaller. Expanding the squared Euclidean norm, yields
$$
= T \sum_ {n = 1} ^ {N} \sum_ {j = 1} ^ {d _ {x}} \sigma_ {n - 1, j} ^ {2} (\boldsymbol {z} _ {n} (t _ {n} ^ {*})).
$$
Write  $\tilde{\sigma}_{n,j}^{2}(\boldsymbol{z}) \stackrel{\mathrm{def}}{=} \operatorname{Var}[f_{j}(\boldsymbol{z}) \mid \dot{\boldsymbol{y}}_{\tilde{A}_{N}}]$  whereas  $\sigma_{n,j}^{2}(\boldsymbol{z}) = \operatorname{Var}[f_{j}(\boldsymbol{z}) \mid \dot{\boldsymbol{y}}_{A_{N}}]$ . The variance is monotonically decreasing as one conditions on more observations,  $\tilde{\sigma}_{n,j}^{2}(\boldsymbol{z}) \geq \sigma_{n,j}^{2}(\boldsymbol{z}) (\forall n, j, \boldsymbol{z})$ , and hence,
$$
\leq T \sum_ {n = 1} ^ {N} \sum_ {j = 1} ^ {d _ {x}} \tilde {\sigma} _ {n - 1, j} ^ {2} \left(\boldsymbol {z} _ {n} \left(t _ {n} ^ {*}\right)\right).
$$
Again, this bound is tight when only one measurement is obtained per episode, but may be loose otherwise. Lemma 15 of Curi et al. (2020) shows that  $\tilde{\sigma}_{n,j}^{2}(z)\leq \bar{\sigma}\log (1 + \sigma^{-2}\tilde{\sigma}_{n,j}^{2}(z))$  for any  $n,j,z$ . Applying this inequality, we obtain
$$
\leq \bar {\sigma} T \sum_ {j = 1} ^ {d _ {x}} \sum_ {n = 1} ^ {N} \log \left(1 + \frac {\tilde {\sigma} _ {n - 1 , j} ^ {2} \left(\boldsymbol {z} _ {n} \left(t _ {n} ^ {*}\right)\right)}{\sigma^ {2}}\right).
$$
By Lemma 12,
$$
\begin{array}{l} = 2 \bar {\sigma} T \sum_ {j = 1} ^ {d _ {x}} \mathrm {I} \left(\dot {\boldsymbol {y}} _ {\hat {A} _ {N}}; f _ {j}\right) \\ \leq 2 \bar {\sigma} T d _ {x} \gamma_ {N}. \\ \end{array}
$$
Proof of Theorem 3. We derive the regret bound for the adaptive MSS. The regret bounds for the other MSSs follow analogously.
From Corollary 11, recall the model complexity bound
$$
\mathcal{I}_{N}(\boldsymbol{f}^{*},S^{\mathrm{ADP}})\leq \max_{\substack{\boldsymbol{\pi}_{1},\ldots ,\boldsymbol{\pi}_{N}\\ \boldsymbol{\pi}_{n}\in \Pi}}9\sum_{n = 1}^{N}\frac{T}{m_{n}}\sum_{i = 1}^{m_{n}}\| \boldsymbol{\sigma}_{n - 1}\left(\boldsymbol{z}_{n}(t_{n,i})\right)\|^{2}.
$$
By Lemma 13, we can further bound this by
$$
\leq 1 8 \bar {\sigma} T d _ {x} \gamma_ {N}.
$$
Combining this bound with Proposition 1, with high probability the regret is bounded by
$$
R _ {N} \left(S ^ {\mathrm {A D P}}\right) \leq \mathcal {O} \left(\beta_ {N} T ^ {2} e ^ {L _ {f} (1 + L _ {\pi}) T} \sqrt {N \gamma_ {N}}\right).
$$
A.4 Useful Facts and Inequalities
Fact 14 (Gronwall's inequality, theorem 1.1 in chapter 3 of Hartman (2002)). Let  $g(t)$  be a nonnegative continuous function on the interval  $[a,b] \subset \mathbb{R}$ , and let  $C, K$  be a pair of non-negative constants. If the function  $g$  satisfies
$$
g (t) \leq C + K \int_ {a} ^ {t} g (s) d s, \quad \forall t \in [ a, b ], \tag {26}
$$
$$
g (t) \leq C e ^ {K (t - a)}, \quad \forall t \in [ a, b ]. \tag {27}
$$
Lemma 15. For any two metric spaces  $(\mathcal{X},d_{\mathcal{X}})$  and  $(\mathcal{Y},d_{\mathcal{Y}})$ , and any two Lipschitz continuous functions  $f:\mathcal{X}\times \mathcal{X}\to \mathcal{Y}$  and  $g:\mathcal{X}\rightarrow \mathcal{X}$  with Lipschitz constants  $L_{f}$  and  $L_{g}$  respectively, we have that  $f(\cdot ,g(\cdot))$  is  $L$ -Lipschitz continuous with respect to  $d_{\mathcal{Y}}$  with  $L\stackrel {\text{def}}{=}L_{f}(1 + L_{g})$ .
Proof. Fix any  $\pmb{x}, \pmb{x}' \in \mathcal{X}$ . By the triangle inequality,
$$
\begin{array}{l} d _ {\mathcal {Y}} \left(f (\boldsymbol {x}, g (\boldsymbol {x})), f \left(\boldsymbol {x} ^ {\prime}, g \left(\boldsymbol {x} ^ {\prime}\right)\right)\right) \leq d _ {\mathcal {Y}} \left(f \left(\boldsymbol {x} ^ {\prime}, g (\boldsymbol {x})\right), f \left(\boldsymbol {x}, g (\boldsymbol {x})\right)\right) + d _ {\mathcal {Y}} \left(f \left(\boldsymbol {x} ^ {\prime}, g (\boldsymbol {x})\right), f \left(\boldsymbol {x} ^ {\prime}, g \left(\boldsymbol {x} ^ {\prime}\right)\right)\right) \\ \leq L _ {f} d _ {\mathcal {X}} \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) + L _ {f} d _ {\mathcal {X}} \left(g (\boldsymbol {x}), g \left(\boldsymbol {x} ^ {\prime}\right)\right) \\ \leq L _ {f} d _ {\mathcal {X}} \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) + L _ {f} L _ {g} d _ {\mathcal {X}} \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) \\ = L _ {f} \left(1 + L _ {g}\right) d _ {\mathcal {X}} \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right). \\ \end{array}
$$
B Solving Optimistic Optimal Control Problem
The optimal control problem solved by OCORL is constrained to  $L_{f}$ -Lipschitz continuous dynamics. When the dynamics are modeled with a GP using a linear kernel, the control problem can easily be solved directly in weight-space. Beyond this special case, it is commonly assumed that there exists an oracle which solves this optimization problem exactly (see, e.g., assumption 1 of Kakade et al. (2020)).