SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this work, we do not explicitly address the problem of approximating the optimal control problem reasonably well. Kakade et al. (2020) give a brief overview of gradient- and sampling-based methods which may be useful (Jacobson and Mayne, 1970; Todorov and Li, 2005; Mordatch et al., 2012; Williams et al., 2017; Wagener et al., 2019). They alternatively suggest to use Thompson sampling (Thompson, 1933; Osband and Van Roy, 2014), that is, to sample  $f_{n - 1}$  from  $\mathcal{M}_{n - 1}$  (which is straightforward when the dynamics are modeled by a GP, see Williams and Rasmussen (2006)) and then to compute and execute the optimal policy  $\pi_n = \mathrm{argmin}_{\pi \in \Pi} C(\pi, f_{n - 1})$  using a planning oracle. Kakade et al. (2020) conjecture that corresponding regret bounds can be derived using standard techniques for analyzing Bayesian regret of Thompson sampling (Russo and Van Roy, 2014, 2016).
B.1 Application to Arbitrary Discretizations
In this section, we briefly show that our guarantees carry over to discretized dynamics and costs for arbitrary discretizations. Importantly, in the discrete-time setting the lemma mirroring Lemma 4 does not require the assumption that the models  $\pmb{f}_n$  are  $L_{\pmb{f}}$ -Lipschitz continuous, hence, simplifying the optimistic optimal control problem solved by OCoRL. In the discrete-time setting, it is sufficient to assume that  $\pmb{f}^*$  is  $L_{\pmb{f}}$ -Lipschitz continuous.
Consider the dynamical system
$$
\boldsymbol {x} _ {k + 1} = \boldsymbol {h} ^ {*} (\boldsymbol {z} _ {k}, \tau_ {k}), \quad \text {w h e r e} \quad \boldsymbol {z} _ {k} \stackrel {\text {d e f}} {=} (\boldsymbol {x} _ {k}, \boldsymbol {\pi} (\boldsymbol {x} _ {k}))
$$
with initial state  $\pmb{x}_0 \in \mathcal{X}$  where  $\tau_k \in \mathbb{R}_{>0}$  measures the "time" between control points  $\pmb{x}_k$  and  $\pmb{x}_{k+1}$ . We make the assumption that  $h^*$  is  $L_h$ -Lipschitz continuous in all arguments. The (discrete-time) control problem is
$$
\boldsymbol {\pi} ^ {*} \stackrel {\text {d e f}} {=} \underset {\boldsymbol {\pi} \in \Pi} {\operatorname {a r g m i n}} C (\boldsymbol {\pi}, \boldsymbol {h} ^ {*}) = \underset {\boldsymbol {\pi} \in \Pi} {\operatorname {a r g m i n}} \sum_ {k = 1} ^ {T} c \left(\boldsymbol {z} _ {k}\right). \tag {28}
$$
In many applications, the cost to be minimized is defined at discrete control points to begin with. In this case, the continuous-time control of Equation (1) can be approximated arbitrarily well with Equation (28) by choosing a sufficiently dense discretization. Commonly, the discretization is taken to be equidistant, i.e.,  $\tau_{k} \equiv \tau (\forall k)$ , but we remark that our results hold more generally for any discretization.
B.1.1 Bounding Regret with the Model Complexity
In the following, we denote by  $\pmb{x}_{n,k}$  the  $k$ -th state visited during episode  $n$ .
Lemma 16. Assuming  $\pmb{x}_{n,0} = \widehat{\pmb{x}}_{n,0}$ , the distance between the true and the hallucinated trajectory at any iteration  $k \geq 0$  is bounded with high probability by
$$
\left\| \widehat {\boldsymbol {x}} _ {n, k} - \boldsymbol {x} _ {n, k} \right\| \leq 2 \beta_ {n} \left(1 + L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {n} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {T - 1} \sum_ {l = 0} ^ {k - 1} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, l}\right) \| \tag {29}
$$
where we write  $L_h^\prime \stackrel {def}{=} L_h(1 + L_\pi)$  and  $L_{\sigma}^{\prime}\stackrel {def}{=}L_{\sigma}(1 + L_{\pi})$
Proof (based on Lemma 4 of Curi et al. (2020)). We begin by showing by induction that for any  $k \geq 0$ ,
$$
\left\| \widehat {\boldsymbol {x}} _ {n, k} - \boldsymbol {x} _ {n, k} \right\| \leq 2 \beta_ {n} \sum_ {l = 0} ^ {k - 1} \left(L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {n} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {k - 1 - l} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, l}\right) \|. \tag {30}
$$
The base case is implied trivially. For the induction step, assume that Equation (30) holds at iteration  $k$ . We have
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {x}} _ {n, k + 1} - \boldsymbol {x} _ {n, k + 1} \right\| = \left\| \boldsymbol {h} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n, k}, \tau_ {k}\right) - \boldsymbol {h} ^ {*} \left(\boldsymbol {z} _ {n, k}, \tau_ {k}\right) \right\| \\ \leq \left\| \boldsymbol {h} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n, k}, \tau_ {k}\right) - \boldsymbol {h} ^ {*} \left(\boldsymbol {z} _ {n, k}, \tau_ {k}\right) \right\| \\ \leq \left\| \boldsymbol {h} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n, k}, \tau_ {k}\right) - \boldsymbol {h} ^ {*} \left(\widehat {\boldsymbol {z}} _ {n, k}, \tau_ {k}\right) \right\| + \left\| \boldsymbol {h} ^ {*} \left(\widehat {\boldsymbol {z}} _ {n, k}, \tau_ {k}\right) - \boldsymbol {h} ^ {*} \left(\boldsymbol {z} _ {n, k}, \tau_ {k}\right) \right\| \\ \leq 2 \beta_ {n} \| \boldsymbol {\sigma} _ {n - 1} (\widehat {\boldsymbol {z}} _ {n, k}) \| + L _ {\boldsymbol {h}} (1 + L _ {\boldsymbol {\pi}}) \| \boldsymbol {x} _ {n, k} - \widehat {\boldsymbol {x}} _ {n, k} \| \\ \end{array}
$$
where the final inequality follows from Definition 2 (with high probability) and Lemma 15.
$$
\begin{array}{l} = 2 \beta_ {n} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n, k}) + \boldsymbol {\sigma} _ {n - 1} (\widehat {\boldsymbol {z}} _ {n, k}) - \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n, k}) \| \\ + L _ {\boldsymbol {h}} \left(1 + L _ {\boldsymbol {\pi}}\right) \| \boldsymbol {x} _ {n, k} - \widehat {\boldsymbol {x}} _ {n, k} \| \\ \end{array}
$$
Using that  $\sigma_{n - 1}$  is  $L_{\sigma}$ -Lipschitz continuous and applying Lemma 15,
$$
\begin{array}{l} \leq 2 \beta_ {n} \left(\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, k}\right) \| + L _ {\boldsymbol {\sigma}} \left(1 + L _ {\boldsymbol {\pi}}\right) \| \widehat {\boldsymbol {x}} _ {n, k} - \boldsymbol {x} _ {n, k} \|\right) \\ + L _ {\boldsymbol {h}} \left(1 + L _ {\boldsymbol {\pi}}\right) \| \boldsymbol {x} _ {n, k} - \widehat {\boldsymbol {x}} _ {n, k} \| \\ = 2 \beta_ {n} \| \boldsymbol {\sigma} _ {n - 1} (\boldsymbol {z} _ {n, k}) \| + (L _ {\boldsymbol {h}} (1 + L _ {\boldsymbol {\pi}}) + 2 \beta_ {n} L _ {\boldsymbol {\sigma}} (1 + L _ {\boldsymbol {\pi}})) \| \boldsymbol {x} _ {n, k} - \widehat {\boldsymbol {x}} _ {n, k} \| \\ \end{array}
$$
By the induction hypothesis,
$$
\leq 2 \beta_ {n} \sum_ {l = 0} ^ {k} \left(L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {n} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {k - l} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, l}\right) \|.
$$
Thus, Equation (30) holds. Since  $k \leq T$ , we have
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {x}} _ {n, k} - \boldsymbol {x} _ {n, k} \right\| \leq 2 \beta_ {n} \sum_ {l = 0} ^ {k - 1} \left(L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {n} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {k - 1 - l} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, l}\right) \| \\ \leq 2 \beta_ {n} \sum_ {l = 0} ^ {k - 1} \left(1 + L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {n} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {k - 1 - l} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, l}\right) \| \\ \leq 2 \beta_ {n} \left(1 + L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {n} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {T - 1} \sum_ {l = 0} ^ {k - 1} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, l}\right) \|. \\ \end{array}
$$
Lemma 17. For any MSS  $S$ , the regret of any episode  $n$  is bounded with high probability by
$$
r _ {n} (S) \leq 2 \beta_ {n} T \left(1 + L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {n} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {T - 1} L _ {c} ^ {\prime} \sum_ {k = 1} ^ {T} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, k}\right) \| \tag {31}
$$
where  $L_c^\prime \stackrel {\text{def}}{=} L_c(1 + L_\pi)$ .
$$
\begin{array}{l} r _ {n} (S) = C \left(\boldsymbol {f} ^ {*}, \boldsymbol {\pi} _ {n}\right) - C \left(\boldsymbol {f} ^ {*}, \boldsymbol {\pi} ^ {*}\right) \\ \leq C \left(\boldsymbol {f} ^ {*}, \boldsymbol {\pi} _ {n}\right) - C \left(\boldsymbol {f} _ {n - 1}, \boldsymbol {\pi} _ {n}\right) \\ = \sum_ {k = 0} ^ {T} c \left(\boldsymbol {z} _ {n, k}\right) - c \left(\widehat {\boldsymbol {z}} _ {n, k}\right). \\ \end{array}
$$
$$
r _ {n} (S) \leq L _ {c} ^ {\prime} \sum_ {k = 1} ^ {T} \left\| \boldsymbol {x} _ {n, k} - \widehat {\boldsymbol {x}} _ {n, k} \right\|.
$$
By Lemma 16 (with high probability), we further have that
$$
\begin{array}{l} r _ {n} (S) \leq 2 \beta_ {n} \left(1 + L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {n} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {T - 1} L _ {c} ^ {\prime} \sum_ {k = 1} ^ {T} \sum_ {l = 0} ^ {k - 1} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, l}\right) \| \\ = 2 \beta_ {n} T \left(1 + L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {n} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {T - 1} L _ {c} ^ {\prime} \sum_ {k = 1} ^ {T} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, k}\right) \right\|. \\ \end{array}
$$
The above results allow us to prove a general regret bound which is analogous to Proposition 1.
Lemma 18. Let  $S$  be any MSS. If we run OCoRL with arbitrary discretization  $\pmb{h}^*$ , we have with high probability,
$$
R _ {N} (S) \leq 2 \beta_ {N} T ^ {\frac {3}{2}} \left(1 + L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {N} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {T - 1} L _ {c} ^ {\prime} \sqrt {N \tilde {\mathcal {I}} _ {N} \left(\boldsymbol {h} ^ {* , S}\right)} \tag {32}
$$
where we define the discrete-time model complexity,
$$
\tilde {\mathcal {I}} _ {N} \left(\boldsymbol {h} ^ {*}, S\right) \stackrel {{d e f}} {{=}} \max  _ {\substack {\boldsymbol {\pi} _ {1}, \dots , \boldsymbol {\pi} _ {N} \\ \boldsymbol {\pi} _ {n} \in \Pi}} \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {T} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, k}\right) \right\| ^ {2}. \tag{33}
$$
Proof. Let us first bound  $R_N^2(S)$ . By the Cauchy-Schwarz inequality,
$$
\begin{array}{l} R _ {N} ^ {2} (S) \leq N 4 T ^ {2} L _ {c} ^ {\prime 2} \sum_ {n = 1} ^ {N} \beta_ {n} ^ {2} \left(1 + L _ {\mathbf {h}} ^ {\prime} + 2 \beta_ {n} L _ {\sigma} ^ {\prime}\right) ^ {2 (T - 1)} \left(\sum_ {k = 1} ^ {T} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, k}\right) \|\right) ^ {2} \\ \leq N 4 \beta_ {N} ^ {2} T ^ {3} \left(1 + L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {N} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {2 (T - 1)} L _ {c} ^ {\prime 2} \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {T} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, k}\right) \right\| ^ {2}. \\ \end{array}
$$
$$
R _ {N} (S) \leq 2 \beta_ {N} T ^ {\frac {3}{2}} \left(1 + L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {N} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {T - 1} L _ {c} ^ {\prime} \sqrt {N \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {T} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n , k}\right) \right\| ^ {2}}.
$$
$$
\sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {T} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n, k}\right) \right\| ^ {2} \leq \tilde {\mathcal {I}} _ {N} \left(\boldsymbol {h} ^ {*}, S\right).
$$
B.1.2 Bounding Model Complexities
The model complexity bounds from the continuous-time setting (c.f., Appendix A.2) extend seamlessly to the discrete-time setting. In the following we briefly sketch the bound for the oracle MSS.
Lemma 19 (Discrete-time oracle model complexity). For any  $N \geq 1$ ,
$$
\tilde {\mathcal {I}} _ {N} \left(\boldsymbol {h} ^ {*}, S ^ {O R A}\right) \leq \max  _ {\substack {\boldsymbol {\pi} _ {1}, \dots , \boldsymbol {\pi} _ {N} \\ \boldsymbol {\pi} _ {n} \in \Pi}} T \sum_ {n = 1} ^ {N} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, 1}\right)\right) \right\| ^ {2}. \tag{34}
$$
$$
\begin{array}{l} \tilde{\mathcal{I}}_{N}(\boldsymbol{h}^{*},S^{\mathrm{ORA}}) = \max_{\substack{\boldsymbol{\pi}_{1},\ldots ,\boldsymbol{\pi}_{N}\\ \boldsymbol{\pi}_{n}\in \Pi}}\sum_{n = 1}^{N}\sum_{k = 1}^{T}\left\| \boldsymbol{\sigma}_{n - 1}\left(\boldsymbol{z}_{n,k}\right)\right\|^{2} \\ \leq \max_{\substack{\boldsymbol{\pi}_{1},\ldots ,\boldsymbol{\pi}_{N}\\ \boldsymbol{\pi}_{n}\in \Pi}}T\sum_{n = 1}^{N}\max_{k\in \{1,\ldots ,T\}}\left\| \boldsymbol{\sigma}_{n - 1}\left(\boldsymbol{z}_{n,k}\right)\right\|^{2} \\ \leq \max_{\substack{\boldsymbol{\pi}_{1},\ldots ,\boldsymbol{\pi}_{N}\\ \boldsymbol{\pi}_{n}\in \Pi}}T\sum_{n = 1}^{N}\| \boldsymbol{\sigma}_{n - 1}\left(\boldsymbol{z}_{n}(t_{n,1})\right)\|^{2}. \\ \end{array}
$$
Note that the MSS  $S$  can (but not has to) be restricted to the discretization of  $[0, T]$  used for the dynamics and costs. The derivations for the equidistant and adaptive MSSs follow analogously, where we note that the bucket length  $\Delta_{n}$  is to be defined with respect to a modified constant  $\Gamma_{n}$ .
Finally, applying the measurement uncertainty bound presented in Appendix A.3, we obtain the following theorem.
Theorem 20. Fix an arbitrary discretization  $\pmb{h}^{*}$  and assume that  $\pmb{h}^{*} \in \mathcal{H}_{n,B}^{d_{x}}$ , the observation noise is i.i.d.  $\mathcal{N}(0; \sigma^2 I)$ , and let  $\| \cdot \|$  be the Euclidean norm. We model  $\pmb{h}^{*}$  with the GP model. The regret for the adaptive MSS is with probability at least  $1 - \delta$  bounded by
$$
R _ {N} (S ^ {A D P}) \leq \mathcal {O} \left(\beta_ {N} T ^ {2} \left(1 + L _ {\boldsymbol {h}} ^ {\prime} + 2 \beta_ {N} L _ {\boldsymbol {\sigma}} ^ {\prime}\right) ^ {T - 1} \sqrt {N \gamma_ {N}}\right), \quad m _ {n} ^ {A D P} = \mathcal {O} (\beta_ {n}).
$$
C Experimental Setup
C.1 System's dynamics
Here we either give reference to the equations of the dynamical system or provide their equations:
Cancer Treatment:
$$
\dot {x} (t) = r x (t) \log \left(\frac {1}{x (t)}\right) - \delta u (t) x (t)
$$
$$
x (0) = 0. 9 7 5, r = 0. 3, \delta = 0. 4 5
$$
- Glucose in Blood:
$$
\dot {x} _ {0} (t) = - a x _ {0} (t) - b x _ {1} (t)
$$
$$
\dot {x} _ {1} (t) = - c x _ {1} (t) + u (t),
$$
$$
a = 1, b = 1, c = 1, x (0) = (0. 7 5, 0)
$$
Pendulum:
$$
\dot {x} _ {0} (t) = x _ {1} (t)
$$
$$
\dot {x} _ {1} (t) = \frac {g}{l} \sin (x _ {0} (t)) + u (t)
$$
$$
g = 9. 8 1, l = 5. 0
$$
- Mountain Car:
$$
\dot {x} _ {0} (t) = 1 0 x _ {1} (t)
$$
$$
\dot {x} _ {1} (t) = 3. 5 u (t) - 2. 5 \cos (3 x _ {0} (t))
$$
- Cart Pole: Equation 6.1 and 6.2 of Kelly (2017)
- Furuta Pendulum: Equation 18 of Gafvert (2016)
- Bicycle: Dynamics from (Singh and Theers, 2021)
- Quadrotor 2D: Equations 1, 2, 3 of Paining et al. (2020)
- Quadrotor 3D: Equations 12, 13, 14 of Thomas et al. (2017)
C.2 System's tasks
In all considered settings the running cost has the following quadratic form:
$$
c (\boldsymbol {x}, \boldsymbol {u}) = (\boldsymbol {x} - \boldsymbol {x} _ {T}) ^ {\top} \boldsymbol {Q} (\boldsymbol {x} - \boldsymbol {x} _ {T}) + (\boldsymbol {u} - \boldsymbol {u} _ {T}) ^ {\top} \boldsymbol {R} (\boldsymbol {u} - \boldsymbol {u} _ {T}).
$$
Here,  $\boldsymbol{x}_T$  is the target state, and  $\boldsymbol{u}_T$  is the target control. Descriptively, in cancer treatment and glucose in Blood systems, our objective is to reduce the prevalence of unhealthy cells, while striving to be as minimally invasive as possible. For the pendulum, Furuta pendulum, and cart pole systems our goal is to skillfully swing the pendulum upwards from its lowest point. In the mountain car system, the challenge is to guide the car from the valley's depth to the hill's summit. Finally, in systems bicycle, quadrotor 2D, and quadrotor 3D our objective is to adeptly navigate the agent to reach a desired target state.
Table 3: Cost specifications for considered systems.
System	Running Q	Running R	Target xT	Target uT
Cancer Treatment	I1	I1	[0.54]	01
Glucose in Blood	I2	I1	[0.47, 0.33]	01
Pendulum	I2	I1	[0, 0]	01
Mountain Car	I2	I1	[π/6, 0]	01
Cart Pole	I4	I1	04	01
Furuta Pendulum	I4	I1	04	01
Bicycle	I4	I2	[2, 1, π/6, 0]	02
Quadrotor 2D	I6	10 · I2	04	[g · m, 0]
Quadrotor 3D	diag([1, 1, 1, 1, 1, 1, 0.1, 0.1, 0.1, 1, 0.1, 0.1, 0.1])	diag([5, 0.8, 0.8, 0.3])	[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]	[0.15, 0, 0, 0, 0]
C.3 Adaptive MSSs
In this section, we describe in detail the two adaptive MSSs from Section 5.
Greedy Max Determinant In the Greedy Max Determinant adaptive strategy we solve the following objective:
$$
\max  _ {S \subset [ 0, T ], | S | = M} \det  \left(\boldsymbol {K} _ {S} ^ {n}\right). \tag {35}
$$
Here,  $\pmb{K}_S^n \stackrel{\mathrm{def}}{=} [k_n(\widehat{\pmb{z}}_n(t), \widehat{\pmb{z}}_n(t'))]_{t,t' \in S}$  and  $k_n(z, z') \stackrel{\mathrm{def}}{=} k(z, z') - k_n^\top(z) (\pmb{K}_n + \sigma^2\pmb{I})^{-1}\pmb{k}_n(z')$ . The goal here is to find at every episode  $n$  a set  $S_n$  of  $M$  time points from  $[0,T]$ . Solving problem (35) is NP-hard and we approximate it by solving it greedily:
Algorithm 1 GreEDy MAX DETERMINANT
$\mathbf{Init}$  ..  $S_{n} = \{\mathrm{argmax}_{t\in [0,T]}k_{n}(\widehat{\mathbf{z}}_{n}(t),\widehat{\mathbf{z}}_{n}(t))\}$
for  $k = 2,\ldots ,M$  do
$$
S_{n} = S_{n}\cup \left\{\operatorname *{argmax}_{t\in [0,T]}\det \left(\boldsymbol{K}_{S_{n}\cup \{t\}}^{n}\right)\right\}
$$
Greedy Max Kernel Distance The second adaptive MSS we consider in Section 5 is the Greedy Max Kernel Distance. Here we greedily solve the metric  $M$ -center problem in space  $[0, T]$  with the kernel metric:  $d_{n}(t, t')^{2} \stackrel{\mathrm{def}}{=} k_{n}(\widehat{\mathbf{z}}_{n}(t), \widehat{\mathbf{z}}_{n}(t)) + k_{n}(\widehat{\mathbf{z}}_{n}(t'), \widehat{\mathbf{z}}_{n}(t')) - 2k_{n}(\widehat{\mathbf{z}}_{n}(t), \widehat{\mathbf{z}}_{n}(t'))$  on the hallucinated trajectory.
Algorithm 2 GreEDy MAX KERNEL DISTANCE
$\mathbf{Init}$  ..  $S_{n} = \{\mathrm{argmax}_{t\in [0,T]}k_{n}(\widehat{\boldsymbol{z}}_{n}(t),\widehat{\boldsymbol{z}}_{n}(t))\}$
$$
S_{n} = S_{n}\cup \left\{\operatorname *{argmax}_{t\in [0,T]}\min_{t^{\prime}\in S_{n}}d_{n}(t,t^{\prime})\right\}
$$
C.4 Episodes Hyperparameters
Table 4: We take a few tens of measurements per episode in each environment. We run the experiments for at most 40 episodes in each environment.
System	Number of episodes N	Number of Measurements per episodes M	Time horizon T
Cancer Treatment	20	10	20
Glucose in Blood	20	10	0.45
Pendulum	20	10	10
Mountain Car	40	10	1
Cart Pole	40	10	10
Furuta Pendulum	40	10	10
Bicycle	20	10	10
Quadrotor 2D	20	10	8
Quadrotor 3D	25	20	15
C.5 Practical Implementation
In practice, we approach solving the optimal control problem (2) with offline planning and online tracking strategy. Before the episode starts we obtain an open loop trajectory  $\tilde{\pmb{x}}_n(t),\tilde{\pmb{u}}_n(t)$  by offline planning:
$$
\tilde {\boldsymbol {x}} _ {n} (t), \tilde {\boldsymbol {u}} _ {n} (t) = \underset {\boldsymbol {x} (t), \boldsymbol {u} (t) \eta (t)} {\operatorname {a r g m i n}} \underset {\eta (t)} {\min } \int_ {0} ^ {T} c (\boldsymbol {x} (t), \boldsymbol {u} (t)) d t \tag {36}
$$
$$
\text {s . t .} \dot {\boldsymbol {x}} (t) = \boldsymbol {\mu} _ {n} (\boldsymbol {x} (t), \boldsymbol {u} (t)) + \beta_ {n} \sigma_ {n} (\boldsymbol {x} (t), \boldsymbol {u} (t)) \boldsymbol {\eta} (t)
$$
$$
\boldsymbol {\eta} (t) \in [ - 1, 1 ] ^ {d _ {x}}, \quad \forall t \in [ 0, T ].
$$
We solve the optimization problem (36) with Iterative Linear Quadratic Regulator (ILQR) (Li and Todorov, 2004). Then we track the open loop trajectory  $\tilde{\pmb{x}}_n(t)$  online with MPC:
$$
\boldsymbol {u} (t) = \underset {\boldsymbol {u} (t), \boldsymbol {x} (t)} {\operatorname {a r g m i n}} \int_ {s} ^ {s + T _ {M P C}} \left(\left\| \boldsymbol {x} (t) - \tilde {\boldsymbol {x}} (t) \right\| _ {2} ^ {2} + \left\| \boldsymbol {u} (t) - \tilde {\boldsymbol {u}} (t) \right\| _ {2} ^ {2}\right) d t \tag {37}
$$
$$
\text {s . t .} \dot {\boldsymbol {x}} (t) = \boldsymbol {\mu} _ {n} (\boldsymbol {x} (t), \boldsymbol {u} (t))
$$
Here, horizon  $T_{MPC}$  depends on the system.
Table 5: The MPC horizon for considered systems.
System	TMPC
Cancer Treatment	5.0
Glucose in Blood	0.2
Pendulum	6.0
Mountain Car	1.0
Cart Pole	5.0
Furuta Pendulum	5.0
Bicycle	5.0
Quadrotor 2D	3.0
Quadrotor 3D	5.0
We solve MPC tracking problem (37) also with the ILQR.
D Additional Experiments on Number of Measurements