SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lenart Treven
ETH Zürich  
trevenl@ethz.ch
Jonas Hubotter
ETH Zürich  
jhuebotter@student.ethz.ch
Bhavya Sukhija
ETH Zürich sukhijab@ethz.ch
Florian Dörfler ETH Zürich lörfler@ethz.ch
Andreas Krause ETH Zürich krausea@ethz.ch
Reinforcement learning algorithms typically consider discrete-time dynamics, even though the underlying systems are often continuous in time. In this paper, we introduce a model-based reinforcement learning algorithm that represents continuous-time dynamics using nonlinear ordinary differential equations (ODEs). We capture epistemic uncertainty using well-calibrated probabilistic models, and use the optimistic principle for exploration. Our regret bounds surface the importance of the measurement selection strategy (MSS), since in continuous time we not only must decide how to explore, but also when to observe the underlying system. Our analysis demonstrates that the regret is sublinear when modeling ODEs with Gaussian Processes (GP) for common choices of MSS, such as equidistant sampling. Additionally, we propose an adaptive, data-dependent, practical MSS that, when combined with GP dynamics, also achieves sublinear regret with significantly fewer samples. We showcase the benefits of continuous-time modeling over its discrete-time counterpart, as well as our proposed adaptive MSS over standard baselines, on several applications.
Real-world systems encountered in natural sciences and engineering applications, such as robotics (Spong et al., 2006), biology (Lenhart and Workman, 2007; Jones et al., 2009), medicine (Panetta and Fister, 2003), etc., are fundamentally continuous in time. Therefore, ordinary differential equations (ODEs) are the natural modeling language. However, the reinforcement learning (RL) community predominantly models problems in discrete time, with a few notable exceptions (Doya, 2000; Yildiz et al., 2021; Lutter et al., 2021). The discretization of continuous-time systems imposes limitations on the application of state-of-the-art RL algorithms, as they are tied to specific discretization schemes.
Discretization of continuous-time systems sacrifices several essential properties that could be preserved in a continuous-time framework (Nesic and Postoyan, 2021). For instance, exact discretization is only feasible for linear systems, leading to an inherent discrepancy when using standard discretization techniques for nonlinear systems. Additionally, discretization obscures the inter-sample behavior of the system, changes stability properties, may result in uncontrollable systems, or requires an excessively high sampling rate. Multi-time-scale systems are particularly vulnerable to these issues (Engquist et al., 2007). In many practical scenarios, the constraints imposed by discrete-time modeling are undesirable. Discrete-time models do not allow for the independent adjustment of measurement and control frequencies, which is crucial for real-world systems that operate in different regimes. For example, in autonomous driving, low-frequency sensor sampling
and control actuation may suffice at slow speeds, while high-speed driving demands faster control. How to choose aperiodic measurements and control is studied in the literature on the event and self-triggered control (Astrom and Bernhardsson, 2002; Anta and Tabuada, 2010; Heemels et al., 2012). They show that the number of control inputs can be significantly reduced by using aperiodic control. Moreover, data from multiple sources is often collected at varying frequencies (Ghysels et al., 2006) and is often not even equidistant in time. Discrete-time models struggle to exploit this data for learning, while continuous-time models can naturally accommodate it.
Continuous-time modeling also offers the flexibility to determine optimal measurement times based on need, in contrast to the fixed measurement frequency in discrete-time settings, which can easily miss informative samples. This advantage is particularly relevant in fields like medicine, where patient monitoring often requires higher frequency measurements during the onset of illness and lower frequency measurements as the patient recovers (Kaandorp and Koole, 2007). At the same time, in fields such as medicine, measurements are often costly, and hence, it is important that the most informative measurements are selected. Discrete-time models, limited to equidistant measurements, therefore often result in suboptimal decision-making and their sample efficiency is fundamentally limited. In summary, continuous-time modeling is agnostic to the choice of measurement selection strategy (MSS), whereas discrete-time modeling often only works for an equidistant MSS.
Contributions Given the advantages of continuous-time modeling, in this work, we propose an optimistic continuous-time model-based RL algorithm – OCoRL. Moreover, we theoretically analyze OCoRL and show a general regret bound that holds for any MSS. We further show that for common choices of MSSs, such as equidistant sampling, the regret is sublinear when we model the dynamics with GPs (Williams and Rasmussen, 2006). To our knowledge, we are the first to give a no-regret algorithm for a rich class of nonlinear dynamical systems in the continuous-time RL setting. We further propose an adaptive MSS that is practical, data-dependent, and requires considerably fewer measurements compared to the equidistant MSS while still ensuring the sublinear regret for the GP case. Crucial to OCoRL is the exploration induced by the optimism in the face of uncertainty principle for model-based RL, which is commonly employed in the discrete-time realm (Auer et al., 2008; Curi et al., 2020). We validate the performance of OCoRL on several robotic tasks, where we clearly showcase the advantages of continuous-time modeling over its discrete-time counterpart. Finally, we provide an efficient implementation<sup>1</sup> of OCoRL in JAX (Bradbury et al., 2018).
2 Problem setting
In this work, we study a continuous-time deterministic dynamical system  $f^{*}$  with initial state  $\pmb{x}_0 \in \mathcal{X} \subset \mathbb{R}^{d_x}$ , i.e.,
$$
\boldsymbol {x} (t) = \boldsymbol {x} _ {0} + \int_ {0} ^ {t} \boldsymbol {f} ^ {*} (\boldsymbol {x} (s), \boldsymbol {u} (s)) d s.
$$
Here  $\pmb{u}:[0,\infty)\to \mathbb{R}^{d_u}$  is the input we apply to the system. Moreover, we consider state feedback controllers represented through a policy  $\pi :\mathcal{X}\rightarrow \mathcal{U}\subset \mathbb{R}^{d_u}$ , that is,  $\pmb {u}(s) = \pmb {\pi}(\pmb {x}(s))$ . Our objective is to find the optimal policy with respect to a given running cost function  $c:\mathbb{R}^{d_x}\times \mathbb{R}^{d_u}\to \mathbb{R}$ . Specifically, we are interested in solving the following optimal control (OC) problem over the policy space  $\Pi$ :
$$
\boldsymbol {\pi} ^ {*} \stackrel {\text {d e f}} {=} \underset {\boldsymbol {\pi} \in \Pi} {\operatorname {a r g m i n}} C (\boldsymbol {\pi}, \boldsymbol {f} ^ {*}) = \underset {\boldsymbol {\pi} \in \Pi} {\operatorname {a r g m i n}} \int_ {0} ^ {T} c (\boldsymbol {x}, \boldsymbol {\pi} (\boldsymbol {x})) d t \tag {1}
$$
$$
\begin{array}{l l} \text {s . t .} & \dot {\boldsymbol {x}} = \boldsymbol {f} ^ {*} (\boldsymbol {x}, \boldsymbol {\pi} (\boldsymbol {x})), \quad \boldsymbol {x} (0) = \boldsymbol {x} _ {0}. \end{array}
$$
The function  $f^{*}$  is unknown, but we can collect data over episodes  $n = 1, \dots, N$  and learn about the system by deploying a policy  $\pi_{n} \in \Pi$  for the horizon of  $T$  in episode  $n$ . In an (overly) idealized continuous time setting, one can measure the system at any time step. However, we consider a more practical setting, where we assume taking measurements is costly, and therefore want as few measurements as necessary. To this end, we formally define a measurement selection strategy below.
<https://github.com/lenarttreven/ocorl>
Definition 1 (Measurement selection strategy). A measurement selection strategy  $S$  is a sequence of sets  $(S_{n})_{n\geq 1}$ , such that  $S_{n}$  contains  $m_{n}$  points at which we take measurements, i.e.,  $S_{n}\subset [0,T],|S_{n}| = m_{n}.^{2}$
During episode  $n$ , given a policy  $\pmb{\pi}_n$  and a MSS  $S_n$ , we collect a dataset  $\mathcal{D}_n \sim (\pmb{\pi}_n, S_n)$ . The dataset is defined as
$$
\begin{array}{l} \mathcal {D} _ {n} \stackrel {\text {d e f}} {=} \left\{\left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right), \dot {\boldsymbol {y}} _ {n} \left(t _ {n, i}\right)\right) \mid t _ {n, i} \in S _ {n}, i \in \{1 \dots , m _ {n} \} \right\} \quad \text {w h e r e} \\ \boldsymbol {z} _ {n} (t _ {n, i}) \stackrel {\text {d e f}} {=} (\boldsymbol {x} _ {n} (t _ {n, i}), \boldsymbol {\pi} _ {n} (\boldsymbol {x} _ {n} (t _ {n, i}))), \quad \dot {\boldsymbol {y}} _ {n} (t _ {n, i}) \stackrel {\text {d e f}} {=} \dot {\boldsymbol {x}} _ {n} (t _ {n, i}) + \boldsymbol {\epsilon} _ {n, i}. \\ \end{array}
$$
Here  $\boldsymbol{x}_n(t), \dot{\boldsymbol{x}}_n(t)$  are state and state derivative in episode  $n$ , and  $\epsilon_{n,i}$  is i.i.d,  $\sigma$ -sub-Gaussian noise of the state derivative observations. Note, even though in practice only the state  $\boldsymbol{x}(t)$  might be observable, one can estimate its derivative  $\dot{\boldsymbol{x}}(t)$  (e.g., using finite differences, interpolation methods, etc. Cullum (1971); Knowles and Wallace (1995); Chartrand (2011); Knowles and Renka (2014); Wagner et al. (2018); Treven et al. (2021)). We capture the noise in our measurements and/or estimation of  $\dot{\boldsymbol{x}}(t)$  with  $\epsilon_{i,n}$ .
In summary, at each episode  $n$ , we deploy a policy  $\pi_{n}$  for a horizon of  $T$ , observe the system according to a proposed MSS  $S_{n}$ , and learn the dynamics  $\pmb{f}^{*}$ . By deploying  $\pi_{n}$  instead of the optimal policy  $\pi^{*}$ , we incur a regret,
$$
r _ {n} (S) \stackrel {\text {d e f}} {=} C \left(\boldsymbol {\pi} _ {n}, \boldsymbol {f} ^ {*}\right) - C \left(\boldsymbol {\pi} ^ {*}, \boldsymbol {f} ^ {*}\right).
$$
Note that the policy  $\pi_{n}$  depends on the data  $\mathcal{D}_{1:n - 1} = \cup_{i <   n}\mathcal{D}_i$  and hence implicitly on the MSS  $S$
Performance measure We analyze OCoRL by comparing it with the performance of the best policy  $\pi^{*}$  from the class  $\Pi$ . We evaluate the cumulative regret  $R_{N}(S)\stackrel {\mathrm{def}}{=}\sum_{n = 1}^{N}r_{n}(S)$  that sums the gaps between the performance of the policy  $\pi_{n}$  and the optimal policy  $\pi^{*}$  over all the episodes. If the regret  $R_{N}(S)$  is sublinear in  $N$ , then the average cost of the policy  $C(\pi_n,f^*)$  converges to the optimal cost  $C(\pi^{*},f^{*})$ .
Any meaningful analysis of cumulative regret for continuous time, state, and action spaces requires some assumptions on the system and the policy class. We make some continuity assumptions, similar to the discrete-time case (Khalil, 2015; Curi et al., 2020; Sussex et al., 2023), on the dynamics, policy, and cost.
Assumption 1 (Lipschitz continuity). Given any norm  $\| \cdot \|$ , we assume that the system dynamics  $f^{*}$  and cost  $c$  are  $L_{f}$  and  $L_{c}$ -Lipschitz continuous, respectively, with respect to the induced metric. Moreover, we define  $\Pi$  to be the policy class of  $L_{\pi}$ -Lipschitz continuous policy functions and  $\mathcal{F}$  a class of  $L_{f}$  Lipschitz continuous dynamics functions with respect to the induced metric.
We learn a model of  $f^{*}$  using data collected from the episodes. For a given state-action pair  $z = (x, u)$ , our learned model predicts a mean estimate  $\mu_{n}(z)$  and quantifies our epistemic uncertainty  $\sigma_{n}(z)$  about the function  $f^{*}$ .
Definition 2 (Well-calibrated statistical model of  $\pmb{f}^{*}$ , Rothfuss et al. (2023)). Let  $\mathcal{Z} \stackrel{\text{def}}{=} \mathcal{X} \times \mathcal{U}$ . An all-time well-calibrated statistical model of the function  $\pmb{f}^{*}$  is a sequence  $\{\mathcal{M}_n(\delta)\}_{n\geq 0}$ , where
$$
\mathcal {M} _ {n} (\delta) \stackrel {{d e f}} {{=}} \left\{\boldsymbol {f}: \mathcal {Z} \rightarrow \mathbb {R} ^ {d _ {x}} \mid \forall \boldsymbol {z} \in \mathcal {Z}, \forall j \in \{1, \dots , d _ {x} \}: | \mu_ {n, j} (\boldsymbol {z}) - f _ {j} (\boldsymbol {z}) | \leq \beta_ {n} (\delta) \sigma_ {n, j} (\boldsymbol {z}) \right\},
$$
if, with probability at least  $1 - \delta$ , we have  $\pmb{f}^{*}\in \bigcap_{n\geq 0}\mathcal{M}_{n}(\delta)$ . Here,  $\mu_{n,j}$  and  $\sigma_{n,j}$  denote the  $j$ -th element in the vector-valued mean and standard deviation functions  $\pmb{\mu}_n$  and  $\pmb{\sigma}_n$  respectively, and  $\beta_{n}(\delta)\in \mathbb{R}_{\geq 0}$  is a scalar function that depends on the confidence level  $\delta \in (0,1]$  and which is monotonically increasing in  $n$ .
Assumption 2 (Well-calibration). We assume that our learned model is an all-time well-calibrated statistical model of  $\mathbf{f}^*$ . We further assume that the standard deviation functions  $(\sigma_n(\cdot))_{n\geq 0}$  are  $L_{\sigma}$ -Lipschitz continuous.
2Here, the set  $S_{n}$  may depend on observations prior to episode  $n$  or is even constructed while we execute the trajectory. For ease of notation, we do not make this dependence explicit.
This is a natural assumption, which states that we are with high probability able to capture the dynamics within a confidence set spanned by our predicted mean and epistemic uncertainty. For example, GP models are all-time well-calibrated for a rich class of functions (c.f., Section 3.2) and also satisfy the Lipschitz continuity assumption on  $(\sigma_{n}(\cdot))_{n\geq 0}$  (Rothfuss et al., 2023). For Bayesian neural networks, obtaining accurate uncertainty estimates is still an open and active research problem. However, in practice, re-calibration techniques (Kuleshov et al., 2018) can be used.
By leveraging these assumptions, in the next section, we propose our algorithm OCoRL and derive a generic bound on its cumulative regret. Furthermore, we show that OCoRL provides sublinear cumulative regret for the case when GPs are used to learn  $f^{*}$ .
3 Optimistic Continuous-time RL Algorithm
OCoRL: OPTIMISTIC CONTINUOUS-TIME RL
Init: Statistical model  $\mathcal{M}_0$ , Simulator SIM, MSS  $S$ , Probability  $\delta$
for episode  $n = 1,\dots ,N$  do
$$
\boldsymbol{\pi}_{n} = \operatorname *{argmin}_{\boldsymbol {\pi}\in \Pi}\min_{\boldsymbol {f}\in \mathcal{M}_{n - 1}\cap \mathcal{F}}C(\boldsymbol {\pi},\boldsymbol {f})
$$
/* Select optimistic policy */
$$
\mathcal {D} _ {n} = \left\{\left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right), \dot {\boldsymbol {y}} _ {n} \left(t _ {n, i}\right)\right) \mid t _ {n, i} \in S _ {n} \right\} \leftarrow \operatorname {S I M} \left(\boldsymbol {\pi} _ {n}, S _ {n}\right)
$$
/* Measurement collection */
$$
\mathcal {M} _ {n} \leftarrow \left(\boldsymbol {\mu} _ {n}, \boldsymbol {\sigma} _ {n}, \beta_ {n} (\delta)\right) \leftarrow \mathcal {D} _ {1: n}
$$
/* Update statistical model/*
Optimistic policy selection There are several strategies we can deploy to trade-off exploration and exploitation, e.g., dithering (ε-greedy, Boltzmann exploration (Sutton and Barto, 2018)), Thompson sampling (Osband et al., 2013), upper-confidence RL (UCRL) (Auer et al., 2008), etc. OCoRL is a continuous-time variant of the Hallucinated UCRL strategy introduced by Chowdhury and Gopalan (2017); Curi et al. (2020). In episode  $n$ , the optimistic policy is obtained by solving the optimal control problem:
$$
\left(\boldsymbol {\pi} _ {n}, \boldsymbol {f} _ {n}\right) \stackrel {\text {d e f}} {=} \underset {\boldsymbol {\pi} \in \Pi , \boldsymbol {f} \in \mathcal {M} _ {n - 1} \cap \mathcal {F}} {\operatorname {a r g m i n}} C (\boldsymbol {\pi}, \boldsymbol {f}) \tag {2}
$$
Here,  $f_{n}$  is a dynamical system such that the cost by controlling  $f_{n}$  with its optimal policy  $\pi_{n}$  is the lowest among all the plausible systems from  $\mathcal{M}_{n-1} \cap \mathcal{F}$ . The optimal control problem (2) is infinite-dimensional, in general nonlinear, and thus hard to solve. For the analysis, we assume we can perfectly solve it. In Appendix B, we present details on how we solve it in practice. Specifically, in Appendix B.1, we show that our results seamlessly extend to the setting where the optimal control problem of Equation (1) is discretized w.r.t. an arbitrary choice of discretization. Moreover, in this setting, we show that theoretical guarantees can be derived without restricting the models to  $f \in \mathcal{M}_{n-1} \cap \mathcal{F}$ . Instead, a practical optimization over all models in  $\mathcal{M}_{n-1}$  can be performed as in Curi et al. (2020); Pasztor et al. (2021).
Model complexity We expect that the regret of any model-based continuous-time RL algorithm depends both on the hardness of learning the underlying true dynamics model  $\pmb{f}^{*}$  and the MSS. To capture both, we define the following model complexity:
$$
\mathcal {I} _ {N} \left(\boldsymbol {f} ^ {*}, S\right) \stackrel {\text {d e f}} {=} \max  _ {\substack {\boldsymbol {\pi} _ {1}, \dots , \boldsymbol {\pi} _ {N} \\ \boldsymbol {\pi} _ {n} \in \Pi}} \sum_ {n = 1} ^ {N} \int_ {0} ^ {T} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (t)\right) \right\| ^ {2} d t. \tag{3}
$$
The model complexity measure captures the hardness of learning the dynamics  $f^{*}$ . Intuitively, for a given  $N$ , the more complicated the dynamics  $f^{*}$ , the larger the epistemic uncertainty and thereby the model complexity. For the discrete-time setting, the integral is replaced by the sum over the uncertainties on the trajectories (c.f., Equation (8) of Curi et al. (2020)). In the continuous-time setting, we do not observe the state at every time step, but only at a finite number of times wherever the MSS  $S$  proposes to measure the system. Accordingly,  $S$  influences how we collect data and update our calibrated model. Therefore, the model complexity depends on  $S$ . Next, we first present the regret bound for general MSSs, then we look at particular strategies for which we can show convergence of the regret.
Proposition 1. Let  $S$  be any MSS. If we run OCORL, we have with probability at least  $1 - \delta$ ,
$$
R _ {N} (S) \leq 2 \beta_ {N} L _ {c} \left(1 + L _ {\pi}\right) T ^ {\frac {3}{2}} e ^ {L _ {f} \left(1 + L _ {\pi}\right) T} \sqrt {N \mathcal {I} _ {N} \left(\boldsymbol {f} ^ {*}, S\right)}. \tag {4}
$$
We provide the proof of Proposition 1 in Appendix A. Because we have access only to the statistical model and any errors in dynamics compound (continuously in time) over the episode, the regret  $R_{N}(S)$  depends exponentially on the horizon  $T$ . This is in line with the prior work in the discrete-time setting (Curi et al., 2020). If the model complexity term  $\mathcal{I}_N(\pmb {f}^*,S)$  and  $\beta_{N}$  grow at a rate slower than  $N$ , the regret is sublinear and the average performance of OCoRL converges to  $C(\pmb {\pi}^{*},\pmb{f}^{*})$ . In our analysis, the key step to show sublinear regret for the GP dynamics model is to upper bound the integral of uncertainty in the model complexity with the sum of uncertainties at the points where we collect the measurements. In the next section, we show how this can be done for different measurement selection strategies.
3.1 Measurement selection strategies (MSS)
In the following, we present different natural MSSs and compare the number of measurements they propose per episode. Formal derivations are included in Appendix A.2.
Oracle Intuitively, if we take measurements at the times when we are the most uncertain about dynamics on the executed trajectory, i.e., when  $\| \sigma_{n-1}(z_n(t)) \|$  is largest, we gain the most knowledge about the true function  $f^*$ . Indeed, when the statistical model is a GP and noise is homoscedastic and Gaussian, observing the most uncertain point on the trajectory leads to the maximal reduction in entropy of  $f^*(z_n(t))$  (c.f., Lemma 5.3 of Srinivas et al. (2009)). In the ideal case, we can define an MSS that collects only the point with the highest uncertainty in every episode, i.e.,  $S_n^{\mathrm{ORA}} \stackrel{\mathrm{def}}{=} \{t_{n,1}\}$  where  $t_{n,1} \stackrel{\mathrm{def}}{=} \operatorname{argmax}_{0 \leq t \leq T} \| \sigma_{n-1}(z_n(t)) \|^2$ . For this MSS we bound the integral over the horizon  $T$  with the maximal value of the integrand times the horizon  $T$ :
$$
\mathcal {I} _ {N} \left(\boldsymbol {f} ^ {*}, S _ {n} ^ {\text {O R A}}\right) \leq \max  _ {\substack {\boldsymbol {\pi} _ {1}, \dots , \boldsymbol {\pi} _ {N} \\ \boldsymbol {\pi} _ {n} \in \Pi}} T \sum_ {n = 1} ^ {N} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, 1}\right)\right) \right\| ^ {2} \tag{5}
$$
The oracle MSS collects only one measurement per episode, however, it is impractical since it requires knowing the most uncertain point on the true trajectory a priori, i.e., before executing the policy.
Equidistant Another natural MSS is the equidistant MSS. We collect  $m_{n}$  equidistant measurements in episode  $n$  and upper bound the integral with the upper Darboux integral.
$$
\mathcal {I} _ {N} \left(\boldsymbol {f} ^ {*}, S _ {n} ^ {\mathrm {E Q I}}\right) \leq \max  _ {\substack {\boldsymbol {\pi} _ {1}, \dots , \boldsymbol {\pi} _ {N} \\ \boldsymbol {\pi} _ {n} \in \Pi}} \sum_ {n = 1} ^ {N} \frac {T}{m _ {n}} \sum_ {i = 1} ^ {m _ {n}} \left(\left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right)\right) \right\| ^ {2} + \frac {T L _ {\boldsymbol {\sigma} ^ {2}}}{m _ {n}}\right). \tag{6}
$$
Here  $L_{\sigma^2}$  is the Lipschitz constant of  $\| \pmb{\sigma}_{n - 1}(\cdot)\|^2$ . To achieve sublinear regret, we require that  $\sum_{n = 1}^{N}\frac{1}{m_n}\in o(N)$ . Therefore, for a fixed equidistant MSS, our analysis does not ensure a sublinear regret. This is because we consider a continuous-time regret which is integrated (c.f., Equation (1)) while in the discrete-time setting, the regret is defined for the equidistant MSS only (Curi et al., 2020). Accordingly, we study a strictly harder problem. Nonetheless, by linearly increasing the number of observations per episode and setting  $m_{n} = n$ , we get  $\sum_{n = 1}^{N}\frac{1}{m_n}\in \mathcal{O}(\log (N))$  and sublinear regret. The equidistant MSS is easy to implement, however, the required number of samples is increasing linearly with the number of episodes.
Adaptive Finally, we present an MSS that is practical, i.e., easy to implement and at the same time requires only a few measurements per episode. The core idea of receding horizon adaptive MSS is simple: simulate (hallucinate) the system  $\pmb{f}_n$  with the policy  $\pmb{\pi}_n$ , and find the time  $t$  such that  $\| \pmb{\sigma}_{n-1}(\widehat{\pmb{z}}_n(t)) \|$  is largest. Here,  $\widehat{\pmb{z}}_n(t)$  is the state-action pair at time  $t$  in episode  $n$  for the hallucinated trajectory.
However, the hallucinated trajectory can deviate from the true trajectory exponentially fast in time, and the time of the largest variance on the hallucinated trajectory can be far away from the time of the largest variance on the true trajectory. To remedy this technicality, we utilize a receding horizon MSS. We split, in episode  $n$ , the time horizon  $T$  into uniform-length intervals of  $\Delta_{n}$  time, where  $\Delta_{n} \in \Omega\left(\frac{1}{\beta_{n}}\right)$ , c.f., Appendix A. At the beginning of every time interval, we measure the true state-action pair  $z$ , hallucinate with policy  $\pi_{n}$  starting from  $z$  for  $\Delta_{n}$  time on the system  $f_{n}$ , and calculate the time when the hallucinated trajectory has the highest variance. Over the next time horizon  $\Delta_{n}$ , we collect a measurement only at that time. Formally, let  $m_{n} = \lceil T / \Delta_{n} \rceil$  be the number of measurements in episode  $n$  and denote for every  $i \in \{1, \dots, m_{n}\}$  the time with the highest variance on the hallucinated trajectory in the time interval  $[(i - 1)\Delta_{n}, i\Delta_{n}]$  by  $t_{n,i}$ :