Efficient Exploration in Continuous-time Model-based Reinforcement Learning
Authors: Lenart Treven
Jonas Hübotter
Bhavya Sukhija
Florian Dorfler
Andreas Krause
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Optimal Control
Continuous Time
TL;DR: We study reinforcement learning in continuous time.

Abstract: Reinforcement learning algorithms typically consider discrete-time dynamics, even though the underlying systems are often continuous in time. In this paper, we introduce a model-based reinforcement learning algorithm that represents continuous-time dynamics using nonlinear ordinary differential equations (ODEs). We capture epistemic uncertainty using well-calibrated probabilistic models, and use the optimistic principle for exploration. Our regret bounds surface the importance of the measurement selection strategy (MSS), since in continuous time we not only must decide how to explore, but also when to observe the underlying system. Our analysis demonstrates that the regret is sublinear when modeling ODEs with Gaussian Processes (GP) for common choices of MSS, such as equidistant sampling. Additionally, we propose an adaptive, data-dependent, practical MSS that, when combined with GP dynamics, also achieves sublinear regret with significantly fewer samples. We showcase the benefits of continuous-time modeling over its discrete-time counterpart, as well as our proposed adaptive MSS over standard baselines, on several applications.

Efficient Exploration in Continuous-time Model-based Reinforcement Learning
Lenart Treven ETH Zürich
Jonas Hübofter
Bhavya Sukhija ETH Zürich
ETH Zürich
treven1@ethz.cr ch
Jhusbotterrsttudent ethz. ch
sukhijabb@thhz.c ch
Florian Dürfler
Andreas Krause
ETH Zürich dorflerGethz.ch
ETH Zürich
krauseaßethz.ccm ch
Abstract
Reinforcement learning algorithms typically consider discrete-time dynamics, even though the underlying systems are often continuous in time. In this paper, we introduce model-based reinforcement learning algorithm that represents continuous-time dynamics using nonlinear ordinary differential equations (ODEs). We capture epistemic uncertainty using well-calibrated probabilistic models, and use the optimistic principle for exploration. Our regret bounds surface the importance of the measurement selection strategy (MSS), since in continuous time we not only must decide how to explore, but also when to observe the underlying system. Our analysis demonstrates that the regret is sublinear when modeling ODEs with Gaussian Processes (GP) for common choices of MSS, such as equidistant sampling. Additionally, we propose an adaptive, data-dependent, practical MSS that, when combined with GP dynamics, also achieves sublincar regret with significantly fewer samples. We showcase the benefits of continuous-time modeling over its discrete-time counterpart, well as our proposed adaptive MSS over standard baselines, on several applications.
Introduction
Real-world systems encountered in natural sciences and engineering applications, such as robotics (Spong al., 2006), biology (Lenhart and Workman, 2007: Jones et al., 2009), medicine (Panetta and Fister, 2003), etc., are fundamentally continuous in time. Therefore, ordinary differential equations (ODEs) are the natural modeling language. However, the reinforcement learning (RL) community predominantly models problems in discrete time, with a few notable exceptions (Doya, 2000; Yildiz al., 2021; Lutter al., 2021). The discretization of continuous-time systems imposes limitations the application of state-of of-the-art RL algorithms, as they are tied to specific discretization schemes.
Discretization of continuous-time systems sacrifices several essential properties that could be preserved in continuous-time framework (Nesic and Postoyan, 2021). For instance, exact discretization is only feasible for linear systems, leading to an inherent discrepancy when using standard discretization techniques for nonlinear systems. Additionally, discretization obscures the inter-sample behavior of the system, changes stability properties, may result in uncontrollable systems, or requires an excessively high sampling rate. Multi-time-see systems are particularly vulnerable to these issues (Engquist et al., 2007). In many practical scenarios, the constraints imposed by discrete-tinn time modeling are undesirable. Discrete-time models do not allow for the independent adjustment of measurement and control frequencies, which is crucial for real-world systems that operate in different regimes. For example, autonomous driving, wow-frequency sensor sampling
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

and control actuation may suffice at slow speeds, while high-speed driving demands faster control. How to choose aperiodic measurements and control is studied in the literature on the event and self-triggered control (Astrom and Bernhardsson, 2002; Anta and Tabuada, 2010; Heemels et al., 2012). They show that the number of control inputs can be significantly reduced by using aperiodic control. Moreover, data from multiple sources is often collected at varying frequencies (Ghysels et al., 2006) and is often not even equidistant in time. Discrete-time models struggle to exploit this data for learning, while continuous-time models can naturally accommodate it.
Continuous-timm modeling also offers the flexibility to determine optimal measurement times based on need, in contrast to the fixed measurement frequency in discrete-time settings, which can easily miss informative samples. This advantage is particularly relevant in fields like medicine, where patient monitoring often requires higher frequency measurements during the onset of illness and lower frequency measurements as the patient recovers (Kaandorp and Koole, 2007). At the same time, in fields such as medicine, measurements are often costly, and hence, it is important that the most informative measurements are selected. Discrete-time models, limited to equidistant measurements, therefore often result in suboptimal decision-makee and their sample efficiency is fundamentally limited. In summary, continuous- modeling is agnostic to the choice of measurement selection strategy (MSS), whereas discrete-time modeling often only works for an equidistant MSS.
Contributions Given the advantages of continuous- time modeling, in this work, we propose an optimistic continuous-time model-based RL algorithm OCORL. Moreover, we theoretically analyze OCORL and show a general regret bound that holds for any MSS. We further show that for common choices of MSSs, such equidistant sampling, the regret is sublinear when we model the dynamics with GPs (Williams and Rasmussen, 2006). To our knowledge, we are the first to give a no-regret algorithm for a rich class of nonlinear dynamical systems in the continuous-time RL setting. We further propose an adaptive MSS that is practical, data-dependent, and requires considerably fewer measurements compared to the equidistant MSS while still ensuring the sublinear regret for the GP case. Crucial to OCoRL is the exploration induced by the optimism in the face of uncertainty principle for model-based RL, which is commonly employed in the discrete-time realm (Auer et al., 2008; Curi et al., 2020). We validate the performance of OCoRL on several robotic tasks, where we clearly showcase the advantages of continuous-time modeling over its discrete-time counterpart. Finally, we provide an efficient implementation? of OCORL in JAX (Bradbury et al., 2018).
2 Problem setting
In this work, we study continuous-tin time deterministic dynamical system f with initial state 2o E X' C Rdd i.c.
(t) 20 + ( f" (ar(s), ds.
Here u : 00) Rd is the input we apply to the system. Moreover, we consider state feedback controllers represented through policy : X' + U Rd that is, u(s) π (xxs)). Our objective is to find the optimal policy with respect to given running cost function e R2dz x Rd - IR. Specifi- cally, we are interested in solving the following optimal control (OC) problem over the policy space 11:
Jo c(æ, π (ar)) dl /'
def n argmin C(T,, f*) = argmin TEll
HEll
s.t. Φ = f((xTT())))
(00))= Bo
The function f is unknown, but we collect data over episodes 1...,N and learn about the system by deploying policy n, E IT for the horizon of T in episode n. In an (overly) idealized continuous time setting, one can measure the system any time step. However, we consider more practical setting, where we assume taking measurements is costly, and therefore want as few measurements as necessary. To this end, we formally define measurement selection strategy below.

---

Definition 1 (Measurement selection strategy). A measurement selection strategy S is se- quence of sets (S.)n21, such that Sn contains m, points at which we take measurements, i.e. Sp C [0,T), ma."
During episode n. given a policy n and a MSS Sa, we collect a dataset Dr ~ (T Sa). The dataset is defined as
Dn (En((nn)) yn (l(,))))) bn E Sn, E ,..,,
where
2n (4n,)) (In (t TI (2 (fm,))),; (n,,n,)) def dr, (fm,s) + 6n,i.
Here F, (t), 2b, (t) are state and state derivative in episode n, and Gn is i...d, --sub-Gaussian noise of the state derivative observations. Note, even though in practice only the state æ(t) might be observable, one can estimate its derivative (t) (e.g., using finite differences, interpolation methods, etc. Cullum (1971); Knowles and Wallace (1995); Chartrand (201 I); Knowles and Renka (2014); Wagner et al. (2018); Treven et al. (2021)). We capture the noise in our measurements and/or estimation of (t) with Ei,n-
In summary, each episode we deploy policy ,, for horizon of T, observe the system according to proposed MSS Sa, and learn the dynamics f*. By deploying T. instead of the optimal policy π.", incur regret,
(S) def C(TT, f*) - C(TT"ff))
Note that the policy n depends on the data Dim-1 UsenD D, and hence implicitly on the MSS S.
Performance measure We analyze OCORL by comparing it with the performance of the best policy π" from the class II. We evaluate the cumulative regret Rx Σn=1 (S) that sums the gaps between the performance of the policy n, and the optimal policy π over all the episodes. If the regret (x(S) is sublinear in N, then the average cost of the policy C(T,,, f*) converges to the optimal cost C(TT', π f*).
2.1 Assumptions
Any meaningful analysis of cumulative regret for continuous time, state, and action spaces requires some assumptions on the system and the policy class. We make some continuity assumptions, similar to the discrete-time case (Khalil, 2015; Curi et al., 2020; Sussex et al., 2023), on the dynamics, policy, and cost.
Assumption 1 (Lipschitz continuity). Given any norm ||-||, we assume that the system dynamics f and cost are Lg and Le-Lipschitz continuous, respectively, with respect to the induced metric. Moreover, define II to be the policy class of -ipisscciitz continuous policy functions and F class of Lg Lipschitz continuous dynamics functions with respect to the induced metric.
We learn model of f using data collected from the episodes. For given state-action pair (z, u), our learned model predicts mean estimate (z)()) and quantifies our epistemic uncertainty (2) about the function f"
Definition 2 (Well-calibratee statistical model of f", Rothfuss et al. (2023)). Let Z def X x U. An all-time well-calibrated statistical model of the function f is a sequence (Mn(6))n20 where
M)) def {s : - R Vz E 2,Vj (1....,dz) ((m)(z)) (=))())))=()(()))) S (n
if. with probability least 1 8, we have f -n=o Mn). Here, pnnj and nn. denote the j-th element in the vector-valued mean and standard deviation functions th, and on respectively, and (6) E Ix is a scalar function that depends on the confidence level E (0, 1] and which is monotonically increasing in
A Assumption 2 velllcaaibratiion) statistical model of f*. We further assume that the standard deviation functions (a,,.))) are Lo Lipsshiiz continuous.
We assume that our learned model is all-time well-calibrated
Heer, the set S,, may depend on observations prior to episode or is even constructed while we execute the trajectory. For ease of notation, do not make this dependence explicit.
3

---

This is natural assumption, which states that we are with high probability able to capture the dynamics within a confidence set spanned by our predicted mean and epistemic uncertainty. For example, GP models are all-time well-calibrated for rich class of functions (c.f., Section 3.2) and also satisfy the Lipsehitz continuity assumption on (a,, (.),(.))))) ()) (Rothfuss et al., 2023). For Bayesian neural networks, obtaining accurate uncertainty estimates is still an open and active research problem. However, in practice, re-calibration techniques (Kuleshov et al., 2018) can be used.
By leveraging these assumptions, in the next section, we propose our algorithm OCORL and derive generic bound on its cumulative regret. Furthermore, we show that OCoRL provides sublinear cumulative regret for the case when GPs are used to learn f
3 Optimistic Continuous-time RL Algorithm
OCoRL: OPTIMISTIC CONTINUOUS-TIME RI. Init: Statistical model Mo. Simulator SIM, MSS S, Probability 8 for episode n = 1,...; ...... N do
= argmin min well feM
C(T,F
1* Select optimistic policy /*
Dn = (tzn(tn,), Un (tna)) tnis E Sn} + SIM(T Sn)
/* Measurement collection /
Mn f (Mn, On, (x()) + Din
Update statistical model/*
Optimistic policy selection There several strategies can deploy trade-off exploration and exploitation, c.g.. dithering (s-greedy, Boltzmann exploration (Sutton and Barto, 2018)). Thompson sampling (Osband et al., 2013), upper-confidence RL (UCRL) (Auer al., 2008), etc. OCoRL is continuous- time variant of the Hallucinated UCRL strategy introduced by Chowdhury and Gopalan (2017); Curi et al. (2020). In episode n, the optimistic policy is obtained by solving the optimal control problem:
(Tna fn)
argmin TElL, fEMa-inF
C(T, f)
(2)
Here, fn. is a dynamical system such that the cost by controlling fn with its optimal policy T, is the lowest among all the plausible systems from Mn n N.F. The optimal control problem (2) is dimenssnallesional, in general nonlinear, and thus hard to solve. For the analysis, we assume we can perfectly solve In Appendix B. we present details on how we solve it in practice. Specifically, in Appendix B.1, we show that our results seamlessly extend to the setting where the optimal control problem of Equation (1) is discretized w.r.t. an arbitrary choice of discretization. Moreover, in this setting, we show that theoretical guarantees can be derived without restricting the models to ff E Mn-1 O F. Instead, a practical optimization over all models in Mn-1 can be performed as in Curi et al. (2020); Pasztor et al. (2021).
Model complexity We expect that the regret any model-based continuous-time RL algorithm depends both on the hardness of learning the underlying true dynamics model f and the MSS. To capture both, we define the following model complexity:
N Σ ==1
IN (f",
S) def
max JT ...... AN ElI
( dt.
The model complexity measure captures the hardness of learning the dynamics f Intuitively, for given N, the more complicated the dynamics f", the larger the epistemic uncertainty and thereby the model complexity. For the discrete-ting setting, the integral replaced by the sum over the uncertain- ties on the trajectories (c.f., Equation (8) of Curi et al. (2020)). In the continuous-ting setting, we do not observe the state at every time step, but only at a finite number of times wherever the MSS S pro- poses to measure the system. Accordingly, S influences how we collect data and update our calibrated model. Therefore, the model complexity depends on S. Next, we first present the regret bound for general MSSs, then we look at particular strategies for which
show convergence of the regret.

---

Proposition 1. Let S be any MSS. we run OCORL, we have with probability at least 1 - 8.
(y( S 28 Lc L (NIN(f*.S).
(4)
We provide the proof of Proposition 1 in Appendix A. Because we have access only to the statistical model and any errors in dynamics compound (continuously in time) over the episode, the regret Ry(S) depends exponentially on the horizon T. This is in line with the prior work in the discrete-time setting (Curi et al., 2020). If the model complexity term (x( , S) and 3N grow at rate slower than N, the regret is sublinear and the average performance of OCORL converges to C( ). In our analysis, the key step to show sublinear regret for the GP dynamics model to upper bound the integral of uncertainty in the model complexity with the sum of uncertainties
at the points where we collect the measurements. the next section, we show how this can be done for different measurement selection strategies.
3.1 Measurement selection strategies (MSS)
In the following, we present different natural MSSs and compare the number of measurements they propose per episode. Formal derivations are included in Appendix A.2.
Oracle Intuitively, if we take measurements at the times when we are the most uncertain about dynamics on the executed trajectory, i.e., when ||o,-1 (zn (f))l is largest, we gain the most knowledge about the true function f*. Indeed, when the statistical model is a GP and noise is homoscedastic and Gaussian, observing the most uncertain point on the trajectory leads to the maximal reduction in entropy of f (=,,(()) (c.f., Lemma 5.3 Srinivas et al. (2009)). In the ideal case, define MSS that collects only the point with the highest uncertainty in every episode, def del {ln,1) where 1n.1 argmaxoşren || (t))l. For this MSS we bound the integral over the horizon T with the maximal value of the integrand times the horizon T::
i.e., SOKA
(f",, SORA) S max
T T 1a (Zn (1n,)))1
The oracle MSS collects only one measurement per episode, however, it impractical since it requires knowing the most uncertain point on the true trajectory priori, i.e., before executing the policy.
Equidistant Another natural MSS is the equidistant MSS. We collect m, equidistant measurements in episode and upper bound the integral with the upper Darboux integral.
T Min
TL mn
If
SPOT
max W., EIT mn
[|on-1 (zn
(6)
Here Liga the Lipschitz constant of [|on- 1-) To achieve sublinear regret, we require that En 1 E o(N). Therefore, for a fixed equidistant MSS, our analysis does not ensure a sublinear regret. This is because we consider contimuous-time regret which is integrated (c.f., Equation (1)) while in the discrete-time setting, the regret is defined for the equidistant MSS only (Curi et al., 2020). Accordingly, we study a strictly harder problem. Nonetheless, by linearly increasing the number of
observations per episode and setting m. n, we get N 1 E O(lg((N) (N)) and sublinear regret.
mn The equidistant MSS is easy to implement, however, the required number of samples is increasing linearly with the number of episodes.
Adaptive Finally, we present an MSS that is practical, i.e., easy to implement and at the same time requires only few measurements per episode. The core idea of receding horizon adaptive MSS is simple: simulate (hallucinate) the system fn with the policy n,, and find the time t such that ||o,., (2, (t)) is largest. Here, (t) is the acteonction pair time in episode for the hallucinated trajectory.
5

---

However, the hallucinated trajectory can deviate from the true trajectory exponentially fast time, and the time of the largest variance the halluci- nated trajectory can be far away from the time of the largest variance on the true trajectory. To rem- edy this technicality, we utilize a receding horizon MSS. We split, in episode the time horizon T into uniform-length intervals of An time, where An S2(
En(tan)
In (3Am)
z (mma)))
2, f (2 T (((())))
xa (2am)
valla))
(Th) = 0 (Tm)
8.. ). c.f., Appendix A. At the beginning of every time interval. we measure the true state- action pair z, hallucinate with policy nu starting from for An time on the system fn, and cal- culate the time when the hallucinated trajectory has the highest variance. Over the next time hori- zon A.. we collect a measurement only at that time. Formally, let m, [T/A,] be the num- ber of measurements in episode n and denote for every i {L...., m the time with the highest variance on the hallucinated trajectory in the time interval [[i 1) iA,,] by En
2nn (fn.))
Ir (An)
fu (Zni (T.))
24.!
(. (0) (.,(0)) 20
Figure 1: In episode n we split the horizon T into intervals of A, time. We hallucinate the trajectory in every interval and select time 6, in the interval where the uncertainty on the hallucinated trajectory is the highest.
b del (i - 1) + argmax ( Osisa,
ān = fn n (3n)) (0) 2,,(i 1)
s.t.
For this MSS, we show in Appendix A the upper bound
nn
IN(f", SAPP) S max
9
Σ
|| |o (zn (In())))))
(7)
Mn
The
model complexity upper bound for the adaptive MSS in Equation (7) does not have the a dependent term from the equidistant MSS (6) and only depends on the sum of uncertainties the col- lected points. Further, the number of points we collect in episode n is O (B,), and 8,,, e.g., if we model the dynamics with GPs with radial basis function (RBF) kernel, of order polylog(n) (c.f., Lemma 2).
3.2 Modeling dynamics with a Gaussian Process
We can prove sublinear regret for the proposed three MSSs coupled with optimistic exploration when we model the dynamics using GPs with standard kernels such as RBF, linear, etc. We consider the vector-value dynamics model (z) (z) fā, (z)), where vallur functions f E Hy are elements of Reproducing Kernel Hilbert Space (RKHS) Hr with kernel function k;
and their norm is bounded || ll. < B. We write f H
def
|1fll S B}.
To learn f we fit GP model with a zero mean and kernel & to the collected data D For ease of notation, we denote by vl.n the concatenated j-th dimension of the state derivative observations from D The posterior means and variances of the GP model have closed form (Williams and Rasmussen (2006)):
(n)(z) = k (z) Kn + +021) - Vi.
(z) k(z, k (z) (Kn + 0221) kz
where we write K2, = 1k(1,, 2m ) 2.. Eee...DDi . and k (z)= [k(Z), 2)),, =)),,EDL... The posterior means and variances with the right scaling factor (5) satisfy the all-time well-calibrated Assumption 2.
Lemma 2 (Lemma 3.6 from Rothfuss et al. (2023)). Let f E Hdd and & E (0, Then there exist (5) E O(V7 ( such that the confidence sets Mn built from the triplets (Mh, nn 3n (5)) form an all time well-calibrated statistical model.

---

Here, ine is the maximum information gain after observing points (Srinivas et al., 2009), as defined in Appendix A. where we also provide the rates for common kernels. For example, for the RBF kernel, 7n O (log(n)d=+uur
Finally, show sublinear regret for the proposed MSSs for the case when we model dynamics with GPs. The proof of Theorem 3 is provided in Appendix A.
Theorem 3. Assume that f HE. the observation noise is ii.d. NN o² I), and ||-Il is the Euclidean norm. We model f with the GP model. The regret for different MSSs is with probability at least ò bounded by
( S 0
(BNT2 (NTN) (SNT2 VN (7N + log(()))
ORA m n.
=
Rx (S < 0
FOF m
(y s O (SNT4 (Nw))
m = O(B,)
The optimistic exploration coupled with any of the proposed MSSs achieves regret that depends on the maximum information gain. All regret bounds from Theorem 3 are sublinear for common kernels, like
linear and RBF. The bound for the adaptive MSS with RBF kernel is O V N (2),, +d. where we hide the dependence on T in the O notation. We reiterate that while the number of observations per episode of the oracle MSS is optimal, cannot implement the oracle MSS in practice. In contrast, the equidistant MSS is easy to implement, but the number of measurements grows linearly with episodes. Finally, adaptive MSS is practical, i..., easy to implement, and requires only a few measurements per episode, i.e., polylog(n) in episode n for RBF.
Summary OCORL consists of two key and orthogonal components; (i) optimistic policy selection and (ii) measurement selection strategies (MSSs). In optimistic policy selection, we optimistccally, w.r.t plausible dynamics. plan trajectory and rollout the resulting policy. We study different MSSs, such as the typical equidistant MSS, for data collection within the framework of continuous time modeling- Furthermore, we propose an adaptive MSS that measures data where we have the highest uncertainty on the planned (hallucinated) trajectory. We show that OCoRL suffers no regret for the equidistant, adaptive, and oracle MSSs.
Related work
Model-based Reinforcement Learning Model-based reinforcement learning (MBRL) has been an active area of research in recent years, addressing the challenges of learning and exploiting environment dynamics for improved decision-making Among the seminal contributions, Deisenroth and Rasmussen (2011) 1) proposed the PILCO algorithm which uses Gaussian processes for learning system dynamics and policy optimization. Chua al. (2018) used deep ensembles as dynamics models. They coupled MPC (Morari and Lee, 1999) to efficiently solve high dimensional tasks with considerably better sample complexity compared to the state-of-the-ar the-art model-free methods SAC (Haarooa al., 2018), PPO (Schulman al., 2017). and DDPG (Lillicrap et al., 2015). The aforementioned model-based methods use more or less greedy exploitation that is provably optimal only in the case of the linear dynamics (Simchowitz and Foster, 2020). Exploration methods based on Thompson sampling (Dearden et al., 2013; Chowdhury and Gopalan, 2019) and Optimism (Auer et al., 2008: Abbasi-Yadkori and Szepesvári, 2011: Luo et al., 2018: Curi al., 2020), however, provably converge to the optimal policy also for a large class of nonlinear dynamics if modeled with GPs. Among the discrete-time RL algorithms, our work is most similar the work of Curi al. (2020) where they use the reparametrization trick to explore optimistically among all statistically plausible discrete dynamical models.
Continuous-time time Reinforcement Learning Reinforcement learning in continuous-timm has been around for several decades (Doya, 2000; Vrabie and Lewis, 2008, 2009; Vanvoudakis et al., 2009). Recently, the field has gained more traction following the work on Neural ODEs by Chen et al. (2018). While physies biases such as Lagrangian (Cranmer et al., 2020) or Hamiltonian (Greydanus et al., 2019) mechanics can be enforced in continuous-ting modeling, different challenges such
7

---

as vanishing Q-function (Tallec et al., 2019) need to be addressed in the continuous time setting. Yildiz al. (2021) introduce a practical episodic model-based algorithm continuous time. In each episode, they use the learned mean estimate of the ODE model to solve the optimal control task with a variant of a continuous-ting solve the optimal control task using the optimistic principle. Moreover, we thoroughly motivate optimistic planning from a theoretical standpoint. Lutter et al. (2021) introduce a continuous fitted value iteration and further show a successful hardware application of their continuous- time algorithm. A vast literature exists for continuous-timg
actor-critic algorithm. Compared to Yildiz et al. (2021) we
RL with linear systems (Modares and Lewis, 2014; Wang et al., 2020), but few, only for linear systems, provide theoretical guarantees (Mohammadi et al., 2021; Basei et al., 2022). To the best of our knowledge, we are the first to provide theoretical guarantees of convergence to the optimal cost in continuous time for large class of RKHS functions.
Aperiodic strategies Aperiodic MSSs and controls (event and self-triggered control) are mostly neglected in RL since most RL works predominantly focus on discrete-time modeling. There exists considerable amount of literature on the event and self-triggered control (Astrom and Bernhardsson, 2002; Anta and Tabuada, 2010; Heemels et al., 2012). However, compared to periodic control, its theory is far from being mature (Heemels et al., 2021). In our work, we assume that we can control the system continuously, and rather focus on when to measure the system instead. The closest to our adaptive MSS is the work of Du al. (2020), where they empirically show that by optimizing the number of interaction times, they can achieve similar performance (in terms of cost) but with fewer interactions. Compared to us, they do not provide any theoretical guarantees. Umlauft and Hirche (2019); Lederer et al. (2021) consider the non-episodic setting where they can continuously monitor the system. They suggest taking measurements only when the uncertainty of the learned model on the monitored trajectory surpasses the boundary ensuring stability. They empirically show, for feedback linearizable systems, that by applying their strategy the number of measurements reduces drastically and the tracking error remains bounded. Compared to them, we consider general dynamical systems and also don't assume continuous system monitoring.
5 Experiments
We now empirically evaluate the performance of OCORL on several environments. We test OCORL on Cancer Treatment and Glucose in blood systems from Howe et al. (2022), Pendulum, Mountain Car and Cart Pole from Brockman et al. (2016), Bicycle from Polack et al. (2017), Furuta Pendulum from Lutter al. (2021) and Quadrotor 2D and 3D from Nonami et al. (2010). The details of the systems' dynamics and tasks are provided in Appendix C.
Comparison methods To make the comparison fair, we adjust methods so that they all collect the same number of measurements per episode. For the equidistant setting, we collect M points per episode (we provide values of A for different systems in Appendix C). For the adaptive MSS, we assume A,, 2 T, and instead of one measurement per episode we collect a batch of M measurements such that they (as batch) maximize the variance on the hallucinated trajectory. To this end, we consider the Greedy Max Determinant and Greedy Max Kernel Distance strategies of Holzmüller et al. (2022). We provide details of the adaptive strategies in Appendix C. We compare OCoRL with the described MSSs to the optimal discrete-time zero hold control, where we assume the access to the true discretized dynamics fā u) x + _ (T/(M-1) f (æ(t), u) dt. We further also compare with the best continuous-time control policy, i.e., the solution Equation (I).
Does the continuous-time control policy perform better than the discrete-time control policy? In the first experiment, we test whether learning a continuous-ting coupled with continuous- control policy on the learned model can outperform the discrete-time zero-order hold control on the true system. We conduct the experiment on all environments and report the cost after running OCORL for a few tens of episodes (the exact experimental details are provided in Appendix C). From Table 1, we conclude that the OCORL outperforms the discrete-time zero-order hold control on the true model on every system we use the adaptive MSS, while achieving lower cost on 7 out of 9 systems if we measure the system equidistantly.
model from the finite data
8

---

Table 1: OCoRL with adaptive MSSs achieves lower final cost C( compared the discrete-time control on the true system on all tested enviromments while converging towards the best continuous- control policy. While equidistant MSS achieves higher cost compared to the adaptive MSS, it still outperforms the discrete-time zero-order hold control on the true model for most systems.
Known True Model
Optimistic Exploration with different MSS
System
Continuous time OC
Discrete zero order hold OC
Max Kernel
Max
Equidistant
Distance
Determinant
Cancer Treatment
20.57
21.05
20.70 = 0.06
20.68 + 0.05
21.03 + 1.60
Glucose in Blood
15.23
15.30
15.23 0.01
15.24 0.01
15.23 0.01
Pendulum
20.16
20.59
20.20 = 0.02
20.29 0.03
Mountain Car
34.63
35.04
34.63 + 0.01
34.63 +.01
34.64 0.01
Cart Pole
17.49
19.96
17.52 0.04
17.53
17.63 0.05
Bicycle
9.45
10.24
9.53 + 0.02
9.03
9.67 = 0.05
Furuta Pendulum
23.31
25.11
23.64 + 0.22
23.52
314.77 + 411.01
Quadrotor 2D
3.54
4.01
3.54 0.01
3.54
3.57 = 0.01
0.01
Quadrotor 3D
7.38
7.84
7.51 + 0.21
7.54
9.41 + 1.43
Does the adaptive MSS perform better than equidistant MSS? We compare the adaptive and equidistant MSSs on all systems and observe that the adaptive MSSs consistently perform better than the equidistant MSS. To better illustrate the difference between the adaptive and equidistant MSSs we study a a- 2-dimensional Pendulum system (c.f., Figure 2). First, we see that if we use adaptive MSSs we consistently achieve lower per-episode costs during the training. Second, we observe that while equidistant MSS spreads observations equidistantly in time and collects lots of measurements with almost the same state-action input the dynamical system, the adaptive MSS spreads the measurements have diverse state-action input pairs for the dynamical system on the executed trajectory and collects higher quality data.
x10
x10
102
MAX KERNEL DISTANCE
oz(x(t)) sf(x(!))
Equidi MSS
MAX DETERMINANT
Adaptive MSS
2.5
101
Best continuous
EQUIDISTANE
Variance
Σ ox(tt))
2.0
Cost
Best discrete
10
2
1.5
1.0
10-1
-3
24
10
15
0.0
2.5
5.0
7.5
10.0
2
Episode
Time
Figure 2: All MSSs coupled with continuous-tin timm control achieve lower cost than the optimal discrete zero-order hold control on the true model. Adaptive MSSs (Greedy Max Kernel Distance and Greedy Max Determinant) reduce the suffered cost considerably faster than equidistant MSS and are converging towards the best possible continuous time control. Whereas equidistant MSS spreads the measurements uniformly over time, the adaptive MSSs spread the data over the state-action space (dynamical system's 's input) and collect higher quality data.
Does optimism help? For the final experiment, we examine whether planning optimistically helps. In particular, we compare our planning strategy to the mean planner that is also used by Yildiz et al. (2021). The mean planner solves the optimal control problem greedily with the learned mean model fn in every episode MSSs. We observe that planning optimistically results in reducing the cost faster and achieves better performance for all MSSs.
We evaluate the performance of this model on the Pendulum system for all

---

EQUIDISTANT
MAX KERNEL DISTANCE
MAX DETERMINANT
102
GREEDY
OPTIMISM
Best cont nuous
Cost
10
10
10
15
0
5
10
15
0
5
10
15
Episode
Episode
Episode
Figure 3: Both greedy and optimistic continuous-time control strategies outperform the discrete- time zero-order hold control on the true model. However, the optimistic strategy is particularly notable as it expedites cost reduction across all MSSs.
Conclusion
We introduce a model-based continuous-time RL algorithm OCORL that uses the optimistic paradigm provably achieve sublinear regret, with respect the best possible continuous-time performance, for several MSSs if modeled with GPs. Further, we develop a practical adaptive MSS that, compared the standard equidistant MSS, drastically reduces the number of measurements per episode while retaining the regret guarantees, Finally. we showcase the benefits of continuous time compared to discrete-time modeling in several environments (c.f. Figure 1, Table 1), and demonstrate the benefits of planning with optimism compared to greedy planning.
In this work, considered the setting with deterministic dynamics where obtain noisy measure- ment of the state's 's derivative. We leave the more practical setting, where only noisy measurements of the state, instead of its derivative, available as well as stochastic, delayed differential equations, and partially observable systems to future work.
Our aim with this work is catalyze further research within the RL community continuous-timg modeling. We believe that this shift in perspective could lead to significant advancements in the field and look forward to future contributions.
Acknowledgments and Disclosure of Funding
We would like to thank Yarden As, Scott Sussex and Dominik Baumann for their feedback. This project has received funding from the Swiss National Science Foundation under NCCR Automation, grant agreement 5INF40 180545, and the Microsoft Swiss Joint Research Center.
10

---

References
Abbasi- Abbasi-Yadkori, and Szepesvári, C. (2011), Regret bounds for the adaptive control linear quadratic systems. In Proceedings of the 24th Annual Conference Learning Theory, pages 1-26. JMLR Workshop and Conference Proceedings.
Anta, A. and Tabuada, P. (2010). To sample or not to sample: Self-triggered control for nonlinear systems. IEEE Transactions on automatic control, 55(9):2030-200 2042.
Astrom, K. J. and Bernhardsson, B. M. (2002). Comparison of riemann and lebesgue sampling for first order stochastic systems. In Proceedings of the st IEEE Conference Decision and Control, 2002., volume 2, pages 2011-2016. IEEE.
Auer, P., Jaksch, T., and Ortner, R. (2008). Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21.
Basei, M., Guo, X., Hu, A., and Zhang, (2022). Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over finite-time horizon. Journal of Machine Learning Research, 23(178):1
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+Nummy programs.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv preprint 1606::6606.015600
Chartrand, R. (2 Numerical differentiation of noisy, nonsmooth data. International Scholarly Research Notices, 2011.
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. (2018). Neural ordinary differential equations. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Chowdhury, S. R. and Gopalan, A. (2017). On kernelized multi-armed bandits. In International Conference on Machine Learning, pages 844-853. PMLR.
Chowdhury, S. R. and Gopalan, A. (2019). Online learning in kernelized markov decision processes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3197-3205. PMLR.
Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018). Deep reinforcement learning in handful trials using probabilistic dynamics models. Advances in neural information processing systems, 31.
Cover, T. M. (1999). Elements of information theory. John Wiley & Sons.
Cranmer, M., Greydanus, S., Hoyer, S., Battaglia, P., Spergel, D., and Ho, S. (2020). Lagrangian neural networks. arXiv preprint arXiv:2003.04630
Cullum, (1971). Numerical differentiation and regularization. Journal on numerical analysis, 8(2):254-265.
Curi, S., Berkenkamp, F. and Krause, A. (2020). Efficient model-based reinforcement learning through optimistic policy search and planning. Advances Neural Information Processing Systems, 15:14156-14170.
Dearden, R., Friedman, N., and Andre, D. (2013). Model-based bayesian exploration. arXiv preprint 17X.v:1301.6690.
Deisenroth, M. and Rasmussen, C. E. (2011). Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML--1)), pages 465-472.

---

Doya, K. (2000), Reinforcement learning in continuous time and space. Neural computation. 12(1):219-22)
Du, J., Futoma, J., and Doshi-Velez, (2020). Model-based reinforcement learning for semi- markov decision processes with neural odes. Advances in Neural Information Processing Systems, 33:19805-19816,
Engquist, B., Li, X., Ren, W., Vanden- Eijnden, E., et al. (2007). Heterogeneous multiscale methods: review. Communications in Computational Physics, 2(3):367-455
Gäfvert, M. (2016). Modelling the furuta pendulum. Department of Automatic Control, Lund Institute of Technology (LTH)..
Ghysels, E., Santa- Clara, P., and Valkanov, R. (2006). Predicting volatility: getting the most out of return data sampled at different frequencies. Journal of Econometrics, 13((1-2):59-95
Greydanus, S., Dzamba, M., and Yosinski, (2019). Hamiltonian neural networks. Advances in neural information processing systems, 32.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha. S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018). Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.0528
Hartman, P. (2002). Ordinary differential equations. SIAM.
Heemels, W., Johansson, K. H., and Tabuada, P. (2021). Event-triggered and self-triggered control. In Encyclopedia of Systems and Control, pages 724-730. Springer.
Heemels, W. P., Johansson, K. H., and Tabuada, P. (2012). An introduction to event-triggered and self trigeree control. 2012 ieee 51st ieee conference on decision and control (cdc), pages 2270-3285. IEEE.
Holzmüller, D., Zaverkin, V., Kästner, J., and Steinwart, I. (2022). A framework and benchmark for deep batch active learning for regression. arXiv preprint (xX.xx22233091100
Howe, N., Dufort-Labbé, S., Rajkumar, N., and Bacon, P.-L. (2022). Myriad: a real-world testbed to bridge trajectory optimization and deep learning. Advances in Neural Information Processing Systems, 15:29801-298155
Jacobson, D. H. and Mayne, D. Q- (1970). Differential dynamic programming. Number 24. Elsevier Publishing Company.
Jones, D. S., Plank, M., and Sleeman, B. D. (2009). Differential equations and mathematical biology. CRC press.
Kaandorp, G. C. and Koole, G. (2007). Optimal outpatient appointment scheduling. Health care management science, 10:217-229.
Kakade, S., Krishnamurthy, A., Lowrey, K., Ohnishi, M., and Sun, W. (2020). Information theoretic regret bounds for online nonlinear control. Advances in Neural Information Processing Systems, 33:15312-15325.
Kelly, M. (2017). An introduction to trajectory optimization: How to do your own direct collocation. SIAM Review, 5944):849990.
Khalil, H. K. (2015). Nonlinear control, volume 406. Pearson New York.
Kirschner, and Krause, A. (2018). Information directed sampling and bandits with heteroscedastic noise. In Conference On Learning Theory, pages 358-384. PMLR.
Knowles, I. and Renka, R. J. (2014). Methods for numerical differentiation of noisy data. Electron. J. Differ: Equ, 21:235-2444
Knowles, I. and Wallace, R. (1995). variational method for numerical differentiation. Numerische Mathematik, 70:91-110.
12

---

Kuleshov, V., Fenner, N., and Ermon, S. (2018). Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning, pages 2796-2804, PMLR.
Lederer, A., Conejo, A. O., Maier, K. A., Xiao, W., Umlauft, J., and Hirche, S. (2021). Gaussian process-based real-time learning for safety critical applications. In International Conference Machine Learning, pages
PMLR.
Lenhart, S. and Workman, J. T. (2007). Optimal control applied to biological models. CRC press.
Li, W. and Todorov, E. (2004). Iterative linear quadratic regulator design for nonlinear biological movement systems. ICINCO (1), pages 222-229. Citeseer.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv: (ttp:=:599097777
Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T. (2018). Algorithmic framework for model- based deep reinforcement learning with theoretical guarantees. arXiv preprint
Lutter, M., Mannor, S., Peters, J., Fox, D., and Garg, A. (2021). Value iteration in continuous actions, states and time. arXiv preprint arXiv:210S.0400
Modares, H. and Lewis, F. L. (2014). Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Transactions on Automatic control, 59(11):3051-3056.
Mohammadi, H., Zare, A., Soltanolkotabi, M., and Jovanović, M. R. (2021). Convergence and sample complexity of gradient methods for the model-free linear-quadratic regulator problem. IEEE Transactions on Automatic Control, 67(5):2435-24500
Morari, M. and Lee, H. (1999). Model predictive control: past, present and future. Computers & Chemical Engineering, 236-65)667.-82.
Mordatch, Todorov, E., and Popović, Z. (2012). Discovery of complex behaviors through contact- invariant optimization. ACM Transactions on Graphics (ToG), 11(4::-8..
Nesic, D. and Postoyan, R. (2021). Nonlinear sampled-data systems. In Encyclopedia of Systems and Control, pages 1477-1483, Springer.
Nonami, K., Kendoul, F., Suzuki, S., Wang, W., and Nakazawa, D. (2010). Autonomous flying robots:
unmanned aerial vehicles and micro aerial vehicles. Springer Science & Business Media.
Osband, I., Russo, D., and Van Roy, B. (2013). (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26.
Osband, I. and Van Roy, B. (2014), Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems, 27.
Paing, H. S., Schagin, A. V., Win, K. S., and Linn, Y. H. (2020). New designing approaches for quadcopter using 2d model modelling a cascaded pid controller. In 2020 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), pages 2370- 2373. IEEE
Panetta, J. C. and Fister, K. R. (2003). Optimal control applied to competing chemotherapeutic cell-kill strategies. SIAM Journal on Applied Mathematics, 1354-1995-
Pasztor, B., Bogunovic, I., and Krause, (2021). Efficient model-based multi-agent mean-fiel reinforcement learning. arXiv preprint arXiv:2107.04050
Polack, P., Altché, F., d' Andréa Novel, B., and de La Fortelle, A. (2017). The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles? In 2017 IEEE intelligent vehicles symposium (IV), pages 812-818. IEEE.
Rothfuss, J., Sukhija, B., Birchler, T., Kassraie, P., and Krause, A. (2023). Hallucinated adversarial control for conservative offline policy evaluation. arXiv preprint arXiv:2303.01076
13

---

Russo, D. and Van Roy, B. (2014). Learning to optimize via posterior sampling, Mathematics of Operations Research, 39(4):121--1243.
Russo, D. and Van Roy, B. (2016). An nnformation-theorett analysis of thompson sampling. The Journal of Machine Learning Research, 17(1):2442-2471.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint 177.v:170.
Simchowitz, M. and Foster, D. (2020). Naive exploration is optimal for online lqr. In International Conference Machine Learning, pages 8937-8948. PMLRR.
Singh, M. and Theers, M. (2021). Kinematic bicycle model. ttttps://thomasfeeee .ee..giitin github... i Algorithms for- Automated wut.o//ttw...iiii////o//oo////// 09.
html. Accessed: 2023-
Spong, M. W., Hutchinson, S., Vidyasagar, M., al. (2006). Robot modeling and control, volume 3. Wiley New York.
Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. (2009). Gaussian process optimization the bandit setting: No regret and experimental design. arXiv preprint arXiv:09/2.3999.
Sussex, S., Makarova, A., and Krause, A. (2023). Model-based causal bayesian optimization. In International Conference on Learning Representations (ICLR).
Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
Tallec, C., Blier, L., and Ollivier, Y. (2019). Making deep q-learning methods robust to time discretization. In International Conference on Machine Learning, pages
PMLR.
Thomas, J., Welde, J., Loianno, G., Danillidis, K., and Kumar, (2017). Autonomous flight for detection, localization, and tracking of moving targets with small quadrotor. IEEE Robotics and Automation Letters, 2(3):1762-176))
Thompson, W. R. (1933). On likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-29)
Todorov, E. and Li, W. (2005). A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems. Proceedings of the 2005, American Control Conference, 2005., pages 300-306. IEEE.
Treven, L., Wenk, P., Dorfler, F., and Krause, A. (2021). Distributional gradient matching for learning uncertain neural dynamics models. Advances in Neural Information Processing Systems, 34:29780-29799
Umlauft, J. and Hirche, S. (2019). Feedback linearization based gaussian processes with event- triggered online learning. IEEE Transactions on Automatic Control, 61444)445544416
Vakili, S., Khezeli, K., and Picheny, V. (2021). On information gain and regret bounds in gaussian process bandits. In International Conference Artificial Intelligence and Statistics, pages 82-90. PMLR.
Vamvoudakis, K., Vrabie, D., and Lewis, F. (2009), Online policy iteration based algorithms to solve the continuous- infinite horizon optimal control problem. 2009 JEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, pages 36 66-41. IEEE.
Vrabie, D. and Lewis, F. (2009). Neural network approach to continuous direct adaptive optimal control for partially unknown nonlinear systems. Neural Networks, 22(3):237-246.
Vrabie, D. and Lewis, F. L. (2008). Adaptive optimal control algorithm for continuous-tin nonlinear systems based on policy iteration. In 2008 47th IEEE Conference on Decision and Control, pages 73-79. IEEE.
Wagener, N., Cheng, C.-A., Sacks, J., and Boots, B. (2019). An online learning approach to model predictive control. arXiv preprint 172.97190..08967
14

---

Wagner. J. Mazurek, P., Miękina, A., and Morawski, R. Z. (2018). Regularised differentiation of measurement data in systems for monitoring of human movements. Biomedical Signal Processing and Control, 43:265-277.
Wang, H., Zariphopoulou, T., and Zhou, X. Y. (2020). Reinforcement learning in continuous time and space: stochastic control approach. The Journal of Machine Learning Research, (11888845-8178.
Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning, volume 2.
MIT press Cambridge, MA.
Williams, G., Aldrich, A., and Theodorou, E. A. (2017). Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2):344-357.
Yildiz, C.,, Heinonen, M., and Lähdesmäki, H. (2021). Continuous-time model-based reinforcement learning. In International Conference Machine Learning, pages 12009-12018 PMLR.
15

---

Contents of Appendix
Theory
17
A.1 Bounding Regret with the Model Complexity
17
A.2 Bounding Model Complexities
18
A.3 Bounding Measurement Uncertainty of GPs
IID.............................
A.4 Useful Facts and Inequalities
22
B Solving Optimistic Optimal Control Problem
23
B.1 Application to Arbitrary Discretizations
23
Experimental Setup
26
C.I System's dynamics
................................
26
C.2 System's tasks
................................
27
C.3 Adaptive MSSs
27
C.4 Episodes Hyperparameters
28
C,5 Practical Implementation
28
D Additional Experiments on Number of Measurements
29
16

---

A Theory
We begin in Appendix A.1 by deriving the cumulative regret bound of Proposition which depends on the model complexity. In Appendix A.2, we bound the model complexities of the presented MSSs. Finally, in Appendix A.3, we bound the measurement uncertainty when GP is used as the statistical model, yielding sublinear regret guarantees for many commonly used kernels. We include useful facts and inequalities Appendix A.4.
A.1 Bounding Regret with the Model Complexity
The main goal of this section is to derive Proposition I which relates the cumulative regret Ry(S) to the model complexity In (f -). In what follows, the quantifier "with high probability" refers to "with probability at least 5 as in Definition 2".
Lemma 4. Assuming In (0) = Tn (0), the distance between the true and the hallucinated trajectory at any time t 2 0 is bounded with high probability by
(,,( zn S 23, ( ( |lon- (zx())) ds
(8)
and
[t) 2x(()]l s 23, 23,c4r((thr)) ( 1 ds.
(9)
Proof.
[t),(t) - xt,(()]| 1// fn fn-1(2n(s)) - (z( ds
/ | (2n (2)())) f*(2n()))l ds
$ S/" (2n)((2)))) (z- + Il (l ff (zz(s))) ds
$ Ly(1+ LT) S/ [Imn(()) En)lll ds + 23n L 11on-1 (zn()))
where the final inequality follows from Lemma 15 and Definition (with high probability). We obtain Equation (8) by applying Grönwall's 's inequality (Fact 14).
Equation (9) obtained analogously by expanding with f (2)( rather than fn-1(zn(s))
Lemma 5. For any MSS S, the regret of any episode is bounded with high probability by
r. (S) < 2,, Te ["" ( dt.
(10)
Proof. By definition of in OCORL we have with high probability, C(f", 2 C(fn-1, T.). Therefore, with high probability.
(S) = C(f n.) - -Cff,,t"
C(S ,T) - C(fn-1, TT)) = =1 c((())) dt.
By Lemma 15, we further have that
TS) < Lo + L.) /' || - Zx,()) dt.
17

---

By Lemma (with high probability), we further have that
Tn (S) 23,
L Lx) (
10 (zn(s)) ds dt
/ llon-1 (Zn ds dt
pT
< 23 + Lim (1+
0
23, L Te
10 (zn (t)] dt.
Now we are ready to prove Proposition 1.
Proof of Proposition 1. Let us first bound (S). By the Cauchy-Schwarr inequality,
R ( N Σ (S).
By Lemma (with high probability), we have that
N
% lo (t)) dt
(S) - N4L L 22 2 (1+ yr
Σ
32
m=1
T
L1 TT 21
(
10 n- (t))l² dt.
n=1
Taking the square root, obtain
N
RN(S) S 23x L
| (zn(t)))( de.
n=1
The result follows by noting that
Σ [ [|n- di s II((f
A.2 Bounding Model Complexities
In this section, we derive the model complexity bounds for the presented MSSs. Our bounds are based on the approximation of integrals using an upper Darboux sum.
Fact 6 (Upper Darboux approximation). Given any b, a function .R + R, and a partition P (a.... (a....,b) b) of [a, bj into & sub-intervals,
[. f(x) de
Σ A max f(x)
(11)
zEPli
1=1
where P[i] denotes the i-th sub-interval and A; denotes the length of P(i).
We are now ready bound the model complexities. Lemma' (Oracle model complexity). For any N 2 1,
IN S <
T ETT
o (zn (tm,,)))))
(12)
18

---

Proof.
IN [N : SOORA)
lon-1 (tn(1)) dr.
max TLONT
For each episode upper Darboux approximation (Fact 6) with k to obtain
N
| (zn (t))
max T1 Tioom Wa Ell
max 0StST
max
T
|o (3n (3n((n,1))11=
R.
Lemma 8 (Equidistant model complexity). For any N 2 1,
T Mm
The
(f St S
max n. ETT
| (Z (zn(tn))))1
(13)
Mpn
mn
Proof.
IN (f", SERI)
|on-1 (1)) de.
Ell
n=1
For each episode n, let Pa def (0, 24
24, ....... T) with An T be a partition of [0, 7] into m.
my
sub-intervals, each of length An. Using the upper Darboux approximations (Fact 6) of || 1.))(())11 with respect to Pr, we have
m <
A
Io (Zn (t)]1
BE 1)A..iA..]
ETT
n=1
Observe that tn, E - 1 inn). and hence, using that |Ion-1 (-)1 is Lip-schhht continuous,
Σ An Σ
(zn ( Lo
max
m.
In the remainder of this section, we study the model complexity of the adaptive MSS. In each episode n, we partition [0, 7) into equally sized sub-intervals (called "buckets") of length A,, where A, is the solution to
A, = 1
(14)
(A,
where I,, (A,) def 2,,, Lo + L As I', a monotonically increasing function of A, it is clear that a suitable A,, exists. Moreover, it follows from A, < T' that I',, E ((3)). and hence that m,, = T/A,, E O(B.). Without loss generality, we assume m,, E N.
The definition of the adaptive MSS can be interpreted as follows: In every bucket, we take two measurements. One measurement is saved and used later for updating the statistical model. The other measurement is taken at the beginning of every bucket, and used to inform OCoRL of the current state position. For the purposes of our theoretical analysis, these initial measurements are not saved and not used for the construction of the statistical model.
19

---

Lemma 9. For any episode bucket i E {1...., mah and t E [[ 1)An, iAn] where the real
and hallucinated trajectories are "synced" initially, i.e., ((i 1)An) nn ((i 1) 1) we have with high probability that
( - & I'
|0n (s)]l ds and (15)
(i-1)A,
- - I'n
(0n-1(2n(s))
(16)
(i-1)An
Proof. Using that an-1 is Lippscchitzzz continuous and applying Lemma 15,
( 1 Lo (1 L,,) [t, x,, (t)ll
The result then follows from Lemma 4, where the lower integral bounds can be tightened to (i 1) the real and hallucinated trajectories are "synced" in the beginning of the bucket.
Lemma 10. For any episode and i {l,-.., mn), if in is selected according to the adaptive MSS then with high probability,
max lE E
(
(17)
Proof. By applying Lemma 9 we have (with high probability) for every t E - 1)4 An).
1 < 0n-1 (2, + +
(i-1)A. Ion-1 (2n())) ds.
Note that |oon-1 (2n(1)) < o (2n (t :))l by the definition of tn,i, and hence,
(zn (t)]l s |In-1 (2, (fn,:))) r, || (tn((n))))
(1+
(1 + 1 1 ||on-1 (tn,:)))
1
(18)
Similarly, by applying Lemma 9, we have (with high probability) that
||on-1 (2n (1n,)))) 2 Ion-1 (2n (Zn,)n))) - rn
||an-1 (2n())) ds
(1-1)An
2 |In (tn,))) -T, ||an-1 (2n (fn,)))l
1 o (Zn (tn.)))))
(19)
Combining Equations (18) and (19), we obtain
1 + 7
| (x)()
||an-1 (Zn (2n(tn,))) = 9l (2n(tn.)))
1 - !
Corollary 11 (Adaptive model complexity). For any N 2 1.
N
IN SAAP)) <
max mi ...... W.....N Ell
9
1 onn 1 (zn (11.)))1
(20)
n=1
Proof. The result follows analogously to the proof of Lemma 8, using Lemma 10 in the last step.
20

---

A.3
Bounding Measurement Uncertainty of GPs
Our goal in this section is prove Theorem 3.
The informativeness of a set of sampling points A C about a function f ~ (0,00, k) with the noisy observations Yd fa εд be measured by the information gain (c.f., Cover (1999); Srinivas et al. (2009)),
f) def T(yA: fa) H(yA) H(yA fA);
(21)
which quantifies the reduction in entropy of fA when observing Here, we write fA = [f(x])= and сΛ N o For a Gaussian, H(N (p,, Σ)) 2 log dot so that when is a
Gaussian process, I(ya: f) = log det(I - AA) where KAA = [k( x') x'),, We denote the maximal information gain by observing n points by
Tn
()) -
(22)
Clearly, The depends on the kernel k. In Table 2, we state the magnitude of on for different kernels. We take the magnitudes from Theorem 5 of Srinivas et al. (2009) and Remark of Vakili et al. (2021).
Table 2:
Here we present different magnitudes of 7h. The magnitudes hold under the assumption that Z is compact. Here, B, is the modified Bessel function.
Kernel
k(x, x')
In
Linear
x T x''
(d (dlog(n)
1--2112
RBF
log dog++1p (n))
Matérn
20
Br
v
20
a
n zuira log (n)
12222--1
The maximum information gain can be interpreted analogously the setting where f Hr.B. see Srinivas et al. (2009) and section 5.2 of Kirschner and Krause (2018). The following result relates mutual information and epistemic uncertainty.
Lemma 12 (Lemma 5.3 from Srinivas et al. (2009)). For any c Z, GP(O,k) or f E Hr.B. and if the observation noise is i.i.d. (0; then
log 1+ Varl/( y(
T(ya: 5) =
(23)
vn,) def A.
where (xI,...,,.,,
In our setting, denote by An del (Si,..., ,N) the set of times of observations DI:N until episode N. We define for every episode n,
argmax iE(I....m...
||
(24)
We write t" del tn and Av del comprises exactly N observations.
(tt Note that Áw(al S Ay(r) (vn) and Än
Lemma 13. ff CP(O, k) for all j e (l.....,d. d,) or if Hds and if the observation noise is Lid. o0;002 then
o (2. S 27
(25)
n=l
where δ 1og(1+a
max a.jEz.. o0,5(2) and yN is with respect to kernel k.
2 maxe
max
21

---

Proof. We have
Mhn (2- S T Σ O (2,
Σ
This inequality is tight when my = 1 (Vn). When more than one measurement is obtained per episode then the regret may be smaller. Expanding the squared Euclidean norm, yields
o (2n
(Zn (1,))
Write (z) def (z) UAy. whereas (z) = Vz) JANl. The variance is mono-
tonically decreasing one conditions on more observations, nn (2) 2 (z) (Vn,j.=), and hence,
(Zn (f,)).
==1j=1
Again, this bound is tight when only one measurement is obtained per episode, but may be loose otherwise. Lemma 15 of Curi et al. (2020) shows that o (z) ā log(1 + 2 (z)) for any n. Applying this inequality, we obtain
Φ2 n 11- 1
(Zn (ta))
T
log
j=1 n=1
By Lemma 12,
25'
j=1
s 20Tdz7N-
Proof of Theorem 3. We derive the regret bound for the adaptive MSS. The regret bounds for the other MSSs follow analogously.
From Corollary 1,, recall the model complexity bound
T ma
IN(f", SADP)
9
|o
(2n (I,,)))))
mn
EI
t=1
By Lemma 13, we can further bound this by
< 180T'....
Combining this bound with Proposition 1, with high probability the regret is bounded by RN @ VNyN)
A.4 Useful Facts and Inequalities
Fact 14 (Grönwall's inequality, theorem 1.1 in chapter of Hartman (2002)). Let g(t) be a non- negative continuous function the interval (a, b) C IR, and let C, K be pair of non-negative constants. If the function g satisfies
o(1) S SCCK L g(s)ds, Vi ( b).
(26)
then we have
s(t) < ((t-a)))
Vt E [a,
(27)
22

---

Lemma 15. For any two metric spaces (X, dx) and (V.dy), and any two Lipschitz continuous
functions f : N x - and g X X with Lipschitz constants Lg and Lg respectively, we have that (()) is L--Lipschitz continuous with respect to dy with I. def 11((1 + Lg).
Proof. Fix any a, R. By the triangle inequality, f(a)) g(a')))
& dy(f(z', g(x))()()))))))))))) f(x, + dylf(a', (())) f(2',g(x())) S Lfdx(2 (xxx((()))((())
S L + L1Ladd((0,
L1 + Lg) (g)
B Solving Optimistic Optimal Control Problem
The optimal control problem solved by OCORL. is constrained to L.p-siiccitzz continuous dynamics. When the dynamics are modeled with a GP using a linear kernel, the control problem can easily be solved directly in weight-space. Beyond this special case, it is commonly assumed that there exists an oracle which solves this optimization problem exactly (see, e.g., assumption of Kakade al. (2020)).
In this work, do not explicitly address the problem of approximating the optimal control problem reasonably well. Kakade et al. (2020) give brief overview gradient- and sampling- based methods which may be useful (Jacobson and Mayne, 1970; Todorov and Li, 2005; Mordatch et al., 2012: Williams et al., 2017; Wagener et al., 2019). They alternatively suggest to use Thompson sampling (Thompson, 1933; Oshand and Van Roy, 2014), that is, sample fa-1 from Mn- (which is straightforward when the dynamics are modeled by a GP, see Williams and Rasmussen (2006)) and then compute and execute the optimal policy nr argmin C(T,, f1- using planning oracle. Kakade et al. (2020) conjecture that corresponding regret bounds can be derived using standard techniques for analyzing Bayesian regret of Thomposon sampling (Russo and Van Roy, 2014, 2016).
B.1 Application Arbitrary Discretizations
In this section, we briefly show that our guarantees carry over to discretized dynamics and costs for arbitrury discretizations. Importantly, in the discrete-time setting the lemma mirroring Lemma does not require the assumption that the models fn are Ly Lipsschhit continuous, hence, simplifying the optimistic optimal control problem solved by OCoRL. In the discrete-time setting, it is sufficient assume that f is Lipischhit continuous.
Consider the dynamical system
#h++ 1-"h0"11 h2k,,, Tk), where
Zk (æx,, (xx))
with initial state Dy E where Tk E Ry0 measures the "time" between control points Ty and #++1- We make the assumption that h" is LnpLiicciitz continuous in all arguments. The (discrete-time) control problem is
T Σ c(z))
π def
argmin C( h²) argmin wEll
(28)
#Ell
In many applications, the cost to be minimized is defined at discrete control points to begin with. In this case, the continuous-time control of Equation (1) can be approximated arbitrarily well with Equation (28) by choosing a sufficiently dense discretization. Commonly, the discretization is taken to be equidistant, i.e., тı = (Vk), but we remark that our results hold more generally for any discretization.
23

---

B.1.1 Bounding Regret with the Model Complexity
In the following, we denote by Ln.k the k-th state visited during episode n.
Lemma 16. Assuming tn,0 = Zn,0. the distance between the true and the hallucinated trajectory at any iteration k 2 0 is bounded with high probability by
R-1
|in,k - Lnnkl 1 2B, + I/ + 2,, 1-111-11
Σ 0n-1 (zm,)ll
(29)
1=0
where we write Ln del Ln(1 + L and L' dif Lo (1 + Lim)
Proof (based on Lemma 4 of Curi et al. (2020)). We begin by showing by induction that for any k 220 O,
Innk Fn,Aul S 23,, Σ (L + 23n n- (zn,))l
(30)
The base case is implied trivially. For the induction step, assume that Equation (30) holds at iteration k. We have
[n,,+1 - 1n,R+11
||
Pn- (Zn,ks rk) (z(,n, Tk)
h kn Tk) h h'(,n, Tk)
| (2 Tk) - h- (2 Tx)ll + (2n, Tk) h- (Zn,ks Th) I 23, |lon-1 (2n, + LA (1 Lx) || T..A
where the final inequality follows from Definition 2 (with high probability) and Lemma 15.
= 23, |lon-1 (Zn,k) + 0n (2n,s) on-1 (zn,x)l
+ +La(1 + Lm) | &n.k
Using that an-1 is L. Lipschitz continuous coutiuuuu and applying Lemma 15,
s 2,3 On (zn,k)) + Lo L || Fn,
+ Ln + Lm) 1n, kn.kll
23, |lon-1 (Zn,k)ll + (L + Lx) 2,, La (1 Lx)) ||
By the induction hypothesis,
h Σ (Lh 1 (Zn.))l
23,
1=0
Thus, Equation (30) holds. Since h < T, we have
k-1
|| - 2n,ull S 28,, Σ
(L, 23n (1-1 100-11inn11111 (2n,))l
1=0
k-1
< 2pn Σ
(1 + L/ + 2B (- 110 (zn.))l
1=0
28, LA 28,
|I (zn,))l
Lemma 17. For any MSS S, the regret of any episode n. is bounded with high probability by
(S) < 23,T(11 + L4, + +
2 1 L
o (zn,k)
(31)
k=1
where L' def Lc(1 + LT).
24

---

Proof. By definition of n, in OCoRL we have with high probability, C(f", 2 C(fn-1, Therefore, with high probability,
(5) = C(f" na) - C(f", x²) (f Taa) - C(fn-1: TTa) c(Zn,k) c(2n,k).
By Lemma 15, we further have that
(S) S 1o Σ Donk Bnn Il
By Lemma 16 (with high probability), we further have that
T k-1
(n( & 23, + 1A + 23,
0n-1
(zn.1)l
k= 1=0
211 + T/ + 22
T%
The above results allow us to prove general regret bound which is analogous to Proposition 1. Lemma 18. Let S be any MSS. If high probability,
we we run OCoRL with arbitrary discretization h", we have with
RN(S) 23 I22 + +23 LL, NIN(h",
(32)
where we define the discrete-time model complexity,
N
(h S) def
max M
o (2n,
(33)
1==1
Proof. Let us first bound (S). By the Cauchy-Schwarz
N R2((S) S N Σ (S).
n=1
By Lemma (with high probability), we have that
T ((T-11
RS(() S N4T Σ B L/2 28
| (zn
N
S T (1 14 + 2
2(
12
O (zn,
10
1=1
Taking the square root, obtain
Rx(S) S 28 (1+ L/ + 23 L'oy
N
o (21.
n=
1=1
The result follows by noting that
T
o (z1, IIn("",
25

---

B.1.2 Bounding Model Complexities
The model complexity bounds from the continuous-time setting (c.f., Appendix A.2) extend seam- lessly the discrete-time setting. the following we briefly sketch the bound for the oracle MSS.
Lemma (Discrete-ting) oracle model complexity). For any N 2 I,
IN SORA)) 2
T
Σ
no-11 (Zn(4n.
(34)
ell
Proof.
T
IN (h', , SORA)
nn (2n,))]
max
* M ma Ell
n= k=1
| (2n,
max
ke (1 (I...T))
ell
max
T
nn- (Z (z)(tn,)))1
T TN
ETI n=1
Note that the MSS S can (but not has to) be restricted to the discretization of [0, T'I used for the dynamics costs. The derivations for the equidistant and adaptive MSSs follow analogously, where we note that the bucket length A,, is to be defined with respect to modified constant I...
Finally, applying the measurement uncertainty bound presented in Appendix A.3, we obtain the following theorem.
Theorem 20. Fix an arbitrary discretization h" and assume that h E H the observation noise i.i.d. (O; o² o² I), and let |I-Il be the Euclidean norm. We model h" with the GP model. The regret for the adaptive MSS is with probability at least 8 bounded by
(SAP)) So (3xT Ln 23x 11) "NNN)
my m
((3))
C Experimental Setup
C.1 System's dynamics
Here we either give reference to equations of the dynamical system or provide their equations:
Cancer Treatment:
1 [ )u(t)x(()
a(t) = rx(() log
x((()
=x0) 0.975, 0.3, 8 0.45
Glucose in Blood:
(n(() ax (1)
(1) = - -cx (1) + u(1),
a 1,b 1,c 1, (0) (0.75, 0)
Penndulum:
2o(t) z (t) (1 0
9.81, = 5.0
26

---

Mountain Car:
(o(() 10rr i (t) = 3.5u(t) 2.5 cos (3zo
Cart Pole: Equation 6.1 and 6.2 of Kelly (2017)
Furuta Pendulum: Equation 18 of Gäfvert (2016)
Bicycle: Dynamics from (Singh and Theers, 2021)
Quadrotor 2D: Equations 1, 2, of Paing et al. (2020)
Quadrotor 3D: Equations 12, 13, 14 Thomas et al. (2017)
C.2 System's tasks
In all considered settings the running cost has the following quadratic form:
c(z, u) = (x - xT) Q( - xT) + (u - wT) "((u-r)) ur).
Here, æy is the target state, and uy is the target control. Descriptively, in cancer treatment and glucose in Blood systems, our objective is to reduce the prevalence of unhealthy cells, while striving be as minimally invasive possible. For the pendulum, Furuta pendulum, and cart pole systems our goal to skillfully swing the pendulum upwards from its lowest point. In the mountain car system, the challenge is guide the car from the valley's depth to the hill's summit. Finally, in systems bicycle, quadrotor 2D, and quadrotor 3D our objective is to adeptly navigate the agent to reach a desired target state.
Table 3: Cost specifications for considered systems.
System
Running Q
Running R
Target ØT
Target uy
Cancer Treatment
1
I
[0.54]
0,
Glucose in Blood
I
I
0..47, 0.33]
0
Pendulum
I2
I
0.
0
Mountain Car
I
Ij
[
0
Cart Pole
I4
I1
0
Furuta Pendulum
Ia
0,
0
Bicycle
I
Ig
[2, 6.0
Og
Quadrotor 2D
Is
10 12
0,
[g m.
[0, 0, 0,0,
Quadrotor 3D
diag((5, 0.8,
00.15, 0, 0
0.1,0.1,1,011,00111
0, 0, 0,0,1,0,0,0)
C.3
Adaptive MSSs
In this section, we describe in detail the two adaptive MSSs from Section
Greedy Max Determinant the Greedy Max Determinant adaptive strategy we solve the following objective:
max det (K SC
(35)
Here, Ky def
(t,, 2, (t))),, and (2, def k(z, k (z) (K,, +021)= kn (z').
The goal here is to find at every episode n a set S, time points from [0, T). Solving problem (35) is NP- uarhard and approximate it by solving it greedily:
Algorithm GREEDY MAX DETERMINANT
Init: S, (argmax.e(o.7)
kn (Z,, (()))
for = 2. 2...., M do
S, S, U argmaa det ( (KS.u(e)) } tE(o,T]
27

---

Greedy Max Kernel Distance Max Kernel Distance. Here we greedily solve the metric M-center problem in space [0, T) with
The second adaptive MSS we consider in Section is the Greedy
the kemnel metric: dl,(t,)) t')2 del hallucinated trajectory.
(l,, (1) kn (2. (1'), (()), 2h,, (2. Zn (t))) the
Algorithm 2 GREEDY MAX KERNEL DISTANCE
Init: Sn = (argmax4e(oor) kin (1)(),, (t)))
for 2. 2........ .M do
Sn = Sn U
argmax min du(l,()) [E(0,7] d'ES.
C.4 Episodes Hy pyperaarameters
Table 4: We take few tens of measurements per episode in each environment. We run the
experiments for at most 40 episodes in each environment.
Number of episodes N
Number of Measurements per episodes M
Time horizon T
System
Cancer Treatment
20
10
20
Glucose in Blood
20
10
0.45
Pendulum
20
10
10
Mountain Car
40
10
1
Cart Pole
40
10
10
Furuta Pendulum
40
10
10
Bicycle
20
10
10
Quadrotor 2D
20
10
8
Quadrotor 3D
25
20
15
C.5 Practical Implementation
In practice, we approach solving the optimal control problem (2) with offline planning and online tracking strategy. Before the episode starts obtain open loop trajectory क. (t), ū, by offline planning:
/ c(x((), u(t)) dt
(t),), at (t) = argmin min
=()) v(e)
(36)
+(t) ph, (t) u(t) + o, (()))((t) (tt) E 11 VI E [0
We solve the optimization problem (36) with Iterative Linear Quadratic Regulator (ILQR) (Li and Todorov, 2004). Then we track the open loop trajectory H, (t) online with MPC:
TTMMP
(l=(()) x(1)l2 + [|u(t) a(()l)) de
(()) argmin
(xt)),a(t))
(37)
s.t. a(t) μh, (z(()) ut))
Here, horizon TMPC depends the system.
28

---

Table The MPC horizon for considered systems.
System
ТMPC
Cancer Treatment
5.0
Glucose in Blood
0.2
Pendulum
6.0
Mountain Car
1.0
Cart Pole
5.0
Furuta Pendulum
5.0
Bicycle
5.0
Quadrotor 2D
3.0
Quadrotor 3D
5.0
We solve MPC tracking problem (37) also with the ILQR.
D
Additional Experiments on Number of Measurements
EQUIDISTANT
MAX KERNEL DISTANCE
MAX DETERMINANT
102
10
Cost
10
10
0
10
10
10
Episode
Episode
Episode
M=
M
M
8
M
10
Best continuous .... Best discrete (M=10)
(a) We vary number of measurements we take per episode for the Pendulum system. As expected the more observation results in lower achieved cost for every MSS.
-
M -
M
- 10
103
Cost
10a
10
0
10
10
10
10
Episode
Episode
Episode
Episode
EQUIDISTANT
MAX KERNEL DISTANCE
MAX WAX_DETERR DETERNMNNT
Best continuous
....
Best discrete (M:10)
(b) For the number of measurments compare different MSSs. We that equidistant MSS repeatedly suffers higher cost compare the adaptive Kernel Distance Max Deteriminant MSS.
29


Reviews:
[{"review_id": "bfT3cRE43H", "reviewer": "Reviewer_ffit", "summary": "This paper introduces a novel algorithm for efficient exploration in continuous-time model-based reinforcement learning. The algorithm represents continuous-time dynamics using nonlinear ODEs and captures epistemic uncertainty using probabilistic models. The analysis shows that the approach achieves sublinear regret with significantly fewer samples, making it a promising solution for various applications. The paper also develops a practical adaptive measurement selection strategy that reduces the number of measurements per episode while retaining regret guarantees. The benefits of continuous-time modeling and planning with optimism are demonstrated in several environments. The authors aim to catalyze further exploration within the RL community regarding the potential of modeling dynamics in a continuous-time framework.", "questions": "In which scenarios are measurements considered costly? Have you discussed in the Introduction whether there are enough scenes that truly require continuous-time systems due to costly measurements?\n\nIn the problem setting, the computation of cumulative regret relies on an optimal policy. However, if the optimal policy is unattainable for a particular problem, can your method still be applied? Please provide an explanation.\n\nRegarding Line 122, is it possible for re-calibration techniques to provide accurate uncertainty estimation while maintaining a low time complexity?\n", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper appears to be one of the pioneering works in the field of continuous-time reinforcement learning (RL) applied to nonlinear dynamical systems.\n\nFurthermore, the utilization of epistemic uncertainty for measurement selection, a technique commonly employed in active learning but rarely explored in RL, adds a unique and valuable dimension to this study.\n\nIn addition to these notable contributions, the paper introduces several innovative techniques and investigates their efficacy across various experimental environments. \n", "weaknesses": "The paper encompasses a wide range of techniques, including model-based RL, continuous-time RL, and aperiodic strategies. However, the abundance of new content may make it challenging to write and comprehend the relationship between each technique and the overall idea. To enhance clarity and understanding, it would be beneficial to reorganize the paper and provide clearer connections between the different techniques employed.\n\nRegarding Table 1, it appears that, under a known true model, discrete-time control outperforms continuous-time OC without the inclusion of MSS. Consequently, it may not be fair to directly compare continuous-time OC with MSS and discrete-time control since the conditions and components differ significantly between these approaches. A more appropriate comparison could be made between continuous-time OC and discrete-time control without MSS.\n\nTo provide a more comprehensive evaluation, it is advisable to compare this method with a broader range of other proposed RL methods. The current results may not fully demonstrate the performance of OCoRL, and including additional comparisons would enhance our understanding of its capabilities.\n\nIn Figure 3, it would be beneficial to consider incorporating more environments for experimentation. Relying solely on a single environment may not provide enough evidence to draw robust conclusions. Including a broader set of environments would strengthen the validity and generalizability of the findings.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, thank you a lot for your positive and valuable feedback! We will indeed incorporate the proposed ideas in the updated version of the paper. \n\n## Weaknesses\n\n1. *Enhance Clarity:*\nWe thank the reviewer for the feedback. We have added the following summary of our method at the end of section 3:\n*OCORL consists of two key and orthogonal components; (i) optimistic policy selection and (ii) measurement selection strategies (MSSs). In optimistic policy selection, we optimistically, w.r.t. plausible dynamics, plan a trajectory and rollout the resulting policy. We study different MSSs, such as the typical equidistant MSS, for data collection within the framework of continuous time modeling. Furthermore, we propose an adaptive MSS that measures data where we have the highest uncertainty on the planned (hallucinated) trajectory. We show that OCORL suffers no regret for the equidistant, adaptive, and oracle MSSs.*\n\n2. *Comparisons in Table 1:*\nNote that the numbers in the table represent the cost, so the smaller the better. Hence continuous time OC under the true model is the best (lowest) value any controller can achieve (up to numerical calculations). Since in the discrete-time control setting, we are only allowed to change the controller at equidistant times, the cost (computed on the realized continuous trajectory) is strictly larger (worse). \n\n3. *Comparison to other methods:*\nWe could also compare it to any other discrete-time algorithm SAC, PPO, etc., but since our algorithm controls the systems continuously, it outperforms even the best possible discrete-time control problem (i.e., the case when we know the true dynamics and can solve the optimal control problem with any solver) we decided to show the comparison only to the best possible control in discrete time (with given discretization). \n\n4. *Further experiments on optimism vs greedy exploration*\nThe comparison between greedy and optimistic planning has been done in several related works, e.g. in [1], where they show that optimistic planning performs better than greedy planning, especially in environments with scarce rewards or large action penalties. Since this is a well-studied problem and not within the focus of our paper we decided not to add any additional environments for the demonstration due to space restrictions. If the reviewer still insists on other environments we can add them to the final version. \n\n\n ## Questions:\n1. *Costly Measurements:*\nThere is plenty of scenarios when taking measurements is costly, in the introduction we consider a case when a patient is coming to the doctor for medical checks. We don't want him to come too often (every appointment at the doctor is very costly), but only when it is necessary (the uncertainty about the development of the disease is large). Another instance can be found in wireless control systems, where there are constraints on energy, computation, and communication capacity. In such systems, communication should take place only when there's pertinent information to share [2].\n\t\n2. *Unattainable optimal policy:*\nNot sure that we understand the second question exactly, please let us know if the following is what you meant: It can happen that the solution to the optimal control problem is not attainable (in a sense of inf, but the minimum is not attained). For the analysis, we assume the minimum is attained. Just for running the algorithm, we don't need that assumption. \n\n3. *Recalibration techniques:*\nWhen Modelling dynamics with GPs we don't need to recalibrate the model since GPs provide statistically sound confidence sets by design (with the right value of $\\beta_n$). For the recalibration of an Ensemble of Neural Networks, we need to solve $d_x$ (output of a neural network) optimization problem over a scalar variable, which can be solved very fast in practice, in our experiments recalibration usually takes only around 10ms.  \n\n## References\n\n[1] Curi, S., Berkenkamp, F., and Krause, A. (2020). Efficient model-based reinforcement learning through optimistic policy search and planning. Advances in Neural Information Processing Systems, 33:14156–14170.\n\n[2] Anta, Adolfo, and Paulo Tabuada. \"To sample or not to sample: Self-triggered control for nonlinear systems.\" IEEE Transactions on automatic control 55.9 (2010): 2030-2042.\n"}, {"review_id": "xLYasQ7Pm5", "reviewer": "Reviewer_7bYe", "summary": "This paper proposes a continuous-time model-based reinforcement learning method for controlling fully observed dynamical system environments where there is a cost to take a sample of the state. A Gaussian process dynamics model is used, and a novel adaptive measurement selection strategy is proposed to determine when to take a sample of the state, such that the overall optimization converges to the optimal policy in the limit of infinite trials. The proposed method (OCoRL) is theoretically analyzed to show a general regret bound holds for any measurement selection strategy and is empirically verified across a range of dynamical system environments.", "questions": "* Could this approach be generalized to other environments that are not implicitly defined by an ODE, such as other types of differential equations, e.g., delayed differential equations, stochastic differential equations, etc? Furthermore, can this approach work for partially observable environments?\n* What is the trade-off of varying the number of measurements taken, i.e., an ablation of varying M for all experiments? Does the algorithm still hold under these settings? Can this be demonstrated empirically?\n* Could the OCoRL method be benchmarked against the closest related works of Yildiz et al. (2021) and Du et al. (2020)?\n* (From above): Can OCoRL be adapted to skip taking a sample in some $\\Delta_n$ of time, i.e., where they are not needed or informative?\n* Typo: Line 339: \"Figure 1\" -> \"Figure 2\".", "limitations": "The limitations are addressed with the assumptions outlined in Section 2.1.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method OCoRL and associated general regrets bound that holds for any measurement selection strategy appear novel.\n* The paper is well-written and clearly laid out.\n* The no-regret algorithm for nonlinear dynamical systems in the continuous-time RL setting seems widely applicable and relevant to the RL and ML community.\n* The code is reproducible and easily extendable, being well documented throughout.", "weaknesses": "* How does the proposed method perform when the state differential is not observed $\\dot{x}_n(t)$ and has to be inferred? Could you perform an ablation of this to show that the proposed method is still practical? \n* Why is the noise only added to the observed state derivatives $\\dot{x}_n(t)$? Perhaps it could be more realistic to consider noise added to both the observed state $x_n(t)$ and the observed state derivative $\\dot{x}_n(t)$?\n* Line 118: \"predicted mean and epistemic uncertainty\". How does the model guarantee that you only measure the \"epistemic uncertainty\" and not the \"epistemic uncertainty and aleatoric uncertainty\"? This was not clear, and I presume the model learns both. If so, how can you split the uncertainty to only use the epistemic uncertainty, as outlined in the method?\n* The adaptive MSS assumes that $m_n=\\left \\lceil{T/ \\Delta_n}\\right \\rceil$, i.e., that a sample must be taken in each uniform interval $\\Delta_n$ of time. This seems overly restrictive. Can OCoRL be adapted to skip taking a sample in some $\\Delta_n$ of time, i.e., where they are not needed or informative?\n  * On this, can a further ablation be performed where $m_n$ is varied across all the environments and baselines to empirically verify the adaptive MSS claims for wide ranges of $m_n$?\n* Unsubstantiated claim of \"We compare the adaptive and equidistant MSSs on all systems and observe that the adaptive MSSs consistently perform better than the equidistant MSS\". Table 1, shows that adaptive MSSs can perform better than equidistant MSS only in certain environments and that equidistant MSS achieves the same performance (final cost) within error bounds for the environments of Cancer Treatment, Pendulum, and Mountain Car.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the positive and valuable feedback!\n## Weaknesses\n1. *Case when state derivatives $\\\\dot{x}(t)$ are not observed:*\nWe thank the reviewer for this very interesting question. When the state derivatives are not observed one can apply several techniques to obtain them (e.g., using finite differences, interpolation methods, etc, [1-3]). We consider the incorporation of the derivative estimation inside the RL loop as an interesting avenue for future work. \n2. *Noise also in the state $x(t)$:*\nThe noise can be added to the states $x(t)$ as well (only as a measurement noise though). However, in the analysis, we would then need to transfer the noise from the inputs to the outputs via Taylor approximation and also update the $\\sigma$ of the sub-Gaussian noise assumption. For simplicity and disposition flow we assumed noise only on the outputs of learned function, i.e. $\\dot{x}(t)$.\n3. *Measuring uncertainty:*\nYes, the model learns both, aleatoric and epistemic uncertainty. We split noise into aleatoric and epistemic parts and use only epistemic part for planning. In the GP case, we learn homoscedastic aleatoric uncertainty by optimizing noise variance $\\sigma^2_{ale}(x) = \\sigma^2$ in the negative log-likelihood term: $\\frac{1}{2}\\dot{\\mathbf{y}}^\\top (\\mathbf{K} + \\sigma^2I)^{-1}\\dot{\\mathbf{y}} + \\frac{1}{2}\\text{logdet}(\\mathbf{K} + \\sigma^2I)$, the epistemic uncertainty (epistemic variance) we obtain from the formula: $\\sigma^2_{epi}(x) = k(x, x) - k(x, \\mathbf{X})(\\mathbf{K} + \\sigma^2I)^{-1}k(\\mathbf{X}, x) $. Here we denoted by $\\mathbf{K}$ the covariance matrix built from the observations, and $k(x, \\mathbf{X})$ is a vector of kernel evaluations between point $x$ and all observations. When we use deep ensembles for modeling dynamics, every member of the ensemble learns to predict mean and aleatoric (heteroscedastic) uncertainty (variance). To obtain (approximate) epistemic standard deviation, we take, as is commonly done, a standard deviation of the predicted means of the ensemble.\n4. *Skipping measurement in* $\\Delta_n$ *interval time*:\nWith the assumptions of the paper, we were not able to get rid of sampling in every interval $\\Delta_n$ of time. However, if we for example assume some kind of stability (so that the trajectory doesn't have the option to deviate exponentially fast in time) we can alleviate this dependence. The second possible approach is to consider event-triggered sampling, where we would sample (take a measurement) only when the epistemic uncertainty at the true state would surpass a certain value. Again, for this kind of approach, we would need another assumption, namely to be able to continuously monitor the system (which one could argue can be done with hardware). Regarding changing the $m_n$ in our experiments: If we let $m_n$ be large, the difference between all MSS would become negligible. The main difference between the different MSSs arises when the number of collected data samples is small. We chose $m_n$ small enough so that the difference is visible. However, upon the reviewer's request, we performed an ablation study for different values of $m_n$ on the pendulum environment. We have attached a figure in the Author rebuttal section with our results. It is visible from the figure that for small values of $m_n$ there is a significant difference between the MSSs, while for larger MSSs this difference vanishes.  \n5. *Unsubstantiated claim based on Table 1:*\nThanks for spotting that, we have corrected the claim in the revised version. From the table, it is only possible to claim that adaptive MSS outperforms equidistant MSS on the Cancer Treatment, Pendulum, Bicycle, Furuta Pendulum, Quadrotor 2D, and Quadrotor 3D environments while performing on par (within the overlapping confidence sets) on Glucose in Blood, Mountain Car, and Cart Pole environments.\n## Questions\n1. *Generalization to other environments:*\nIn this work, we only consider systems driven by an ODE. Other types of systems like delayed differential equations, stochastic differential equations, and partially observable environments are still an open problem and we leave them as exciting future work.\n2. *Varying number of taken measurements M:*\nWith a small M (small number of observations) adaptive MSS performs much better compared to equidistant MSS. When the number of measurements grows the adaptive and equidistant MSS should result in similar performance (on our experiments) since we collect enough data in all regions (important and unimportant). We refer the reviewer to the newly added figure above for an ablation study.\n3. *Comparison to Yildiz et al. (2021) and Du et al. (2020):*\nThe setting of Yildiz et al. (2021) and Du et al. (2020) is a bit different since they don't consider access to noisy derivatives as we do. To compare with the work of Yildiz et al. (2021) we run experiments with a mean (greedy) planner (this is the planner used by Yildiz et al (2021)) instead of our optimistic one. As we can see in Figure 3 in the paper the optimistic planner is faster at finding the optimal policy.\n4. *Skipping observations:*\nIn practice (experimentally), we can indeed skip taking the measurements in some $\\Delta_n$ interval of time, if the uncertainty there is not high (in hallucination), however with this we lose theoretical guarantees.\n5. *Typo:*\nThanks for spotting that, we were indeed referencing Figure 1. We updated the typo in the updated version of the paper\n## References\n[1] Treven, L., Wenk, P., Dorfler, F., and Krause, A. (2021). Distributional gradient matching for learning uncertain neural dynamics models. Advances in Neural Information Processing Systems.\n\n[2] Chartrand, R. (2011). Numerical differentiation of noisy, nonsmooth data. International Scholarly Research Notices, 2011.\n\n[3] Knowles, I. and Renka, R. J. (2014). Methods for numerical differentiation of noisy data. Electron. J. Differ. Equ, 21:235–246.\n"}, {"review_id": "Sc2voBDFsL", "reviewer": "Reviewer_xqxW", "summary": "This paper proposes a continuous time framework for model based reinforcement learning. Their algorithm OCoRL solves the optimal control problem eq (1) by: 1, selecting optimistic policy. 2, rollout to collect data. 3, update model estimation and statistics. Specifically, they study the measurement selection strategy (sampling state-action in the continuous time framework), and show both theoretically and empirically the effect of different measurement selection strategy.\n", "questions": "See weakness", "limitations": "See weakness", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper models the optimal control or reinforcement learning problem in continuous time (eq (1)), which is elegant and also convenient for algorithm design and analysis. The realization of state-action at discrete time steps is naturally modeled as discrete-time sampling from the continuous time trajectory (the measurement selection strategy (MSS)). The proposed algorithm alternates between: optimistic planning given current statistical model and data collection for a more accurate model. They further proposed the adaptive MSS that samples state-action time points based variance of model estimate in the state-action space (rollout trajectory).\n\nBoth theoretical and empirical analysis are provided and showcase the impact of different discrete sampling strategies.\n", "weaknesses": "The algorithm enjoys elegance and nice theoretical properties, but it can be difficult to realize in practice. Especially, the first step in OCoRL that solves optimistic policy can be very time-consuming in practice. The authors mentioned in Appendix C that it is approximately solved by Iterative Linear Quadratic Regulator, but still it can be impractical.\n\nThe experiments mainly focus on investigating the impact of measurement selection strategies. A comparison in terms of performance: time/computation complexity and over all cost with existing approach (PPO, etc) would better help readers evaluate this approach", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and valuable feedback!\n\n# Weaknesses and Questions\n\n1. *A comparison in terms of performance: time/computation complexity and over all cost with existing approach (PPO, etc) would better help readers evaluate this approach*:\n As you correctly notice the first step of OCORL, namely solving optimization problem (2) from the paper is challenging. We tried to solve optimzation problem (2) using black box optimal control optimizers. In particular, we used ILQR coupled with MPC. We solved the continuous-time optimal control problem (2) by discretizing it with a small time step such that the change in the achieved cost was negligible with coarser discretization. \nIn Table 1, we compare our work with the ground truth continuous-time dynamics and its discretization. The baselines suggested by the reviewer (PPO etc.) operate in discrete time. Since we already compare our algorithm to the ground-truth discrete-time system, we did not include other discrete-time baselines. We would also like to highlight that the number of collected transitions is only in the range of [200, 500], this is a considerable gain in sample efficiency compared to model-free methods such as SAC or PPO.\n"}, {"review_id": "mxgkciIkea", "reviewer": "Reviewer_knGi", "summary": "The submission considers online RL, while the true dynamics is continuous-time. To deal with this issue, the authors provide a continuous-time model-based method. The proposed method is special by \n1. iteratively fitting ode-based models and deriving/learning the control from the fitted models \n2. having the option to adaptively selecting the sampling timing. \n\nThe proposed method is able to well handle problems with continuous-time true dynamics, with both theoretical guarantees and empirical evidence.", "questions": "1. In the experiments, are the cost function collected in a discrete-time manner or continuous-time manner? More specifically, are the results summation of costs or the integral of costs over time? \n\n2. What difference might be, if we replace the ODE model by a discrete-time model, for example, like iLQR. Please demonstrate the differences both theoretically and empirically. \n\n3. In experiments why does the proposed method with equidistant perform better than discrete zero-order hold OC? Are the two policy implemented using the same time interval on the same grid? Or does the proposed method calculates the integral over time of costs? Or does it use a different time interval than the competing discrete-time policy? What is the definition of the discrete zero-order hold OC?  \n\n4. Is it possible to implement MSS with a discrete-time assumed model? For example, I can assume that the true model is a discrete-time model which is just same as the Euler discretized version of the ODE in the submission, and then conduct adaptive MSS like in the submission. What is the problem of this naive ablation?\n\n5. Would it be possible to explicitly list the dynamics of the experiments? The cost function is provided in the supplements but not the dynamics. I am concerned that the considered experiments are too simple like all linear ODEs, which may make the method not general enough.   \n\nAs a summary, the submission definitely has novelty and contributions. However, I feel that the current method is not demonstrated clear enough, especially on what role the continuous-time modeling is playing in the method.", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The submission is novel. The proposed method incorporates continuous-time models with adaptive sampling, which to the best of my knowledge is the first. \n\nThe submission provides thorough results with both theoretical guarantees and also empirical experiments. \n\nThe studied problem is well motivated: there are problems where the data is generated in continuous-time in need of specialized methodology like the submission. \n\nThe writing is clear and easy to follow.", "weaknesses": "The submission is solid, and the adaptive MSS seems to be an important and useful technique. However, there is one concern bugging me, which seems a little crucial: it is not clear whether the continuous modeling is indeed needed. \n\nMy question is twofold: 1. whether it is necessary or practical to assume a continuous-time ground-truth dynamic; 2. whether it is necessary or helpful to use continuous-time models in the method for $f$ \n\nFirst of all, the continuous dynamics in section 2 and equation (1) may never be empirically achieved or used. There are very rare cases where one can really implement a policy in a continuous-manner as that equation. As a result, the policy functions considered will often time be piece-wise constant, which does not fit (1). It is not clear how this may affect the theory. When evaluating a policy, equation (1) may also need to be discretized. Therefore, in each episode, one can also fit a discrete-time model instead of a continuous-time one as suggested by the authors. I am wondering how the proposed method compares to this discrete-time version of the proposed method. \n\nSecond, the theoretical results like proposition 1 also holds for the discrete dynamics setting. The adaptive MSS may also be feasible in the discrete-time setting, which itself now is implemented in a discretized version. \n\nTherefore, the observations above make me wonder whether the continuous-time part is really needed in the submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and valuable feedback!\n## Weaknesses\n1. *Why continuous-time modeling:*\nContinuous time learning has several benefits over discrete-time modeling. For example, all systems in natural sciences are continuous in nature, therefore introducing priors into the learning problem is easier in continuous time (discrete-time priors are susceptible to the choice of discretization). Furthermore, when learning discrete-time dynamics, the sampling and control frequency is fixed to the discretization of the problem formulation. For a new choice of discretization, generally one has to relearn the dynamics. In continuous time, once the ODE is learned, the learned model can be used for any discretization.\nWe appreciate the comment of the reviewer that real-world systems are controlled in discrete time and not continuously. In our analysis, we consider a general policy class. A special case of this policy class is piece-wise constant policy functions, which as the reviewer acknowledges are more practical (we do not explicitly include the time-dependency of the policy in eq (1) for simplicity). Furthermore, as we show in our results, we can obtain continuous-time performance with our algorithm empirically. Furthermore, we compare our algorithm with the true discrete-time models in Table 1 and show that our method outperforms the discrete-time case. Since we already outperform the ground truth discrete-time model, we did not evaluate a discrete-time model-based RL method additionally.\nIn summary, continuous time modeling has several advantages over its discrete-time counterpart as we discuss above. Our problem formulation also considers the more practical class of piecewise constant policy functions, and in our results, we show that continuous time modeling achieves better performance. We hope we could convince the reviewer about continuous time modeling with our response.\n## Questions\n1. *Cost function evaluation:*\nThe cost is always evaluated in the continuous-time setting. That means that no matter whether we control the system continuously or with a discrete zero-order hold, we evolve the dynamics in continuous time and compute the continuous-time cost (up to ODE solver precision).\n2. *Difference between ODE vs discrete dynamics:*\nWe are not sure if we understood the question exactly. Here is a potential response; we compare our continuous time modeling approach to the ground truth discrete-time model in Table 1. Our experiments show that our method outperforms the discrete-time case. Learning discrete-time dynamics is a well-studied problem, where similar theoretical guarantees exist [1].\n3. *Equidistant performs better than discrete zero-order hold OC:*\nWe refer the reviewer to the Author Rebuttal on how the costs are calculated in our experiments. As discussed in the authors rebuttal section discrete-time modeling and control are limited to the discretization schema. On the contrary continuous-time modeling benefits from representing the dynamics between the discretization nodes and adapting the controls accordingly. With equidistant MSS we take measurements equidistantly in time. However, we still learn a continuous time model and use it for continuous-time control.\nFrom Table 1, we see that the performance of our learned continuous-time controller is better than the discretized zero-order hold controller obtained using the true dynamics. \n4. *MSS for discrete-time models:*\nWe are not entirely sure we understood your question here. Here is a tentative answer:\nMSS can be used in the setting where we can only observe the state at some fixed discretized times. However, it still requires learning dynamics in continuous time. This is because discrete-time modeling requires observing transitions $(x_{k}, u_{k}, x_{k+1})$ at each time step. Did we provide the answer you had in mind or were you interested in something else?\n5. *List of dynamics models:*\nAll considered systems except for the Glucose in the blood system are not linear and standard in the literature. For the sake of completeness, we added them to the updated version of the paper.\n\nHaving addressed all of the questions provided by the reviewer, and given the contributions of this paper, we would appreciate it if the reviewer would increase their score for our paper. We would be happy to answer any remaining questions or concerns.\n## References\n[1] Curi, Sebastian, Felix Berkenkamp, and Andreas Krause. \"Efficient model-based reinforcement learning through optimistic policy search and planning.\" Advances in Neural Information Processing Systems 33 (2020)."}]

Author rebuttal:
We thank all the reviewers for their valuable and useful feedback. We believe there is been some confusion around the discrete-time control setting we consider in our work. Accordingly, we have clarified this further in the appendix of the updated paper. We include a summary below:

1. When we learn a system that follows the ODE $\dot{x} = f^*(x, u)$ in discrete time, we can only learn the discretized dynamics of the system, i.e., $x_{k+1} = f(x_k, u_k)$. This approach is severely limited to the choice of discretization.  
2. For the learned discretized system $x_{k+1} = f(x_k, u_k)$, we *cannot model (predict) the behavior of the system between two time steps.* Therefore, our control inputs can only be changed at the discretization frequency.
3. Nonetheless, the underlying system is still evolving continuously over time. Hence, even if we model and control the system with the discrete model, we evaluate the state evolution and the corresponding cost with an ODE integrator, where the control is changing at the discretization frequency. 

Lastly, we have attached a pdf of an additional experiment that studies different measurement selection strategies as requested by the reviewers. We included the results of the additional experiment in the appendix of the updated version of the paper. We would be happy to further update the paper if there are any remaining questions or feedback from the reviewers.

Meta-review: The discussion is inclined towards acceptance.


We encourage the authors to consider the feedback provided in the review process.
Decision: Accept (poster)