Title: Efficient Exploration in Continuous-time Model-based Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Optimal Control

Continuous Time
Abstract: Reinforcement learning algorithms typically consider discrete-time dynamics, even though the underlying systems are often continuous in time. In this paper, we introduce a model-based reinforcement learning algorithm that represents continuous-time dynamics using nonlinear ordinary differential equations (ODEs). We capture epistemic uncertainty using well-calibrated probabilistic models, and use the optimistic principle for exploration. Our regret bounds surface the importance of the measurement selection strategy (MSS), since in continuous time we not only must decide how to explore, but also when to observe the underlying system. Our analysis demonstrates that the regret is sublinear when modeling ODEs with Gaussian Processes (GP) for common choices of MSS, such as equidistant sampling. Additionally, we propose an adaptive, data-dependent, practical MSS that, when combined with GP dynamics, also achieves sublinear regret with significantly fewer samples. We showcase the benefits of continuous-time modeling over its discrete-time counterpart, as well as our proposed adaptive MSS over standard baselines, on several applications.

Efficient Exploration in Continuous-time Model-based Reinforcement Learning

Abstract

Reinforcement learning algorithms typically consider discrete-time dynamics, even though the underlying systems are often continuous in time. In this paper, we introduce a model-based reinforcement learning algorithm that represents continuous-time dynamics using nonlinear ordinary differential equations (ODEs). We capture epistemic uncertainty using well-calibrated probabilistic models, and use the optimistic principle for exploration. Our regret bounds surface the importance of the measurement selection strategy (MSS), since in continuous time we not only must decide how to explore, but also when to observe the underlying system. Our analysis demonstrates that the regret is sublinear when modeling ODEs with Gaussian Processes (GP) for common choices of MSS, such as equidistant sampling. Additionally, we propose an adaptive , data-dependent, practical MSS that, when combined with GP dynamics, also achieves sublinear regret with significantly fewer samples. We showcase the benefits of continuous-time modeling over its discrete-time counterpart, as well as our proposed adaptive MSS over standard baselines, on several applications.

1 Introduction

Real-world systems encountered in natural sciences and engineering applications, such as robotics (Spong et al., 2006), biology (Lenhart and Workman, 2007; Jones et al., 2009), medicine (Panetta and Fister, 2003), etc., are fundamentally continuous in time. Therefore, ordinary differential equations (ODEs) are the natural modeling language. However, the reinforcement learning (RL) community predominantly models problems in discrete time, with a few notable exceptions (Doya, 2000; Yildiz et al., 2021; Lutter et al., 2021). The discretization of continuous-time systems imposes limitations on the application of state-of-the-art RL algorithms, as they are tied to specific discretization schemes.

Discretization of continuous-time systems sacrifices several essential properties that could be preserved in a continuous-time framework (Nesic and Postoyan, 2021). For instance, exact discretization is only feasible for linear systems, leading to an inherent discrepancy when using standard discretization techniques for nonlinear systems. Additionally, discretization obscures the inter-sample behavior of the system, changes stability properties, may result in uncontrollable systems, or requires an excessively high sampling rate. Multi-time-scale systems are particularly vulnerable to these issues (Engquist et al., 2007). In many practical scenarios, the constraints imposed by discrete-time modeling are undesirable. Discrete-time models do not allow for the independent adjustment of measurement and control frequencies, which is crucial for real-world systems that operate in different regimes. For example, in autonomous driving, low-frequency sensor sampling

and control actuation may suffice at slow speeds, while high-speed driving demands faster control. How to choose aperiodic measurements and control is studied in the literature on the event and self-triggered control (Astrom and Bernhardsson, 2002; Anta and Tabuada, 2010; Heemels et al., 2012). They show that the number of control inputs can be significantly reduced by using aperiodic control. Moreover, data from multiple sources is often collected at varying frequencies (Ghysels et al., 2006) and is often not even equidistant in time. Discrete-time models struggle to exploit this data for learning, while continuous-time models can naturally accommodate it.

Continuous-time modeling also offers the flexibility to determine optimal measurement times based on need, in contrast to the fixed measurement frequency in discrete-time settings, which can easily miss informative samples. This advantage is particularly relevant in fields like medicine, where patient monitoring often requires higher frequency measurements during the onset of illness and lower frequency measurements as the patient recovers (Kaandorp and Koole, 2007). At the same time, in fields such as medicine, measurements are often costly, and hence, it is important that the most informative measurements are selected. Discrete-time models, limited to equidistant measurements, therefore often result in suboptimal decision-making and their sample efficiency is fundamentally limited. In summary, continuous-time modeling is agnostic to the choice of measurement selection strategy (MSS), whereas discrete-time modeling often only works for an equidistant MSS.

Contributions Given the advantages of continuous-time modeling, in this work, we propose an optimistic continuous-time model-based RL algorithm - OCORL. Moreover, we theoretically analyze OCORL and show a general regret bound that holds for any MSS. We further show that for common choices of MSSs, such as equidistant sampling, the regret is sublinear when we model the dynamics with GPs (Williams and Rasmussen, 2006). To our knowledge, we are the first to give a no-regret algorithm for a rich class of nonlinear dynamical systems in the continuous-time RL setting. We further propose an adaptive MSS that is practical, data-dependent, and requires considerably fewer measurements compared to the equidistant MSS while still ensuring the sublinear regret for the GP case. Crucial to OCORL is the exploration induced by the optimism in the face of uncertainty principle for model-based RL, which is commonly employed in the discrete-time realm (Auer et al., 2008; Curi et al., 2020). We validate the performance of OCORL on several robotic tasks, where we clearly showcase the advantages of continuous-time modeling over its discrete-time counterpart. Finally, we provide an efficient implementation 1 of OCORL in JAX (Bradbury et al., 2018).

2 Problem setting

In this work, we study a continuous-time deterministic dynamical system f ∗ with initial state x 0 ∈ X ⊂ R d x , i.e.,

Here u : [0 , ∞ ) → R d u is the input we apply to the system. Moreover, we consider state feedback controllers represented through a policy π : X → U ⊂ R d u , that is, u ( s ) = π ( x ( s )) . Our objective is to find the optimal policy with respect to a given running cost function c : R d x × R d u → R . Specifically, we are interested in solving the following optimal control (OC) problem over the policy space Π :

The function f ∗ is unknown , but we can collect data over episodes n = 1 . . . , N and learn about the system by deploying a policy π n ∈ Π for the horizon of T in episode n . In an (overly) idealized continuous time setting, one can measure the system at any time step. However, we consider a more practical setting, where we assume taking measurements is costly, and therefore want as few measurements as necessary. To this end, we formally define a measurement selection strategy below.

Definition 1 (Measurement selection strategy) . A measurement selection strategy S is a sequence of sets ( S n ) n ≥ 1 , such that S n contains m n points at which we take measurements, i.e., S n ⊂ [0 , T ] , | S n | = m n . 2

During episode n , given a policy π n and a MSS S n , we collect a dataset D n ∼ ( π n , S n ) . The dataset is defined as

Here x n ( t ) , ˙ x n ( t ) are state and state derivative in episode n , and ϵ n,i is i.i.d, σ -sub-Gaussian noise of the state derivative observations. Note, even though in practice only the state x ( t ) might be observable, one can estimate its derivative ˙ x ( t ) (e.g., using finite differences, interpolation methods, etc. Cullum (1971); Knowles and Wallace (1995); Chartrand (2011); Knowles and Renka (2014); Wagner et al. (2018); Treven et al. (2021)). We capture the noise in our measurements and/or estimation of ˙ x ( t ) with ϵ i,n .

In summary, at each episode n , we deploy a policy π n for a horizon of T , observe the system according to a proposed MSS S n , and learn the dynamics f ∗ . By deploying π n instead of the optimal policy π ∗ , we incur a regret,

Note that the policy π n depends on the data D 1: n -1 = ∪ i<n D i and hence implicitly on the MSS S .

Performance measure We analyze OCORL by comparing it with the performance of the best policy π ∗ from the class Π . We evaluate the cumulative regret R N ( S ) def = ∑ N n =1 r n ( S ) that sums the gaps between the performance of the policy π n and the optimal policy π ∗ over all the episodes. If the regret R N ( S ) is sublinear in N , then the average cost of the policy C ( π n , f ∗ ) converges to the optimal cost C ( π ∗ , f ∗ ) .

2.1 Assumptions

Any meaningful analysis of cumulative regret for continuous time, state, and action spaces requires some assumptions on the system and the policy class. We make some continuity assumptions, similar to the discrete-time case (Khalil, 2015; Curi et al., 2020; Sussex et al., 2023), on the dynamics, policy, and cost.

Assumption 1 (Lipschitz continuity) . Given any norm ∥·∥ , we assume that the system dynamics f ∗ and cost c are L f and L c -Lipschitz continuous, respectively, with respect to the induced metric. Moreover, we define Π to be the policy class of L π -Lipschitz continuous policy functions and F a class of L f Lipschitz continuous dynamics functions with respect to the induced metric.

We learn a model of f ∗ using data collected from the episodes. For a given state-action pair z = ( x , u ) , our learned model predicts a mean estimate µ n ( z ) and quantifies our epistemic uncertainty σ n ( z ) about the function f ∗ .

Definition 2 (Well-calibrated statistical model of f ∗ , Rothfuss et al. (2023)) . Let Z def = X × U . An all-time well-calibrated statistical model of the function f ∗ is a sequence {M n ( δ ) } n ≥ 0 , where if, with probability at least 1 -δ , we have f ∗ ∈ ⋂ n ≥ 0 M n ( δ ) . Here, µ n,j and σ n,j denote the j -th element in the vector-valued mean and standard deviation functions µ n and σ n respectively, and β n ( δ ) ∈ R ≥ 0 is a scalar function that depends on the confidence level δ ∈ (0 , 1] and which is monotonically increasing in n .

Assumption 2 (Well-calibration) . We assume that our learned model is an all-time well-calibrated statistical model of f ∗ . We further assume that the standard deviation functions ( σ n ( · )) n ≥ 0 are L σ -Lipschitz continuous.

This is a natural assumption, which states that we are with high probability able to capture the dynamics within a confidence set spanned by our predicted mean and epistemic uncertainty. For example, GP models are all-time well-calibrated for a rich class of functions (c.f., Section 3.2) and also satisfy the Lipschitz continuity assumption on ( σ n ( · )) n ≥ 0 (Rothfuss et al., 2023). For Bayesian neural networks, obtaining accurate uncertainty estimates is still an open and active research problem. However, in practice, re-calibration techniques (Kuleshov et al., 2018) can be used.

By leveraging these assumptions, in the next section, we propose our algorithm OCORL and derive a generic bound on its cumulative regret. Furthermore, we show that OCORL provides sublinear cumulative regret for the case when GPs are used to learn f ∗ .

3 Optimistic Continuous-time RL Algorithm

OCoRL: OPTIMISTIC CONTINUOUS-TIME RL Init: Statistical model M 0 , Simulator SIM, MSS S , Probability δ for episode n = 1 , . . . , N do π n = argmin π ∈ Π min f ∈M n -1 ∩F C ( π , f ) /* Select optimistic policy /* D n = { ( z n ( t n,i ) , ˙ y n ( t n,i )) | t n,i ∈ S n } ← SIM ( π n , S n ) /* Measurement collection /* M n ← ( µ n , σ n , β n ( δ )) ←D 1: n /* Update statistical model/*

Optimistic policy selection There are several strategies we can deploy to trade-off exploration and exploitation, e.g., dithering ( ε -greedy, Boltzmann exploration (Sutton and Barto, 2018)), Thompson sampling (Osband et al., 2013), upper-confidence RL (UCRL) (Auer et al., 2008), etc. OCORL is a continuous-time variant of the Hallucinated UCRL strategy introduced by Chowdhury and Gopalan (2017); Curi et al. (2020). In episode n , the optimistic policy is obtained by solving the optimal control problem:

Here, f n is a dynamical system such that the cost by controlling f n with its optimal policy π n is the lowest among all the plausible systems from M n -1 ∩ F . The optimal control problem (2) is infinite-dimensional, in general nonlinear, and thus hard to solve. For the analysis, we assume we can perfectly solve it. In Appendix B, we present details on how we solve it in practice. Specifically, in Appendix B.1, we show that our results seamlessly extend to the setting where the optimal control problem of Equation (1) is discretized w.r.t. an arbitrary choice of discretization. Moreover, in this setting, we show that theoretical guarantees can be derived without restricting the models to f ∈ M n -1 ∩ F . Instead, a practical optimization over all models in M n -1 can be performed as in Curi et al. (2020); Pasztor et al. (2021).

Model complexity We expect that the regret of any model-based continuous-time RL algorithm depends both on the hardness of learning the underlying true dynamics model f ∗ and the MSS. To capture both, we define the following model complexity:

The model complexity measure captures the hardness of learning the dynamics f ∗ . Intuitively, for a given N , the more complicated the dynamics f ∗ , the larger the epistemic uncertainty and thereby the model complexity. For the discrete-time setting, the integral is replaced by the sum over the uncertainties on the trajectories (c.f., Equation (8) of Curi et al. (2020)). In the continuous-time setting, we do not observe the state at every time step, but only at a finite number of times wherever the MSS S proposes to measure the system. Accordingly, S influences how we collect data and update our calibrated model. Therefore, the model complexity depends on S . Next, we first present the regret bound for general MSSs, then we look at particular strategies for which we can show convergence of the regret.

Proposition 1. Let S be any MSS. If we run OCORL , we have with probability at least 1 -δ ,

We provide the proof of Proposition 1 in Appendix A. Because we have access only to the statistical model and any errors in dynamics compound (continuously in time) over the episode, the regret R N ( S ) depends exponentially on the horizon T . This is in line with the prior work in the discrete-time setting (Curi et al., 2020). If the model complexity term I N ( f ∗ , S ) and β N grow at a rate slower than N , the regret is sublinear and the average performance of OCORL converges to C ( π ∗ , f ∗ ) . In our analysis, the key step to show sublinear regret for the GP dynamics model is to upper bound the integral of uncertainty in the model complexity with the sum of uncertainties at the points where we collect the measurements. In the next section, we show how this can be done for different measurement selection strategies.

3.1 Measurement selection strategies (MSS)

In the following, we present different natural MSSs and compare the number of measurements they propose per episode. Formal derivations are included in Appendix A.2.

Oracle Intuitively, if we take measurements at the times when we are the most uncertain about dynamics on the executed trajectory, i.e., when ∥ σ n -1 ( z n ( t )) ∥ is largest, we gain the most knowledge about the true function f ∗ . Indeed, when the statistical model is a GP and noise is homoscedastic and Gaussian, observing the most uncertain point on the trajectory leads to the maximal reduction in entropy of f ∗ ( z n ( t )) (c.f., Lemma 5.3 of Srinivas et al. (2009)). In the ideal case, we can define an MSS that collects only the point with the highest uncertainty in every episode, i.e., S ORA n def = { t n, 1 } where t n, 1 def = argmax 0 ≤ t ≤ T ∥ σ n -1 ( z n ( t )) ∥ 2 . For this MSS we bound the integral over the horizon T with the maximal value of the integrand times the horizon T :

The oracle MSScollects only one measurement per episode, however, it is impractical since it requires knowing the most uncertain point on the true trajectory a priori, i.e., before executing the policy.

Equidistant Another natural MSS is the equidistant MSS. We collect m n equidistant measurements in episode n and upper bound the integral with the upper Darboux integral.

Here L σ 2 is the Lipschitz constant of ∥ σ n -1 ( · ) ∥ 2 . To achieve sublinear regret, we require that ∑ N n =1 1 m n ∈ o ( N ) . Therefore, for a fixed equidistant MSS, our analysis does not ensure a sublinear regret. This is because we consider a continuous-time regret which is integrated (c.f., Equation (1)) while in the discrete-time setting, the regret is defined for the equidistant MSS only (Curi et al., 2020). Accordingly, we study a strictly harder problem. Nonetheless, by linearly increasing the number of observations per episode and setting m n = n , we get ∑ N n =1 1 m n ∈ O (log( N )) and sublinear regret. The equidistant MSS is easy to implement, however, the required number of samples is increasing linearly with the number of episodes.

Adaptive Finally, we present an MSS that is practical, i.e., easy to implement and at the same time requires only a few measurements per episode. The core idea of receding horizon adaptive MSS is simple: simulate (hallucinate) the system f n with the policy π n , and find the time t such that ∥ σ n -1 ( ̂ z n ( t )) ∥ is largest. Here, ̂ z n ( t ) is the state-action pair at time t in episode n for the hallucinated trajectory.

However, the hallucinated trajectory can deviate from the true trajectory exponentially fast in time, and the time of the largest variance on the hallucinated trajectory can be far away from the time of the largest variance on the true trajectory. To remedy this technicality, we utilize a receding horizon MSS. We split, in episode n , the time horizon T into uniform-length intervals of ∆ n time, where ∆ n ∈ Ω( 1 β n ) , c.f., Appendix A. At the beginning of every time interval, we measure the true stateaction pair z , hallucinate with policy π n starting from z for ∆ n time on the system f n , and calculate the time when the hallucinated trajectory has the highest variance. Over the next time horizon ∆ n , we collect a measurement only at that time. Formally, let m n = ⌈ T/ ∆ n ⌉ be the number of measurements in episode n and denote for every i ∈ { 1 , . . . , m n } the time with the highest variance on the hallucinated trajectory in the time interval [( i -1)∆ n , i ∆ n ] by t n,i :

t

2∆

n

f

n

(

̂

x

n

,

π

n

(

̂

x

n

))

˙

x

n

=

f

∗

(

x

n

,

π

n

(

x

n

))

̂

x

n

(

T

n

) =

x

n

(

T

n

)

‖

σ

n

-

1

(

̂

z

n

(

t

))

‖

x

n

(

t

n,

1

)

x

n

(

t

n,

2

)

x

n

(

t

n,

3

)

x

n

(

t

n,

4

)

t

n,

2

x

n

(∆

n

)

x

n

(2∆

n

)

x

n

(3∆

n

)

∆

n

For this MSS, we show in Appendix A the upper bound

The model complexity upper bound for the adaptive MSS in Equation (7) does not have the L σ 2 -dependent term from the equidistant MSS (6) and only depends on the sum of uncertainties at the collected points. Further, the number of points we collect in episode n is O ( β n ) , and β n , e.g., if we model the dynamics with GPs with radial basis function (RBF) kernel, is of order polylog ( n ) (c.f., Lemma 2).

3.2 Modeling dynamics with a Gaussian Process

We can prove sublinear regret for the proposed three MSSs coupled with optimistic exploration when we model the dynamics using GPs with standard kernels such as RBF, linear, etc. We consider the vector-valued dynamics model f ∗ ( z ) = ( f ∗ 1 ( z ) , . . . , f ∗ d x ( z )) , where scalar-valued functions f ∗ j ∈ H k are elements of a Reproducing Kernel Hilbert Space (RKHS) H k with kernel function k , and their norm is bounded ∥ ∥ f ∗ j ∥ ∥ k ≤ B . We write f ∗ ∈ H d x k,B def = { ( f 1 , . . . , f d x ) | ∥ f j ∥ k ≤ B } . To learn f ∗ we fit a GP model with a zero mean and kernel k to the collected data D 1: n . For ease of notation, we denote by ˙ y j 1: n the concatenated j -th dimension of the state derivative observations from D 1: n . The posterior means and variances of the GP model have a closed form (Williams and Rasmussen (2006)):

where we write K n = [ k ( z l , z m )] z l , z m ∈D 1: n , and k n ( z ) = [ k ( z l , z )] z l ∈D 1: n . The posterior means and variances with the right scaling factor β n ( δ ) satisfy the all-time well-calibrated Assumption 2.

Lemma 2 (Lemma 3.6 from Rothfuss et al. (2023)) . Let f ∗ ∈ H d x k,B and δ ∈ (0 , 1) . Then there exist β n ( δ ) ∈ O ( √ γ n +log(1 /δ )) such that the confidence sets M n built from the triplets ( µ n , σ n , β n ( δ )) form an all-time well-calibrated statistical model.

<latexi sh

1_b

64="8GupmU

LHZSIvWo

3O7rC

P

AQ

>

n

c

VFd

M2

X

qj

0

K

D

T

z

Y

y

k

+w

E

f

/

R

9

g

B

N

5

J

<latexi sh

1_b

64="H3MvBINC9

fkF+u

S

ORn7Jj

DU

>A

c

Z

E

X

q

d

0g pT

z

Q

w

K

W

Y

L

2

r

V

G

/

5

y

P

m

8

o

<latexi sh

1_b

64="GRI02F/JYP

NWv

S7jCo

U

m

>A

3

c

Z

9w

E

T

y

p

L

z

H

5f

K

Br+

n

kVO

d

8

D

q

M

g

u

Q

X

<latexi sh

1_b

64="nOdr3f8

5GM

o

cQSAW

L

>

C

ZF

9

w

I

DX

V

ERUkzYJqm

j

N

K

p

H

P

T

+

v

u

y

/

B

7

g

2

0

<latexi sh

1_b

64="E9

HdZX

5vj

8CqO

uo

>A

c

FR

Mw

I

GB0

Dp

V

S

m

Q

PTN

k

U

J

3

W

yn

f

/

2

7

+

z

r

K

g

Y

L

<latexi sh

1_b

64="+j7SVTP

d

No5zMY3

0Q

GH

>A

C2

c

ZF

9w

EI

fq

u

K

R

D

k

8ng

v

W

r

/

U

m

y

X

O

p

J

L

B

<latexi sh

1_b

64="ORdT

7QXyv

FWZ

DPn3q

c

>A

C

f

9

w

MUV

u

+

L

2

Gz mBk

o

j

p

r

0

E

N

K

H

5

g

/

8

Y

I

J

S

<latexi sh

1_b

64="

0KXkByRI2

p

Q

S

DgoM

v

Z

>A

C

3

c

FN

9

wE

u

r

O

JW

V

q

T

m8

UP

H

5

L

/

G

+

Y

7d

z

f

n

j

<latexi sh

1_b

64="

LQyFN5pHJW

S7

m+TZogwR

ck

>A

C

n

d

9

GIUV

u

P

z

3

0

OB

f

2K

r

D

/

M8X

q

E

v

Y

j

<latexi sh

1_b

64="gCd qK

+yu vLAVMZ

2

n

7Q

>

H

c

F

9

w

IY

5

N

B

0

m

G

k

W

S

E/

3

p

J

f

P

D

O

j

8

R

o

z

<latexi sh

1_b

64="v

LUV

Y/oH+

9X

RZrm

g

cM

>A

DKn

JNj wEI

d8

W

y

u

S

0

2

5

f

3

p

Q

G

7

k

C

T

O

B

P

F

<latexi sh

1_b

64="j mM

C

P

O9

Uc n+

Y

/

o

>A

DG

ZF

w

d8

Ef

0

u

R

V

q

SBNI

J

H

T

py

K

v

X

r

5

Q3

W

g

2

L

z

<latexi sh

1_b

64="vKjW

JCX+VGk2

Mr

A

NSq3D8

>

H

c

ZF

EI

d5

y

Qu

o

0

U9g

z

n

P

O

f

/w

7

Y

m

R

p

B

L

T

<latexi sh

1_b

64="BEN

XufkKy2LdV

IG

>A

C

3

c

ZF

w

7

q

R

T

W

o

Q

JP

S

5M

0

mn z/

j

g

O

v

r

8

U

D

H

<latexi sh

1_b

64="

dyC9

S

Km50wunv3cRDo

>A

X

ZF

M

I

G

q

N

r

LQ

k

7

VT

O

W

z

2

/

p

g

8

H

+

J

Y

B

j

P

E

<latexi sh

1_b

64="2TyYo

9prPZKG

8

7NC

Wz v3M

>A

B

H

c

VDLSg

EO

dQ

U

nJ

m

RX

k

/w

I

q

u

0

F

j

5

+

f

Here, γ n is the maximum information gain after observing n points (Srinivas et al., 2009), as defined in Appendix A, where we also provide the rates for common kernels. For example, for the RBF kernel, γ n = O ( log( n ) d x + d u +1 ) .

Finally, we show sublinear regret for the proposed MSSs for the case when we model dynamics with GPs. The proof of Theorem 3 is provided in Appendix A.

Theorem 3. Assume that f ∗ ∈ H d x k,B , the observation noise is i.i.d. N ( 0 ; σ 2 I ) , and ∥·∥ is the Euclidean norm. We model f ∗ with the GP model. The regret for different MSSs is with probability at least 1 -δ bounded by

The optimistic exploration coupled with any of the proposed MSSs achieves regret that depends on the maximum information gain. All regret bounds from Theorem 3 are sublinear for common kernels, like linear and RBF. The bound for the adaptive MSS with RBF kernel is O ( √ N log( N ) 2( d x + d u +1) ) ,

where we hide the dependence on T in the O notation. We reiterate that while the number of observations per episode of the oracle MSS is optimal, we cannot implement the oracle MSS in practice. In contrast, the equidistant MSS is easy to implement, but the number of measurements grows linearly with episodes. Finally, adaptive MSS is practical, i.e., easy to implement, and requires only a few measurements per episode, i.e., polylog ( n ) in episode n for RBF.

Summary OCORL consists of two key and orthogonal components; (i) optimistic policy selection and (ii) measurement selection strategies (MSSs). In optimistic policy selection, we optimistically, w.r.t. plausible dynamics, plan a trajectory and rollout the resulting policy. We study different MSSs, such as the typical equidistant MSS, for data collection within the framework of continuous time modeling. Furthermore, we propose an adaptive MSS that measures data where we have the highest uncertainty on the planned (hallucinated) trajectory. We show that OCORL suffers no regret for the equidistant, adaptive, and oracle MSSs.

4 Related work

Model-based Reinforcement Learning Model-based reinforcement learning (MBRL) has been an active area of research in recent years, addressing the challenges of learning and exploiting environment dynamics for improved decision-making. Among the seminal contributions, Deisenroth and Rasmussen (2011) proposed the PILCO algorithm which uses Gaussian processes for learning the system dynamics and policy optimization. Chua et al. (2018) used deep ensembles as dynamics models. They coupled MPC (Morari and Lee, 1999) to efficiently solve high dimensional tasks with considerably better sample complexity compared to the state-of-the-art model-free methods SAC (Haarnoja et al., 2018), PPO (Schulman et al., 2017), and DDPG (Lillicrap et al., 2015). The aforementioned model-based methods use more or less greedy exploitation that is provably optimal only in the case of the linear dynamics (Simchowitz and Foster, 2020). Exploration methods based on Thompson sampling (Dearden et al., 2013; Chowdhury and Gopalan, 2019) and Optimism (Auer et al., 2008; Abbasi-Yadkori and Szepesvári, 2011; Luo et al., 2018; Curi et al., 2020), however, provably converge to the optimal policy also for a large class of nonlinear dynamics if modeled with GPs. Among the discrete-time RL algorithms, our work is most similar to the work of Curi et al. (2020) where they use the reparametrization trick to explore optimistically among all statistically plausible discrete dynamical models.

Continuous-time Reinforcement Learning Reinforcement learning in continuous-time has been around for several decades (Doya, 2000; Vrabie and Lewis, 2008, 2009; Vamvoudakis et al., 2009). Recently, the field has gained more traction following the work on Neural ODEs by Chen et al. (2018). While physics biases such as Lagrangian (Cranmer et al., 2020) or Hamiltonian (Greydanus et al., 2019) mechanics can be enforced in continuous-time modeling, different challenges such

as vanishing Q -function (Tallec et al., 2019) need to be addressed in the continuous-time setting. Yildiz et al. (2021) introduce a practical episodic model-based algorithm in continuous time. In each episode, they use the learned mean estimate of the ODE model to solve the optimal control task with a variant of a continuous-time actor-critic algorithm. Compared to Yildiz et al. (2021) we solve the optimal control task using the optimistic principle. Moreover, we thoroughly motivate optimistic planning from a theoretical standpoint. Lutter et al. (2021) introduce a continuous fitted value iteration and further show a successful hardware application of their continuous-time algorithm. A vast literature exists for continuous-time RL with linear systems (Modares and Lewis, 2014; Wang et al., 2020), but few, only for linear systems, provide theoretical guarantees (Mohammadi et al., 2021; Basei et al., 2022). To the best of our knowledge, we are the first to provide theoretical guarantees of convergence to the optimal cost in continuous time for a large class of RKHS functions.

Aperiodic strategies Aperiodic MSSs and controls (event and self-triggered control) are mostly neglected in RL since most RL works predominantly focus on discrete-time modeling. There exists a considerable amount of literature on the event and self-triggered control (Astrom and Bernhardsson, 2002; Anta and Tabuada, 2010; Heemels et al., 2012). However, compared to periodic control, its theory is far from being mature (Heemels et al., 2021). In our work, we assume that we can control the system continuously, and rather focus on when to measure the system instead. The closest to our adaptive MSS is the work of Du et al. (2020), where they empirically show that by optimizing the number of interaction times, they can achieve similar performance (in terms of cost) but with fewer interactions. Compared to us, they do not provide any theoretical guarantees. Umlauft and Hirche (2019); Lederer et al. (2021) consider the non-episodic setting where they can continuously monitor the system. They suggest taking measurements only when the uncertainty of the learned model on the monitored trajectory surpasses the boundary ensuring stability. They empirically show, for feedback linearizable systems, that by applying their strategy the number of measurements reduces drastically and the tracking error remains bounded. Compared to them, we consider general dynamical systems and also don't assume continuous system monitoring.

5 Experiments

We now empirically evaluate the performance of OCORL on several environments. We test OCORL on Cancer Treatment and Glucose in blood systems from Howe et al. (2022), Pendulum , Mountain Car and Cart Pole from Brockman et al. (2016), Bicycle from Polack et al. (2017), Furuta Pendulum from Lutter et al. (2021) and Quadrotor in 2D and 3D from Nonami et al. (2010). The details of the systems' dynamics and tasks are provided in Appendix C.

Comparison methods To make the comparison fair, we adjust methods so that they all collect the same number of measurements per episode. For the equidistant setting, we collect M points per episode (we provide values of M for different systems in Appendix C). For the adaptive MSS, we assume ∆ n ≥ T , and instead of one measurement per episode we collect a batch of M measurements such that they (as a batch) maximize the variance on the hallucinated trajectory. To this end, we consider the Greedy Max Determinant and Greedy Max Kernel Distance strategies of Holzmüller et al. (2022). We provide details of the adaptive strategies in Appendix C. We compare OCORL with the described MSSs to the optimal discrete-time zero hold control, where we assume the access to the true discretized dynamics f ∗ d ( x , u ) = x + ∫ T/ ( M -1) 0 f ∗ ( x ( t ) , u ) dt . We further also compare with the best continuous-time control policy, i.e., the solution of Equation (1).

Does the continuous-time control policy perform better than the discrete-time control policy? In the first experiment, we test whether learning a continuous-time model from the finite data coupled with a continuous-time control policy on the learned model can outperform the discrete-time zero-order hold control on the true system. We conduct the experiment on all environments and report the cost after running OCORL for a few tens of episodes (the exact experimental details are provided in Appendix C). From Table 1, we conclude that the OCORL outperforms the discrete-time zero-order hold control on the true model on every system if we use the adaptive MSS, while achieving lower cost on 7 out of 9 systems if we measure the system equidistantly.

±

±

±

Does the adaptive MSS perform better than equidistant MSS? We compare the adaptive and equidistant MSSs on all systems and observe that the adaptive MSSs consistently perform better than the equidistant MSS. To better illustrate the difference between the adaptive and equidistant MSSs we study a 2-dimensional Pendulum system (c.f., Figure 2). First, we see that if we use adaptive MSSs we consistently achieve lower per-episode costs during the training. Second, we observe that while equidistant MSS spreads observations equidistantly in time and collects lots of measurements with almost the same state-action input of the dynamical system, the adaptive MSS spreads the measurements to have diverse state-action input pairs for the dynamical system on the executed trajectory and collects higher quality data.

5

10

15

Episode

10

1

10 0

10 1

10 2

Cost

MAX_KERNEL_DISTANCE

MAX_DETERMINANT

EQUIDISTANT

Best continuous

Best discrete

0.0

2.5

5.0

7.5

10.0

Time

0

1

2

3

×10

3

2

0

(

x

(

t

))

2

1

(

x

(

t

))

i

=0

2

i

(

x

(

t

))

0

1

2

3

x

1

3

2

1

0

x

2

Equidistant MSS

Adaptive MSS

0.5

1.0

1.5

2.0

2.5

×10

3

Does optimism help? For the final experiment, we examine whether planning optimistically helps. In particular, we compare our planning strategy to the mean planner that is also used by Yildiz et al. (2021). The mean planner solves the optimal control problem greedily with the learned mean model µ n in every episode n . We evaluate the performance of this model on the Pendulum system for all MSSs. We observe that planning optimistically results in reducing the cost faster and achieves better performance for all MSSs.

0

5

10

15

Episode

10

1

10 0

10 1

10 2

Cost

EQUIDISTANT

GREEDY

OPTIMISM

Best continuous

Best discrete

0

5

10

15

Episode

MAX_KERNEL_DISTANCE

0

5

10

15

Episode

MAX_DETERMINANT

6 Conclusion

We introduce a model-based continuous-time RL algorithm OCORL that uses the optimistic paradigm to provably achieve sublinear regret, with respect to the best possible continuous-time performance, for several MSSs if modeled with GPs. Further, we develop a practical adaptive MSS that, compared to the standard equidistant MSS, drastically reduces the number of measurements per episode while retaining the regret guarantees. Finally, we showcase the benefits of continuous-time compared to discrete-time modeling in several environments (c.f. Figure 1, Table 1), and demonstrate the benefits of planning with optimism compared to greedy planning.

In this work, we considered the setting with deterministic dynamics where we obtain a noisy measurement of the state's derivative. We leave the more practical setting, where only noisy measurements of the state, instead of its derivative, are available as well as stochastic, delayed differential equations, and partially observable systems to future work.

Our aim with this work is to catalyze further research within the RL community on continuous-time modeling. We believe that this shift in perspective could lead to significant advancements in the field and look forward to future contributions.

Acknowledgments and Disclosure of Funding

We would like to thank Yarden As, Scott Sussex and Dominik Baumann for their feedback. This project has received funding from the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545, and the Microsoft Swiss Joint Research Center.

References

Abbasi-Yadkori, Y. and Szepesvári, C. (2011). Regret bounds for the adaptive control of linear quadratic systems. In Proceedings of the 24th Annual Conference on Learning Theory , pages 1-26. JMLR Workshop and Conference Proceedings.

Anta, A. and Tabuada, P. (2010). To sample or not to sample: Self-triggered control for nonlinear systems. IEEE Transactions on automatic control , 55(9):2030-2042.

Astrom, K. J. and Bernhardsson, B. M. (2002). Comparison of riemann and lebesgue sampling for first order stochastic systems. In Proceedings of the 41st IEEE Conference on Decision and Control, 2002. , volume 2, pages 2011-2016. IEEE.

Auer, P., Jaksch, T., and Ortner, R. (2008). Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21.

Basei, M., Guo, X., Hu, A., and Zhang, Y. (2022). Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon. Journal of Machine Learning Research , 23(178):1-34.

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv preprint arXiv:1606.01540 .

Chartrand, R. (2011). Numerical differentiation of noisy, nonsmooth data. International Scholarly Research Notices , 2011.

Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. (2018). Neural ordinary differential equations. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc.

Chowdhury, S. R. and Gopalan, A. (2017). On kernelized multi-armed bandits. In International Conference on Machine Learning , pages 844-853. PMLR.

Chowdhury, S. R. and Gopalan, A. (2019). Online learning in kernelized markov decision processes. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3197-3205. PMLR.

Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems , 31.

Cover, T. M. (1999). Elements of information theory . John Wiley & Sons.

Cranmer, M., Greydanus, S., Hoyer, S., Battaglia, P., Spergel, D., and Ho, S. (2020). Lagrangian neural networks. arXiv preprint arXiv:2003.04630 .

Cullum, J. (1971). Numerical differentiation and regularization. SIAM Journal on numerical analysis , 8(2):254-265.

Curi, S., Berkenkamp, F., and Krause, A. (2020). Efficient model-based reinforcement learning through optimistic policy search and planning. Advances in Neural Information Processing Systems , 33:14156-14170.

Dearden, R., Friedman, N., and Andre, D. (2013). Model-based bayesian exploration. arXiv preprint arXiv:1301.6690 .

Deisenroth, M. and Rasmussen, C. E. (2011). Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11) , pages 465-472.

Doya, K. (2000). Reinforcement learning in continuous time and space. Neural computation , 12(1):219-245.

Du, J., Futoma, J., and Doshi-Velez, F. (2020). Model-based reinforcement learning for semimarkov decision processes with neural odes. Advances in Neural Information Processing Systems , 33:19805-19816.

Engquist, B., Li, X., Ren, W., Vanden-Eijnden, E., et al. (2007). Heterogeneous multiscale methods: a review. Communications in Computational Physics , 2(3):367-450.

Gäfvert, M. (2016). Modelling the furuta pendulum . Department of Automatic Control, Lund Institute of Technology (LTH).

Ghysels, E., Santa-Clara, P., and Valkanov, R. (2006). Predicting volatility: getting the most out of return data sampled at different frequencies. Journal of Econometrics , 131(1-2):59-95.

Greydanus, S., Dzamba, M., and Yosinski, J. (2019). Hamiltonian neural networks. Advances in neural information processing systems , 32.

Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018). Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 .

Hartman, P. (2002). Ordinary differential equations . SIAM.

Heemels, W., Johansson, K. H., and Tabuada, P. (2021). Event-triggered and self-triggered control. In Encyclopedia of Systems and Control , pages 724-730. Springer.

Heemels, W. P., Johansson, K. H., and Tabuada, P. (2012). An introduction to event-triggered and self-triggered control. In 2012 ieee 51st ieee conference on decision and control (cdc) , pages 3270-3285. IEEE.

Holzmüller, D., Zaverkin, V., Kästner, J., and Steinwart, I. (2022). A framework and benchmark for deep batch active learning for regression. arXiv preprint arXiv:2203.09410 .

Howe, N., Dufort-Labbé, S., Rajkumar, N., and Bacon, P.-L. (2022). Myriad: a real-world testbed to bridge trajectory optimization and deep learning. Advances in Neural Information Processing Systems , 35:29801-29815.

Jacobson, D. H. and Mayne, D. Q. (1970). Differential dynamic programming . Number 24. Elsevier Publishing Company.

Jones, D. S., Plank, M., and Sleeman, B. D. (2009). Differential equations and mathematical biology . CRC press.

Kaandorp, G. C. and Koole, G. (2007). Optimal outpatient appointment scheduling. Health care management science , 10:217-229.

Kakade, S., Krishnamurthy, A., Lowrey, K., Ohnishi, M., and Sun, W. (2020). Information theoretic regret bounds for online nonlinear control. Advances in Neural Information Processing Systems , 33:15312-15325.

Kelly, M. (2017). An introduction to trajectory optimization: How to do your own direct collocation. SIAM Review , 59(4):849-904.

Khalil, H. K. (2015). Nonlinear control , volume 406. Pearson New York.

Kirschner, J. and Krause, A. (2018). Information directed sampling and bandits with heteroscedastic noise. In Conference On Learning Theory , pages 358-384. PMLR.

Knowles, I. and Renka, R. J. (2014). Methods for numerical differentiation of noisy data. Electron. J. Differ. Equ , 21:235-246.

Knowles, I. and Wallace, R. (1995). A variational method for numerical differentiation. Numerische Mathematik , 70:91-110.

Kuleshov, V., Fenner, N., and Ermon, S. (2018). Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning , pages 2796-2804. PMLR.

Lederer, A., Conejo, A. J. O., Maier, K. A., Xiao, W., Umlauft, J., and Hirche, S. (2021). Gaussian process-based real-time learning for safety critical applications. In International Conference on Machine Learning , pages 6055-6064. PMLR.

Lenhart, S. and Workman, J. T. (2007). Optimal control applied to biological models . CRC press.

Li, W. and Todorov, E. (2004). Iterative linear quadratic regulator design for nonlinear biological movement systems. In ICINCO (1) , pages 222-229. Citeseer.

Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y ., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 .

Luo, Y., Xu, H., Li, Y., Tian, Y ., Darrell, T., and Ma, T. (2018). Algorithmic framework for modelbased deep reinforcement learning with theoretical guarantees. arXiv preprint arXiv:1807.03858 .

Lutter, M., Mannor, S., Peters, J., Fox, D., and Garg, A. (2021). Value iteration in continuous actions, states and time. arXiv preprint arXiv:2105.04682 .

Modares, H. and Lewis, F. L. (2014). Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Transactions on Automatic control , 59(11):3051-3056.

Mohammadi, H., Zare, A., Soltanolkotabi, M., and Jovanovi´ c, M. R. (2021). Convergence and sample complexity of gradient methods for the model-free linear-quadratic regulator problem. IEEE Transactions on Automatic Control , 67(5):2435-2450.

Morari, M. and Lee, J. H. (1999). Model predictive control: past, present and future. Computers & Chemical Engineering , 23(4-5):667-682.

Mordatch, I., Todorov, E., and Popovi´ c, Z. (2012). Discovery of complex behaviors through contactinvariant optimization. ACM Transactions on Graphics (ToG) , 31(4):1-8.

Nesic, D. and Postoyan, R. (2021). Nonlinear sampled-data systems. In Encyclopedia of Systems and Control , pages 1477-1483. Springer.

Nonami, K., Kendoul, F., Suzuki, S., Wang, W., and Nakazawa, D. (2010). Autonomous flying robots: unmanned aerial vehicles and micro aerial vehicles . Springer Science & Business Media.

Osband, I., Russo, D., and Van Roy, B. (2013). (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems , 26.

Osband, I. and Van Roy, B. (2014). Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems , 27.

Paing, H. S., Schagin, A. V., Win, K. S., and Linn, Y. H. (2020). New designing approaches for quadcopter using 2d model modelling a cascaded pid controller. In 2020 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus) , pages 23702373. IEEE.

Panetta, J. C. and Fister, K. R. (2003). Optimal control applied to competing chemotherapeutic cell-kill strategies. SIAM Journal on Applied Mathematics , 63(6):1954-1971.

Pasztor, B., Bogunovic, I., and Krause, A. (2021). Efficient model-based multi-agent mean-field reinforcement learning. arXiv preprint arXiv:2107.04050 .

Polack, P., Altché, F., d'Andréa Novel, B., and de La Fortelle, A. (2017). The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles? In 2017 IEEE intelligent vehicles symposium (IV) , pages 812-818. IEEE.

Rothfuss, J., Sukhija, B., Birchler, T., Kassraie, P., and Krause, A. (2023). Hallucinated adversarial control for conservative offline policy evaluation. arXiv preprint arXiv:2303.01076 .

Russo, D. and Van Roy, B. (2014). Learning to optimize via posterior sampling. Mathematics of Operations Research , 39(4):1221-1243.

Russo, D. and Van Roy, B. (2016). An information-theoretic analysis of thompson sampling. The Journal of Machine Learning Research , 17(1):2442-2471.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 .

Simchowitz, M. and Foster, D. (2020). Naive exploration is optimal for online lqr. In International Conference on Machine Learning , pages 8937-8948. PMLR.

Singh, M. and Theers, M. (2021). Kinematic bicycle model. https://thomasfermi.github.io/ Algorithms-for-Automated-Driving/Control/BicycleModel.html . Accessed: 2023-1009.

Spong, M. W., Hutchinson, S., Vidyasagar, M., et al. (2006). Robot modeling and control , volume 3. Wiley New York.

Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. (2009). Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995 .

Sussex, S., Makarova, A., and Krause, A. (2023). Model-based causal bayesian optimization. In International Conference on Learning Representations (ICLR) .

Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction . MIT press.

Tallec, C., Blier, L., and Ollivier, Y. (2019). Making deep q-learning methods robust to time discretization. In International Conference on Machine Learning , pages 6096-6104. PMLR.

Thomas, J., Welde, J., Loianno, G., Daniilidis, K., and Kumar, V. (2017). Autonomous flight for detection, localization, and tracking of moving targets with a small quadrotor. IEEE Robotics and Automation Letters , 2(3):1762-1769.

Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294.

Todorov, E. and Li, W. (2005). A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005. , pages 300-306. IEEE.

Treven, L., Wenk, P., Dorfler, F., and Krause, A. (2021). Distributional gradient matching for learning uncertain neural dynamics models. Advances in Neural Information Processing Systems , 34:29780-29793.

Umlauft, J. and Hirche, S. (2019). Feedback linearization based on gaussian processes with eventtriggered online learning. IEEE Transactions on Automatic Control , 65(10):4154-4169.

Vakili, S., Khezeli, K., and Picheny, V. (2021). On information gain and regret bounds in gaussian process bandits. In International Conference on Artificial Intelligence and Statistics , pages 82-90. PMLR.

Vamvoudakis, K., Vrabie, D., and Lewis, F. (2009). Online policy iteration based algorithms to solve the continuous-time infinite horizon optimal control problem. In 2009 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning , pages 36-41. IEEE.

Vrabie, D. and Lewis, F. (2009). Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neural Networks , 22(3):237-246.

Vrabie, D. and Lewis, F. L. (2008). Adaptive optimal control algorithm for continuous-time nonlinear systems based on policy iteration. In 2008 47th IEEE Conference on Decision and Control , pages 73-79. IEEE.

Wagener, N., Cheng, C.-A., Sacks, J., and Boots, B. (2019). An online learning approach to model predictive control. arXiv preprint arXiv:1902.08967 .

Wagner, J., Mazurek, P., Mi˛ ekina, A., and Morawski, R. Z. (2018). Regularised differentiation of measurement data in systems for monitoring of human movements. Biomedical Signal Processing and Control , 43:265-277.

Wang, H., Zariphopoulou, T., and Zhou, X. Y. (2020). Reinforcement learning in continuous time and space: A stochastic control approach. The Journal of Machine Learning Research , 21(1):8145-8178.

Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA.

Williams, G., Aldrich, A., and Theodorou, E. A. (2017). Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics , 40(2):344-357.

Yildiz, C., Heinonen, M., and Lähdesmäki, H. (2021). Continuous-time model-based reinforcement learning. In International Conference on Machine Learning , pages 12009-12018. PMLR.

Contents of Appendix

A Theory

We begin in Appendix A.1 by deriving the cumulative regret bound of Proposition 1 which depends on the model complexity. In Appendix A.2, we bound the model complexities of the presented MSSs. Finally, in Appendix A.3, we bound the measurement uncertainty when a GP is used as the statistical model, yielding sublinear regret guarantees for many commonly used kernels. We include useful facts and inequalities in Appendix A.4.

A.1 Bounding Regret with the Model Complexity

The main goal of this section is to derive Proposition 1 which relates the cumulative regret R N ( S ) to the model complexity I N ( f ∗ , S ) . In what follows, the quantifier 'with high probability' refers to 'with probability at least 1 -δ as in Definition 2'.

Lemma 4. Assuming x n (0) = ̂ x n (0) , the distance between the true and the hallucinated trajectory at any time t ≥ 0 is bounded with high probability by and

Proof.

where the final inequality follows from Lemma 15 and Definition 2 (with high probability). We obtain Equation (8) by applying Grönwall's inequality (Fact 14).

Equation (9) is obtained analogously by expanding with f ∗ ( ̂ z n ( s )) rather than f n -1 ( z n ( s )) . Lemma 5. For any MSS S , the regret of any episode n is bounded with high probability by

Proof. By definition of π n in OCORL we have with high probability, C ( f ∗ , π ∗ ) ≥ C ( f n -1 , π n ) . Therefore, with high probability,

By Lemma 15, we further have that

By Lemma 4 (with high probability), we further have that

Now we are ready to prove Proposition 1.

Proof of Proposition 1. Let us first bound R 2 N ( S ) . By the Cauchy-Schwarz inequality,

By Lemma 5 (with high probability), we have that

Taking the square root, we obtain

The result follows by noting that

A.2 Bounding Model Complexities

In this section, we derive the model complexity bounds for the presented MSSs. Our bounds are based on the approximation of integrals using an upper Darboux sum.

Fact 6 (Upper Darboux approximation) . Given any a ≤ b , a function f : R → R , and a partition P = ( a, . . . , b ) of [ a, b ] into k sub-intervals,

where P [ i ] denotes the i -th sub-interval and ∆ i denotes the length of P [ i ] .

We are now ready to bound the model complexities.

Lemma 7 (Oracle model complexity) . For any N ≥ 1 ,

Proof.

For each episode n , use an upper Darboux approximation (Fact 6) with k = 1 to obtain

Lemma 8 (Equidistant model complexity) . For any N ≥ 1 ,

Proof.

For each episode n , let P n def = (0 , ∆ n , 2∆ n , . . . , T ) with ∆ n def = T m n be a partition of [0 , T ] into m n sub-intervals, each of length ∆ n . Using the upper Darboux approximations (Fact 6) of ∥ σ n -1 ( · ) ∥ 2 with respect to P n , we have

Observe that t n,i ∈ [( i -1)∆ n , i ∆ n ] , and hence, using that ∥ σ n -1 ( · ) ∥ 2 is L σ 2 -Lipschitz continuous,

In the remainder of this section, we study the model complexity of the adaptive MSS. In each episode n , we partition [0 , T ] into equally sized sub-intervals (called 'buckets') of length ∆ n , where ∆ n is the solution to

where Γ n (∆ n ) def = 2 β n L σ (1 + L π ) e L f (1+ L π )∆ n . As Γ n is a monotonically increasing function of ∆ n it is clear that a suitable ∆ n exists. Moreover, it follows from ∆ n ≤ T that Γ n ∈ O ( β n ) , and hence that m n = T/ ∆ n ∈ O ( β n ) . Without loss of generality, we assume m n ∈ N .

The definition of the adaptive MSS can be interpreted as follows: In every bucket, we take two measurements. One measurement is saved and used later for updating the statistical model. The other measurement is taken at the beginning of every bucket, and used to inform OCORL of the current state position. For the purposes of our theoretical analysis, these initial measurements are not saved and not used for the construction of the statistical model.

Lemma 9. For any episode n , bucket i ∈ { 1 , . . . , m n } , and t ∈ [( i -1)∆ n , i ∆ n ] where the real and hallucinated trajectories are 'synced' initially, i.e., ̂ z n (( i -1)∆ n ) = z n (( i -1)∆ n ) , we have with high probability that

Proof. Using that σ n -1 is L σ -Lipschitz continuous and applying Lemma 15,

∥ σ n -1 ( ̂ z n ( t )) -σ n -1 ( z n ( t )) ∥ ≤ L σ (1 + L π ) ∥ ̂ x n ( t ) -x n ( t ) ∥ . The result then follows from Lemma 4, where the lower integral bounds can be tightened to ( i -1)∆ n as the real and hallucinated trajectories are 'synced' in the beginning of the bucket.

Lemma 10. For any episode n and i ∈ { 1 , . . . , m n } , if t n,i is selected according to the adaptive MSS then with high probability,

Proof. By applying Lemma 9 we have (with high probability) for every t ∈ [( i -1)∆ n , ∆ n ] ,

Combining Equations (18) and (19), we obtain

Corollary 11 (Adaptive model complexity) . For any N ≥ 1 ,

Proof. The result follows analogously to the proof of Lemma 8, using Lemma 10 in the last step.

A.3 Bounding Measurement Uncertainty of GPs

Our goal in this section is to prove Theorem 3.

The informativeness of a set of sampling points A ⊂ Z about a function f ∼ GP (0 , k ) with the noisy observations y A = f A + ϵ A can be measured by the information gain (c.f., Cover (1999); Srinivas et al. (2009)),

which quantifies the reduction in entropy of f A when observing y A . Here, we write f A = [ f ( x )] x ∈ A and ϵ A ∼ N ( 0 , σ 2 I ) . For a Gaussian, H( N ( µ , Σ )) = 1 2 log det(2 πe Σ ) , so that when f is a Gaussian process, I( y A ; f ) = 1 2 log det( I + σ -2 K AA ) where K AA = [ k ( x , x ′ )] x , x ′ ∈ A . We denote the maximal information gain by observing n points by

Clearly, γ n depends on the kernel k . In Table 2, we state the magnitude of γ n for different kernels. We take the magnitudes from Theorem 5 of Srinivas et al. (2009) and Remark 2 of Vakili et al. (2021).

The maximum information gain can be interpreted analogously in the setting where f ∈ H k,B , see Srinivas et al. (2009) and section 5.2 of Kirschner and Krause (2018). The following result relates mutual information and epistemic uncertainty.

Lemma 12 (Lemma 5.3 from Srinivas et al. (2009)) . For any A ⊂ Z , if f ∼ GP (0 , k ) or f ∈ H k,B , and if the observation noise is i.i.d. N (0; σ 2 ) then

where { x 1 , . . . , x n } def = A .

In our setting, denote by A N def = ( S 1 , . . . , S N ) the set of times of observations D 1: N until episode N . We define for every episode n ,

We write t ∗ n def = t n,i ∗ n and ˜ A N def = ( { t ∗ 1 } , . . . , { t ∗ N } ) . Note that ˜ A N [ n ] ⊆ A N [ n ] ( ∀ n ) and ˜ A N comprises exactly N observations.

Lemma 13. If f j ∼ GP (0 , k ) for all j ∈ { 1 , . . . , d x } or if f ∈ H d x k,B , and if the observation noise is i.i.d. N ( 0 ; σ 2 I ) then

where ¯ σ = σ 2 max log(1+ σ -2 σ 2 max ) , σ 2 max = max z ∈Z ,j ∈{ 1 ,...,d x } σ 2 0 ,j ( z ) , and γ N is with respect to kernel k .

Proof. We have

This inequality is tight when m n = 1 ( ∀ n ) . When more than one measurement is obtained per episode then the regret may be smaller. Expanding the squared Euclidean norm, yields

Write ˜ σ 2 n,j ( z ) def = Var[ f j ( z ) | ˙ y ˜ A N ] whereas σ 2 n,j ( z ) = Var[ f j ( z ) | ˙ y A N ] . The variance is monotonically decreasing as one conditions on more observations, ˜ σ 2 n,j ( z ) ≥ σ 2 n,j ( z ) ( ∀ n, j, z ) , and hence,

Again, this bound is tight when only one measurement is obtained per episode, but may be loose otherwise. Lemma 15 of Curi et al. (2020) shows that ˜ σ 2 n,j ( z ) ≤ ¯ σ log(1 + σ -2 ˜ σ 2 n,j ( z )) for any n, j, z . Applying this inequality, we obtain

Proof of Theorem 3. We derive the regret bound for the adaptive MSS. The regret bounds for the other MSSs follow analogously.

From Corollary 11, recall the model complexity bound

By Lemma 13, we can further bound this by

Combining this bound with Proposition 1, with high probability the regret is bounded by

A.4 Useful Facts and Inequalities

Fact 14 (Grönwall's inequality, theorem 1.1 in chapter 3 of Hartman (2002)) . Let g ( t ) be a nonnegative continuous function on the interval [ a, b ] ⊂ R , and let C, K be a pair of non-negative constants. If the function g satisfies then we have

Lemma 15. For any two metric spaces ( X , d X ) and ( Y , d Y ) , and any two Lipschitz continuous functions f : X ×X → Y and g : X → X with Lipschitz constants L f and L g respectively, we have that f ( · , g ( · )) is L -Lipschitz continuous with respect to d Y with L def = L f (1 + L g ) .

Proof. Fix any x , x ′ ∈ X . By the triangle inequality,

B Solving Optimistic Optimal Control Problem

The optimal control problem solved by OCORL is constrained to L f -Lipschitz continuous dynamics. When the dynamics are modeled with a GP using a linear kernel, the control problem can easily be solved directly in weight-space. Beyond this special case, it is commonly assumed that there exists an oracle which solves this optimization problem exactly (see, e.g., assumption 1 of Kakade et al. (2020)).

In this work, we do not explicitly address the problem of approximating the optimal control problem reasonably well. Kakade et al. (2020) give a brief overview of gradient- and sampling-based methods which may be useful (Jacobson and Mayne, 1970; Todorov and Li, 2005; Mordatch et al., 2012; Williams et al., 2017; Wagener et al., 2019). They alternatively suggest to use Thompson sampling (Thompson, 1933; Osband and Van Roy, 2014), that is, to sample f n -1 from M n -1 (which is straightforward when the dynamics are modeled by a GP, see Williams and Rasmussen (2006)) and then to compute and execute the optimal policy π n = argmin π ∈ Π C ( π , f n -1 ) using a planning oracle. Kakade et al. (2020) conjecture that corresponding regret bounds can be derived using standard techniques for analyzing Bayesian regret of Thomposon sampling (Russo and Van Roy, 2014, 2016).

B.1 Application to Arbitrary Discretizations

In this section, we briefly show that our guarantees carry over to discretized dynamics and costs for arbitrary discretizations. Importantly, in the discrete-time setting the lemma mirroring Lemma 4 does not require the assumption that the models f n are L f -Lipschitz continuous, hence, simplifying the optimistic optimal control problem solved by OCORL. In the discrete-time setting, it is sufficient to assume that f ∗ is L f -Lipschitz continuous.

Consider the dynamical system

with initial state x 0 ∈ X where τ k ∈ R > 0 measures the 'time' between control points x k and x k +1 . We make the assumption that h ∗ is L h -Lipschitz continuous in all arguments. The (discrete-time) control problem is

In many applications, the cost to be minimized is defined at discrete control points to begin with. In this case, the continuous-time control of Equation (1) can be approximated arbitrarily well with Equation (28) by choosing a sufficiently dense discretization. Commonly, the discretization is taken to be equidistant, i.e., τ k ≡ τ ( ∀ k ) , but we remark that our results hold more generally for any discretization.

B.1.1 Bounding Regret with the Model Complexity

In the following, we denote by x n,k the k -th state visited during episode n .

Lemma 16. Assuming x n, 0 = ̂ x n, 0 , the distance between the true and the hallucinated trajectory at any iteration k ≥ 0 is bounded with high probability by where we write L ′ h def = L h (1 + L π ) and L ′ σ def = L σ (1 + L π )

Proof (based on Lemma 4 of Curi et al. (2020)). We begin by showing by induction that for any k ≥ 0 ,

The base case is implied trivially. For the induction step, assume that Equation (30) holds at iteration k . We have

Using that σ n -1 is L σ -Lipschitz continuous and applying Lemma 15,

By the induction hypothesis,

Thus, Equation (30) holds. Since k ≤ T , we have

Lemma 17. For any MSS S , the regret of any episode n is bounded with high probability by

where L ′ c def = L c (1 + L π ) .

Proof. By definition of π n in OCORL we have with high probability, C ( f ∗ , π ∗ ) ≥ C ( f n -1 , π n ) . Therefore, with high probability,

By Lemma 15, we further have that

By Lemma 16 (with high probability), we further have that

, we have with

The above results allow us to prove a general regret bound which is analogous to Proposition 1. Lemma 18. Let S be any MSS. If we run OCORL with arbitrary discretization h ∗ high probability, where we define the discrete-time model complexity,

Proof. Let us first bound R 2 N ( S ) . By the Cauchy-Schwarz inequality,

By Lemma 5 (with high probability), we have that

Taking the square root, we obtain

The result follows by noting that

B.1.2 Bounding Model Complexities

The model complexity bounds from the continuous-time setting (c.f., Appendix A.2) extend seamlessly to the discrete-time setting. In the following we briefly sketch the bound for the oracle MSS.

Lemma 19 (Discrete-time oracle model complexity) . For any N ≥ 1 ,

Proof.

Note that the MSS S can (but not has to) be restricted to the discretization of [0 , T ] used for the dynamics and costs. The derivations for the equidistant and adaptive MSSs follow analogously, where we note that the bucket length ∆ n is to be defined with respect to a modified constant Γ n .

Finally, applying the measurement uncertainty bound presented in Appendix A.3, we obtain the following theorem.

Theorem 20. Fix an arbitrary discretization h ∗ and assume that h ∗ ∈ H d x n,B , the observation noise is i.i.d. N ( 0 ; σ 2 I ) , and let ∥·∥ be the Euclidean norm. We model h ∗ with the GP model. The regret for the adaptive MSS is with probability at least 1 -δ bounded by

C Experimental Setup

C.1 System's dynamics

Here we either give reference to the equations of the dynamical system or provide their equations:

Cancer Treatment:

Glucose in Blood:

Pendulum:

Mountain Car:

Cart Pole: Equation 6.1 and 6.2 of Kelly (2017)

Furuta Pendulum: Equation 18 of Gäfvert (2016)

Bicycle: Dynamics from (Singh and Theers, 2021)

Quadrotor 2D: Equations 1, 2, 3 of Paing et al. (2020)

Quadrotor 3D: Equations 12, 13, 14 of Thomas et al. (2017)

C.2 System's tasks

In all considered settings the running cost has the following quadratic form:

Here, x T is the target state, and u T is the target control. Descriptively, in cancer treatment and glucose in Blood systems, our objective is to reduce the prevalence of unhealthy cells, while striving to be as minimally invasive as possible. For the pendulum, Furuta pendulum, and cart pole systems our goal is to skillfully swing the pendulum upwards from its lowest point. In the mountain car system, the challenge is to guide the car from the valley's depth to the hill's summit. Finally, in systems bicycle, quadrotor 2D, and quadrotor 3D our objective is to adeptly navigate the agent to reach a desired target state.

C.3 Adaptive MSSs

In this section, we describe in detail the two adaptive MSSs from Section 5.

Greedy Max Determinant In the Greedy Max Determinant adaptive strategy we solve the following objective:

Here, K n S def = [ k n ( ̂ z n ( t ) , ̂ z n ( t ′ ))] t,t ′ ∈ S and k n ( z , z ′ ) def = k ( z , z ′ ) -k ⊤ n ( z ) ( K n + σ 2 I ) -1 k n ( z ′ ) . The goal here is to find at every episode n a set S n of M time points from [0 , T ] . Solving problem (35) is NP-hard and we approximate it by solving it greedily:

Algorithm 1 GREEDY MAX DETERMINANT

Greedy Max Kernel Distance The second adaptive MSS we consider in Section 5 is the Greedy Max Kernel Distance . Here we greedily solve the metric M -center problem in space [0 , T ] with the kernel metric: d n ( t, t ′ ) 2 def = k n ( ̂ z n ( t ) , ̂ z n ( t )) + k n ( ̂ z n ( t ′ ) , ̂ z n ( t ′ )) -2 k n ( ̂ z n ( t ) , ̂ z n ( t ′ )) on the hallucinated trajectory.

Algorithm 2 GREEDY MAX KERNEL DISTANCE

C.4 Episodes Hyperparameters

C.5 Practical Implementation

In practice, we approach solving the optimal control problem (2) with offline planning and online tracking strategy. Before the episode starts we obtain an open loop trajectory ˜ x n ( t ) , ˜ u n ( t ) by offline planning:

We solve the optimization problem (36) with Iterative Linear Quadratic Regulator (ILQR) (Li and Todorov, 2004). Then we track the open loop trajectory ˜ x n ( t ) online with MPC:

Here, horizon T MPC depends on the system.

We solve MPC tracking problem (37) also with the ILQR.

D Additional Experiments on Number of Measurements

0

5

10

Episode

10

1

10 0

10 1

10 2

Cost

EQUIDISTANT

0

5

10

Episode

MAX_KERNEL_DISTANCE

0

5

10

Episode

MAX_DETERMINANT

M = 2

M = 5

M = 8

M = 10

Best continuous

Best discrete (M=10)

(a) We vary number of measurements we take per episode for the Pendulum system. As expected the more observation results in lower achieved cost for every MSS.

0

5

10

Episode

10

1

10 0

10 1

10 2

Cost

M = 2

0

5

10

Episode

M = 5

0

5

10

Episode

M = 8

0

5

10

Episode

M = 10

EQUIDISTANT

MAX_KERNEL_DISTANCE

MAX_DETERMINANT

Best continuous

Best discrete (M=10)

(b) For the same number of measurments we compare different MSSs. We see that equidistant MSS repeatedly suffers higher cost compare to the adaptive Max Kernel Distance and Max Deteriminant MSS.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission considers online RL, while the true dynamics is continuous-time. To deal with this issue, the authors provide a continuous-time model-based method. The proposed method is special by \n1. iteratively fitting ode-based models and deriving/learning the control from the fitted models \n2. having the option to adaptively selecting the sampling timing. \n\nThe proposed method is able to well handle problems with continuous-time true dynamics, with both theoretical guarantees and empirical evidence.\n\nSTRENGTHS:\nThe submission is novel. The proposed method incorporates continuous-time models with adaptive sampling, which to the best of my knowledge is the first. \n\nThe submission provides thorough results with both theoretical guarantees and also empirical experiments. \n\nThe studied problem is well motivated: there are problems where the data is generated in continuous-time in need of specialized methodology like the submission. \n\nThe writing is clear and easy to follow.\n\nWEAKNESSES:\nThe submission is solid, and the adaptive MSS seems to be an important and useful technique. However, there is one concern bugging me, which seems a little crucial: it is not clear whether the continuous modeling is indeed needed. \n\nMy question is twofold: 1. whether it is necessary or practical to assume a continuous-time ground-truth dynamic; 2. whether it is necessary or helpful to use continuous-time models in the method for $f$ \n\nFirst of all, the continuous dynamics in section 2 and equation (1) may never be empirically achieved or used. There are very rare cases where one can really implement a policy in a continuous-manner as that equation. As a result, the policy functions considered will often time be piece-wise constant, which does not fit (1). It is not clear how this may affect the theory. When evaluating a policy, equation (1) may also need to be discretized. Therefore, in each episode, one can also fit a discrete-time model instead of a continuous-time one as suggested by the authors. I am wondering how the proposed method compares to this discrete-time version of the proposed method. \n\nSecond, the theoretical results like proposition 1 also holds for the discrete dynamics setting. The adaptive MSS may also be feasible in the discrete-time setting, which itself now is implemented in a discretized version. \n\nTherefore, the observations above make me wonder whether the continuous-time part is really needed in the submission.\n\nQUESTIONS:\n1. In the experiments, are the cost function collected in a discrete-time manner or continuous-time manner? More specifically, are the results summation of costs or the integral of costs over time? \n\n2. What difference might be, if we replace the ODE model by a discrete-time model, for example, like iLQR. Please demonstrate the differences both theoretically and empirically. \n\n3. In experiments why does the proposed method with equidistant perform better than discrete zero-order hold OC? Are the two policy implemented using the same time interval on the same grid? Or does the proposed method calculates the integral over time of costs? Or does it use a different time interval than the competing discrete-time policy? What is the definition of the discrete zero-order hold OC?  \n\n4. Is it possible to implement MSS with a discrete-time assumed model? For example, I can assume that the true model is a discrete-time model which is just same as the Euler discretized version of the ODE in the submission, and then conduct adaptive MSS like in the submission. What is the problem of this naive ablation?\n\n5. Would it be possible to explicitly list the dynamics of the experiments? The cost function is provided in the supplements but not the dynamics. I am concerned that the considered experiments are too simple like all linear ODEs, which may make the method not general enough.   \n\nAs a summary, the submission definitely has novelty and contributions. However, I feel that the current method is not demonstrated clear enough, especially on what role the continuous-time modeling is playing in the method.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a continuous time framework for model based reinforcement learning. Their algorithm OCoRL solves the optimal control problem eq (1) by: 1, selecting optimistic policy. 2, rollout to collect data. 3, update model estimation and statistics. Specifically, they study the measurement selection strategy (sampling state-action in the continuous time framework), and show both theoretically and empirically the effect of different measurement selection strategy.\n\nSTRENGTHS:\nThe paper models the optimal control or reinforcement learning problem in continuous time (eq (1)), which is elegant and also convenient for algorithm design and analysis. The realization of state-action at discrete time steps is naturally modeled as discrete-time sampling from the continuous time trajectory (the measurement selection strategy (MSS)). The proposed algorithm alternates between: optimistic planning given current statistical model and data collection for a more accurate model. They further proposed the adaptive MSS that samples state-action time points based variance of model estimate in the state-action space (rollout trajectory).\n\nBoth theoretical and empirical analysis are provided and showcase the impact of different discrete sampling strategies.\n\nWEAKNESSES:\nThe algorithm enjoys elegance and nice theoretical properties, but it can be difficult to realize in practice. Especially, the first step in OCoRL that solves optimistic policy can be very time-consuming in practice. The authors mentioned in Appendix C that it is approximately solved by Iterative Linear Quadratic Regulator, but still it can be impractical.\n\nThe experiments mainly focus on investigating the impact of measurement selection strategies. A comparison in terms of performance: time/computation complexity and over all cost with existing approach (PPO, etc) would better help readers evaluate this approach\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a continuous-time model-based reinforcement learning method for controlling fully observed dynamical system environments where there is a cost to take a sample of the state. A Gaussian process dynamics model is used, and a novel adaptive measurement selection strategy is proposed to determine when to take a sample of the state, such that the overall optimization converges to the optimal policy in the limit of infinite trials. The proposed method (OCoRL) is theoretically analyzed to show a general regret bound holds for any measurement selection strategy and is empirically verified across a range of dynamical system environments.\n\nSTRENGTHS:\n* The proposed method OCoRL and associated general regrets bound that holds for any measurement selection strategy appear novel.\n* The paper is well-written and clearly laid out.\n* The no-regret algorithm for nonlinear dynamical systems in the continuous-time RL setting seems widely applicable and relevant to the RL and ML community.\n* The code is reproducible and easily extendable, being well documented throughout.\n\nWEAKNESSES:\n* How does the proposed method perform when the state differential is not observed $\\dot{x}_n(t)$ and has to be inferred? Could you perform an ablation of this to show that the proposed method is still practical? \n* Why is the noise only added to the observed state derivatives $\\dot{x}_n(t)$? Perhaps it could be more realistic to consider noise added to both the observed state $x_n(t)$ and the observed state derivative $\\dot{x}_n(t)$?\n* Line 118: \"predicted mean and epistemic uncertainty\". How does the model guarantee that you only measure the \"epistemic uncertainty\" and not the \"epistemic uncertainty and aleatoric uncertainty\"? This was not clear, and I presume the model learns both. If so, how can you split the uncertainty to only use the epistemic uncertainty, as outlined in the method?\n* The adaptive MSS assumes that $m_n=\\left \\lceil{T/ \\Delta_n}\\right \\rceil$, i.e., that a sample must be taken in each uniform interval $\\Delta_n$ of time. This seems overly restrictive. Can OCoRL be adapted to skip taking a sample in some $\\Delta_n$ of time, i.e., where they are not needed or informative?\n  * On this, can a further ablation be performed where $m_n$ is varied across all the environments and baselines to empirically verify the adaptive MSS claims for wide ranges of $m_n$?\n* Unsubstantiated claim of \"We compare the adaptive and equidistant MSSs on all systems and observe that the adaptive MSSs consistently perform better than the equidistant MSS\". Table 1, shows that adaptive MSSs can perform better than equidistant MSS only in certain environments and that equidistant MSS achieves the same performance (final cost) within error bounds for the environments of Cancer Treatment, Pendulum, and Mountain Car.\n\nQUESTIONS:\n* Could this approach be generalized to other environments that are not implicitly defined by an ODE, such as other types of differential equations, e.g., delayed differential equations, stochastic differential equations, etc? Furthermore, can this approach work for partially observable environments?\n* What is the trade-off of varying the number of measurements taken, i.e., an ablation of varying M for all experiments? Does the algorithm still hold under these settings? Can this be demonstrated empirically?\n* Could the OCoRL method be benchmarked against the closest related works of Yildiz et al. (2021) and Du et al. (2020)?\n* (From above): Can OCoRL be adapted to skip taking a sample in some $\\Delta_n$ of time, i.e., where they are not needed or informative?\n* Typo: Line 339: \"Figure 1\" -> \"Figure 2\".\n\nLIMITATIONS:\nThe limitations are addressed with the assumptions outlined in Section 2.1.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel algorithm for efficient exploration in continuous-time model-based reinforcement learning. The algorithm represents continuous-time dynamics using nonlinear ODEs and captures epistemic uncertainty using probabilistic models. The analysis shows that the approach achieves sublinear regret with significantly fewer samples, making it a promising solution for various applications. The paper also develops a practical adaptive measurement selection strategy that reduces the number of measurements per episode while retaining regret guarantees. The benefits of continuous-time modeling and planning with optimism are demonstrated in several environments. The authors aim to catalyze further exploration within the RL community regarding the potential of modeling dynamics in a continuous-time framework.\n\nSTRENGTHS:\nThis paper appears to be one of the pioneering works in the field of continuous-time reinforcement learning (RL) applied to nonlinear dynamical systems.\n\nFurthermore, the utilization of epistemic uncertainty for measurement selection, a technique commonly employed in active learning but rarely explored in RL, adds a unique and valuable dimension to this study.\n\nIn addition to these notable contributions, the paper introduces several innovative techniques and investigates their efficacy across various experimental environments.\n\nWEAKNESSES:\nThe paper encompasses a wide range of techniques, including model-based RL, continuous-time RL, and aperiodic strategies. However, the abundance of new content may make it challenging to write and comprehend the relationship between each technique and the overall idea. To enhance clarity and understanding, it would be beneficial to reorganize the paper and provide clearer connections between the different techniques employed.\n\nRegarding Table 1, it appears that, under a known true model, discrete-time control outperforms continuous-time OC without the inclusion of MSS. Consequently, it may not be fair to directly compare continuous-time OC with MSS and discrete-time control since the conditions and components differ significantly between these approaches. A more appropriate comparison could be made between continuous-time OC and discrete-time control without MSS.\n\nTo provide a more comprehensive evaluation, it is advisable to compare this method with a broader range of other proposed RL methods. The current results may not fully demonstrate the performance of OCoRL, and including additional comparisons would enhance our understanding of its capabilities.\n\nIn Figure 3, it would be beneficial to consider incorporating more environments for experimentation. Relying solely on a single environment may not provide enough evidence to draw robust conclusions. Including a broader set of environments would strengthen the validity and generalizability of the findings.\n\nQUESTIONS:\nIn which scenarios are measurements considered costly? Have you discussed in the Introduction whether there are enough scenes that truly require continuous-time systems due to costly measurements?\n\nIn the problem setting, the computation of cumulative regret relies on an optimal policy. However, if the optimal policy is unattainable for a particular problem, can your method still be applied? Please provide an explanation.\n\nRegarding Line 122, is it possible for re-calibration techniques to provide accurate uncertainty estimation while maintaining a low time complexity?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}