Sample-efficient and Scalable Exploration in Continuous-Time RL
Authors: Klemens Iten
Lenart Treven
Bhavya Sukhija
Florian Dorfler
Andreas Krause
Venue: ICLR 2026 Poster (2026)
Keywords: continuous-time reinforcement learning
model-based RL
intrinsic rewards
epistemic uncertainty
exploration-exploitation trade-off
Primary area: reinforcement learning
TL;DR: We propose COMBRL, a scalable continuous-time RL algorithm that balances reward and epistemic uncertainty using intrinsic rewards, achieving sublinear regret and strong performance in both supervised and unsupervised settings.

Abstract: Reinforcement learning algorithms are typically designed for discrete-time dynamics, even though the underlying real-world control systems are often continuous in time. In this paper, we study the problem of continuous-time reinforcement learning, where the unknown system dynamics are represented using nonlinear ordinary differential equations (ODEs). We leverage probabilistic models, such as Gaussian processes and Bayesian neural networks, to learn an uncertainty-aware model of the underlying ODE. Our algorithm, COMBRL, greedily maximizes a weighted sum of the extrinsic reward and model epistemic uncertainty. This yields a scalable and sample-efficient approach to continuous-time model-based RL. We show that COMBRL achieves sublinear regret in the reward-driven setting, and in the unsupervised RL setting (i.e., without extrinsic rewards), we provide a sample complexity bound. In our experiments, we evaluate COMBRL in both standard and unsupervised RL settings and demonstrate that it scales better, is more sample-efficient than prior methods, and outperforms baselines across several deep RL tasks.

Published as a conference paper at ICLR 2026
SAMPLE-EFRICIEN AND SCALABLE EXPLORATION IN CONTINUOUS- TIME RL
Klemens Iten, Lenart Treven, Bhavya Sukhija, Florian Dörfler, Andreas Krause ETH Zürich, Zürich, Switzerland (kiten, trevenl, sukhijab, dorfler, krausea] @ethz.
ABSTRACT
Reinforcement learning algorithms are typically designed for discrete-time dynam- ics, even though the underlying real-world control systems are often continuous in time. In this paper, we study the problem of continuous time reinforcement learn- ing, where the unknown system dynamics are represented using nonlinear ordinary differential equations (ODEs). We leverage probabilistic models, such as Gaussian processes and Bayesian neural networks, to learn an uncertainty-on underlying ODE. Our algorithm, COMBRL, greedily maximizes a weighted sum of the extrinsic reward and model epistemic uncertainty. This yields scalable and sample-efficien approach to continuous- time model-based RL. We show that COM- BRI. achieves sublinear regret in the reward-driven setting. and in the unsupervised RL setting (i.e., without extrinsic rewards). In our experiments, we evaluate COMBRL in both standard and unsupervised RL settings and demonstrate that it scales better, is more -efpeeeffficient than prior methods, and outperforms baselines across several deep RL tasks.
model of the
provide sample complexity bound.
INTRODUCTION
Reinforcement learning (RL) has proven be flexible paradigm for learning control policies through interaction, with success stories across robotics (Levine et al. 2016 Hwangbo et al. 2019)
Spiridonov et al.] 2024), games (Schritwwieser al., 2020 Hafner al. 2025). J. and applications in medicine and energy Yu et al. 2021] Degrave et al. 2022). In most RL algorithms, time discretized: agents select actions at fixed intervals and the system evolves in discrete steps. However, many real-world control systems, from physical robots biological processes, naturally modeled by continuous-timg dynamics governed by ordinary differential equations (ODEs). Discretization can obscure key temporal behaviours and limit control flexibility, whereas continuous-time models align better with weal-world sensing and actuation.
In this work, we study continuous-tin time model-based reinforcement learning, where the goal is to interact with an unknown dynamical system and use the data collected on the system to learn the underlying ODE. We address two settings: (i) reward-driven RL, where the goal is to solve a specific task; and (ii) unsupervised RI, where the objective is to learn the system dynamics globally. The latter requires accurate global modeling. while the former demands task-relevant accuracy.
We propose COMBRL, continuous time model-based RL algorithm that uses probabilistic models, e.g., Gaussian processes and Bayesian neural networks, to capture the epistemic uncertainty of the learned model. Policies are selected by maximizing weighted sum of extrinsic reward and epistemic uncertainty, following the optimism-in iiwiiiss-m---h----eee offe-----e--off------o--
ofce-of-uncertaint principle (Curi al., 2020 Kakade et al. 2020 Treven et al. 2023 Sukhija et al. 2025b). In the unsupervised setting olten encountered
in system identification (Astrmm and Eykhoff 1971 and in unsupervised exploration (Aubret et al.
2019: Sekar et al.] 2020 Sukhija al. (2023). COMBRL reduces to active learning (Taylor et al.
202T). specifically uncertainty sampling (Lewis and Catleti 1994) and guides the agent toward
regions where the model is most uncertain.
We show that COMBRL achieves sublinear regret in the woward-driven case and provide sample complexity bounds in the unsupervised setting. Empirically, we evaluate COMBRL on several deep continuous-time RL tasks, demonstrating improved performance and sample efficiency compared to baselines.

---

Published as a conference paper at ICLR 2026
Unlike discrete-time RL, where interaction occurs fixed grid and uncertainty is evaluated only visited steps, continuous-time reinforcement learning requires the agent to additionally decide when to observe and control the system. As a result, uncertainty accumulates along entire trajectories, and regret depends not only the policy but also the measurement schedule. Prior continuous approaches typically enforce optimism by optimizing jointly over policies and plausible dynamies, which is computationally demanding. In contrast, COMBRL realizes optimism through an intrinsic reward and directly trades off reward and epistemic uncertainty, yielding a scalable exploration mechanism whose guarantees explicitly depend on the measurement strategy.
Contributions
We propose COMBRL, a continuous-time optimistic model-based RL algorithm that balances task performance and model exploration. Unlike prior continuous methods, which either purely exploitative or rely on costly co-optimization, COMBRL uses a single sealar to balance reward and epistemic uncertainty, supporting both reward-driven and unsupervised learning.
We provide theoretical guarantees, showing sublinear regret in the reward-driven case and offer sam- ple complexity bounds the unsupervised setting, with explicit dependence on the measurement selection strategy.
We demonstrate strong empirical results across continuous-time deep RL benchmarks, showing that COMBRL scales better than prior methods and generalizes to unseen downstream tasks.
PROBLEM SETTING
Consider an unknown continuous- time dynamical system f (x((), utt) with initial state x(0) = 27p X' Rdd and control input u [0, + U C Rdd, The state at time t obtained by integrating the deterministic dynamics:
xxt) = =x0 xo + (' f (x(s), u(s)) ds.
Furthermore, consider policy π X 11 so that the control input follows said policy, i.e., u(l) π (t)). Typically in optimal control (OC, cf. Luenberger (1979)), we consider the associated finite-time OC problem over the policy space II to solve for the optimal policy such that objective function J JT,, f*) is maximized:
π def arg max J (TT, f*) = arg max
ds
TE
TEll
s.t. a(1) = f(x(), (<)())),
x00) - æo, (x(11, u(() E x' xu, LE [0,
To gather information about the unknown dynamics (ax(), u())), we collect data over episodes 1, I.....N. N. In each episode, we select a policy π. E II by optimizing an objective (e.g., task reward exploration) and deploy it for the horizon T.
Optimizing solely for the reward function r(=(t), ut)) during learning introduces a directional bias, since f is mostly explored in high-reward regions of the state-action space, resulting in limited exploration and poor generalization of the leamed dynamics. However, in many practical scenarios, such as unsupervised RL or system identification, the objective is not to optimize a performance criterion, but to learn the underlying nonlinear ODE f as fast and accurately possible.
In practice, during an episode n., we query f (xx(), ut)) by taking sequence of measurements at m,, selected time points specified by a measurement selection strategy (MSS) S = (Sa)n21
Definition 1 (Measurement selection strategy, Treven et al. (2023)) measurement selection strategy S is sequence of sets (S,),2t. 11 such that S,, contains m., points at which we take measurements, i.e., S, C [0. 71, S., m.
'erre, the S., may depend on observations prior episode is even constructed while execute the trajectory. For case of notation, do not make this dependence explicit.
2

---

Published as a conference paper at ICLR 2026
This gives dataset of measurements Dn (Tn: as follows:
Dn def
((Zn (In,))) Dr (bn,)) In S (1 ...., mnn} where
Zn (tm,)
def (In (tn,))), ne (tn E = = xxu, u (t((),)) def (tn,ss))) + Cn.i-
Since direct measurement of ē may not be feasible, it is typically estimated via finite differences or filtering, and is subject to measurement noise En Thus, at episode n, the collected dataset up to the current episode Dinn- == - D; informs the policy n, deployed for horizon T. Note that, because we defined a continuous-time setting, we are not restricted to a fixed sampling rate and COMBRL allows flexible, event-driven sampling and control.
2.1 RELATED WORK
We briefly mention the most relevant related works below. An extended discussion of related works provided in Appendix A
Yildiz (2021 introduce continuous-time model-based RL approach that greedily maximizes the reward integral. [Teven et al.] (2023) show that greedy exploration performs suboptimally and propose leveraging optimistic dynamics to encourage exploration in uncertain regions. The resulting algorithm, OCORL, enjoys convergence guarantees under common continuity assumptions on the underlying ODE. However, the method does not scale to higher dimensional systems since it relies on challenging optimization
the set of plausible dynamics enforce optimism.
Moreover, both aforementioned works rely on an external reward signal. When no extrinsic reward is available, or when accurate dynamics modeling is prioritized over extrinsic performance, these methods cannot be applied. On the other hand, intrinsic motivation techniques have long been used for this purpose (Salge et al. 2014, Bellemare et al. 2016; Pathak et al. 2017]] Sekar et al.) 2020). In particular, Sukhija et al. (2023) show convergence of intrinsic exploration methods in discrete time. However, their continuous-tima counterpart much less understood.
To this end, we present COMBRL, a scalable and efficient continuous-tine model-based RI, algo- rithm with a flexible optmismmdrrvvn objective that enables natural transition between reward -divee (extrinsic) and incertainty-driverr (intrinsic) exploration. Unlike prior continuous-time methods, which are only designed for extrinsic exploration, COMBRL supports both extrinsic and intrinsic settings. In addition, unlike Treven et al. (023)) COMBRI also scales to higher dimensional settings since it does not require optimizing over the set of plausible dynamics. Our work closely related to (Sukhija et al. (2025b), who study the problem in the discrete-time setting. However, our focus on continuous- time systems requires different theoretical analysis and experimental design, discuss in Sections andd
2.2 PERFORMANCE MEASURE
For the supervised RL setting, a natural performance measure is given by the cumulative regret that sums the gaps between the performance of the policy n, episode and the optimal policy πt" over all the episodes:
N
N
R
def
def
J(T,
(2)
#=1
R=1
the cumulative regret Ry is sublinear in N, then the average reward of the policy converges to the optimal reward, and by extension to the optimal policy n².
2.3 ASSUMPTIONS
In the following, we make some common assumptions that allow us to theoretically analyse the regret Ry and prove a regret bound. We first make an assumption on the continuity of the underlying system and the observation noise.
Assumption 1 (Lipschitz continuity). The dynamics model f", reward and all policies π E 11 are Ls, L, and I.r Lipschitz-continnuni respectively.
Assumption 2 (Sub-Gane Lub-Gaussian noise). We assume that the measurement noise En,i is i.i.d. o-sub Gaussian.
3

---

Published as a conference paper at ICLR 2026
The Lipschitz assumption is commonly made for analysing nonlinear systems (Khalil 2014) and is satisfied for many weal-world applications. Furthermore, assuming ø-suu Gaussian noise (Rigollet and Hütter 2023) is also fairly general and is common in both RL and Bayesian optimization literature (Srinivas al. 2012 Chowdhury and Gopalan 2017).
In COMBRI, we learn an uncertainty-awarr model of the underlying dynamics. Therefore, we obtain a mean estimate (x) and quantify our epistemic uncertainty o (z) about the function f":
Definition 2 (Well-calibrated statistical model of f*, Rothfuss al. (202)). Let Z def X U. An all-time nell-calibrated statistical model of the function f is a sequence {Mn where
(5) def (s 2 2 Vz E 2, (1...,,da)) - S Ba(())n.)())}
if. with probability least 1 8, we have f E Mn( Here, pn.j and nn denote the j-th element in the vector-valued mean and standard deviation functions phn and nn respectively, and Bn(5) E I2 a scalar function that depends on the confudence level E (0, 1] and which is monotonically increasing in n.
Assumption 3 Well-calibration). We assume that our learned model is an all-time well-calibrated statistical model of f³. We further assume that the standard deviation functions (orn())))2) are L -Lipschiiz continuous.
Intuitively, Assumption 3 states that we are, with high probability, able capture the true dynamics f within a confidence set spanned by our predicted mean μn, and epistemic uncertainty on and in turn, COMBRL learns probabilistic model fn that provides both mean predictions (z) and uncertainty estimates o, (z). For Gaussian process (GP) models, the assumption is satisfied (Rothfuss et al. 2023] Lemma 3.6), and for more general classes of models such as Bayesian neural networks (BNNs), re-calibration techniques (Kuleshov et al.] 2018) can be used. Thus, COMBRL is -gnesssg: the statistical model can be instantiated using GPs (Rasmussen and Williams 2005 Deisenroth et al., 2015), BNNs (MacKay) 1992), ensembles (Lakshminarayanan al. 2011 or other estimators that capture epistemic uncertainty.
Lastly, we make an assumption on the regularity of the dynamics by placing them in a reproducing kernel Hilbert space (RKHS), which enforces smoothness and boundedness:
Assumption 4 (RKHS Prior on Dynamics). We assume that the functions fj, j E {l...., dæ) lie in a RKHS with kernel k and have a bounded norm B, that is
f E H
with H.. = {f |lfall BBj= =..==1.... LB.jj
Moreover, we assume that k(z, 2) S Fmax for all æ E X.
3 COMBRL: CONTINUOUS TIME OPTIMISTIC MODEL-BASED RL
We now present COMBRL, a continuous-time, optimistic model-based reinforcement learning (MBRL) algorithm. COMBRL proceeds in continuous-tings episodic setting, alternating between learning predictive model of the dynamics from data and selecting policies that trade off extrinsic reward and epistemic uncertainty. The method assumes only access to a simulator or physical system for episodic rollouts and measurements.
3.1 OPTIMISTIC PLANNING OBJECTIVE
In each episode n, select any Ly-isscchizz model from the confidence set M, of the previous episode, i.e., fn E Mn n F, where F is the set of Lipsshhtt functions. We then choose a policy π,, that maximizes a reward-augmented optimistic objective under the current model f.:
L (2'2(s), us)) An || 1 (x (s), u(s)))l ds
n. = arg max *1
(3)
1+ An
s.t. ='(t) fn (z (t), u(t)), u(t) न (x'(t)))
(z(1), ut)) E = X' xll C +d te [0
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 COMBRL: Continuous- Optimistic MBRL
1: Initialize: Statistical model Mu. Simulator SIM(:,), :) Dataset Do 0, measurement selection
strategy 5 = (Sn) intrinsic reward weights (An ) confidence level 8
2: for episode I..... do
3: n + OPTIMIZEPLLICY
Solve optimistic objective from Equation G
subject to - dynamics fn E F
4: Dn t SIM(TT, S.)
Collect rollout using S,
5:
Mn f UPDATEMODEL (Dom 6)
Fit mean Me, and uncertainty dn
The key feature of COMBRL is its pataard-plu-uunnertte objective in continuous time, which enables a principled trade-off between exploration and exploitation. The scalar X, balances reward and model uncertainty. and is treated as a tunable hyperparameter in practice. The epistemic uncertainty term 0- (2) in Equation (3) encourages the agent visit poorly understood regions of the state-action space.
Earlier continuous-ting optimistic methods addressed exploration by jointly optimizing over policies and dynamics f., M I. This is intractable and heuristically addressed using a reparametriza- tion trick from Curi et al. (2020), which increases input dimensionality from d, to d, + dz, limiting scalability in high-dimensional settings, e.g., control from pixels. COMBRL avoids this by selecting any model from M, n F; in practice, using f4, works well Unlike optimistic dynamics or clas- sical control, COMBRL balances exploration and exploitation via single scalar A, and encourages exploration of uncertain regions to improve model fidelity. COMBRL is summarized Algorithm
3.2 THE INTERNAL REWARD WEIGHT Xn
The scalar value X, in Equation 3 determines the trade-off between maximizing extrinsic reward and exploring regions of high model uncertainty. We differentiate between three key scenarios that affect the agent's behaviour:
Greedy (A, 0): Pure exploitation with respect to the given reward function. The model is only updated passively through whatever data results from seward-seeking behaviour, as in prior continuous-tine MBRL approaches (Yildiz et al. 2021).
Balanced (0 < An < oc): The agent balances task reward and model uncertainty. This regime leads to goal-directed yet exploratory behaviour and is the most relevant in practice, reducing epistemic uncertainty and improving model quality over time. We offer some strategies to select
An subsequently.
Unsupervised (A,, oo): The agent ignores the reward and acts purely to reduce uncertainty. This corresponds to an unsupervised RL or active learning setting. This similar to some discrete-time methods that use the model disagreement or epistemic uncertainty an intrinsic reward (Pathak) Sukhija et al. 2023).
et al. 2019 Sekar al., 2020
How to choose A, in the balanced case? For the case An oo, study several practical strategies for setting or adapting An;
Static (hyperparameter): Set Xn = À to a fixed value tuned via cross-validation or grid search.
This is simple and often effective.
Scheduled (annealing): Decrease \n over time, for example A, Ag (1-n/N). This encourages more exploration early in training and more exploitation as the model improves.
Auto-tuned: Use an information-theons
approach such as the auto-tuning procedure described by (Sukhija et al. (2025a), which selects X,, adaptively based on maximizing mutual information gain or related criteria.
In this work, we primarily treat An as a tunable hyperparameter with scheduling. In Section 4 also study the auto-tuning approach and show its effectiveness, as well as the unsupervised RI. case.
'Even though the mean model might not lie in M.-1 F.F. For GP dynamies, show how to pick model from M,- 1 in Appendix B.2
5

---

Published as a conference paper at ICLR 2026
3.3 THEORETICAL RESULTS
The regret any continuous- model-based RL algorithm depends on the hardness of learning the true dynamics f and on the measurement selection strategy (MSS) S from Definition [ Crucially, in continuous-ting RL, measurements they occur; for example, an equidistant MSS chooses uniformly spaced times. For the underlying dynamics f and MSS S, the model complexity is defined as:
be taken at arbitrary times, with the MSS S specifying when
N Σ
IN(s" 5) def
|lon-1 (zn(1)) dt.
mIx W WIHERRNN RN
Ell
Intuitively, for a given NN, the more complicated the dynamics fr, the larger the epistemic uncertainty and thereby the model complexity. Curi et al. (2020) study the model complexity for the discrete-time setting, where the integral is replaced by the sum over uncertaintics. In continuous- time, the MSS S proposes when we measure the system and influences how we collect data to update our model.
Unlike discrete time, where uncertainty is assessed on fixed grid, the continuous-tine model complexity integrates uncertainty along the entire trajectory. To compete with the optimal continuous- time policy, one needs increasingly dense observations along trajectories; hence [S,,] must increase with n. For example, Treven al. (2023) show that for an equidistant MSS with [S,, n, the model complexity is bounded by
[n S) = O(T + log N),
where "YN is the maximum information gain of N noisy observations under the chosen kernel. This growth condition is necessary because, asymptotically, recovering the full ODE requires ever denser coverage of the trajectory, while with fixed per-episode budget the uncertainty integral cannot shrink quickly enough. With this intuition in place, we can now formalize how the regret of COMBRL scales with the model complexity.
Theorem 1. Under regularity assumptions (Assumptions 1. the following holds with a probability of at least 8:
Ry S (OVRR((
Theorem bounds the regret of COMBRL w.r.t. the model complexity. Importantly, this shows that the learning performance depends also on how efficiently the MSS collects informative measurements.
For GP dynamics, where the well-calibration assumption is true and the monotonicity of the variance holds, Treven al. (2023) show that the model complexity In SS) is sublinear for many common kernels and MSSs, e.g., grows with poly log(N) for the RBF kernel and equidistant MSS. Therefore, for common kernels and MSSs, COMBRL enjoys sublinear regret in the GP setting, and the policy converges the optimal policy ".
For the unsupervised setting without an external reward signal, the regret term as defined in Equa- tion (2) does not apply, and instead we analyze the sample complexity of reducing epistemic uncer- tainty.
Theorem 2. Consider the unsupervised setting (An + 6o) and let Assumptions 7 to hold. (n). nn E Z, j & d and 0, then
TEll '' max
( [R( N
T (m))) (x(())) ds 50
(m(ss)
Theorem 2 provides a sample complexity bound for the unsupervised case. Effectively, this shows that pure intrinsic exploration with X,, - co reduces our model epistemic uncertainty with rate of / /N. To the best of our knowledge, we are the first to show this for continuous-tim time RL.
We provide the proofs for all theoretical results in Appendix B

---

Published as a conference paper at ICLR 2026
Pendulum-GP. Swing-up
Mountaincar-GP. Go up
1.0
MPC (true dynamics)
Avg. ep return (normalized)
COMBRI (Ours)
0.5
OCORL
PETS
0.0
Mcan Planning
Episode
10 Ep.
5
10 15
Figure 1: GP dynamics. Learning curves for baselines, COMBRL and OCORL with fixed internal reward weight An using GP dynamics and iCEM planning, averaged over 5 seeds. We report the mean and the standard error bands, and additionally the performance of MPC under the true dynamics as an estimate for the performance of the optimal policy. COMBRL achieves higher asymptotic returns than PETS and mean, while matching or exceeding OCORL at roughly 3x lower computational cost (Appendix
EXPERIMENTS
We evaluate the COMBRL algorithm on various environments from the OpenA//Farama gymnasium benchmark suite (Gym. Brockman et al., 2016;: Towers et al. 2024) and the DeepMind control suite (DMC, Tassa et al. 2018: Tunyasuvunakool et al. 20200. For the dynamics model fn, we use Gaussian processes (GPs) and probabilistic ensembles (PEs) to capture uncertainty about well- calibrated statistical models. Since COMBRL models continuous-time dynamics directly, it also remains agnostic to the solver or discretization scheme, and can in principle accommodate adaptive strategies.
Earlier work has shown that continuous time formulations can outperform discrete-time counterparts in sample efficiency. Building this, our experiments are designed show both (i) the core behavior of COMBRI under standard equidistant sampling and fixed-rate control, and (ii) that principle COMBRL can also be used in the time-adaptive setting, where it requires fewer interactions.
Concretely, we first evaluate COMBRL with equidistant MSS and fixed control rates, solving the continuous-tin time planning problem using discrete-time solvers with fine-grained discretization to ensure accurate approximation of the underlying ODE. Unlike discrete-time RI, however, f.. learns the ODE of the true system f We then extend our evaluation the time-adaptive setting, comparing adaptive MSS against fixed-rate control.
We use fn to generate simulated trajectories for policy training using soft actor-critic (SAC. Haarnoja al. 2018) for closed-loop control and the improved cross-entropy method (iCEM, Pinneri et al. 2021 for real-time trajectory optimization. For the time-adaptive control experiments, we additionally employ the time-adaptive control & sensing (TaCoS) framework of [Treeen et al. (2024) as an alternative planner.
Brief descriptions of SAC and iCEM as well as additional implementation details are provided in Appendix C. and C.4. while further details on the TaCoS framework are given in Appendix C.7
Baselines We compare COMBRL with two basclines with different planning approaches. The mean planner uses the mean estimate Sh, of the statistical model and greedily maximizes the extrinsic reward for the task hand (i.e., A = 0), akin to Yildiz et al. (2021).
Furthermore, we compare our method with the trajectory sampling scheme (TS-1) proposed by Chua et al. (2018), subsequently referred to PETS. PETS samples trajectories from PEs for state propagation and thus inherently captures the epistemic uncertainty during planning.
Does COMBRL scale better than the state of the art? In Figure D we empirically validate our theoretical insights using GPs. We consider two classic continuous control tasks the pendulum swing-up and the mountainear problem (Moore) 1990)- - and use iCEM for real-time planning. We compare COMBRL with the baselines above and with the ofte-of-tthearr continuous-time RL algorithm OCORL from Treven et al. (2023). these experiments, An is held constant throughout training and treated as static, hand-tuned hyperparameter.
After each episode, evaluate the learned model by computing the return using its mean prediction f.n on the original task. As a lower bound for the performance of the optimal policy n", we report the best performance achieved by MPC (iCEM) on the true system when the dynamical system f is known. Thus, the MPC can plan with respect to the task/reward hand subject the true dynamics.
7

---

Published as a conference paper at ICLR 2026
COMBRL (Ous)
PETS
Mem Planning
Gyym-Pandulum: Swing
Gym--MounaiaaCar Go up
Quadnuped:
DMC-CanPPle: Swing-up
DMC- Humamood: Stand
-1090
0.
-11
65
Gym-Hali Run
Hopper: Hop
Reachee target
Gyys- Pushe target
Humaanou Run
Averrga
2000
100
Environment stops
Effeett of intrinsic rewards. Learning curves for COMBRL (with auto-tuned X₂) and baselines on different continuous control tasks. We report the mean return when evaluating the learned model on the task at hand, averaged across 10 random seeds along with the standard error. COMBRL achieves the largest performance gains in sparse underactuated tasks, and consistent improvements in higher-dimensional domains.
Figure 2:
Results, averaged over five random seeds with standard error bands, show that COMBRL consistently achieves higher asymptotic returns than the baselines, confirming the benefits of intrinsic rewards for guiding exploration. For the Pendulum experiments, all algorithms solve the task of swinging up. However, the co-optimization over the reward and the optimistic dynamics as well as the reparametrization trick used by the OCORL algorithm are computationallyy prohibitive, and require around 3x the compute time compared to COMBRL (see Appendix C.2). Thus our algorithm scales better and is more computationally efficient than the state of the art.
How does the intrinsic reward affect learning? To assess the effect of intrinsic rewards, we compare COMBRL with An 0 against PETS and the mean planner. Figure 2 shows learning curves on several environments from Gym and DMC. these experiments, we model dynamics using PEs and use SAC to train the policy. We compare COMBRL with nonzero intrinsic reward weight An against PETS and the mean planner (A,, 0). For COMBRL, the internal reward weight X, is automatically tuned following the strategy proposed by Sukhija et al. (2025a). Details of the tuning procedure and the resulting evolution of Xn, over training are provided in Appendix C.3 We further show in Appendix C.6 that this auto-tuned intrinsic reward drives exploration towards epistemically uncertain regions of the state-action space.
Across range of environments, COMBRL achieves higher asymptotic returns, particularly in sparse- reward or underactuated settings such as MountainCar and CartPole, highlighting that optimism- driven exploration significantly accelerates learning. In digher-dimensional environments such as HalfCheetah, Hopper, and Humanoid, COMBRL improves performance by encouraging exploration of uncertain regions, which helps uncover effective behaviors in complex, high-degree-on defree-of-freeddoe freedom systems, even when these behaviors are not directly tied to high immediate reward signals.
How does COMBRL perform in the unsupervised RL setting? We evaluate whether exploration driven by model uncertainty improves generalization to unseen tasks. Specifically, we train policies on primary task (e.g., reaching target) and assess zero-shot performance a downstream task not encountered during training (e.g., moving away from the target). The downstream tasks are implemented by modifying the reward per environment and are summarized Appendix C.5)
Figure 3 compares standard COMBRL well as its unsupervised variant (An, - cf. Section 3.2 the baselines. To ensure a fair comparison, we let each agent explore the environment for several episodes, and then periodically evaluate the learned model on downstream tasks. While COMBRL generally achieves the best performance the primary task especially in environments which offer sparse rewards, its unsupervised variant performs best on the downstream task across all seven evaluated domains.
This suggests that exploration guided by model uncertainty encourages the agent to cover more diverse state-action space and highlights the trade-off between task focus and exploration breadth: Ignoring the reward signal entirely leads the agent to explore broadly, acquiring a globally accurate model that generalizes better unseen tasks for X, 00.
8

---

Published as a conference paper at ICLR 2026
Mean Planning
PETS
COMBRL (Ours)
Unsupervised COMBRL (Ours)
MuuntainCar [go up right)
Hallcheetah forward)
Hopper
Pendulum (balance)
Pendulum (keep
Pusher
thop
(reach target)
(push maget)
1.5
0.0
-0.6-
Printry tolk
7,5-
-1.0
1.0-
5.0-
0.5
2,5-
Average evalurtion
(go left)
(sun backward) le3
(hop backward) 1e2
(swing down)) 1e2
(swing
(go away)
(push away)
lel
Dowstteen took
-1.75-
3,0-
5.0-
2.00
-2255
4.5-
0.0
0.0 2.50
Figure 3: Generalization to downstream tasks. Final returns at convergence on primary (trained) and downstream (unseen) tasks across seven Gym environments. We report the mean return well the standard error for the primary and downstream task. For COMBRL, we differentiate between the balanced case with static or annealing schedule for An. or the unsupervised case with Xy. + DO.
Primary task
Downstream task
Hopper
Reacher
HalfChectah
Pusher
Average evaltattio retum (normalized)
1.0
0.5
0.0
0.1
1000
0.1
10
10
100
0.1
reward weight (A= equals mean planning, equals unsupervised COMBRL)
Figure 4: Ablating the internal reward weight Ay. Final performance at convergence for different environments and tasks with varying X. We ablate over different choices of À and report the mean return and standard error on a primary task which the proposed algorithm was trained on, as well as previously unseen downstream task.
How does the choice of An trade off directed exploration the extrinsic reward and global exploration? We ablate different values of the internal reward weight Ay for the Gym implementa- tion of the HalfCheetah [Wawrzyáski] 2009). Hopper (Erez al. 2012). well the Reacher and Pusher, which are part of the MuloCo environments (Todorov et al. 2012), Figure shows shows that grow- ing nonzero A, values improve downstream generalization while maintaining strong performance on the primary task. In contrast, large X, may overly favor exploration and degrade performance on the primary task. This suggests that there exists an intermediate value A, that balances goal-directed behaviour with model uncertainty reduction, achieving the best of both COMBRL (lask-optimal) and its unsupervised variant (exploration-optina))
How does COMBRL perform the time-adaptive setting? A key advantage of continuous- time RL over its discrete-time counterpart is that enables us to deploy policies varying control frequencies. To illustrate this benefit, we extend our evaluation the time- adaetidaeiive control & sensing (TaCoS) framework from Treven et al. (2024), where interactions with the system incur transition costs and the agent jointly optimizes both actions and their durations, i.e.. the control frequency. We offer details on the TaCoS framework in Appendix C.7
We compare COMBRL with time-adaptive MSS (COMBRI-TACOS) with OTaCoS (the variant of OCORL for TaCoS), Mean- TaCoS, and PETS- TaCoS. Furthermore. compare these iimeaadaptive methods to COMBRL. with fixed control rate and equidistant MSS such as the one used in the previous experiments. Across benchmarks. COMBRL- R achieves competitive or superior returns while using fewer interactions compared the fixed control rate variant, which is shown by

---

Published as a conference paper at ICLR 2026
Pendulum: Swing-up
RC Car: Go to target
COMBRL-TaCoS (Ours)
COMBRI. (fixed sampling mite)
OCORI. (OTaCoS)
PFTS- TFCo-
Mean Planning (Mean-TaCoS)
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Average normalized episode return at convergence
Figure 5: Adaptive TaCoS setting. Final performance at convergence for COMBRL-TaCoS compared to OTaCoS, Mean-TaCoS, PETS-TaCos and COMBRL with fixed control rate (equidistant MSS). Final returns convergence are averaged over 10 random seeds and reported as mean with standard error over 10 random seeds. COMBRI-TaCoS achieves competitive or superior returns while requiring fewer interactions than its fixed-rate variant, and matches exceeds the performance of OTaCoS.
the higher returns in Figure and in turn, demonstrates sample efficiency. Compared to the previous experiments, this shows that optimism-driven exploration extends naturally to the adaptive-control setting and that COMBRL is, in principle, applicable independently of not only the dynamics model, but also the measurement selection strategy.
5 CONCLUSION
In this work, we introduced COMBRL, a continuous time model-based reinforcement learning algorithm that uses epistemic uncertainty guide exploration through intrinsic reward, Our approach provides a principled and flexible mechanism balance exploration and exploitation via internal reward weight Ån, generalizing the classical optimism-in in-ttee-feee the-thce-oee face-off-urneeee cee---ffffueeanntt paradigm to continuous-time domains in way that is scalable, easy implement, and agnostic the statistical model, trajectory planner, as well as the measurement and control strategy.
Our experiments demonstrate that COMBRL. is strong at goal-directed learning the task at hand, while its unsupervised RL variant (i.e., A, + оo) particularly effective at generalizing to unseen downstream tasks. This highlights a core trade-off: Reward directed exploration for appropriate X values and exploration across the entire reachable domain for X - co. We differentiate between three cases for the internal reward weight that affect the agent's 's behaviour greedy, balanced, and unsupervised. Empirical ablations confirm that there exists a range for X, that enables generalizable yet sample-efficient learning. Furthermore, the unsupervised version of COMBRL acts unsupervised system identification strategy, enabling strong zero-shot adaptation to new objectives.
LLM USAGE STATEMENT
Parts of this text revised with the assistance of large language model to aid polish writing and to improve grammar and clarity; the authors remain responsible for all content.
REPRODUCIBILITY STATEMENT
All algorithmic details of COMBRL, including the problem formulation, regret analysis. and assumptions, are presented the main text, with complete proofs provided in Appendix B The experimental setup, including environment details, model architectures, and hyperparameters, is described in Appendix
C
The source code for the conducted experiments is available under https omb
7/github. com/
ACKNOWLEDGEMENTS
We would like to thank Bruce D. Lee for the insightful feedback this work. This project has received funding from the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545, Numerical simulations were performed on the ETH Zürich Euler cluster.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Mare Abeille and Alessandro Lazaric. Efficient optimistic exploration linear-quadratic regulators via lagrangian relaxation. International Conference on Machine Learning (ICML), 2020. URL
https::/proceee: ngs ml www.sppwsss/v/eeeeee ottp://eeiniiaan....preee///////
Adolfo Anta and Paulo Tabuada.
To sample or not to sample: Self-triggered control for nonlinear
systems. IEEE Transactions on Automatic Control, 55(9):2030-2042, 2010. URL https
//doi.org/ 100 ww1.11999.....00 2000 2042980
Karl Johan Åström and Bo M. Bernhardsson, Comparison of riemann and lebesgue sampling for first order stochastic systems. In JEEE Conference on Decision and Control (CDC), pages 2011-2011 2002. httttt:/s///oo// org/10
htt.:pppe//bii.orro0.1.11/////// 11tp:///11.111./1.11111/ 2wwp://CC....2....... 184924
Karl Johan Áström and Pieter Eykhoff. System identification - survey. Automatica, 7(2):123-162,
1971. URL
http://...o.oooooooo 10. 1016/ 11tp://011111111000--00011
(71) 90059-8
Arthur Aubret, Lactitia Matignon, and Salima Hassas. A survey on intrinsic motivation in reinforce- ment learning. 2019.
: / arxiv wtwpg///////// 1908 06976
Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-tima analysis of the multiarmed bandit problem. Machine Learning, 47(2):235-256, 2002. URL htttp:://ooi org/ 0r..2..01/3//.............
1023/A
1013689704352
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Conference on Neural Information Processing Systems (NIPS), 2016. URL hRLpps:s::/axi wwt.:///g/aaoo..oooooooo 01868
Felix Berkenkamp. Safe Exploration in Reinforcement Learnigg: ane in Robotics. Doctoral thesis. ETH Zurich, 2019. URL http:://reea.rrc-coolectttttttte ethz. ch/
handle 500. www.11118378883
Greg Brockman, Vicki Cheung, Ludwig Wojeiech Zaremba. OpenAl Gym. 2016. URL /ttp:
Schnecccee John Schulman, Jie Tang, and 01540
Nicolò Cesa-Biancie Claudio Gentile, Gábor Lugosi, and Gergely Neu. Boltzmann exploration done right. Conference on Neural Information Processing Systems (NIPS), 2017. URL https
//arxiv. haxggabb/17005 10257
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Conference on Neural Information Processing Systems (NeurIPS), 2018.
URL htttps://arxiv/ org/ abs.aass/./0 07366
Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. International Con- ference on Machine Learning (ICML), 2017. URL http:/sss/ppocceeainggs... mlr prr.sesss/
v70/ chowdhury172 html
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Conference on Neural Information Processing Systems (NeurIPS), 2018. /ttt:://aaxii/////
1800
12114
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks, 2020. URL B http:/ss/s//xiv..oooot hbtpg//aabs//00.... 24
abs/
Sebastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient model-based reinforcement learning through optimistic policy search and planning. Conference on Neural Information Processing Systems (NeurIPS), 2020.
http://axxvvvooo www.or///bsbaeeoeoeoooo 2006 08684
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz. Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli. Jackie Kay, Antoine Merle, Jean-Mare Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897:4414419, 2022. URL https: / doi org 1038/ 11884-18600000 021 021-00301----000-0

---

Published as a conference paper at ICLR 2026
Mare Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data-efficien learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intellgence, 37(2):408-423, 2015. URL https:://eeeeppp.rrrtt leee. org/ document /66teet/t/66//11 66541656111
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio bar- rier. International Conference on Learning Representations (ICLR), 2023. URL https 9aBBVJe
//openreview.net. wwt.eeiwwwwweeeeeeeeeeeeeeeeeeee www.:etttttdidaaaeooooo.
Kenji Doya. Reinforcement learning in continuous time and space. Neural computation, 12(1): 219-245, 2000. https://t: httppp/pp.....oooooooooooooooo/a
ieee. org /ttp///tttmettmmco5.5.5
Tom Erez, Yuval Tassa, and Emanuel Todorov. Infinit-horizonn model predictive control for periodic tasks with contacts. Robotics: Science and systems (RSS), 2012. URL https
/ oos.
Nicolas Frémaux, Henning Sprekeler. and Wulfram Gerstner. Reinforcement learning using a continuous time actor-critic framework with spiking neurons. PLoS Computational Biology, 94): el003024, 2013. URL htRp::/ts: htgp:///oooooo/////// /00. 1371 / .ppri 1003024
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Conference on Neural Information Processing Systems (NeurIPS), 2019. URL https: /ttpiv/.//xx/vo////// /00s/1966. 01563
Riccardo A. Grimaldi, Giacomo Baggio, Ruggero Carli, and Gianluigi Pillonetto. The bayesian separation principle for data-driven control, 2024. URL 16717
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-eritic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. International Conference Machine Learning (ICML), 2018. URL httpps://prceeednggs mll. www..mrre.........mmmmmm.mmmmm.e v
haarnoja html
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640(8059):647-653, 2025.
1038/s41588 025 2
Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. Mo- Dem: Accelerating visual masel-based reinforcement learning with demonstrations, International net/
Conference on Learning Representations (ICLR), 2023.
Eorum??dd-ucInc3q477
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. International Conference on Learning Representations (ICLR), 2024. URL Sittps://openrest
net htte://oommmmooooossttataa
Wilhelmus P. M. H. Heemels, Karl Henrik Johansson, and Paulo Tabuada. An introduction to event-triggered and self-triggered control. In IEEE Conference on Decision and Control (CDC), pages 3285, 2012. URL btRpstps:: // wrw/10.../....o...oo.oo. 1109/CDC. 2012. 6425820
Wilhelmus P. M. H. Heemels, Karl Henrik Johansson, and Paulo Tabuada. Event-triggered and self-triggered control. In Encyclopedia of Systems and Control, pages 724-730. Springer, 2021.
URL https::////ooo http///oo.ooooo////////oo /00 wtt.://////1110001111. 000100000000000004011111----1110
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis. Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Sci- ence Robotics, 4(26))caau5872, 2019. htthttt:: http://wwwwc.cm scwence.ee...........oo.....eees org/ wttg//o////oooooooooooooo/oooooo haas//b01000
1126/ robot
aau5872
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Conference Neural Information Processing Systems (NeurIPS), 2019. URL 08253
htttps://axxvv.ott wwwpw/wwwwwwv...o.xot/ao.oootr/t
Sham Kakade, Akshay Krishnamurthy, Kendall I Lowrey, Motoya Ohnishi, and Wen Sun. Information theoretic regret bounds for online nonlinear control. Conference on Neural Information Processing Systems (NeurIPS), 2020. URL htlphhtp:
12466
12

---

Published as a conference paper at ICLR 2026
Motonobu Kanagawa, Philipp Hennig, Dino Sejdinovic, and Bharath K. Sriperumbudur. Gaussian processes and kemne methods: review on connections and equivalences, 2018. URL https
V/arxiv. wttp://rrrraaa//1..0222222 02582
Hassan Khalil. Nonlinear Control. Pearson New York, 2014. ISBN 9781292060507. URL
de/ htt.//ook/////////// ser.
pearson.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning
using calibrated regression. International Conference on Machine Learning (ICML), 2018. URL
https://proceeiings mlr wtt.:/s.ww.r.rr.s..eeeeeeeeeeeee wwtppppessssppppeee http://///ookkllcselleea
htm_
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Conference on Neural Information Processing
Systems (NIPS), 2017. htLp:t::: http://axvvvv... http://xxxxv....g...... abs/ 1612. 01474
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39): 1-00, 2016. URL URthttpp
//jmlr.org/papers.ee.
522 .tmll
David D. Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning.
Machine Learning Proceedings 1994, pages 148-156, 1994. URL httpp:://oi.org///
1016 2016//978-1-5555
335- 6. 50026-X
David G. Luenberger. Optimal control. In Introduction to Dynamic Systems, pages 393-435, John
Wiley & Sons, New York, 1979. ISBN 0-471-02594-11
David J. C. MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4(3):448 1992. URL Uttphhttsss//bb..rrr/00.111////0 1992.4....9.......
448
Andrew William Moore. Efficient memory-based learning for robot control. Technical report.
University of Cambridge, 1990. URL https::/www. cl. cam. ac. uk/techreports/
UCAM-CL- C01-CLL-TRE TR- 209.209.00 pdf
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. International Conference on Machine Learning (ICML), 2017. URL
httpss://proceecinggg http:///ws......peeeeee//aaaaaaa
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement.
International Conference on Machine Learning (ICML), 2019. URL httlittss//ppcceednngs
mlr ww.press/vv. wathaaaaaaaaka 9a. html
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. Conference Robot Learning (CoRL), 2021. URL.tttp:://prceeeiingg mlr www.:ress...../...mmmmm press/
w155/pinneri211 21m.hmm...mm...
Carl Edward Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Machhne Learn- ing. The MIT Press, 2005. ISBN 026218.53X. URL
gpml7
Philippe Rigollet and Jan-Christian Hütter. High-dimensional statistics, 2023. URL http:// www.vvorrr/bb///// 19244
Jonas Rothfuss, Bhavya Sukhija, Tobias Birchler, Parnian Kassraie, and Andreas Krause. Halluci- nated adversarial control for conservative offline policy evaluation. Conference Uncertainty in Artificial Intelligence (UAI), 2023. URL htttps:// htttps::/proceedinngn www.sanng..r...p..... www.sa/s..................
fosth ht.m
Jonas Rothfuss, Bhavya Sukhija, Lenart Treven, Florian Dörfler, Stelian Coros, and Andreas Krause. Bridging the sim-to-real gap with bayesian inference. JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2024. URL https::/leeexplore..eee. org
Reuven Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology And Computing In Applied Probability, 1(2):127-190, 1999. http::::: /doi
org/10.
13

---

Published as a conference paper at ICLR 2026
Christoph Salge, Cornelius Glackin, and Daniel Polani. Guided Self Organization: Inception, 2014. URL
Empowerment- an introduction.
978 3- 1142-53734-2
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588 (7839):604 609, 2020. URL https: /tt:::/doii..oo. or0 10110100//1116811 00005-8000220005-22
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. International Conference on Machine Learning (ICML), 2020. URL httph:ttp//// htttts:://proceedingg.. mlr httpppoceecccoos..o.ppppeessse/s html
Alexander Spiridonov, Fabio Buehler, Moriz Berclaz, Valerio Schelbert, Jorit Geurts, Elena Krasnova, Emma Steinke, Jonas Toma, Joschua Wuethrich, Recep Polat, Wim Zimmermann, Philip Arm, Nikita Rudin, Hendrik Kolvenbach, and Marco Hutter. Spacehopper: small legged robot for exploring low-gravity celestial bodies. IEEE International Conference on Robotics and Automation (ICRA), 2024. URL httph: wwteeeeeeepppeeeeeee leee. org/ wtrg//document/161110
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Information-theorer regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions Infor- mation Theory, 2012. URL [ttps::/
ieee. wtg. htrg/document//111 613891
Bhavya Sukhija, Lenart Treven, Cansu Sancaktar, Sebastian Blaes, Stelian Coros, and Andreas Krause. Optimistic Active Exploration of Dynamical Systems. Conference on Neural Information Processing Systems (NeurIPS), 2023. URL
12371
Bhavya Sukhija, Stelian Coros, Andreas Krause, Pieter Abbeel, and Carmelo Sferrazza. Max- InfoRL: Boosting exploration in reinforcement learning through information gain maximiza- tion. International Conference on Learning Representations (ICLR), 2025a. URL https
7/ //openeeviww.o. wwt.:riwwweeeeeeeememmmm Fo
Bhavya Sukhija, Lenart Treven, Carmelo Sferrazza, Florian Dörfler, Pieter Abbeel, and Andreas Krause. SOMBRL: Scalable and optimistic model- RL. Conference Neural Informa- tion Processing Systems (NeurIPS), 2025b. URL
net/ htt.//orrmmm/mmm
eGf 15FF5k
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy Lillicrap, and Martin Riedmiller. DeepMind Control Suite, 2018. UtLtstts::///ax www.ariwwww..gvaaaaa abs 180881.0 00690
Annalisa T. Taylor, Thomas A. Berrueta, and Todd D. Murphey. Active learning in robotics: A review of control principles. Mechatronics. 77:102576, 2021.
mmechat 2021. 102576
Emanuel Todorov, Tom Hrez, and Yuval Tassa. MuloCo: A physics engine for model-based control. International Conference on Intelligent Robots and Systems (IROS), 2012. URL https // V/ieeexplore. leee org/ document/ /cccment//63861000 6386109
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG. Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: A standard interface for reinforcement learning environments, 2024. URL https://arxii/ 2407.17032
/bs/
Lenart Treven, Jonas Hübotter, Bhavya Sukhija, Florian Dörfler, and Andreas Krause. Efficient explo- ration in continuous-time model-based reinforcement learning. Conference on Neural Information Processing Systems (NeurIPS), 2023. URI httt:://aaxiv.o//// wttp.//ggg////////// 2.31 O. 19848
Lenart Treven, Bhavya Sukhija, Yarden As, Florian Dörfler, and Andreas Krause. When to sense and control? A time-adaptive approach for continuous-time RL. Conference on Neural Information Processing Systems (NeurIPS), 2024.
01166
14

---

Published as a conference paper at ICLR 2026
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for con- tinuous control. Software Impacts, 6: 100022, 2020, URL httpp: /www sci encedirect
com/ http://cccccaaaatccccaa///666662
Pawel Wawrzyúski. A cat-like robot real-time learning to International Conference Adaptive and Natural Computing Algorithms (ICANNGA), 2009. URL htKLtttts:///taff.comm pw
edu. http://wwwwz///p//a/aamma
s/0812 LSCLRR. p
Cagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Continuous-time model-based re-
inforcement learning. International Conference Machine Learning (ICML), 2021. URL
httpss:/pproceesinnggs.. m http://rrreeeiiia.....r.rrrsss/s ptrp://.....ps//1111/111111211..
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: survey, ACM Computing Surveys (CSUR), 55(1):1-36, 2021. /ttp:s/////o org/10
1145/3477600
Feiran Zhao, Alessandro Chiuso, and Florian Dörfler. Policy Gradient Adaptive Control for the LQR: Indirect and Direct Approaches, 2025a. URI
03706
Feiran Zhao, Alessandro Chiuso, and Florian Dörfler. Regularization for Covariance Parameterization of Direct Data-Driven LQR Control. IEEE Control Systems Letters, 9:961-966, 2025b. URL https://Leeexploree ieee org/ /cc////t/meeee1001199
15

---

Published as a conference paper at ICLR 2026
APPENDICES AND SUPPLEMENTARY MATERIAL
Additional related work
17
A.1
Model-based reinforcement learning
17
Unsupervised reintorcement learning and intrinsic exploration
17
A.3
Continuous citmu reinforcement learning
17
Theory
18
I Assumptions
18
B.2
Analysis of Gaussian process dynamics
19
Experimental setup
22
GP experiments
22
Computational costs
24
Auto-tuning experiments
24
BNN experiments
24
Downstream tasks
25
C.6
Epistemic uncertainty decrease
26
Time-adaptive experiments
27
16

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
A.1 MODEL-BASED REINFORCHMENT LEARNING
Model-based RL (MBRL) has emerged as sample-efficient alternative to model-free methods, with applications ranging from robotics to online decision-maide makinn (Chua al.) 2018 Janner et 2019) Hansen et al. 2023 Rothfuss et al.) 2024). Recent deep MBRL methods differ primarily in dynamics modeling and planning strategies. yet often rely on naive exploration heuristics such as Boltzmann exploration (Hansen et al. 2024 Hafner et al. (2055). However, such heuristics are suboptimal even
in simple settings (Cesa- Bianchi et al. 2017).
COMBRL addresses this by introducing principled exploration mechanism that combines epistemic uncertainty with extrinsic reward. Unlike prior methods, it model- and planner-agnostic, scalable, and comes with sublinear regret guarantees. We show that this intrinsic reward formulation not only improves theoretical performance but also enables meaningful exploration across deep RL benchmarks, where naive methods fail.
A.2 UNSUPERVISED REINFORCEMENT LEARNING AND INTRINSIC EXPLORATION
System identification (Áström and Eykhoff 1971 and unsupervised exploration (Aubret et al. 2019) require efficient exploration, as the agent must learn accurate global dynamics models without access to extrinsic rewards. In such settings, exploration is essential for covering informative regions of the state-action space. To this end, intrinsic motivation techniques have long been used to encourage exploration in RL. making use of objectives like model prediction error (Pathak et al. (017)) novelty (Bellemare et al. 2016), empowerment (Salge al., 2014), and information gain (Sekar (et al.) 2020 Sukhija et al. 2023). However, such techniques are often used in isolation from extrinsic rewards. COMBRL instead combines epistemic uncertainty, which poses principled intrinsic signal,
with task rewards, aligning exploration with learning progress.
In the unsupervised setting, COMBRL reduces to active learning Taylor et al. (202 1), specifically uncertainty sampling (Lewis and Catlett 1994) and guides the agent toward regions where the model most uncertain. Similar ideas have been explored in bandits (Auer et al. 2002; /Srinivas al. 2012) data-driven control (Grimaldi et al. 2024), and RL (Abeille and Lazaric 2020; Sukhija et al. 2025a), where joint optimization of reward and model uncertainty is shown to improve learning. For the case of data-driven control linear- quearr-ua systems, /Zhao et al. (2025a by use similar weighting mechanism which is initially motivated by robustness and regularization, but interpretable as trading off exploitation and exploration, showing how ancertainty-awart regularization exploration bonuses can systematically balance performance and robustness in continuous control.
COMBRL follows this direction, using model epistemic uncertainty as a reward bonus to guide exploration. Our work is closely related to Sukhija et al. (2025b), who propose similar reward formulation in the discrete-time setting. However, our focus on continuous-time systems leads to significantly different theoretical analysis and experimental design. In particular, we derive regret and sample complexity bounds tailored to the continuous-time domain and analyze the effect of the intrinsic reward weight Xn in greater depth. Unlike prior work, which is mostly empirical or limited to linear systems, COMBRL also provides theoretical guarantees for general nonlinear systems in continuous time and demonstrates scalability high-dimensional tasks.
A.3 CONTINUOUS- REINFORCEMENT LEARNING
While most model-based RL methods are developed in discrete time, continuous-time formula- tions have gained increasing interest due to their relevance for real-world control and physical modelling (Doya] 2000; Frémaux et al. 2013; Yildiz et al. (02T). Recent works explore learning dynamics neural ODES (Chen al. 2018) and thysics-informed priors (Greydanus et al. 2019 Cranmer et al. 2020).
Yildiz et al. (021) propose continuous-time actor-critic method that plans using the posterior mean of the learned ODE model. Treven al. (2023) 2024) derive regret bounds for continuous-time MBRL using optimistic dynamics and highlight the role of different measurement selection strategies, showing that adaptive sampling can substantially reduce the number of interactions needed to match or surpass discrete-tine methods. A complementary line of research is event- and selfftriggered
control (Áström and Bernhardsson 2002 Anta and Tabuada
2010 Heemels et al. 2012 2021).
17

---

Published as a conference paper at ICLR 2026
which seeks efficiency by updating control inputs only when certain conditions met, thereby avoiding unnecessary interventions while maintaining stability.
COMBRL extends this line of work by proposing a flexible framework for both reward-driven and unsupervised exploration in continuous time. In contrast prior methods. COMBRL incorporates optimism directly reward function and thus offers a simple, scalable, and theoretically grounded approach that operates directly in the continuous- timm domain and supports general-purposs puuppse dynamics models.
B THEORY
We provide the assumptions and proofs for Theorems 1 and which formalize the regret and exploration guarantees of COMBRL, in this section. The former bounds regret terms of model complexity, which is sublinear for common GP kernels and MSSs, implying convergence to the optimal policy. The latter shows that intrinsic exploration alone (i.e., - 00) reduces epistemic uncertainty at rate of / To the best of our knowledge, we are the first to show this for continuous- time RL.
B. ASSUMPTIONS
In the following, we restate the assumptions from Section 2.3 completeness. We make some common assumptions (cf. Curi et al.] (2020); Treven et al. (2023)) that allow us to theoretically analyse the regret Ry and prove regret bound. We first make assumption the continuity of the underlying system and the observation noise.
Restated Assumption 1 (Lipschitz continuutty..
The dynamics model f", reward and all policies
I1 are Ly. L, and L Lipschitz-contimuuuss, respectively.
Restated Assumption (Sub- Sub--Gausiaa noise). We assume that the measurement noise Gn is i.i.d. o-sub Gaussian.
The Lipschitz assumption commonly made for analysing nonlinear systems (Khalil 2014) and is
satisfied for many real-world applications. Furthermore, assuming d-sub Gaussian noise (Rigollet) and Hütter (023) is also fairly general and is common in both RL and Bayesian optimization literature (Srinivas al. 2012 Chowdhury and Gopalan 2017
Curi et al.) 2020).
In COMBRL, we learn an uncertainty-awarr model of the underlying dynamics. Therefore, we obtain a mean estimate (z) and quantify epistemic uncertainty (2) about the function f*.
(Well-calibrated statistical model of f", |Rothuuss al. (023)). Let X x U. An all-time well-calibrated statistical model of the function f is a sequence (M)(
Restated Definition 2
where
(.) del
2 Rd= 2,jj (I,..., E
(2n S (B)(())n,,
if. with probability at least 1 8, have f Onzo-Ma() Here, pn.j and on denote the j-th element in the vector-valued mean and standard deviation functions phn and Tn respectively, and (6) E R2 a sealar function that depends on the confidence level (0, 1] and which is monotonically increasing in n.
of the model). The learned model is all-time well- calibrated statistical model of fr, i.e., with probability at least 1- we have f E Efno Mn(5) for confidence sets Mn) as defined in Definition 2 Moreover, the standard deviation functtons o, dd are Lg -ipschiit continuous for all n
Restated Assumption 3 (Well-calibratte
Intuitively, Assumption 3 states that we are, with high probability, able to capture the dynamics within confidence set spanned by predicted mean and epistemic uncertainty. For Gaussian process (GP) models, the assumption is satisfied (Rothfuss al.) 2023 Lemma 3.6) and for more general classes of models such as Bayesian neural networks (BNNs), re-calibration techniques (Kuleshov et al. 2018) can be used.
18

---

Published as a conference paper at ICLR 2026
Lastly, make assumption the regularity of the dynamics by placing them in reproducing kernel Hilbert space (RKHS):
Restated Assumption 4 (RKHS Prior Dynamics). We assume that the functions fz. E (1....,,e, dn)) lie in RKHS with kernel k and have bounded norm B, that is
f H with HEBB= {f 11fsll I B,j 1, ....,,.......................... d.)
Moreover, we assume that k(z,z) z) S. Tmax for all æ E
B.2 ANALYSIS GAUSSIAN PROCESS DYNAMICS
Assumption allows us to model f with GPs. The posterior mean (x(()) = epistemic uncertainty o (z) [on (2) (z)lisde can then be obtained using
and
Mn.j k (z) (K, 121)= Vin. / = k(z, z) kr (Kn + -2 (z),
(5)
Here, vi.. corresponds the noisy measurements of fj. i.e., the noisy derivative observation from the dataset Di:s (z) 1 x1)x, and Kn [k(z =.) is the data kernel matrix.
Moreover, since f has bounded RKHS norm, i.e., ff I B (Assumption 4) follows from Srinivas et al. (2012); Chowdhury and Gopalan (207)) that with probability 1 ò we have for every episode n:
|f Mn ll. S Br.
Instead of planning with the mean, which in general might not be Lipschitz continuous for all n, we select a function fr. that not only approximates the f function well, i.e., satisfies || - fn fnlls,, S fn. but also has an RKHS norm that does not grow with n. To achieve this, we propose solving the following quadratic optimization problem:
fn
arg min / (((1)))) k(...)))
||f 11f-11nlaa ponlla,
(6)
s.t. ||fll < B
Theorem 3. The optimization problem Equation (6) is feasible and we have fl pn Malls, S 2
Proof. Consider the noise-free case, i.e., Gn, = 0, and let fa, be the posterior mean for this setting. For the function Pln, it holds that || Mnl lk, < Bn (Kanagawa et al. 2018 Corollary 3.11) and
| B Kanagaaa al. 2018 Theorem 3.5). Thus it follows that
||,,, Mxll, 1|, 1' + 11 - 1n S 28n-
By representer theorem, it also holds that fa, E span(k(zı,,),.. kizzn
Let a. (K - y I=" and reparametrize f(x) Ei= a,k(,,, We have ||
a Ka. We also have:
If 1 (a an))
T K
I
(a (a-ai) an)
Hence the optimization problem in Equation 6
is equivalent to:
1 (a- aa-
min (a an) K I
K oo
a
S.t. & Ka < B²
This is quadratic program that can be solved using any QP solver. The program finds the closest function to the posterior mean pn that Lipschitz continuous. In particular, note that since f B. for Lipschitz kernels, f. has Lipschitz constant Lg which is independent of n (Berkenkamp 2019). From hereon, let T., max [.y, T.B.
19

---

Published as a conference paper at ICLR 2026
Next, we plan with the dynamics f.. that obtained from solving Equation 66 i.e.,
L'' (r(a'(t), u(t)) + An (o, u()) dt
nR = arg max ET RET
s.t.
8'(1- fn(a' (1), u(()).
Lemma 4, Let Assumption and Assumption 4 hold. Consider the following definitions:
J(n E8 J. ((x(t), dt.
8.1. Φ = (
ap =x))),
J(T, fn) E J
dt.
s.t. 3' (n=
26 x00).
En f*) = E I
dt. 2o == =()),
Φ ('x((1,,
(T, fn) = =E L (0, dt.
s.t. e' = (
d6 = x0).
Furthermore, let An = 23, LA lee
Then, we have for all π E II with probability at least
[ f") - J(T f.)l An En (T, fm)
J(T,ff)) f... S An (n )*).
Proof. We follow Treven et al. (023) and bound the regret with En Sn((TT fn))
[J(T,, f*) J fnnl
- (((())))
$ L, + L E 1/" [|x(() - x x'(1))l dt
$ 23, L. + Ted L
(lon)1(((t)) (xx())) de
(Treven et al. (2023), Lemma 4)
Lemma S. Let Assumption 2 and Assumption Food hold and consider the simple regret at episode n.
To J(x",, f*) J(TT: }*).
The following holds for all n > with probability at least 8:
Tn 5 (2An + x2) TT, f*).
20

---

Published as a conference paper at ICLR 2026
Proof of Lemma
Tn J(T fd) J(T f³)
S J(T fn] + An En (T fn) J( f
(Lemma
S (Tus fn) + Xn ans fn) [ S
(Equation
J( fu) J [Tnn f*) Xn ( fr)
I An (Tn, f¹ + An En nT fn)
(Lemma
2An2in (T f*) + An (Nin (Tn: fn) 2n (T f²))
- (A + 2Am En nn, fr)
In the last line, we used the fact that E,, (..:) is bounded and positive, and thus behaves like a reward. In fact, it is intrinsic reward (Sukhija al. 2025b), which is why Lemma also applies.
Restated Theorem I Reggret bound in the sub-Gaussian noise case). Let Assumption 2 and Assump- tion 4 hold. Then we have for all N > with probability at least 8:
RN S O [ N
Proof of Theorem
N R,
(Equation
n=l
N
(Xg + 22n f-)
(Lemmal
n=1
$ (A + ANn Σ
En (Tn f*)
NV
-A2 + 2AN) Σ
Eg
( dt
n=1
N
[
A (A + 2N NT
E
(x(()) (x)())))) dt
1=1
$ <X + 2AN) NTIN
Treven et (003)), Proposition 1)
Note that by Lemma 4 Ay scales with the confidence parameter SN. which itself depends on the maximum information gain (Chowdhury and Gopalan [017]] Rothfuss et al. 2023) and thus scales in the model complexity X O(Ix), yielding the result.
Restated Theorem (Sample complexity bound in the unsupervised case). Let Assumption 2 and Assumption 4 hold. Consider Algorithm
7
with extrinsic reward 0, i.e.,
J (I,,(a (t), ( (f))
nr arg max Ex #ell
t'(t) f. (t)
Then have for all N 0, with probability at least 1 - 6:
T-1
I3 N
max E f- mEll
| (2)(
00
21

---

Published as a conference paper at ICLR 2026
Proof of Theorem Recall the definitions from Lemma 4 Let Ei max; [ f*) and let i be the corresponding policy. Recall from Lemma that 2n (:,,) behaves like reward.
ΣN
(monotonicity of the variance)
(1+ (T
(Lemma
+1+ 2nn (m)
(T, is the maximizer for dynamics f..)
N n=1
Σ
(1 Xn, (T
(Lemma 4
n=1
(1 2N)
En nnn
VN N
N
< <1+ AN)
Σ n=1
(n,
n f*)
inequality)
I N
0
(c.f. Proof of Theorem
C EXPERIMENTAL SETUP
We provide additional details for our experiments in this section.
C.I GP EXPERIMENTS
We evaluate our method on two low-dimensional continuous control tasks: Pendulum-GP and MountainCar- GP (Moore 1990). Unlike in the other, following experiments, these environments implemented directly by nonlinear ODEs, rather than relying on Gym or DMC implementations. We emulate a continuous- time setting by using a fine time discretization for state propagation. As for measurements, assume that we have direct access to the state derivatives.
continuous-
systems with known physical dynamics given by
In Pendulum-GP. the agent must swing up and stabilize a pendulum in the upright position. The state vector is defined
Zig F1
cOs 0
u E 2-
x2
where 0 E [_ m]] is the pendulum angle and Ó is the angular velocity. The underlying nonlinear ODE is:
d0
dg 3g
8,
sin 0 +
dt dt
28
ml2
with constants g 9.81 m/s?. and f 1. We use Gym-style reward, which penalizes deviations from the target angle f O, angular velocity ô, and control input u:
r(ær u) - 0.1 0 0.02
where 0 arctan 2(x1, mo) and 42.
22

---

Published as a conference paper at ICLR 2026
In MountainCar-GP. the agent must build momentum to propel up steep hill. The state vector defined as
a
u - u E [-1,1]
2/2
where 21 [.1.2,0.6] is the position and 22 E -0.07, 0.07] is the velocity. The underlying
nonlinear ODE is given by:
dr
dx² 0.0015 0.0025 cos(3 3 -x1).
22
di
di
Position and velocity are clipped to their bounds, and backward motion is blocked at #1 = -1.2 32 < The reward includes a terminal bonus of + 100 for reaching the goal and penalizes control effort:
ræ, u) 0.1 u² + 100 - Igaal reacheds
where the goal is reached if the car's position exceeds 0.45 and its velocity is non-negative.
For our GP experiments in Figure D we use the RBF kernel. The kernel parameters are updated online using maximum likelihood estimation (Rasmussen and Williams/ 2005). We use hand-tuned static regime for the internal reward weight, i.c., An = A. For the OCORL baseline, we provide the confidence level function 8,, (6) 3. The hyperparameters for the statistical model well as for the environments and algorithms are given Table
Table 1: Model training hyperparameters and experimental setup for the GP-based experi-
ments in Figure
Environment
[s]
N
v [s
Algorithm
À
B
Learning Rate
COMBRL
1.0
OCORL
0
7.5
Pendulum-Gi
2.5
12
20
0.01
PETS
0
Mean
0
COMBRL
106
OCORL
0
30
MountainCar-GP
200
15
0.01
PETS
0
Mean
0
Episode horizon 7', number of episodes N, and measurement/coontol frequency v shared across algorithms for each environment.
For planning. algorithm. The method iteratively samples action sequences, evaluates them in the dynamics model, and refits Gaussian distribution to the cop-perforiing ("clite") trajectories. This elite set determines the parameters for the next iteration.
the iCEM optimizer (Pinneri et al. 2021) even though it is discrete- time
In contrast to vanilla CEM introduced by |ubbinstein (199), iCEM improves efficiency by sampling temporally correlated action sequences and reusing elite trajectories across iterations and MPC steps. We emulate the continuous-time setting by using fine time discretization (see measurement/control frequency y in Table T) and using the equidistant MSS. The hyperparameters for the planning are given in Table 2
Table 2: iCEM hyperparameters used for planning in the GP-based experiments.
Environment
Horizon
# Particles
Samples
Elites
Steps
a
Exponent
Pendulum-GP
30
10
500
50
10
0.2
2
MountainCar-GP 100
10
500
50
5
0.2
2
Horizon refers to the iCEM planning horizon (in time steps). Steps indicates how many CEM optimization iterations are performed per control decision to refine the action distribution using elite samples. The number of time steps and control decisions are given by the measuremenn/conntrll frequency
23

---

Published as a conference paper at ICLR 2026
COMBRL (Ours) with auto-tuned An
Gym-Pendulum: Swing-up
Gym- Cymm-MouttainCar: Go up
DMC-CartPole: Swing-up
1.0
10
Inurinsic reward weight
0.0
035
-10
0.0
1.0
3.5
2.0
X20
0.5
1.0
-10
Glye-Hal(Cheetah: Run
HopC-Hopper: Hop forward
HMC-Quadruped: Run
1.0
0.5
X20
410"
Environment stops
Figure 6: Evolution of the auto-tuned intrinsic reward weight An- We plot X as a function of environment steps for COMBRL with the auto-tuning objective in Equation (8) across several Gym/DMC tasks, as seen in Figure Overall, X, tends decrease over time, indicating stronger exploration early in training and reduced weighting on the intrinsic reward as learning progresses.
C.2 COMPUTATIONAL COSTS
We give the computational costs for our GP experiments in Table This shows that the co- optimization over the reward and the optimistic dynamics as well as the reparametrization trick used by the OCORL algorithm are computationally prohibitive, and require around 3x the compute time compared to COMBRL.
Table 3: Computation cost comparison (training time) for each algorithm across environments.
Environment
COMBRL
OCORL
Mean
PETS
Pendulum-GP*
10.8 0.13 min 1.29 = 0.01 h
30.6 + 0.4 min 4.55 + 0.1 h
10.4 0.04 min 1.28 0.03 h
30.78 0.27 min 4.67 0.16 h
MountainCar-Gii G
Mean total training time. GPU: NVIDIA GeForce RTX 2080 Ti.
Mean training time per episode. GPU: NVIDIA GeForce RTX 2080 Ti.
C.3 AUTO-TUNING EXPERIMENTS
In Figure 2 we auto-tune the intrinsic reward weight A, following the method proposed by Sukhija al. 2025a), who demonstrate that this approach is effective across range of model-free off off-policy methods discrete time. Specifically, we adjust by minimizing the loss:
L(A)
-~ E
lo u) ox ū)).
(8)
Here, n. denotes slowly updated target policy obtained via Polyak averaging of π. The objective promotes larger values when the current policy explores less than the target policy. F that the auto-tuned intrinsic reward weight An typically decreases over training for several of the tasks shown in Figure [ indicating that exploration is emphasized early on and gradually reduced as model and policy stabilize.
C.4 BNN EXPERIMENTS
In our experiments that do not explicitly use Gaussian Processes, we train an ensemble of 5 neural networks to model forward dynamics. Model epistemic uncertainty estimated via the disagreement among the ensemble members (Pathak et al.) 2019). To further leverage the model, we augment the data by including synthetic transitions. For each policy update, sample real transitions (xx ÿ) from the replay buffer D1.,. and add corresponding model-predicted transitions (æz, u, g'), where z is generated by the mean model pą. This lets us blend real and synthetic rollouts, similar to the strategy used by Janner al. (2019), thereby increasing the update-to-data (UTD) internal ratio.
24

---

Published as a conference paper at ICLR 2026
The policy is optimized using soft actor-critic (SAC. Haarnoja al., 2018), off-policy actor-critic method that learns stochastic policy by maximizing expected return together with an entropy bonus to encourage exploration and stabilize training. We adopt the same hyperparameters as [Sukhija et al. (2025a) for optimizing the loss via stochastic gradient descent in Equation (8) and for configuring the UTD. We also periodically perform soft resets for the policy for training stability (D' (D'Oro et al.) 2023). The hyperparameters for the statistical model and SAC are given in Table 4 For the ensemble based experiments, We use several environments from the Gym and DMC benchmark suites (Brockman
iet al.) 2016; (Tunyasuyunkkool et al., 2020). We adapt them to the continuous-tin setting by approximating the derivatives using finite difference filter. The measuremeen//contro) frequency is given by the duration of environment step dt.
Table 4: Hyperparameters for insemble-based experiments with SAC in Section 4
Policy Critic Architecture
Action Repeat
Learning Rate
Model
Architecture
Batch Size
Environments
Gym Pendulum MountainCar
(256,256)
5x(256,256)
3 10
256
Gym Reacher
(256,256)
5x(512,512)
3 3 10 -4
256
Gym other environments"
2
(256,256)
5x(256,256)
3x 10
256
DMC Quadruped
2
(256,256)
5x(512.512)
3x
256
DMC Humanoid
(512,512)
5x(512,512)
3 10
256
DMC other environments
(256,256)
5x(256,256)
3 10
256
HalfCheetah, Hopper, Pusher.
Hopper, CartPole.
C.5 DOWNSTREAM TASKS
To evaluate zero-shot generalization in Figures and we introduce custom downstream tasks that differ semantically from the primary training objective. While the primary task corresponds to the default reward in each Gym or DMC environment, the downstream task uses a custom reward function that incentivizes behaviour that contrasts with the original task (e.g., moving away instead of toward goal).
We implement each downstream task by overriding the reward computation the Gym or DMC environment using environment wrappers. tasks. also gives the internal reward parameter An for the experiments shown in Figure For said experiments, the downstream rewards are defined as follows:
the evaluated primary and downstream
MountainCar - go up left: Encourages the car to reach the leftmost side the hill, in contrast to the standard goal on the right.
HalfCheetah - run backwards: Reverses the locomotion objective by rewarding backward velocity.
Hopper hop backwards: Rewards hopping backwards while maintaining a healthy posture
Pendulum
Balance upright: Starts upright and rewards maintaining the upright position.
Swing up: Starts with the pendulum pointing downward and rewards swinging up.
Swing down: Starts upright and rewards swinging the pendulum downward.
Keep down: Starts downward and rewards staying down.
Reacher go away: Penalizes proximity to the goal, inverting the standard reaching task.
Pusher push away from target: Encourages the agent to push the object away from the goal location.
"The Hopper environment in Gym introduces heal bealthyyrewarl which is preserved for the downstream
task.
25

---

Published as a conference paper at ICLR 2026
Table 5: Primary and downstream tasks used in evaluation. Each downstream task is defined via custom reward that encourages behaviour contrasting the primary task. We also provide the algorithm hyperparameters used in Figure 3 namely the strategy used and the optimized internal reward weight
An-
Environment
Primary Task
Downstream Task
Strategy"
A
MountainCar
Go up right
Go up left
Annealing
50
HalfCheetah
Run forward
Run backward
Annealing
Hopper
Hop forward
Hop backward
Static
10
Pendulum
Balance upright
Swing down
Annealing
10
Pendulum
Keep down
Swing up
Annealing
50
Reacher
Reach target
Keep away
Static
0.17
Pusher
Push target
Push away
Static
0.56
Static denotes fixed internal reward weight A.. annealing decreasing reward weight
n/N).
Pendulum: Swing-up
DMC-CarPole: Swing sparse
0.075
moun(an)
10
(Ours)
0.050
PETS
Mean Plamning
10"1
0.025
Unsupervised COMBRL
OCORL
2
0 20000 40000 60000 80000 100000
Episode
Enviromment steps
Figure 7: Epistemic uncertainty under COMBRL. Left: Unsupervised COMBRL (X., oe) Pendulum. At the end of each episode, we evaluate the model uncertainty [I)x())] on 100 random samples X' x U and track its evolution, showing that unsupervised exploration systematically reduces epistemic uncertainty across the entire reachable state-action space. Right: COMBRL with auto-tuned intrinsic reward weight An on DMC-CartPole We plot the epistemic uncertainty encountered along the executed trajectories each environment step. Compared to Mean and PETS, COMBRL is guided towards regions of higher epistemic uncertainty while still driving uncertainty down over time, indicating that An effectively shapes exploration towards informative state-action regions.
C.6 EPISTEMIC UNCERTAINTY DECREASE
We visualize how COMBRL reduces epistemic uncertainty over time both the unsupervised and reward-driven regimes in Figure
In the unsupervised setting, on the Pendulum environment and monitor epistemic uncertainty over the entire reachable state- action space. Concretely, at the end of each episode we sample 100 random state-action pairs (x, u) from X' x 24 and evaluate the model uncertainty [(),.())] at these points. This shows that unsupervised exploration alone is sufficient to drive down epistemic uncertainty globally over X' x U, rather than only along the visited trajectories.
COMBRL with a purely intrinsic objective (i.e., A oo)
In the reward-driven setting with auto-tuned >n, we consider the CartPole environment and use the method from Appendix C.3 to auto-tune An. Here we track the epistemic uncertainty encountered along the executed trajectories each environment step. Compared to the Mcan-planning and PETS baselines, COMBRL consistently visits regions with higher epistemic uncertainty, indicating that the intrinsic reward term steers the agent towards informative parts of the state-action space, while the overall level of uncertainty still decreases over time.
26

---

Published as a conference paper at ICLR 2026
C.7 TIME-ADAPTIVE EXPERIMENTS
The time- ande-tideiie TaCoS framework (Treven et al. 2024) generalizes continuous- RL to settings where sensing and control actions are costly. Instead of interacting with the system at a fixed frequency, the agent actively chooses when sense and apply control inputs, thus adapting the sampling rate over time.
Formally, the dynamics are described by the unknown flow Φ*, which maps state-action-time triple the successor state and the integrated reward:
Φ (x, ut (z),6),
where æ' is the next state at time t + At and
++ r(x(s), u(s)) ds
the rransition reward, i.e., the cumulative task reward over the chosen interval At. The length of Ät reflects the measurement and control schedule S, meaning that fewer but longer intervals correspond to less frequent interaction.
Crucially, in TaCoS each sensing or control action incurs unit interaction cost C. The agent must therefore balance two objectives: maximizing cumulative task reward while minimizing the number of costly interactions. This induces joint optimization problem over both the control policy and the measurement schedule. Transition rewards provide a natural way to capture variable step lengths, allowing the planner to evaluate policies under adaptive sampling schemes.
In the model-based RL setting, maintain statistical model M,, of the flow T.. For the mean and PETS baselines "Mean-TaCos and PETS- ToTS),,,,,S,, planning is carried out with respect to the task reward only. In contrast, OCORL-TaCoS applies optimism to the dynamics: at episode n, the policy n, is selected by maximizing the optimistic value over all plausible models in Mn-1-
In this framework. COMBRL applics seamlessly. The statistical model Mn naturally extends to modeling flows with transition rewards, and the intrinsic reward shaping in COMBRL integrates epistemic uncertainty with extrinsic task rewards. As result, COMBRL-TaCos. combines optimism- driven exploration with adaptive mcasurement scheduling, reducing unnecessary interactions while preserving sample efficiency in the adaptive regime.
For the time-adaptive experiments, we evaluate COMBRL on two continuous-time environments. We first reuse the Pendulum-GP environment described in the GP experiments (see Appendix C.I with added interaction cost C.
In addition, include the more challenging RC Car environment, using the implementation of Treven et al.] (024).. The continuous- -timu dynamics are integrated with a small base step size (dt 1/30s by default). The reward function combines a tolerance-based state term (encouraging the car to reach goal pose) with penalties on control effort and smoothness.
The RC Car follows nonlinear bicycle model with blended kinematic and dynamic components. The state is
[
x [pz: Py. O, U2 Uy Wy
where (Pa,P,) Py,) is the position, 0 the heading angle, (uz, vyy) the local velocities, and w the yaw rate. The control inputs are steering angle ê and throttle d.
The kinematic model
P = U2 cos 0 - U. sin
P. = U2 sin 0 + Uy cos θ = w.
The dynamic accelerations are given by
0 1 (fr - fra sin + messw)
0 (Fri + fr.v cos - muael))
3 Stalls cos Smaghr)
27

---

Published as a conference paper at ICLR 2026
where m is the mass, I its moment of inertia, L the distances to the front/rear axles, and far fray: fr,z are the tire forces determined by the nonlinear Pacejka tire model.
The blended model used in practice interpolates between the kinematic and dynamic models depend- ing on the velocity regime (see the implementation by Treven al. 2024 for details).
We give the relevant parameters for the time-adaptive experiments in Table Figure [ we show the final performance at convergence of the evaluated algorithms. For completeness, we also offer the complete learning curves in Figure g
Pendulum: Swing-up
RC Carz
COMBRL-TTCCS (Ours)
(ntemalized)
COMBRT (fixed sampling rate)
OCORL (OTaCoS)
PETS-TaCOS
Meun Plarming (Meun-Taaas
10
15
20
20
60
Episode
Episode
Figure 8: Learning curves of COMBRI-TaCos compared to OTaCoS, Mean-TaCoS, PETS-TaCOS, and COMBRL with fixed control rate (equidistant MSS). Results are averaged over 10 random seeds, with mean and standard error shown. COMBRI-TaCOS achieves competitive superior returns while requiring fewer interactions than its fixed-rate variant, and matches or exceeds the performance of OTaCoS.
Table 6: Experimental setup with environment details and hyperpa- rameters for the TaCoS experiments.
Pendulum
RC Car
Episode horizon T [s]
10
Number of episodes
20
70
Base step size At [s] Max steps per episode Min steps per episode Interaction cost C Confidence level f Internal reward weight X
0.05
1/30
200
100
40
20
0.1
0.4
2
10
Model architecture
(6666644.64
5 (64.64,644
Policy hidden layers
(64,64)
(64,64)
Learning rate
3 10 10-4
x 10-4
Batch size
256
256
We use an annealing reward weight schedule, i.e., An A (1 n/N).
28


Reviews:
[{"review_id": "ZYwmUUhPZB", "reviewer": "Reviewer_G4x3", "summary": "The paper addresses continuous-time reinforcement learning (RL), where system dynamics are modeled by unknown nonlinear ordinary differential equations (ODEs). The authors propose COMBRL, a model-based RL algorithm that uses probabilistic models to learn an uncertainty-aware approximation of the ODE dynamics. COMBRL balances maximizing extrinsic reward and reducing model epistemic uncertainty, enabling efficient exploration.", "questions": "- How were hyperparameters (e.g., the trade-off coefficient $\\lambda$) set (static or annealing) for the experiments in Figure 3? Was tuning performed per environment, and if so, what protocol was used?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Continuous-time model-based RL is an under-explored area with practical relevance, e.g., in robotic control.\n- The paper includes extensive experiments evaluating design choices and hyperparameter sensitivity across multiple benchmarks.\n- Theoretical guarantees are provided.", "weaknesses": "- In Figure 3, COMBRL’s performance on the primary task is modest: in 5 out of 7 tasks, its error bars overlap substantially with those of the Mean Planning baseline, raising questions about practical gains.\n- The paper assumes familiarity with several methods (e.g., SAC, iCEM, and TaCoS) without brief introductions, which hinders accessibility.\n- The novelty is unclear. While continuous-time RL is valuable, the core idea—-balancing reward maximization and epistemic uncertainty in the objective function—-is not novel. The contribution appears to be an integration of existing components rather than a conceptual advance. Clarifying the key insight beyond \"applying model-based RL to continuous time\" would strengthen the paper.\n- Algorithm 1 lacks sufficient detail. Critical aspects, such as how the optimistic objective is solved, how rollouts are collected, and how the dynamics model is updated, are omitted. Including a more descriptive outline would improve reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IrXRSqBZsM", "reviewer": "Reviewer_F13n", "summary": "This paper introduces COMBRL, a continuous-time optimistic model-based reinforcement learning algorithm designed to balance task performance and model exploration. The authors establish theoretical performance guarantees and validate the approach empirically across a suite of continuous-time deep RL benchmarks.", "questions": "See the weakness. The author may consider to work on a simulated environment for which the true values are know, and experiments should report covedrage rates of the true value.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The theoretical studies are solid.", "weaknesses": "The primary weakness is the numerical evaluation. When ground-truth values are available, experiments should report coverage rates of the true value; limiting results to the averaged returns and standard deviations does not provide an adequate assessment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yxYG21Der8", "reviewer": "Reviewer_8afj", "summary": "This paper studies the continuous-time model-based RL problem where the system dynamics are represented by nonlinear ODEs. The proposed algorithm, COMBRL, is built on uncertainty-aware probabilistic models (GPs or BNNs) and aims to optimize a reward-plus-uncertainty objective in continuous time. The work focuses on two settings: reward-driven RL and unsupervised RL. In experiments, the proposed COMBRL achieves strong empirical results across different continuous-time RL benchmarks and is sample-efficient compared to other baselines.", "questions": "The current settings and tasks are too simple, I wonder can this algorithm be applied to some real-world problems?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I am not an expert in this area but I agree that the study of the continuous-time RL is promising. In addition, this work further explores the unsupervised setting, which is important and interesting.\n- The reward-plus-uncertainty objective makes sense and the results look good.", "weaknesses": "1. My main concern is that the continuous-time RL is already studied in previous work such as [1], in which the system is also modelled by ODEs and the model is built on GP. This makes the novelty incremental.\n2. In Figure 2, the baseline approaches are only mean planner and PETS. It would be better to provide more comparisons with other state-of-the-art methods on these GYM and DMC tasks.\n\n\n\n\n[1] Efficient exploration in continuous-time model-based reinforcement learning, NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hcdLiRxtw6", "reviewer": "Reviewer_LpWF", "summary": "The paper introduces COMBRL, a continuous-time optimistic model-based RL framework that learns probabilistic ODE dynamics and plans policies by maximizing a reward-plus-uncertainty objective governed by a single $\\lambda$ schedule. They analyze both reward-driven and unsupervised (intrinsic-only) regimes, proving sublinear regret and sample-complexity bounds, and demonstrate on Gymnasium and DeepMind Control benchmarks that COMBRL improves sample efficiency over OCORL, PETS, and mean-planning baselines while also handling time-adaptive sensing and control.", "questions": "- In the fixed-rate experiments, do the authors use the default Gym/DMC sampling interval, or do they employ a finer measurement-selection schedule? Please quantify any difference (e.g., actual step size) and discuss how sensitive COMBRL is to noisy or finite-difference estimates of $\\dot{x}$ under that schedule.\n- Table 1 and Table 5 list quite different $\\lambda$ values. Figure 2 credits intrinsic bonuses for COMBRL’s gains, yet Figure 4’s ablation sometimes peaks at $\\lambda = 0$, suggesting exploration may hurt. Could the authors reconcile these results by clarifying how the hand-tuned and auto-tuned schedules were obtained, explaining when exploration helps versus harms (e.g., Hopper), and showing how the auto-tuned $\\lambda$ evolves over time (plots)?\n- COMBRL’s continuous-time focus is compelling, yet BRO [1] remains the state-of-the-art optimistic model-free continuous-control method with a similar exploration rationale—what prevents a direct comparison to show whether COMBRL’s advantages persist against that baseline?\n- Since the method hinges on epistemic bonuses, could the authors provide plots showing how model uncertainty decreases over episodes, especially in the unsupervised regime?\n- Given the exploration focus, could the evaluation be extended to harder DMC benchmarks such as Humanoid-run or Dog-run, and to sparse-reward settings like Maze2D, to substantiate the claimed advantages beyond the current tasks?\n\nI find the continuous-time and unsupervised RL framing compelling; if the authors can address the questions above, I would be inclined to raise my score.\n\n[1] Nauman et al., Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control, 2024.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Clean optimism formulation: a single $\\lambda$ schedule unifies reward-driven and intrinsic exploration, allowing practitioners to traverse between task-focused and unsupervised regimes.\n- Theory for both regimes: sublinear regret for the reward-driven case and sample-complexity guarantees for pure intrinsic learning extend discrete-time optimism analysis to continuous-time ODE settings.\n- Model-agnostic dynamics learning: the framework accommodates Gaussian processes, Bayesian neural nets, ensembles, and other uncertainty-aware estimators, and remains solver/discretization agnostic through its continuous-time formulation.\n- Empirical coverage: experiments span standard Gymnasium and DMC control tasks, include time-adaptive sensing/control, and report downstream-task performance gains using the learned models, showing consistent sample-efficiency improvements over OCORL, PETS, and mean-planning baselines.", "weaknesses": "- Limited scope of benchmarks: results stop at modest Gymnasium/DMC tasks; harder domains (e.g., Humanoid, Dog) and sparse-reward environments (e.g., Maze2D) that would stress exploration are absent, so the empirical evidence for better exploration remains less convincing.\n- Missing modern baselines: comparisons exclude recent model-free optimistic or exploration-focused controllers such as BRO [1], so it is unclear whether COMBRL advances the empirical state of the art beyond older baselines like OCORL and PETS.\n- $\\lambda$ sensitivity and exploration role not sufficiently explored: tables in the Appendix show $\\lambda$ varying considerably across tasks. Plots seem to show contradictory results for how exploration strength impacts performance or when higher bonuses help versus hurt.\n\n[1] Nauman et al., Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)