SOMBRL: Scalable and Optimistic Model-Based RL
Authors: Bhavya Sukhija
Lenart Treven
Carmelo Sferrazza
Florian Dorfler
Pieter Abbeel
Andreas Krause
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
RL Regret bounds in Kernelized Setting
Deep RL
Primary area: reinforcement_learning
TL;DR: Scalable, simple, and practical algorithm for model-based RL with regret bounds across several RL settings and experiments on state-based, visual control and hardware tasks.

Abstract: We address the challenge of efficient exploration in model-based reinforcement learning (MBRL), where the system dynamics are unknown and the RL agent must learn directly from online interactions. We propose **S**calable and **O**ptimistic **MBRL** (SOMBRL), an approach based on the principle of optimism in the face of uncertainty. SOMBRL learns an uncertainty-aware dynamics model and *greedily* maximizes a weighted sum of the extrinsic reward and the agent's epistemic uncertainty.  SOMBRL is compatible with any policy optimizers or planners, and under common regularity assumptions on the system, we show that SOMBRL has sublinear regret for nonlinear dynamics in the (*i*) finite-horizon, (*ii*) discounted infinite-horizon, and (*iii*) non-episodic setting. Additionally, SOMBRL offers a flexible and scalable solution for principled exploration.  We evaluate SOMBRL on state-based and visual-control environments, where it displays strong performance across all tasks and baselines.  We also evaluate SOMBRL on a dynamic RC car hardware and show SOMBRL outperforms the state-of-the-art, illustrating the benefits of principled exploration for MBRL.

SOMBRL: Scalable and Optimistic Model-Based RL
Bhavya Sukhija Department of Computer Science ETH Zurich
Lenart Treven Department of Computer Science
ETH Zurich Srreven@@etz..cm
Carmelo Sferrazza
Florian Dörfler
Berkeley Al Research UC Berkeley
Department Electrical Engineering ETH Zurich dorfler@ethz.cl. ch
csf caferraazza@eezeeeeyycooo edu
Pieter Abbeel
Andreas Krause Department of Computer Science ETH Zurich krausea@ethz.ck ch
Berkeley AI Research
UC Berkeley pabbeel@berekeeeey..o edu
Abstract
We address the challenge of efficient exploration in model-based reinforcement learning (MBRL), where the system dynamics are unknown and the RL agent must learn directly from online interactions. We propose Scalable and Optimistic MBRL (SOMBRL), approach based on the principle of optimism in the face of uncertainty. SOMBRL learns an uncertainty-awan and greedily maximizes a weighted sum of the extrinsic reward and the agent's s epistemic uncertainty. SOMBRL is compatible with any policy optimizers planners, and under common regularity assumptions on the system, we show that SOMBRL has sublinear regret for nonlinear dynamics in the (?) finite-horizon, (ii) discounted infinite-horizon, and (iti) non-episodic settings. Additionally, SOMBRL offers flexible and scalable solution for principled exploration. We evaluate SOMBRL on state-based and visual-control environments. where it displays strong performance across all tasks and baselines. We also evaluate SOMBRL on a dynamic RC car hardware and show SOMBRL outperforms the state-of- ofate-of-the-as the-ahht art, illustrating the benefits of principled exploration for MBRL.
dynamics model
1 Introduction
Figure 1: Top: We showcase scalability of SOMBRL visual control tasks from DMC and Atari. Bottom: We evaluate SOMBRL on a highly dynamic RC car where we learn to perform a complex parking maneuver in only 20 real-world episodes.
Reinforcement learning (RL) has been successfully applied to variety of sequential-decinn decission making problems like games (Silver et al., 2017), robotics (Kober et al., 2013; Tang et al., 2025), mobile
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

health interventions (Yom-Tov et al., 2017; Liao et al., 2020), and the-ttning of large language models (Ouyang et al., 2022). RL offers flexible learning paradigm, enabling agents to learn directly by interacting with their environment. However, this potential is often not fully realized in practice, as most widely used RL methods (Schulman et al., 2017) are highly sample-inefficient. This mostly rules out their direct application real-world settings where data scarce expensive acquire.
Model-based RL approaches (Moerland et al., 2023) offer a more sample-efficient alternative and have been successfully used for learning directly in the real-world (Hansen et al., 2022; Wu et al., 2023; Rothfuss et al., 2024). However, these methods are mostly based on naive exploration strategies, such as Boltzmann exploration, which are provably sub-optimal (Cesa-Biie) Dessa-Bincchi et al., 2017 7) and often struggle in the presence of sparse rewards.
Related Works Several works study principled exploration approaches in RL (Even-Dar & Man- & Szepesvári (201 I); Cohen et al. (2019); Dean et al. (2020); Kakade et al. (2020); Curi et al. (2020); Neu & Pike-Burke (2020); Foster & Rakhlin (2023); Wagenmaker et al. (2023); Sukhija et al. (2024c), see Section 3 and Appendix A for more details). In particular, optimism in the face of uncertainty is celebrated exploration principle with strong the- oretical guarantees for model-based RL (Brafman & Tennenholtz, 2002; Jaksch et al., 2010; Kakade et al., 2020; Curi et al., 2020; Moulin & Neu, 2023; Sukhija et al., 2024b). However, in practice, these algorithms are computationally prohibitive. As result, naive exploration techniques remain dominant in real-world applications due to their simplicity. We address this gap between theory and practice and propose a simple yet principled method, SOMBRL, for exploration that enjoys convergence guaran- tees across several RL settings. This is in contrast to prior works, which design and study algorithms only for specific settings, c.g., Curi et al. (2020); Kakade & Langford (2002) study the inite-horizon setting, Sukhija et al. (2024c) the unsupervised RL. setting, and Sukhija et al. (2024b) the nonepisodic one. In addition to the theoretical guarantees, SOMBRL. is also much more scalable and, unlike the sishdiimensional settings such as visual control problems
sour (2001); Jaksch et al. (2010);
aforementioned works, can be applied and real-world hardware platforms. We demonstrate this in our experiments (Section 6).
The differences between SOMBRL and prior work on exploration in MBRL are summarized in Table 1, and we provide a more detailed discussion of related works in Appendix A.
Contributions
1. We propose SOMBRL, a principled yet efficient exploration strategy for model-based RL. SOMBRL is based on the principle of optimism in the face of uncertainty, and greedily maximizes weighted sum of the extrinsic reward and the agent's epistemic uncertainty/ddiagreeennnt Therefore, the agent selects policies that maximize rewards while also exploring less visited areas of the state space where the uncertainty about the system is high.
2. We show, under common regularity assumptions on the dynamics, that combining extrinsic rewards with the agent's epistemic uncertainty gives anytime high probability value-function bounds, which could be of independent interest applications such safe RL (Brunke et al., 2022) and offline RL (Levine et al., 2020). We leverage this key insight and show that SOMBRL has sublinear regret for continuous state and action spaces. Our regret bounds are comparable the ones derived by prior work (Kakade et al., 2020; Curi et al., 2020; Sukhija et al., 2024b), but our algorithm is considerably simpler and more scalable.
discounted infinite- horizon, and nonepisodic settings with
3. We validate SOMBRL on standard deep RL benchmarks, showing that it outperforms several naive exploration baselines and scales effectively to high-dimensional tasks, such as visual control. We also evaluate SOMBRL in the real-world on dynamic RC car (see Figure 1), where it learns an agile parking maneuver in only 20 trials, outperforming the state-of-the-art (Rothfuss et al., 2024) w.r.t. performance and sample efficiency. To the best of our knowledge, this is the first empirical demonstration of optimistic exploration in model-based deep RL for high-dimensional and woal-world settings.
2 Problem Setting
We consider a discrete-time dynamical system of the form 0t+11 = f (ax,, u.) + w., where æ, E X' C de is the state, u, E 24 S 1R the control input, and w. E W C R the process noise'. The dynamics f are unknown.
For theory, assume the process noise to be known, but algorithm can learn it from data.

---

Table 1: Comparison of SOMBRL and prior work on model-based RL (MBRL) in terms of sublinear regret/sample complexity for the kernelized settings scalability. Kakade (2020) give regret bound for the finite-horizon setting with Gaussian noise and Curi et al. (2020) for sub- Gaussian noise case. We show that SOMBRI has sublinear regret for both settings (Theorem 5.4 and Theorem B.18)
Finile Horioon
y-discomtted inlintt-horioon Nonepisodie Uncoervised Rt.
Scalable + Practicl
Groedy, egg Moi planning or
Deisomooth Rosmussen (201 lj; Chus et all (2018)
Curi (2020); Kakade al. (2020)
X
x
Suitija all (20245)
x
X
Sukhija et al. (2024))
SOMBRL (ours)
Task In the hnite-horizon RL setting (Puterman, 2014), we are given a reward function r xux - E. and want to learn a policy that maximizes the following objective
J J( max J(TT) max EST well
Σ r( Ut)
(I)
TEll
where action u, follows policy π, i.e., u, (x;)). Moreover, we consider the episodic RL setting, and study a model-based approach. Accordingly, at the beginning of episode we select and roll out a policy T,, for I steps on the true system. We then use the data collected from the rollouts estimate the true dynamics f* The goal is to find a policy that performs as well as π²' as quickly as possible. Therefore a natural performance metric in this context
with episodes n E (1....,NY N., .N).
is the cumulative regret Rx = En= J(T') ) - J(T.). In the following sections, we show that our proposed algorithm achieves sublinear regret. While in the main text for clarity we focus on the finite-horizon episodic setting, in Section 5, we show that our approach has sublinear regret also for
y y-discounted
episodic:
and average reward, nonepisodic settings:
T--1
(+") max Ex well
Σ >TT(EE (1))
(2) (agg(T") )- max lim limsup sup
T ET Σ
r(ar, ue)
(3)
well T
3 Exploration Strategies in MBRL
In MBRL, we learn a model of the true dynamics f and use our learned model to select/update the next policy for data acquisition. Exploration algorithms for MBRL determine how the policy should be chosen given our learned model. Common strategies for this choice are (i) greedy planning, (ii)
Thompson sampling, and (iii) optimistic exploration. We discuss these in detail below.
Let JTT,, f) be the the expected returns under the policy æ and dynamics f. that is
J( f) = En
Σ r(al u:)
2'41 = f(xl,, us) + We, 26 = 20.
and μ4, our mean estimate of the dynamics f at episode n.
Greedy planning The simplest selection strategy is to pick the policy n., that maximizes the expected returns for our estimated dynamics Pn-
MEAN arg max J(T, pm))
(4)
wenl
This strategy is greedy as it does not directly encourage exploration in areas where we have limited data or where our model has high uncertainty. Instead, it exploits our estimate pen of the dynamics. This the basis of methods such as those of Janner et al. (2019); Hafner et al. (2023), where exploration is induced using a stochastic policy that is optimized with an entropy bonus.
To incorporate epistemic uncertainty in our learned model and avoid overfitting to misestimated dynamics, Deisenroth & Rasmussen (2011); Chua et al. 22018); Rothfuss et al. (2024) learn Bayesian model of f ( Here D Di and D. { Ur T++1,)) is the data collected in episode i. The policy ,, then selected as
GREEDY π
arg max E. Eyprp( (J(T,
*ell
Curi et al. (2020) show that greedy planning may fail to perform well in practice, especially for difficult exploration problems (c.g., in context of action penalties).
3

---

Thompson Sampling In Thompson sampling (TS), we also learn a Bayesian model p(f(Pin)) and pick policies by maximizing the reward under sampled from the posterior
n
arg max J f). f p(f(Din). HEll
(6)
While TS encourages exploration in theoretically grounded manner (Russo et al., 2018), in practice, it is often intractable to sample a function f from (D:nn)
Optimistic Exploration This strategy is based on the principle of optimism in the face of uncer- lainty, Optimistic exploration approaches maintain set of plausible dynamics models Mn each episode n, e.g., the set of functions that have high probability w.r.t. a learned Bayesian model p.m) The policy is then selected according
OE
arg max J(T, f) fEM.
There are several works that study optimistic exploration theoretically (Jaksch et al., 2010; Kakade al., 2020; Curi et al., 2020; Treven al., 2024: Sukhija et al., 2024b). However, optimizing f over Mn. typically a difficult non-convex constraint, is often computationally prohibitive, restricting the application of these methods to fairly low-dimensional settings. The most efficient solvers of the optimization problem (7), to the best of our knowledge, are based on a reparametrization trick which introduces additional hallucinated controls (Curi et al., 2020). 'his increases the total control dimension from du to du + dg. which is prohibitive in high-dimensional domains.
4 SOMBRL: Scalable and Optimistic MBRL
We present SOMBRL, our approach for efficient optimistic exploration in MBRL, which alternates between two steps. First, given a dataset transitions Dim. we learn an uncertainty-awan model of the unknown dynamics f*. That is, after each episode n. we learn a mean estimate μt, of f and quantify our epistemic uncertainty a,, the estimate. Models such Gaussian processes (GPs) (Rasmussen Williams, 2005) can be directly used for this purpose. Bayesian deep learning ap- proaches such as deep ensembles are also commonly used to quantify epistemic uncertainty or model disagreement in RL (Chua al., 2018; Pathak et al., 2019; Curi al., 2020; Sekar al., 2020; Sukhija et al., 2024c). In the second step, we solve the following optimization problem for the policy n,
Σ r(a'. uz) + Am Malllon. (a's ui)l|
nT = arg max E T
a'+1= pen (nn us) ww,,
(8)
Jn (T)
where Xn is a positive constant which is used to trade off maximizing the extrinsic reward and model uncertainty (see Appendix for how X is defined in theory and Section 5.2 and Appendix D for how it is selected empirically). Note that in Equation (8), we use the mean dynamics for planning and only use the epistemic uncertainty as an additional intrinsic reward. Compared the principled exploration strategies from Section 3, our approach does not require sampling from or maximizing over the dynam- ics. This makes SOMBRL much simpler and more scalable. Moreover, SOMBRL can be combined with any model-based algorithm such as those of Janner et al. (2019); Hafner et al. (2023); Rothfuss et al. (2024). The only additional modification we make to these methods is that we add the epistemic uncertainty the extrinsic reward. Also note that without the epistemic uncertainty reward, i.c., An 0, the agent follows the greedy strategy discussed in Section and for À - oo the agent performs unsu- pervised exploration (Pathak et al., 2017; Sekar et al., 2020; Buisson- et al., 2020; Sukhija et al., the model uncertainty to facilitate principled exploration for the agent.
2024c). Therefore,
In the following, we show that by optimizing our objective in Equation (8), we are effectively maximizing optimistic estimate of J i.e., we are also performing optimistic exploration. Accordingly, our approach enjoys the same guarantees as other optimistic MBRL algorithms but is much simpler, computationally cheaper, and scalable to high nigh-dimensional settings.
5 Theoretical Results
For our analysis, make some common assumptions on the underlying dynamics f*.

---

can 5.1 Assumptions
We first make continuity assumptions on the system. These assumptions are common the control theory (Khalil, 2015) and reinforcement learning literature (Curi et al., 2020; Sussex et al., 2023; Sukhija et al., 2024c).
Assumption 5.1 (Continuous closed-loop dynamics, bounded rewards, and Gaussian noise.). The dynamics model f and all π E Il are continuous. Furthermore, we assume that the reward is bounded, i.e., r X' U [0, Rmax], and process noise is i.i.d. Gaussian" with variance o², i.e.,
we A (0,00*1)
SOMBRL learns an uncertainty-awann model of the true dynamics f i.e., mean J4n and uncertainty nN estimate. In our theoretical analysis, we focus on Gaussian Process (GP) dynamics models. For GPs, Mn, and a have a closed-form solution, and our learned model is calibrated (c.f.,Rotffuss et al. (2023) Appendix B.1 for the definition of well-calibrated models Kuleshov al. (2018) () on calibration for BNNs). Generally, our guarantees can be extended to broader classes of well-calibrated models, e.g., BNNs (similar to Curi et al. (2020)).
We assume that f resides in Reproducing Kerne Hilbert Space (RKHS) of vector-valued functions.
Assumption 5.2. We assume that the functions fj. E (1,....da, d2 lie in RKHS with kernel k and have a bounded norm B, that is f E H. with H = {f llf S B.j = I...., d Moreover, we assume that l=2, z) S max for all x E X.
Assumption 5.2 ensures that we can model and learn f with GPs. This assumption is common in the Bayesian optimization (Srinivas al., 2012; Chowdhury Gopalan, 2017) and RL literature (Kakade et al., 2020; Curi et al., 2020). Moreover, GPs are nonparametrie models and can learn very complex classes of nonlinear functions (Rasmussen & Williams, 2005).
The posterior mean p4n (z) = [um_ (2) (=)sssa and epistemic uncertainty o,())) = then be obtained using the following formula
Mn.j(=) = k, (=) 1 00) -
, (2) = k(z,z) k (K) + -0
Here, vinn corresponds to the noisy measurements of fj. i.e., the observed next state from the transitions dataset Dim. kn,z)
kx2 Z.)x,,E)) and Kn [k,,,,,1)]=. = is the data kernel matrix. The restriction on the kernel k(2,z) S mmax implies boundedness of f and has also appeared in works studying the episodic setting for nonlinear dynamics (Mania et al., 2020; Kakade al., 2020; Curi al., 2020: Wagenmaker et al., 2023; Sukhija et al., 2024c). We can also define f such that D2 = - + (2 ux-1) 1)++ + uk-1 in which case the boundedness of f captures many real-world systems.
Our theoretical results depend on the maximum information gain of kernel (Srinivas al., 2012), defined as
1 log I
(k(k)= =
max AC 2
N' is a measure of the complexity for learning f from N episodes and is sublinear for many kernels (e.g., 0(0og4+4d++1(N)) for the squared exponential (RBF) kernel, ((d,, + d.y, log(N)) for the linear kerne).. In Appendix B, we report the dependence of I'N on N in Table 2.
Next, we present the following Lemma, which states that J (T,,) from Equation (8) is an optimistic estimate of J(Tr"..
Lemma 5.3. Let Assumption 5.1 and Assumption 5.2 hold. Then, there exists a An E ((VTN), such that we have Vn > 0, π E 11, with probability at least 1- 8, that J(T)) S (T)... Moreover, we have J(xT) S Jn (-T,)
Proof Sketch: Crucially, we leverage the policy difference lemma (Kakade & Langford, 2002), which bounds the difference between the performance of two different policies with the advantage function. However, we study this lemma for fixed policy but different dynamics, effectively obtaining a simulation lemma (Kearn & Singh, 2002) for our setting. Next, we bound the difference
For clarity of exposition, we focus the setting with Gaussian noise. In Appendix B, we also perform the analysis for the more general Lub-Gaussian noise case.
5

---

in performance between the mean dynamics and the true one and show that this is proportional to the model epistemic uncertainty.
Lemma 5.3 shows that for all policies π I1, Jn, (TT) gives an upper bound on the true return J(x). This result is of independent interest and can be applied to settings beyond online RL, such as safe RL (Brunke et al., 2022; As et al., 2024) and offline RL (Levine et al., 2020; Yu et al., 2020; Rigter al., 2022). The exact bound for An is provided in Lemma B.3 in Appendix B.
Finally, we present our main theorem, which bounds the regret of SOMBRL.
Theorem 5.4 (Finite horizon setting). Let Assumption 5.1 and Assumption 5.2 hold. Then we have
VN > 0 with probability at least 1 - 8, Rx S SO (TW
Proof Sketch: To bound the regret, we first prove that Equation (8) is optimistic estimate of Equation (1). Then, we use Lemma 5.3 to bound the difference in performance for extrinsic rewards between the mean and true dynamics with the epistemic uncertainty of the collected rollout. Next, analyse the intrinsic reward term and also show that it's bounded by the epistemic uncertainty of the collected rollout. This allows us to relate the cumulative regret Ry with the information gain I' and obtain the final bound.
Theorem 5.4 guarantees sublinear regret for rich class of RKHS functions. Accordingly, for many RKHS, our algorithm enjoys the same asymptotic guarantees as Kakade et al. (2020). Note that the regret bound from Kakade et al. (2020) is an order of vT better. On the other hand, SOMBRL a much simpler and more scalable algorithm. In Appendix B, we show that SOMBRI. improves the regret bound from Curi et al. (2020), for the sub- Gabssinn noise case, by an exponential factor of T Below, we also provide our regret bounds for the y-discouneed and the non-epison non-episodic setting?.
Theorem 5.5 (r-discounted, infinite horizon setting). Let Ry = >n (T(')) (T,). Under the Assumptiio 5.1 and Assumption 5.2, we have for the y-discounted infinite (Equation (2)) horizon setting VN > 0 that with probability at least -, 8, RN S. O (F VNN
Proof Sketch: The regret decomposition for this setting is similar to the finite-horizon case (Theorem 5.4). However, in contrast to the moite-horizon case, where each episode has a fixed length T,, we care about the infinite horizon in the y-discounted case. Therefore, to obtain sublinear regret for this setting, we require the agent to observe the system for longer horizons, i.e., the horizon ((n) + oc for n - co. We ensure this by picking T(n) E @(log(n)) and show that this is sufficient to obtain sublinear regret.
In Theorem 5.5, we show that even though we truncate each episode after T(n) steps, SOMBRL has sublinear regret w.r.t. the infinite horizon objective. Moreover, the regret for this setting follows the same structure as for the finite horizon case. To the best of our knowledge, we are the first to give regret bound for optimistic model-based RL. algorithms for the y-discounted setting.
Finally, we give our regret bound for the non-episodic setting. As pointed out in Kakade (2003); Sharma et al. (2021), this is the most challenging and closest setting for learning directly in the real-world, Sukhija et al. (2024b) show that optimistic exploration methods have sublinear regret for the nonepisodic setting. However, their proposed algorithm is intractable in practice.
Theorem 5.6 (Informal statement; nonepisodic average reward case). Let Rx = =EN E[Jan((TT) r(z,, n n (.)))) Under the same assumptions as Sukhija et al. (2024b), we have for the average
reward setting (Equation (3)) VN > 0 that with probability at least 6, RN S SO (1
The regret analysis for this setting is, in spirit, similar to the finite horizon case. However, in the nonepisodic setting, we cannot reset the agent and have to learn from a single trajectory. Therefore, unlike the episodic case, where we update our model and policy after every episode, in the nonepisodic setting we have to decide when to update the agent. For SOMBRL we show that if we
Proof Sketch:
only update once we have accumulated enough information, i.e., -t=0 | (Tt,,. )} C, for a positive constant C, then SOMBRL has sublinear regret. Intuitively, the model uncertainty measures how much information our agent has acquired since its last update at time step T [n 1). We only update the agent once the information exceeds the threshold C.
Both Kakade al. (2020) and Curi et al. (2020) do not provide regret bound for these settings.

---

Nonepisodic Setting
Episodic Setting
SOMBAL (OURS)
NCoRL
NsPETS
NEMEAN
NETB
HUGRL
PETS
MEAN
Pendulum
MountainCar
Pendulum
1.0
100
J(EN)
0.5
0.0
-10
x10
X 11
MountainCar
2.5
1.5
J(mx)
0.5
1,0
0.0
0.0
200
400
600
200
430
500
830
1000
Env steps N
Eny steps IV
Episodes N
Figure 2: Left: Learning curves for the nonepisodic setting with GP dynamics. We report the average reward Javg( Vavg(TN (T)) and regret Ry- The curves are reported with 5 seeds, and we plot the median return with its standard deviation. Right: Learning curves for the episodic setting with GP dynamics. We report the median episode reward ( over an episode with 5 seeds and its standard deviation.
In contrast to Sukhija al. (2024b), SOMBRL is much more tractable, and in Theorem 5.6 show that SOMBRL also has sublinear regret in the nonepisodic setting and therefore offers a theoretically strong and practical alternative for model-basee exploration for this case.
In this section, have shown SOMBRL, which maximizes combination of the extrinsic reward and the model epistemic uncertainty, enjoys sublinear regret for common kernels and RL settings. There are principled exploration algorithms designed individually for these settings, e.g.. (Kakade al., 2020; Curi al., 2020) for the finite-horizon case and Sukhija et al. (2024b) for the non-episodic case. However, they are often intractabeccommutaatoonnll prohibitive. contrast, SOMBRL works across the different RL settings while also being more practical and scalable.
We present additional theoretical results, for example, sample complexity bound for unsupervised RL algorithms such as Sekar et al. (2020); Quisson-Fende al. (2020); Sukhija et al. (2024c) and a regret bound for the sub- Gaussian noise setting in Appendix B. Our detailed proofs are also provided in Appendix B.
5.2 Selecting >, in practice
The parameter An controls the intooratioo--xploitaatiii trade-off for SOMBRL. In Appendix B we provide the theoretical bound for A., however in practice, An is treated as hyperparameter. This similar to other optimistic exploration and intrinsic exploration algorithms (Burda et al., 2018; Kakade et al., 2020; Curi et al., 2020), which also heuristically select the amount of exploration. Sukhija et al. (2024a) empirically study combining extrinsic and intrinsic rewards for model-free algorithms and propose an approach for automatically tuning the intrinsic reward coefficient, i.e.. An We find their approach works well for our state-based and visual control tasks. Moreover, we describe their approach and how choose A, for experiments in Appendix D.
5.3 Application of SOMBRL with GP dynamics
Finally, we empirically validate our theoretical findings for the GP case in Figure 2, where we compare SOMBRL to HUCRL (Curi et al., 2020), PETS (Chua et al., 2018), and greedy (mean) planning in the episodic setting. For the nonepisodic setting, consider their nonepisodic counterparts as proposed in Sukhija et al. (2024b). We evaluate the algorithms on the Pendulum and MountainCar tasks from the OpenAl Gym benchmark (Brockman et al., 2016). From the experiments, we conclude that SOMBRL performs the best across all baselines for both the episodic and the nonepisodic setting. Morcover, while HUCRL and NEORL, which explore according to Equation (7), perform better than other baselines, they are worse than SOMBRI. We believe this is because of the practical challenges associated with solving the optimization problem in Equation (7). Moreover, solving Equation (7) is also computationally more expensive. For instance, in our experiments, HUCRI. requires roughly 3x more compute time than OMBRL (see Appendix D.5).
7

---

6 Experiments
In our experiments, we showcase the flexibility and scalability of SOMBRL by combining it with three different model-based RI. algorithms; (i) MBPO (Janner et al., 2019) for state-based tasks, DREAMER (Hafner et al., 2023) for visual control tasks, and SIMFSVGD (Rothfuss et al., 2024) for our hardware experiment on the RC car. Note that principled exploration methods such as Kakade et al. (2020); Curi et al. (2020) do not scale to the settings, such as visual control tasks, considered in this work. We consider the DeepMind control (DMC) benchmark (Tassa al., 2018) for the state-based and visual control tasks and test on environments with varying dimensionality" We also evaluate on several environments from the Atari benchmark (Bellemare et al., 2013) for the visual control tasks. In all our experiments, we report the episodic returns using the median over 5 seeds along with its standard deviation. We provide additional experiment details in Appendix D.
State-based experiments We refer to the MBPO version of SOMBRL as MBPO-OPTIMISTI The resulting algorithm operates similarly to Janner et al. (2019) and trains a policy from real and model- generated rollouts to maximize the extrinsic and intrinsic rewards. For the policy training, we use the soft actor-critic (SAC) algorithm (Haarnoja et al., 2018), and for the intrinsic reward coefficient, An. we use the auto-tuning approach from Sukhija et al. (2024a). We train an ensemble of dynamics models and use their disagreement to quantify the epistemic uncertainty. As baselines, consider (i) TIBPO-MEA
which maximizes only the extrinsic reward, i.e. An = O. and (ii) MBPO- PETS, which is based on the PETS algorithm (Chua et al.. 2018) 1)) maximizing the extrinsic rewards in expectation over the ensemble dynamics (see Equation (5)). We report the results on the left side of Figure 3. We conclude that across all tasks, OPPIMIOIIIII performs the best. Particularly, in sparse reward tasks such as the Mountaincar and CartPole, MBPO-OPTH PAPO-OPTIMISTIC successfully solves the task whereas the greedy baselines fail. MBPP-OPTIMISTIC also successfully scales high dimensional problems such as the Quadruped and Humamoid environments. We provide additional experiments with P1BPO-PPTMMT and compare it with pure off- policy algorithms SAC and MaxInfoRL (Sukhija et al., 2024a).
in Appendix C, where we evaluate it on more environments
Visual control experiments We investigate the scalability of SOMBRL to challenging and high-dimensional problems by evaluating it on visual control tasks. We combine SOMBRL with DREAMER (Hafner et al., 2023), an MBRL algorithm for visual control problems, and call the resulting algorithm DREAMER- OPTIMISTIC. PRRAMER-OPTMIISTIC quantifying the epistemic uncertainty and for selecting the intrinsic reward coefficient, Xn, we use the auto-tuning approach from Sukhija et al. (2024a). We report the results on the right side of Figure 3. Overall, DREAMER -EEEEROOPTMMSSSCC performs on-par with DREAMER on most tasks and outperforms on the Finger-spin task from DMC and the Venture task from the Atari benchmark. Particularly. for Venture. a sparse reward task, Dreamer fails to achieve any reward. Curi et al. (2020) study the sensitivity of greedy exploration algorithms w.r.t. the action penalties in the reward. Inspired by their experiments, we modify the reward for the CartPole and Finger spin environments by adding an action cost, Taction (a) = -K ||a||| where K controls the penalty for large actions. Curi et al. (2020) show that even for small action costs, greedy cxploration methods fail, converging to the sub-optimal solution of applying small actions. We observe a similar outcome in Figure 4 (left side), where DREAMER fails to solve the tasks for both the Finger spin and CartPole environments. On the other hand, -PE--IIIIOSSTIICC achieves much higher returns due to its optimistic exploration.
We use the same approach as Sekar et al. (2020) for
We provide additional experiments with DREAMER- OPTMMMSST,,,IIIIC including more environments and proprioceptive tasks in Appendix C.
Hardware experiments Rothfuss et al. (2024) propose a novel approach for training deep Bayesian models that incorporates low-fidelity physical priors. Their approach significantly improves sample efficiency, which they illustrate in their hardware experiments on an RC car. Inspired by their experimental setup, we conduct a similar experiment a highly dynamic RC car. The task to perform a complex parking maneuver with drifting as depicted in Figure 1. We use the same experimental setup as Rothfuss et al. (2024).
First, we evaluate our algorithm SIMFSVGD, in simulation. The simulation is based realistic race car simulation from Kabzan et al. (2020). For the simulation experiments, we ablate different choices for the reward parameters, starting with the dense reward configuration from Rothfuss et al. (2024) and adapt parameters to obtain sparser rewards (see Appendix D for more detail). We report the results in the top row of Figure+ 4. For the dense reward setting, SIMFSVGD and SIMFSYGD-OPTLMISTU perform similarly, but for sparser rewards, SIMFS - outperforms SIMFSVGD. In particular,
SETPSVDD-OPTMISSSS
a combination of SOMBRL and
'including the humanoid from DMC: da 67. du 21
8

---

Model based policy optimization
Visual control tasks
XAPО-CO-IIIITT roustic (ouRs)
MIPNO-PFIS
MAPOO-MFA
DAF-MER A TISESTIC (OURS)
DERAMER
MountainCar contrnouus
Cartpole swirgup sparso
Hogper hop
Atre: Montecuma Rovenge
Abart Venture
DMAC: Hopper Hop
130
2003
TERR
203
0.0
0-2
12t
Hamaraid
Hurnaridd
DMC: Fingur Spin
Quadrused
2002
100
230
ctops steps
stops
steps
steps
steps
Figure 3: Left: Learning curves for the state-based tasks from DMC using MBPO as the base algorithm. Across all experiments, MBPO-OPTTMISTIC obtains the best performance compared to its greedy variants. AIBPO-OPTIMIS humanoid environments from DMC. Right: Learning curves for the visual control tasks from DMC and Atari using DREAMER as the base algorithm. DREAMER- PRRAMER-OPTIMISTTE OPTIMISTIC either performs on-par or better than DREAMER in all our experiments. Particularly, the Venture task from the Atari benchmark, where DREAMER fails to obtain any rewards.
also scales figh-ddmensional tasks. specifically the
Visual control learning with action cost
Online RL on RC Car
DRIAMER OPT 5T10 (OURS)
DREaaren
SMFSVGD OPTIMISTIC (OURS)
SWFSV3D
Ki 0.1
0.25
K- 0.4
Donse reward
Sparse roward
Very sparse reward
633
22+
0.
05
13
Eploodes
Episodes
Eplecdes
002
[Sparaw swaas))
hingge
401-
Hardinee
Harcwware
400-
400
-00
Frz stece N:x 10*
Frn stwce N'x 10
Erz staps 04 10
Eiploddes N
Figure 4: Left: Learning curves with action costs, where we compare DREAMER with DREAMER- OPTIMISTIC. DREAMER fails to explore sufficiently with action costs, whereas DREAMER- OPTIMISTIC able to explore and obtain much higher performance. Right: Learning curves for our experiments with SIMFSVGD. Top row: We change the parameters of the reward function from Rothfuss et al. (2024), and make it sparse, starting from their dense reward. We observe that, as the reward gets sparser, SIMFSVGD drops in performance and SEAFSVGD-OPTIMISS outperforms it. Bottom row: We run the sparse reward configuration on hardware (depicted on the right side at the bottom), where we obtain similar results. As opposed to SIMFSVGD-OPPIMISS SIMFSVGD fails solve the task.
for the setting with very sparse rewards, SIMFSVGD completely fails to solve the task. We conduct our hardware experiments using the sparse reward configuration, and report the learning curve in the bottom row of Figure 4. In line with the simulation experiments, we also observe similar behavior SESSSGD-OPPMISTTEEE learns to solve the task, whereas SIMFS VGD completely fails. In fact, out of 5 attempts, SIMFSVGD worked only once, and otherwise converged to local optimum of not moving from the starting position'.
on hardware.
7 Conclusion
In this work, we propose SOMBRL, which maximizes a weighted sum of the extrinsic reward and the agent's 's epistemic uncertainty. We show that SOMBRL effectively performs optimistic exploration and provide regret bounds for it in variety of settings, in particular for continuous state-action spaces
Video is available at https: :tttps://sukke h/sukhijjbb gitgtttthhuuu ht/prooeecs/ssobrl/////

---

and many common classes of RL problems, namely, horizee infinite-horizon, episodic, and non-episodic RL. To the best of our knowledge, are the first to provide these theoretical guarantees, yielding a more flexible, scalable, and principled algorithm for exploration. We illustrate the strengths of SOMBRL in our experiments, where we combine SOMBRL with different model-based RL algorithms, evaluate on tasks of varying dimensionality, including visual control problems, and also illustrate the benefits of optimistic exploration on hardware. In all cases, SOMBRL achieves the best performance, being significantly more scalable and computationally cheaper than prior optimistic exploration methods and stronger exploration than SOTA deep RL baselines.
limitation of SOMBRL is that requires training a probabilistic model, e.g., deep ensembles, for quantifying epistemic uncertainty. We report the computational cost of our method in Appendix D.5 and show that the computational overhead is small compared to the training cost for the actor and critic.
A Acknowledgments
We thank Scott Sussex for the insightful discussion and feedback on this work. This project has received funding from the Swiss National Science Foundation under NCCR Automation, grant agree- ment 51NF40 180545, the Microsoft Swiss Joint Research Center, and the SNSF Postdoc Mobility Fellowship 21 1086. Bhavya Sukhija was gratefully supported by ELSA (European Lighthouse on Secure and Safe AI) funded by the European Union under grant agreement No. 101070617.
References
Abbasi-Yauee Yadkori, Y. and Szepesvári, C. Regret bounds for the adaptive control of linear quadratic systems. In Conference on Learning Theory, 2011.
Abeille, M. and Lazaric, A. Efficient optimistic exploration in linear-quadratic regulators via lagrangian relaxation. In International Conference on Machine Learning, 2020.
As, Y., Sukhija, B., Treven, L., Sferrazza, C., Coros, S., and Krause, A. Actsafe: Active exploration with safety constraints for reinforcement learning. arXiv preprint arXiv:2410.09486. 2024.
Åström, K. and Wittenmark, B. Problems of identification and control. Journal of Mathematical analysis and applications, 34(1):90-113) 113, 1971.
Aubret, A., Matignon, I.., and Hassas, S. An information-theoreti perspective on intrinsic motivation in reinforcement learning: A survey, Entropy, 2023.
Auer, P. Finite-time analysis of the multiarmed bandit problem, 2002.
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based exploration and intrinsic motivation. NeurIPS, 2016.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Berkenkamp, F. Safe exploration in reinforcement learning: Theory and applications in robotics. PhD thesis, ETH Zurich, 2019.
Brafman, R. I. and Tennenholtz, M. R-max-a general polynomial time algorithm for optimal reinforcement learning. JMLR, 2002.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint 1606.00.44 2016.
Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A. P. Safe learning in robotics: From learning- based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 2022.
Buisson-Fenet, M., Solowjow, F., and Trimpe, Actively learning gaussian process dynamics. In L4DDC. PMLR, 2020.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. arXiv preprint arxiv:18100 011.1/110....4. /2894, 2018
Bisa-Bianchi, N., Gentile, C., Lugosi, G., and Neu, G. Boltzmann exploration done right. Advances in neural information processing systems, 30, 2017.
10

---

Chiuso, A., Fabris, M., Breschi, V., and Formentin, S. Harnessing uncertainty for a separation principle in direct data-driven predictive control. arXiv preprini urXiv:23/2.14788, 2023.
Chowdhury, S. R. and Gopalan. A. On kernelized multi-armed bandits. In International Conference on Machine Learning, pp. 844-853, PMLR, 2017.
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning handful of trials using probabilistic dynamics models. In NeurlPS, 2018.
Cohen, A., Koren, T., and Mansour, Y. Learning linear-quadratic regulators efficiently with only V regret. In International Conference on Machine Learning, 2019.
Curi, S., Berkenkamp, F., and Krause, A. Efficient model-based reinforcement learning through optimistic policy search and planning. NeurIPS, 33:14156-14170 2020.
Dean, S., Mania, H., Matni, N.. Recht, B., and Tu, S. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics, 20(4):633-679, 2020.
Deisenroth, M. and Rasmussen, E. Pilco: A model-based and data-efficient approach to policy search. In ICML, pp. 465-472, 2011.
Even-Dar, E. and Mansour, Y. Convergence of optimistic and incremental q-learning. NeurlPS, 14, 2001.
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without reward function. arXiv preprint 1rXiv:1802.00677) 2018.
Faradonbeh, M. K. S., Tewari, A., and Michailidis, G. Optimism-based adaptive regulation of quadda-q systems. IEEE Transactions on Automatic Control, 2020.
Foster, D. and Rakhlin, A. Foundations of reinforcement learning and interactive decision making. arXiv preprini 2rXiv:2312.16720 2023.
Grimaldi, R. A., Baggio, G., Carli, R., and Pillonetto, G. The bayesian separation principle for data-driven control. arXiv preprint arXiv:2409. 16717, 2024.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint 181.0:181 2018.
Hafner, D., Pasukonis. J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.cmm 10044404444,,. 2023.
Hansen, N., Lin, Y., Su, H., Wang, X., Kumar, V., and Rajeswaran, A. Modem: Accelerating visual model-based reinforcement learning with demonstrations. arXiv preprint arXiv:2212.05698 2022.
Hansen, N., Su, H., and Wang, X. Td-mpe2: Scalable, robust world models for continuous control.
arXiv preprint arXiv:2310.1682,, 2023.
Jaksch, T., Ortner, R., and Auer, P. Near-optimal regret bounds for reinforcement learning. JMLR, 2010.
Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-baser policy optimization. NeurlPS, 32, 2019.
Kabzan, J., Valls, M. I., Reijgwart, V. J., Hendrikx, H. F., Ehmke, C., Prajapat, M., Bühler, A., Gosala, N.. Gupta, M., Sivanesan, R., et al. Amz driverless: The full autonomous racing system. Journal of Field Robotics, 2020.
Kakade, S. and Langford, J. Approximately optimal approximate reinforcement lcarning. In ICML, pp. 267-274, 2002.
Kakade, S., Krishnamurthy, A., Lowrey, K., Ohnishi, M., and Sun, W. Information theoretic regret bounds for online nonlinear control. NewrIPS, 33:15312-1533 155325,, 2020.
Kakade, S. M. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003.

---

Kanagawa, M., Hennig, P., Sejdinovic, D., and Sriperumbudur, B. K. Gaussian processes and kernel 2018.
methods: A review connections and equivalences. arXiv preprint
Kearns, M. and Singh, S. Near-optimal reinforcement learning in polynomial time. Machine learning, 2002.
Khalil, H. K. Nonlinear control, volume 406. Pearson New York, 2015.
Klyubin, A. S., Polani, D., and Nehaniv, C. L. Empowerment: A universal agent-centric measure of control. In IEEE congress on evolutionary computation, 2005.
Kober, J., Bagnell, J. A., and Peters, J. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 2013.
Kuleshov, V., Fenner, N., and Ermon, S. Accurate uncertainties for deep learning using calibrated regression. In ICML, pp. 2796-2804 PMLR, 2018.
Levine, S., Kumar, A., Tucker, G., and Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005. 00000000.063,,... 2020.
Liao, P., Greenewald, K., Klasnja, P., and Murphy. S. Personalized heartsteps: A reinforcement learning algorithm for optimizing physical activity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2020.
Mania, H., Jordan, M. I., and Recht, B. Active learning for nonlinear system identification with guarantees. arXiv preprint arXiv:2006. 20207,0020.
Mendonca, R., Rybkin, O., Danilidiis, K., Hafner, D., and Pathak, D). Discovering and achieving goals via world models. NeurlPS, 2021.
Moerland, T. M., Broekens, J., Plaat, A., Jonker, C. M., al. Model-based reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-118, 2023.
Moulin, A. and Neu, G. Optimistic planning by regularized dynamic programming. International Conference on Machine Learning, pp. 25337-25357. PMLR, 2023.
Nair, A. V., Pong, V., Dalal, M., Bahl, S., Lin, S., and Levine, S. Visual reinforcement learning with imagined goals. NewrIPS, 2018
Neu, G. and Pike-Burke, C. A unifying view of optimism in episodic reinforcement learning. NeurIPS, 2020.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2788 PMLR, 2017.
Pathak, D., Gandhi, D., and Gupta, A. Self-supervised exploration via disagreement. In ICML. pp. 5062-5071. PMIR, 2019.
Pinneri, C., Sawant, S., Blaes, S., Achterhold, J., Stueckler, J., Rolinek, M., and Martius, G. Sample- efficient cross-entropy method for real-time planning. In Conference on Robot Learning, pp. 1049-1065. PMLR, 2021.
Pong. V. H., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S. Skew-fit: State-covering self- supervised reinforcement learning, arXiv preprint arXiv: 03698, 2019.
Puterman, M. L.. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Rasmussen, C. E. and Williams, C. K. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
Rigter, M., Lacerda, B., and Hawes, N. Rambo-rl: Robust adversarial model-based offline reinforce- ment learning. NeurlPS, 15006 2022.
12

---

Rothfuss. J., Sukhija, B., Birchler, T., Kassraie, P., and Krause, A. Hallucinated adversarial control for conservative offline policy evaluation. UAI, 2023.
Rothfuss, J., Sukhija, B., Treven, L., Dörfler, F., Coros, S., and Krause, Bridging the sim-to-real gap with bayesian inference. IROS, 2024.
Russo, J.J., Van Roy, B., Kazerouni, A., Osband, I,, Wen, Z., et al. A tutorial on thompson sampling. Foundations and Trends0 in Machine Learning, 2018.
Salge, C.,, Glackin, C., and Polani, D. Empowerment-an introduction. Guided Celf-Organization: Inception, 2014.
Schmidhuber, J. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference simulation of adaptive behavior: From animals to animats, pp. 222-227,11 1991.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.063377,7001
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Planning to explore via supervuper world models. In International conference on machine learning, 2020.
Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-awarc unsupervised discovery of skills. arXiv preprint arXiv:/907.01652/ 2019.
Sharma, A., Xu, K., Sardana, N., Gupta, A., Hausman, K., Levine, S., and Finn, C. Autonomous reinforcement learning: Formalism and benchmarking. arXiv preprint arXiv:2112.09660 2021.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 2017.
Simchowitz, M. and Foster, D. Naive exploration is optimal for online lqr. In International Conference on Machine Learning. PMLR, 2020.
Srinivas, N., Krause, A., Kakade, M., and Seeger, M. W. Information-thherrtiie regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory, 2012.
Stadie, B. C., Levine, S., and Abbeel, P. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814 2015.
Sukhija, B., Coros, S., Krause, A., Abbeel, P., and Sferrazza, C. Maxinforl: Boosting exploration in reinforcement learning through information gain maximization. arXiv preprint arXiv:2412.12091 2024a.
Sukhija, B., Treven, I.,, Dörfler, F., Coros, S., and Krause, Neorl: Efficient exploration for nonepisodic rl. NeurIPS, 2024b.
Sukhija, B., Treven, L.., Sancaktar, C., Blaes, S., Coros, S., and Krause, A. Optimistic active exploration of dynamical systems. NeurlPS, 2024c.
Sussex, S., Makarova, A., and Krause, Model-based causal bayesian optimization. In ICLR, May 2023.
Tang, C., Abbatematteo, B., Hu, J., Chandra, R., Martín-Martin, R., and Stone, P. Deep reinforcement learning for roboties: A survey of real-world successes. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A.,
Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv: 000690, 2018.
Treven, L., Curi, S., Mutný, M., and Krause, A. Learning stabilizing controllers for unstable linear quadratic regulators from a single trajectory. In Learning for Dynamics and Control, 2021.
Treven, I.., Hübotter, J., Sukhija, B., Dörfter, F,, and Krause, A. Efficient exploration in continuous- time model-based reinforcement learning. NeurIPS, 2024.
13

---

Wagenmaker, A., Shi, G., and Jamieson, K. Optimal exploration for model-based rl in nonlinear systems. arXiv preprint arXiv:2306.092/0, 2023.
Wu, P., Escontrela, A., Hafner, D., Abbeel, P. and Goldberg, K. Daydreamer: World models for physical robot learning. CORL. PMLR, 2023.
Yom-Tov, E., Feraru, G., Kozdoba, M., Mannor, S., Tennenholtz, M., and Hochberg, I. Encouraging physical activity in patients with diabetes: intervention using a reinforcement learning system. Journal of medical Internet research, 2017.
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. NeurIPS, 2020.
14

---

A Related Work
Deep model-based RL. Model-based RI. algorithms offer a sample-efficient solution for learning directly in the real world Hansen et al. (2022); Wu et al. (2023); Rothfuss et al. (2024). Most widely applied algorithms (Chua et al., 2018; Janner et al., 2019; Hafner et al., 2023; Hansen et al., 2023) commonly rely on naive exploration techniques such as Boltzmann exploration and differ primarily in the type of dynamics modeling and policy planners. resa-Bianchi et al. (2017) show that Boltzmann exploration is suboptimal even in the simplified setting of stochastic bandits. SOMBRL is agnostic to the choice of modeling and planners, as we demonstrate in Section 5 and 6. Moreover, we focus on the problem of exploration for MBRL and propose a principled exploration approach. We derive regret illustrate the
bounds for our approach, showing that it is theoretically grounded. Furthermore, benefits of principled exploration in our hardware experiment, where the naive exploration baseline fails to obtain any meaningful exploration. To the best of our knowledge, we are the first to propose a simple, flexible, scalable, and theoretically grounded approach for principled exploration and show its benefits directly in the real world.
Theoretical results for Model-based RL dynamical systems theoretically (Abbasi- AAbbasi-Yadkori & Szepesvári, 2011; Cohen et al., 2019; Simchowitz & Foster, 2020; Dean et al., 2020; Faradonbeh et al., 2020; Abeille & Lazaric, 2020; Treven et al., 2021), focusing primarily on the challenges of nonepisodic learning. In the nonlinear case, Kakade et al. (2020); Curi et al. (2020); Mania et al. (2020); Wagenmaker et al. (2023); Treven et al. (2024) analyze the finite-horizon episodic setting and provide regret bounds that are sublinear for many RKHS. Recently, Sukhija et al. (2024b) extended these results to the nonepisodic setting. Crucially, most of these algorithms are based on the principle of optimism in the face of uncertainty and require solving the problem in Equation (7). As highlighted in Section and 3, solving these problems is often intractable or computationally expensive. Therefore, naive exploration techniques, such as Boltzmann exploration are more widely used. SOMBRL addresses this drawback and proposes an alternative optimistic exploration method, which is much simpler and more scalable, Furthermore, enjoys the same asymptotic guarantees as these methods and hence is also theoretically grounded.
There are numerous works that study MBRL for linear
Intrinsic exploration in RL Intrinsic rewards are often used as a surrogate objective for principled exploration in challenging tasks (see Aubret et al., 2023, for a comprehensive survey). Common choices of intrinsic rewards are model prediction error "Curiosity" (Schmidhuber, 1991; Pathak transitionn//sttc-vviiitaaoo visitation counts (Stadie et al., 2015;
et al., 2017; Burda et al., 2018), novelty of transitions/state- Bellemare et al., 2016), diversity of skills/goals (Eysenbach et al., 2018; Sharma et al., 2019; Nair et al., 2018; Pong et al., 2019), empowenment (Klyubin et al., 2005; Salge et al., 2014), and informa- tion gain of the dynamics (Sekar et al., 2020; Mendonca et al., 2021; Sukhija et al., 2024c). However, these rewards are mostly used for pure exploration and rarely considered in combination with the extrinsic reward. We show that combining the model epistemic uncertainty, an intrinsic reward, with the extrinsic one, effectively performs optimistic exploration, thus, providing a theoretical grounding for our approach. There a few works from bandits (Auer, 2002; Srinivas et al., 2012), data-driven control (Aström & Wittenmark, 1971; Chiuso et al., 2023; Grimaldi et al., 2024), and RL. (Abeille & Lazaric, 2020; Sukhija et al., 2024a) that have also proposed maximizing extrinsic rewards jointly with epistemic uncertainty. The data-driven control community refers to this as the separation principle between model identification and control design (Aström & Wittenmark, 1971; Chiuso et al., 2023; Grimaldi et al., 2024). In RL, Abeille & Lazaric (2020) show duality between Equation (7) and Equa- tion (8) for linear systems. For nonlinear systems and deep RL, Sukhija et al. (2024a) empirically study combining extrinsic and intrinsic rewards. However, compared to these works, we demonstrate SOM- BRL's scalability to high-dimensional settings such as visual control tasks, and additionally, ground this approach theoretically, providing regret bounds for nonlinear systems and common RL settings.
15

---

B Proofs
B.1 Definition of Well-calibrated Model
We define well-calibrated statistical model of fr, which captures both the mean prediction Mn, and the uncertainty o, of our learned model. Definition B.1 (Well-calibrated statistical model of f", Rothfuss et al. (2023)). Let def X x A sequence of sets {M (6)) m20, where
(5) (f 2 - de Vz E E d=) - - (,(0))n
an all-tima time well-calibrated statistical model of the function f" if, with probability at least we have f E N2zo (o). Here, fn. Mn.j and n.j denote the j-th element in the vector-valued functions f. Mn, and n respectively, and (8) Rzo is scalar function that depends the confidence level 8 E (0, 1] and which monotonically increasing in n.
In the following, we show that Gaussian processes are well-callibrated models if Assumption 5.2 holds.
Lemma B.2 (Well calibrated confidence intervals for RKHS, Rothfuss et al. (2023)). Let f E H Suppose un and on are the posterior exists (8), for which the tuple (pm, on, (,)) a well-culibrated statistical model of f
and variance of GP with kernel k Equation (9). There
In summary, the RKHS setting, GP is a well-calibrated model.
B.2 Analysis for the finite horizon case Lemma B.3. Let Assumption 5. and Assumption 5.2 hold. Consider the following definitions
I(x,f" E,
(I (xx)))
s.l., x(+1 (x, n (xx))
Wt.
In x(0).
J( = Byp
r(c, (({1))
s.l. x/41 (n π (a)()) + wr:
a6 = a(O).
2n(Ttf") = E.
|| I 22,, π (ax)))
s.l. 2441 = (2 (xx:)) + wt:
To = x(0).
En(T,, = Eyn.
o | (1// (z(101
S.l., 2'441 pn ( we, a/ x(0).
1=0
An =OmaxT (1 + Vd 3n
where Cinxx max Rmaxx mmax) Then have for all 2 E II with probubility at least
[J(T, f*) - J(T, un)l < An (T. Mx) [J(T, J(T un)l En
Proof. We give the proof for [J(T,, ( J(T, un)1 < (Lr pr 2m (TT, pn). The same argument holds for the second inequality. Let J++1 ( f x) denote the cost-to-go from state æ. step + 1 onwards under the dynamics f*. Following the Policy difference Lemma from (Kakade & Langford, 2002) and Sukhija et al. (2024c, Corollary 2.)
Σ JeTI(TT, f", 1) - Jeti f,, #tt11)
J(T J(Trf') = Exi..
with 2441 = fr (a',, π (xx)) + we, and 2'41 = unla T(x))) + wi
Therefore,
((T [TT |-]
Σ
JITI 8", x(+1)) - Jes1 1", (e41)
t=0
16

---

T-1
$ Σ Ew, [Je+I(T, x(41)) Jt+1(T, f", +x+1)]l]
Next, we bound the last term using the derivation from Kakade et al. (2020). Let C(x) = Jet (T,, f ..).
Ew. [JT+I fr ,',,)))))))))))) Je+1 f*,+1))
$ max (ow,(C(x((1)))) Ew. [0(
min { (1f π (ax))) pnn (3r 7 (ar 1]
(Kakade al., 2020, Lemma C.2.)
S Rmaxx (1 vdl) (0)
IIon- (1/ (xx))]
Therefore, have
J(T, Ma) f()
EE Buu (et+((T, f 2(42)) Je-1(T,, f",0+)))]
t=0
An 11/0
Note that Lemma 5.3 follows directly from Lemma B.3.
Lemma B.4. Let Assumption 5.1 and Assumption 5.2 hold and consider the simple regret at episode
J(xT" ) - [TT, f*). The following holds for all >> with probability at leust -
Tn (2An + Em) (T
Proof.
rn = J(T 8 [ 1*)
S J(x pn) + En pm)) J( f*)
(Lemma B.3) (Equation (8))
J(TnY pm) + 2n a Mn) J(T fr)
J(TAy pn) - J J(T.. fr) An Ero (TT pmx
I Dn (TTx f³) + >n En (Tn pkm)
(Lemma B.3)
En (TT, 1*) Ao EEn (T pmn) En, (Tr (*))
S (A 22,, (T f*)
Here in the last inequality, we used the fact that (1 (I)) bounded and positive, therefore, can treat it similar to the reward (it is in fact an intrinsic reward) and use Lemma B.3.
Proof of Theorem 5.4.
Rx
Tn
n=l
(X2 + 21,, (n(nnn f*)
N
< (A2 + Ax)
Σ
En nn f*)
m=1
17

---

N
= (A3y + 2AA)
Σ g
( ((x)))]
n=1
N
$ (AN + 2AN) VAN NT
E
(a, π T(x)))
S C( DAN)TVVNTI
(Curi al. (2020, Lemma 17))
Finally, note that from Lemma B.3 we have AN x T3, and 3n x vnn (Chowdhury Gopalan, 2017). Therefore, RN - O VN)
Table 2: Maximum information gain bounds for common choice of kernels.
Kernel
k(x,,0 x²)
N'
Linear
x _Tx' x
O (dlog(N))
= 22
RBF
o (N))
220
x x-
22v
20 Naota log Ioiu (N)
Matèrn
Bu
In Table we list rates of I' for the most common choice of kernels.
B.3 Analysis for the discounted infinite horizon case
For the infinite horizon case, first study the posterior variance on in the feature space. Moreover, let = (x, u) and = x U.
For the case of notation we denote 2 = (x) Tr (xx(z)) )) For & we define the kernel embedding k- = k(2 The covariance matrix V, 7 -> H in the feature form is:
V1
KI
(10)
1=1
Note that we have 2++1 = (Kz,, f²) wT With the design matrix Mj H + R
M, (kss
k
kz.)
(11)
we have V, = I 3 M,M, and since K, = M, M, we have
1
det (V,) = det
I+ 0
K,
(12)
Corollary B.5 (Lower bound on the posterior log determinant).
T.
log (..)) 2 log ( V2-1l) log
1
Σ
|
(
(13)
In particular, we have
VN] VVl
N
Z
log
log
1
Σ
o (2k
(14)
Proof.
f
log ([Vr)) log (1 (\ + log
I I+o 2 1/2
k Exx
k zunn
> log (n-1l)
18

---

Tr
log
1
// -
ki
Rzc,
1/2
(see (*) below)
Ta
log ((V=-1l) + log
1 0
kzz,o
k
V
A=
Pu
log (1Vn-1l) + log
1 2
0 (21
k=1
prove (+) in the following, first my = 0 -v- 121
kin...
then we have
log
I I++1
kzc h Zk.m
V
= log I+
m
The matrix M = Σx= k==1 m by definition is positive semi-definite. Moreover, | + M Ie (1 + a.), where 0; 2 are the eigenvalues M. Furthermore, since a, 2 and I1 (1
ag) = 1 Ei 04 + + II Cii we get 11 + os) 2 + Σiz1 9z. Finally, since
Σ a; = tr (M), we get I + M 2 + ur (M).
Corollary B.6 (Upper bound on the posterior log determinant).
f,
log ([Vr)) I log (VV-11)) +
Σ
log (1 + -0 (Zk1)(Zk()))
Proof.
log (1V, log (1 11) + log (1I + M))
s log (1Vn-1l) log (diag (I M)))
(Hadamard's inequality for PSD matrices)
=log (lVn-1l)
log (1 + 20 (7
Corollary B.5 will be useful for the discounted horizon case, whereas Corollary B.6 will be applied for the nonepisodic setting.
Next, we show that SOMBRL also performs optimism in the discounted horizon case. Lemma B.7. Let Assumption 1. and Assumption 5.2 hold. Consider the following definitions
=
ra, π (x:))
s.t. a++1 (Tt,TT( + wes Eg = x(0),
J, Mn) = E
7 7 T (21)))
s.t. 2/41 = (1/ (x+))) + wi 26 x(0).
Ej(T, E
|1 (zi, (xx)))
s.r., x:41 f (x (xx))) w9, x(0),
EJ, (m,) = E
n1 || 0 (1),, π (xx5)))
s.t. 2/11 = (nl,,, ((x))) 2 x(0),
1=
(1 / vd ). 1
An Ciux
1-7
19

---

where Cmax = max Rmax, max 5. Then we have for all 2 0, π E II with probability at least
1
(TTTff) - J, pn)) [T, 5*) - (T, uam
I Xn - (n)) Xn (TT,
Proof. We give the proof for -") (T. S An pn ) (T, M.). The same argument holds for the second inequality. We can extend the result from Sukhija et al. (2024a, Corollary 2..) to the discounted case and get
(+ (J,( f", ,( - [T f )
(T, Mn)) J = Ey...
with 1c+1 = (s π (2')) + we and d'41 = un (s π (xx1) + wc
Let Bn 1tva (xx) = =J²( , f x). Note that C(x) S An for all E X. Therefore, we have
JJ, n mn) JT(T,, l')
x0
(Jr (T,, TT, f x441)) [T f ((H1i))
(+1E E E [T, f xb/-)) (T f te4 []]]
Y V max Ew, (C(x'4+1)) }}, Be.. (C(
t=0
[ T (ar))) - pan T ((x)))
x xd min
(Kakade et al., 2020, Lemma C.2.)
8
< An
[+' on ( (x
((Sukhija et al., 2024c, Corollary 3))
t=
Proof of Theorem 5.5. We start with bounding
Ewin. [+' 11on-1224, (xx())
(15)
To achieve this, we use sampling strategy where we increase the horizon of rollouts with each episode n. In the discounted setting, this allows collect data the tails of our rollouts, i.e., make observations with longer rollouts and thus approximate the true discounted value function asymptotically. Moreover, we set T'(n) = log(n)
(note that 7 < 1 and therefore T(n) is positive).
Tox
Eunc- [7' 11on-1 (x()))]
T(n)-1
Ewitt [ ||an-- (2/)))
E Emna
6" ||an-1(0ll, (a))))]²
t= (m)
N [(m)-1
N
2
Ewit- (>' 120
T(n)
n=1
1=
n=1
N
Emnaa 7dt I|onn- TT (2t)))1
n
n=1
20

---

N T(a)
Buna-
[ Illon- (21)))]
Co
max log(N)
1 7
0=0
Next, we bound the term
T(m)-1
Σ 2'0-2 II T (x()))
Sn
Note that, Sn E (0 sn) (Srinivas et al., 2012). Define Cy
dy 0 max
Let Smax
ddoomes we have Sn S 1-y
Bmax
log(1
log( 8mas)
Rmax Tom(n+)
We have,
T(m)-1
Sn S C, log
1+o
Σ
x2 1 (({)))
t=
T(m)-1
3 C. log
1
Σ
| 1 (2//2 क (at))))
0=
Finally, we have
N
N
T(n) -1
Σ
sn
S C.yy
Σ log
1+0
1 x(x()))))
1=1
n=1
C.y SIn T(n)
(Corollary B.5)
$ CTT Nlog(N)
N Ry= =
Tn
n=1
N
(X + 2Xn ) (Tn; )
n=1
$ (A% + 22N) Σ Ey (Tn: f*)
= -A2 + 2
E nT 1²
N
$ (2AN + X
E
yt l| (2 π (2x))]
1=
N
oc
I (2AN + V
X
?
Σ k- 0
7dt 1 ( ((x)))|
N
N
= (2AN + x2x)
x
Σ ==1
E
Yt 110 π (2x)))
C, N log(N)
Co log(NN
$ (2AN + Rw)
1-7
(1 7)-
Since AN x 8v 7, we get
RN S 0
(FW1oR(N) vNN
2/2
21

---

B.4 Analysis for the nonepisodic RL case
In this section, we prove Theorem 5.6. First, we restate the bounded energy assumption from Sukhija et al. (2024b).
Definition B.8 (XCs-ffunciions) The function & R20 + R20 is of class Kox, if it is continuous, strictly increasing, {()) = 0 and {(s) + oo for 8 + oc. Assumption B.9 (Policies with bounded energy). We assume there exists Ç Koo, positive constants K, C..y C with C. Ct. and y E (0, 1) such that for each Il we have,
Bounded energy: There exists a Lyapunov function V X + [0, oo) for which Va, x' E X.
IV - v (x')) S <(I =='l)
(uniform continuity)
CE(([==]) 2 V* (x) - =
(positive definiteness) (drift condition)
EE Em | (V S >V- <+) +x)+K +
where x4 = f (ar, π (x)) + u.
Bounded norm of reward:
r(x, (x))
= 1 V (x) 00
sup
Boundedness of the noise with respect to k:
Ew [<[[[[]]]] < 00, Ew [x ([( <
Sukhija et al. (2024b) argue that this assumption often satisfied in practice. We refer the reader to Sukhija et al. (2024b) for further details. Next, make assumption the underlying system f*. Assumption B.10 (Continous closed-loop dynamics, and Gaussian noise.). The dynamics model f and all π E II are continuous, and process noise is i..d. Gaussian with variance a², i.c., w: 1 ('0,
An important quantity in the average reward setting is the bias
B wo) lim E
Σ r( ut) Javg Jang(T))
(16)
T T0000
The Bellman equation for the average reward setting is given by
B TT, x) + (Tvg(TT) = r(x π (x)) + Ex,, [ ((TT.(+)))))))))))) (0), x, x;
(17)
Sukhija et al. (2024b) show that under Assumption B.9 and Assumption B. the average reward solution and the bias (c.f., Equation (3)) are bounded. Moreover, they show that with Assumption 5.2 the average reward and bias are bounded for all dynamics M, n Mg.
Lemma B.11. Let Assumption 5.2, Assumption B.9, and Assumption 5.2 hold. Consider the following definitions
Jaug(TT f*)= lim
E
Σ r(w,, π (xx1)
s.l., Be+1 = fa (xx:)) + wis Bo = x(0),
T-000
Jang (Tr, f)
lim T
raal π (z())
8.1. 2 = f f(x/, π ((x))) + wi, z6 x(0),
T
1
En(TTf") = lim
F
( T (+x)))
IT 8.1., F1+1 ((( + wi To = r0)),
T T
En EnTTT, lim
E
nan (2// (x())
8.h. 2/41 f(0l,, (x()) w 2 x(0),
T
X, D4(x0, 7, (0),.
22

---

and Da( Y: K) is defined as in Sukhija et al. (2024b, Theorem 3.1), is independent of and increases with || x Ill K and y (see Sukhija et al. (2024b) the exact dependence). Then we have for all n 2 0, E II, f E Mn Mo with probability at least 5
Javg (TT f Jang π, f)l S An (T, f)
Javg( f") Jang , f) v Xx En TT, f*)
Proof.
Jaagl T f) Jayg(T fz)
lim
E
r(a π (a:()) - Jwgg(T fr)
T-+00 T
lim T+x0
R.g
B( π (a;))) - B(#141 T ((+e+1))
S En
(1)
In the second last equality, we used the Bellman equation for the average reward setting (Equa- tion (17)), where d is the next state under the true dynamics f*
For the last incquality, Sukhija et al. (2024b) bound the bias term with A, in Section A.3, on pages 23 24.
We can use the same derivation to show that |Jav f) Javg( ( f*) S An En (TT,
SOMBRL in the non-episodic setting operates similarly to NEORL (Sukhija et al., 2024b). In particular, we update our model and policy every Tr step, where T, defined
[log (C./c,)])
Ta max
Tn.
(18)
log (1/7)
Ta arg max +1
(19)
T21 T de
s.t.
log (1 11+0 1++000
on (1+t-))-1((tmm)) S log(2).
(20)
1=1j=1
Effectively, we update our model and policy only once we have accumulated more than one bit of information, i.e., Ex - log (1 o1=02202 --15 model set Mn. we select any dynamics in fn E M. MMo and pick the policy with arg max Jang fn) + EnT, fn).
log(2). With the updated model and
(21)
#ell
Note that (Sukhija al., 2024b) require maximizing over the dynamics in M,, MM whereas we do not. Moreover, while this optimization is generally intractable, for SOMBRL, we can obtain f using the quadratic program described in Equation (22). However, practice, we just pick the mean model
en Mn. This practical modification is also made in Sukhija et al. (2024b) where they optimize over dynamics in M,, instead of M,, 0 Ma-
Theorem B.12 (Formal Theorem statement for informal Theorem 5.6). Define RN
EN B[Javg(TT)) r( Th (Z))). Let Assumption 5.2, Assumption B.9, and Assumption B.10
hold. Then we have for all N 2 0 with probability at least 1 8
RN S (OTT""
Proof. Let Ey denote the number of episodes after N interactions in the environment.
Rx
E
Jave ( -xxx n., (xz
En Tx
E
Javg( [ .fa) + Dn (T)) r(z
(Lemma B.
k=U
23

---

En T. -1
E
Javg Vavg(TT)) (T fn) + An (TT fn) - r(
(Equation (21))
EN T
E
Javg nz fnn - r(2
n=
k=0
EN T--1
+ A
Dnn a fal
h=
BN Tu-1
s 00 (TNNN)
+B AN
Σ (T
(Theorem 3.1 Sukhija et al. (2024b))
==1
k=
Next, we focus on E
AN EN TT-
En (TT: f)
Ev
Tn
En Tnn 1)
E
Ex T,
E
I π (x))))
n=1
Ey
+ E
En (Tn: (T f) (0,,, (x:)))
EN T.
+CVTNN+E E
En (T f) (0,, (x:)))
(Lemma A. 1. Sukhija al. (2024b))
O
(Sukhija et al. (2024b, Theorem 3.1) with reward o,))
Therefore
Fly T
E
AT
En (n f)
s CO(NINNNN)
10
1==0
5 (TT"//VN
In conclusion,
RN < O 1
B.5 Analysis for pure intrinsic exploration
In the following, we derive a sample complexity bound for a pure intrinsic exploration algorithm. Thereby showing convergence for methods such Buisson-Feed Feenet et al. (2020). Theorem B.13. Let Assumption 5.1 and Assumption 5.2 hold. Consider SOMBRL with extrinsic
reward r = 0, i.e.
n. arg max ES TET
Σlla Clonna' (a',s uilll
2/41 Mnna' u.) + wi
Then we have VN 0, with probability least 8
r3
max E. TEll
Σ || (mi, π (x+)))
50
24

---

Proof. Let Eny- = maxx ΣN(TT, f*) and i the corresponding policy.
N
-
2
(monotoncity of the variance)
N
n=1
1 N
(1+ An) kn)
(Lemma B.3)
N
==1
N
(1 An (Tn Un)
(Tr, is the maximizer for mean dynamics pt))
N
N
(1 + 2in f*)
(Lemma B.3)
R=1
N
(1 + )N) 1
En (T fr)
N
n=1
< (1+ANN)
Ex nh f³)
N
Effectively, Theorem B.13 shows that pure intrinsic exploration reduces our model epistemic uncer- tainty with a rate of V F.e To the best of our knowledge, we are the first to show this. Moreover, Sukhija et al. (2024c) derive similar bound but their algorithm performs optimistic exploration from Equation (7) in addition to maximizing the intrinsic rewards. Our result shows that the optimistic exploration is not necessary for this setting.
B.6 Analysis for the finite horizon setting with sub-Gaussia noise
In the following, we analyse the regret for the setting where the process noise w is o-sub Gaussian. Assumption B.14. The dynamics model f", reward r, and all E II are Ly. L, and I,, Lipschitz, respectively. Furthermore, we assume that process noise is i.i.d. a-sub Gaussian.
We make the same assumptions as other works (Curi et al., 2020; Sussex et al., 2023) that study this setting. Moreover, Lipschitz continuity is a common assumption for nonlinear dynamics (Khalil, 2015) and is satisfied for many real-world systems.
Curi al. (2020) provide regret bound that depends exponentially on the horizon T, i.e., RN O /TENN
(VTRN)
They obtain an exponential dependence because when planning optimistically. i.e.,
solving Equation (7), they consider all plausible dynamics, including those that are not Lipschitz continuous for all n. Solving Equation (7) for only continuous dynamics is intractable. However, for SOMBRL,
do not maximize over the set of dynamics
overcome this limitation.
Moreover, since f has bounded RKHS norm, i.e., || < B AAssumtton 5.2). From Srinivas et al. (2012); Chowdhury & Gopalan (2017) follows that with probability - 8 we have for every n:
||
pm ll. < B,.
For SOMBRL, instead of planning with the mean, which in general might not be Lipschitz continuous for all n, we select a function fn that not only approximates the fr function well, i.e., I|f fulle, < 9n. but also its RKHS norm does not grow with To do that propose to solve the following quadratic optimization problem:
fn
arg min
||f - 1mmlkk..
(22)
s.t. |l < B
25

---

Optimizing for dynamics with bounded RKHS norm
fn
Mp
Observations
B 1.0
B = 2.0
B = 5.0
-2
-2
Figure 5: Solution to Equation (22) for different values for B. Effectively, for larger values for B, Mn and fn coincide.
Theorem B.15. The optimization problem Equation (22) is feasible and we have l| pn lls, S Bu-
Proof. Consider the noise-free case, i.c., w = 0, and let fa,, posterior mean for this setting. For the function fan holds that f fn ll. S S,, (Corollary 3.11 of Kanagawa et al. (2018)) and f&n ll S
B (Theorem 3.5 of Kanagawa al. (2018)). Since || l,, 1 fl +I 2 By representer theorem, it also holds that μ., span(k((21,))
Mn Knlle. S
k(zz.
L
Let a, (K + o²r)-1y E IR" and reparametrize f(x) Eis agklaa, æt). a Ka. We also have:
We have |
1 (a (a-an) an)
I
Mn 11 = (a a.)) [R.))KK
K I+ 0² K
Hence the optimization problem Equation (22) equivalent to:
1 o² (a-aa) an)
min (a an) T aER"
K I+
K
s.t. Ka
This is a quadratic program that can be solved using any QP solver. The program finds the closest function to the posterior mean μa, that is Lipschitz continuous (see Figure 5). In particular, note that since ||fflll B, f. has a Lipschitz constant LB which is independent of (Berkenkamp, 2019).
From hereon, let I., max may, LB).
For the sub-Gause uub-Gaussian case, SOMBRL follows the same strategy as Equation (8) but instead of using the mean dynamics Mr. plan with the dynamics fn that are obtained from solving Equation (22).
T-1
arg max En well
Σ
raa, u.) + Aullo (a', utll
(23)
2 = fn (2 u.) + we,
Lemma B.16. Let Assumption B. and Assumption 5.2 hold. Consider the following definitions
T-1
J(T, 5r) = E
r(w,, π (xx)) Σ
s.t. 0z+1 = f (2 7 (xx)) + Wes 2o = x(0),
J(T,, fn) E
r T (x/))
s.t. I/y1 fn fn TT,,, + w., o x((),
En(TT."
(0 (x)1))
s.t. Be+1 (x (xx))) wi,
ao x(0),
(T, fn)
| T (x()))
s.I. x'41 Salae (x/))) + wi.
o = x(0),
26

---

(1+ + L. TT
Then for all 0, π I1 with probability at least 1 -
- An, fri)
[ - (T
Proof.
- fn) E
Σ ra ((x))) r(z/, ((z\))
S Lr + Ln) )
Σ 1e2 x'||
Next we analyze || æ: for any t. Without loss of generality, assume L. z and define L, = (1 + Lx).
We show that
1 D141 x/41|| S (1+ vd.)s,
Σ L I ((()))l)
k=0
Consider
|2|- - x'1l = II (a/6. mxx/)) - (=6:: π (<bb))]
(1 + vd Io (a's ग (zb)))
Consider any t 1,
|| - x || = If (TI,, - fn (a// π (<x)))]
< ||f" (z/,, T (2 - fn (z/, (<x))) + f (zt, ग (xx1) f (2/. (x(
$ (1 + Vd |1 (2
L. || zi1l
< (1 + Vd (1 (2// ())) (1 + Vdn)
L.
Σ
/t-1-k l1o, (2ks (((h)))
k==0
=11 + Vdn)3n
Σ 1.- l|an( π (<1\\)||
In particular, since L. 2 we have || 2'41 (1+
(Ex lIo, x(a\i))).
In summary, we have
((T - J pm))
r(o,, r(a)
S L1 + LT)
[t x'1l
T
S 11 + (1 + da) x E
3a Tt-1
Σ k=0
I (ake, x (2()))
1=0
< (1+ (1+ (1 L L - En ( pm)
= En
The main difference between our analysis and the analysis from Curi et al. (2020) that for us An x Sn if we plan with fn.
27

---

Lemma B.17. Let Assumption B. 1.14 and Assumption 5.2 hold and consider the simple regret at episode n, Vn Jxx", . [ ). The following holds for all with probability least
1 1-8
(2An + (Ta
Proof.
To J(T", f*) [T f
S J( fn) + An n ( fn) [ f³)
(Lemma B.16) (Equation (23))
S (mas fn) + An Tn Sn) - [ )
a(Tas fn) J nT 1*) An An (T fnn)
I An f*) An En NT::nt fn)
(Lemma B.16)
En (T f³) + Am (Tn: fni E f"))
S (27 + 2An En [T f*).
Theorem B.18 (Finite horizon setting sub- Gaussian case). Let Assumption B.14 and Assumption 5.2 hold. Then we have VN > 0 with probability at least 1 8
RN S O r VNN
Proof. The proof is the same as for Theorem 5.4, since in Lemma B.17 we show that also for the sub-Gaussian case, SOMBRL has the same regret dependence w.r.. An and En aTa f*).
28

---

MAXINFOSAC C Additional Experiments
In this section, we provide additional experiments.
MBPO-OPTIMISS OPTIMISTI
SAC
Humanoid walk
Humanoid stand
Humanoid run
Hopper hop
200
200
Return
500
500
130
250
0
0
0.0
0.5
1.0
x10
x106
X
x 108
Walker run
Cheetah run
Cartpole swingup sparse
Quadruped
600
750
750
Return
800
500
200
250
0
0.0
0.0
0.5
200000
400000
0.0
0.5
1.0
100
x108
x100
Environment steps
Environment steps
Environment steps
Environment steps
Figure 6: Comparison between LIBPO-OPTIMES and MAXINFOSAC and SAC. We observe that MBPO-OPTIMSTIC, being an MBRL algorithm, performs the best in terms of sample efficiency.
Visual Cyntrol Thikk
DREAAEE
CVC Hoppor
CVC: Cheach
DVD: Guing
DUO: Reacher Hard
Asul:
Morannra
Fiveridd
DUC: vose
DMC:
Acraact
DMD:
DMC
Part
Figure 7: Learning curves for the visual control tasks from DMC and Atari using DREAMER the base algorithm. DRREAMER-OPTIMITT either performs on-par or better than DREAMER in all our experiments. Particularly, in the Venture task from the Atari benchmark, where DREAMER fails to obtain any rewards.
Experiments with MBPO In Figure algorithms MAXINFOSAC (Sukhija et al., 2024a) and SAC (Haarnoja et al., 2018). From the figure, performs the best in terms of sample-efficiency. particularly
compare MBPO-OPTIMISTIC with off-policy RL
we conclude that TEBPO-OPTIMPH for the challenging/high-ddmeeee -imeeiioiiiiii humanoid tasks. Moreover, between SAC and MAXINFOSAC. the latter achieves much better performance. We believe this is due to its intrinsic exploration reward.
Experiments with DREAMER In Figure 7 compare DREAMER- CERAMMER-OPIMISTTER OPTIMISTIC with DREAMER on additional environments. Overall, we observe that DREA MER OPTIMISTIC performs either on par or better than DREAMER. However, for certain environments such as Reacher Hard or Finger Turn Hard, DREAMER is sample-efficient. We believe this is because in these settings smaller values for An would suffice for exploration. However, we use a constant value for An across all environments and automatically update it using the approach proposed in Sukhija et al. (2024a). Investigating alternative strategies for An, would generally benefit SOMBRL methods. We think this is a promising direction for future work.
In Figure 8 and Figure we compare DREAMER OPTIMISTIC with DREAMER on proprioceptive tasks. In most environments, DREAMER- -FTERR-OPIIISIIC performs on par. It performs better in the Finger Spin environment. However, when action costs are introduced (Figure 9), in line with our results in Section 6, DREAMER fails to obtain any meaningful rewards.
29

---

Proprioceptive control learning DREAMER-OPTIMISE
DREAMER
Cartpole Swing-Up
Cartpole Swing-Up Sparse
Hopper Hop
Walker Run
Return
750
200
500
500
250
0.0
0.5
1.0
x109
x 10
X10P
1000
Reacher Hard
Finger Spin
Cheetah Run
Finger Turn Hard
1000
1000
Return
500
500
500
500
10
x-10
110
x10
Environment steps
Environment steps
Environment steps
Environment steps
Figure Experiments with COMMR-OPTIMISTIC and DREAMER for proprioceptive tasks. performs on with DREAMER, obtaining slightly better performance on
the Finger Spin task.
Proprioceptive control learning with action cost PREAMER-OPTIMISTIC
DREAMER
Carppon Swing Sparse
Action Cost:
Acsion Cost: 0.2
503
0.50
0.55
0.50
075
1.00
0.00
0.25
0.50
0.75
1.00
Retur
503
290
0.50
0.55
0.00
0.5
1.00
0.00
0.25
0.50
0.75
1.00
x10
Walker Pun
203
200
0.50
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1,00
Envionment stogs
steps
x1d"
Figure 9: Experiments with DREAMER- DREAMER-OPTIMISTIC and DREAMER for proprioceptive tasks with
action costs. DREAMER completely fails to solve the task, whereas DREAMER-OPTIMISTEE does not.
30

---

D Experiment Details
In this section, we provide additional details for our experiments.
D.I MBPO-OPTIMISTIC
For MBPO-OPTIMISTIC we train an ensemble of forward dynamics models We use the disagree- ment between the ensembles to quantify model epistemic uncertainty, similar to Pathak al. (2019); Curi et al. (2020); Sukhija et (2024c). For selecting Ano we use the auto-tuning approach from Sukhija et al. (2024a), where the intrinsic reward weight is optimized by minimizing the following loss with stochastic gradient descent
L(A) --
(an u) (n u)).
(24)
Here T. is a target policy, which is updated using polyak updates of nn- This objective increases À when the policy is under exploring compared to the target policy. Sukhija et al. (2024a) show that this strategy works across several model-free off-policy RL algorithms.
Besides using the model to quantify disagreement, we generate additional data by adding the transitions predicted by our learned model. In particular, for every policy update, we sample a batch of transitions from the data buffer (z) u. u.)) ~ Diss and add (z, u. æ'), transitions predicted by our mean model jun, to the batch. This allows us to combine true rollouts with model generated rollouts, as proposed in Janner et al. (2019). Since we can generate additional data through our learned model, we can efficiently increase our update-to-das ratio (UTD). For all our experiments with MBPO, with use an UTD of 5".
We use the same hyperparameters as Sukhija et al. (2024a) for all our state-based experiments.
D.2 DREAMER-OPTIMISTE
We DREAMERV3 As the base model. For quantifying the model epistemic uncertainty, the same approach as Sekar et al. (2020); Mendonca et al. (2021) and learn an ensemble of MLPs to model the latent dynamics The ensemble is only used for quantifying the model uncertainty/intrinse reward. For the policy optimization, we use the DREAMER backbone, where the agent optimizes the policy using imagined rollouts. For selecting X, we also use the objective in Equation (24). We found adding regularize term [ to the objective worked better with DREAMER. We initialize À with 2 and pick 0.001. For the rest, use the same hyperparemters DREAMER
D.3 SIMESVGD-OPTIMISTIN
Tolerance reward with different margins
1.0
Margin=2
Margin=5
Margin=20
Reward
0.8
0.4
0.2
0.0
0.0
3.0
Distance to target
Figure 10: Tolerance reward function for different values of the margin. For larger margins, the agent receives rewards even if its further away from the target.
We use the same experiment setup, simulation prior, and hyperparameters as Rothfuss et al. (2024) 10 The reward function in Rothfuss et al. (2024) is based on the tolerance reward from Tassa et al. (2018).
'For all tasks use (256, 256) neural network architecture with ensembles, except for the humanoid (512, 512).
and quadruped tasks where
WWe did not tune the UTD and chose to trade-off between computational cost and sample efficiency.
'For all tasks million size model and the official DREAMERY3 implementation btttps://githu....
(512, 512) neural network architecture with 5 ensembles.
WWe use
dani j jar harr/deemeev3// tttee/mamiin
officiil implementation: https:/ / http:://ggthub..c/llagggggg hotp:////aagggoppsmmmmttoooooooo or
31

---

The tolerance function, gives higher rewards when the agent is close to a desired state, i.e., in case of the RC car the target position. The "closesness" quantified using margin parameter for the reward function. In Figure 10 we plot the reward for different margin parameters. As we decrease the margin, the reward becomes sparser. Rothfuss et al. (2024) use margin of 20. In our simulation experiments, than
show that SIMFSVGD performs our hardware experiment, we use a margin of 5, for which SIMFSVGD fails to learn. For An we found that linearly decaying schedule worked the best. Therefore, we linearly interpolated from Ao 0.5 and λιo After the tenth episode, the agent greedily maximized the extrinsic reward.
for smaller margins. For
D.4 GP experiments
For our GP experiments, we use the RBF kernel. The kernel parameters are updated online using maximum likelihood estimation (Rasmussen & Williams, 2005). For all the experiments, An 10 and for planning the iCEM optimizer (Pinneri et al., 2021). We use the same hyperparameters Sukhija et al. (2024b)
D.5
Computational Costs
Table 3: Computation cost comparison for SOMBRL with different base algorithms.
Algorithm
Training time
HUCRL (GPs)
90 +/- min (Pendulum),
31.5 +/ min (MountainCar)
SOMBRL (GPs)
min (Pendulum),
13.8 +/ 0.25 min (MountainCar)
MBPO-MEAN
9.6 0.2 min
(Time per 100k steps. ensemble, GPU: NVIDIA GeForce RTX 2080 Ti)
13.7 +/- 0.35 min
(Time per 100k steps, 5 ensembles, GPU: NVIDIA GeForce RTX 2080 Ti)
DREAMER
42.24 +/- 0.95 min
(Time 100k steps, GPU: NVIDIA GeForce RTX 4090)
DREAMER- DREAMER-OPTIMITEE OPTIMISTIC
46.32 +/- 0.34 min
(Time per 100k steps, 5 ensembles, GPU: NVIDIA GeForce RTX 4090)
official implementation: http:://gitabbboo///gggooooxooxo
32

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We highlight the problem setting, algorithm, and theoretical and empirical results the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section we highlight the assumptions of our work, which also correspond to the limitations of our theoretical analysis and also the setting for which our algorithm yields theoretical guarantees. In our experiments (Section 6), we evaluate our algorithm settings where the assumptions are not necessarily satisfied.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
33

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results are accompanied by the relevant assumptions that are listed in Section 5 and provide all proofs in Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We open-source benchmarks, disclose all hyperparameters and experi- ment details in Appendix D. Furthermore, our algorithm build on top of open-source implementations of the base algorithm as highlighted in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce
34

---

the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We base our algorithm on top of open-source repositories and provide all details and hyperparameters in Appendix D. Our experiments based on different reposi- tories depending on the base algorithm. We will provide our final code (which merges all dependencies and repositories) with the camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines pictps://nips. cc/ for more details.
While we encourage the release code and data, we understand that this might not be possible, so No is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. wwtpcc/ppppppppccccccccccccpcppt /ttp//dudee//oeeuumiiiinn
onnoolccy) for more details.
The authors should provide instructions data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification:
All details are provided in Appendix D.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
35

---

Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We run all our experiments with multiple seeds and report the error bars in every plot.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the compute resources as well as times in Table 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : http:///eeruuueeeeeeeeee http:ppp....ppppcc/ccccccccaaaaa
Answer:
[Yes]
Justification:
Our research conforms, in every respect, to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
36

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper proposes method to improve exploration in reinforcement learning, and is not tied to specific applications. As such, shares the many potential societal consequences that are associated with reinforcement learning and automation as whole, spanning from environmental impact concerns ethics and alignment.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release high-risk data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
37

---

12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite all creators whose code we used in our experiments in Section 6.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide supplementary video including a readme that explains the video in detail.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
38

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https / http///eeerr.ppeeeeee wttiiiisssccceeeee / for what should or should not be described.
39


Reviews:
[{"review_id": "tTDGZWav2a", "reviewer": "Reviewer_2BDK", "summary": "The paper proposes a new model-based RL algorithm that incorporates systematic exploration through intrinsic rewards designed to drive the agent to more uncertain regions of the state space. The authors show that under certain regularity conditions, the proposed algorithm achieves a sublinear regret while being simpler and more scalable that previous contendors. Finally, the authors provide some experimental results on some RL benchmarks to validate their approach.", "questions": "Can you examples of setting where Assumption 5.2 holds?", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strenghts: \n- It is nice that the paper provides explicit theoretical guarantees for the proposed algorithm under the regularity conditions considered. It is a bonus that the guarantees extend to the non-episodic settings. \n- The paper is relatively well written and easy to follow. \n\nWeaknesses: \n- First, the authors claim that they are the first to provide guarantees for RL with continuous state and action spaces. This inaccurate (at least not for the model-based setting); see The Statistical Complexity of Interactive Decision Making by Foster et al 2023. \n- The guarantees provided require strong assumptions on the model. This is not really highlighted as a limitation in the paper. \n- Including a measure of epistemic uncertainty as a bonus term to drive exploration is not new; see for example the paper Exploration by Random Network Distillation by Burda et al. 2018. So I am not really seeing the novelty here. \n- In the theory section, epistemic uncertainty is measure with the help of a kernel. In the experiment, the authors use an ensemble model to measure the uncertainty. This feels like testing a different algorithm to me. \n- The literature survey and comparison with existing work isn't exhaustive enough. The two papers above are a good example, and maybe also the paper Sample Efficient Deep Reinforcement Learning via Local Planning by Yin et al. 2023. \n- The experimental results aren't convincing as far as exploration goes. For example, the score on Montezuma's revenge caps at 2500, which is less the model-free RL algorithms in the last two paper mentioned above. So this is does not make a good case for the model based approach of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for acknowledging our theoretical contribution. We would like to highlight that we do not propose a single algorithm; instead, we theoretically study the framework of combining extrinsic rewards with the model epistemic uncertainty. We add it on top of existing MBRL algorithms such as MBPO, Dreamer and SimFSVGD and show that besides enjoying theoretical grounding, it also yields strong performance. As the reviewer highlighted, people have proposed using the model epistemic uncertainty in the past, e.g., Plan2Explore with Dreamer, RND with PPO, OPAX with model-based RL and MPC, and more (see Section A for more works). However, we are the first to show that this approach has sublinear regret, under the same assumptions as other principled exploration methods such as Kakade et al 2020, Curi et al 2020, and more. As opposed to these works, our approach is much more practical than the aforementioned prior works, and we demonstrate this in the experiments.\n\n**Claim on first to give theoretical guarantees**: We are sorry for any confusion caused. We do not claim to be the first to give theoretical guarantees for model-based RL in continuous state action spaces. In fact, in Table 1, we highlight other works that, under the same assumptions as ours, give theoretical guarantees for the different RL settings. Moreover, we have more detailed related works in Section A, where we also discuss other works that theoretically study model-based RL. We will add Foster et al 2023 to this section. Due to the nine-page limit, we only kept a smaller related work discussion in the main paper. Finally, our central claim is that we are the first to show that leveraging model uncertainty estimates as intrinsic rewards gives sublinear regret for most common RL settings in MBRL (finite horizon, discounted horizon, non-episodic, and unsupervised RL). We believe this is a strong contribution, since it provides theoretical grounding for a much simpler approach for principled exploration in RL. \n\n**Assumptions**: We would like to highlight that there are several works that study MBRL theoretically under the same assumption as ours  (Kakade et al. (2020); Curi et al. (2020); Mania et al. (2020); Wagenmaker et al. (2023); Treven et al. (2024), or Section A). Moreover, the kernelized assumption is also ubiquitous in Bandits with continuous spaces (GPUCB, IDS, TS). Hence, to the best of our knowledge, these assumptions are quite common when studying online decision-making algorithms in continuous spaces. Practically, the RKHS assumption captures any function that can be represented with $f(s, a) = \\phi(s, a) \\theta$, where $\\phi$ is a nonlinear, potentially infinite-dimensional feature map (see Section 1 in Kakade et al 2020, for a discussion on the generality of this assumption). Kernelized models are commonly used for learning and control in the real world (c.f., 1-6). \nExtending these results to deep learning models such as RSSMs and BNNs is still an open research question, even for Bandits (there is some research in this line with NTKs (7)). We will highlight this in the limitations section. \n\n**RND and other intrinsic exploration methods**: In our related works (Section A), we discuss RND and other intrinsic exploration methods. As highlighted above, we are the first to show that combining extrinsic rewards with the model epistemic uncertainty gives sublinear regret for several RL settings. None of the discussed works provides such theoretical grounding, and as compared to the methods that do have theoretical results, our approach is much simpler, as highlighted by the reviewer. In summary, our key novelty is to show for several RL settings that adding model epistemic uncertainty is not only empirically strong but also theoretically backed.\nThis requires a novel analysis as highlighted in our proof sketches in Section 5.\n\n**Kernelized setting and BNNs**: In Figure 2 (Section 5), we report experiments for the kernelized case for both finite-horizon and non-episodic RL settings. In the experiments, we evaluate our approach with SOTA deep learning models such as BNNs and RSSMs, where the well-calibration is not guaranteed as in the kernelized case. Crucially, our exploration strategy of adding epistemic uncertainty to the extrinsic reward is agnostic to what model, planner, or in general, base algorithm we use, e.g., MBPO, Dreamer, or SimFSVGD. However, we empirically show that across all experiments, adding the uncertainty reward helps the agent achieve on par or better performance. Particularly, in our hardware experiments, we illustrate the benefits of the uncertainty bonus in the real world, where the agent fails to explore meaningfully without the additional exploration reward. \n\n**Literature Review**: As discussed in our response above, we do have a more detailed related works section in Section A. We will add Foster et al 2023, and Yin et al 2023 to it. Foster et al 2023 propose a theoretical algorithm, which also requires solving a min-max optimization problem that is intractable in practice. Our RKHS assumption is the same as the reliazbility assumption in Foster et al 2023, where the model class is defined by the RKHS. \nYin et al 2023 empirically study uncertainty-based exploration methods for the setting where the environment can be reset to previously visited states. In comparison, our method is both practical and theoretically grounded for several common RL settings. \n\n**Experiments**: We add our exploration bonus on top of several base RL algorithms (MBPO, Dreamer, SimFSVGD). In the case of Montezuma, we use Dreamer, a well-established and SOTA RL algorithm for sample-efficiency and performance. We believe the performance of our algorithm is affected by the base algorithm itself, i.e., the expressivity of the RSSM model and the planner. Generally, we find that adding our exploration objective helps in obtaining better performance across the different base algorithms we try (see Figure 3 and 4). However, other elements, such as how to learn a model, how to obtain a policy from the learned models, are also important in RL but not the focus of our work.\n\n**Examples where Assumption 5.2 holds**: See discussion on assumptions.\n\n\nWe hope our response clarifies the concerns raised. We believe that our work addresses an important gap for exploration in MBRL by proposing a simple and scalable/tractable alternative for principled exploration. Our theoretical analysis is novel and applied to several important RL settings. In addition, we evaluate our work on state-based, visual control, and real hardware experiments. We think this makes our work original and a strong contribution to RL research. We hope we could bring our opinion across better to the reviewer, and would appreciate it if they would increase our score. \n\n**Additional References**:\n\n1. https://ieeexplore.ieee.org/document/8754713\n\n2. https://ieeexplore.ieee.org/abstract/document/8467518?casa_token=zXxHSxz19hYAAAAA:CZc9OA6MklYZrZ-VNrMK_XvBxeVpCbWAStnYeVdChbyVvU6JVQsfyMXzmXAu6cGknKK73bKNqpc\n\n3. https://arxiv.org/abs/2103.04490\n\n4. https://proceedings.mlr.press/v139/song21b.html\n\n5. https://proceedings.mlr.press/v144/boffi21a.html\n\n6. https://www.science.org/doi/full/10.1126/scirobotics.abm6597\n\n7. https://arxiv.org/abs/1806.07572"}, {"review_id": "ztuWh3gQCp", "reviewer": "Reviewer_ST8B", "summary": "This paper presents SOMBRL, a scalable and optimistic model-based reinforcement learning algorithm that combines extrinsic rewards with epistemic uncertainty to guide exploration. The key idea is to use a weighted sum of the reward and the model uncertainty during planning. The method is simple yet theoretically grounded, achieving reasonably good results across lots of RL settings. This work shows strong empirical performance in both simulation (state-based and visual control tasks) and a real-world RC car setting.", "questions": "- What is the key intuition for why maximizing epistemic uncertainty improves convergence and stability? Can this be interpreted as a kind of regularization?\n- How is the uncertainty model accuracy affect the results?", "limitations": "- The paper briefly mentions the computational overhead of learning a probabilistic model but does not thoroughly discuss the accuracy of uncertainty estimation and its potential impact.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This is an important problem. Exploration in model-based RL is a key challenge for real-world deployment of RL systems.\n- The solution is simple but clear. This work proposes to learn an uncertainty model along with the environment model.\n- This work shows strong empirical results. The method shows consistent performance improvements across various high-dimensional benchmarks.\n\nWeaknesses\n- The abstract suggests that SOMBRL targets realistic settings, but the theoretical results still rely on continuity assumptions. While common in literature, these assumptions may not always hold in complex real-world environments.\n- The guarantees rely on well-calibrated uncertainty models, but how accurate these uncertainty models is not explored. Further, the intuition of why this method works was not clearly stated.", "was_revised": true, "final_justification": "No change", "rebuttal": "Thanks for your review, for highlighting the simplicity of our method and our theoretical as well as empirical results. \n\n**Tackling realistic settings**: We agree with the reviewer that certain continuity assumptions are required for theoretically studying nonlinear systems in continuous spaces. However, we try to bridge the gap between theory and practice by thoroughly evaluating our method across different real-world settings, such as state-based control, visual control, and even real-world hardware experiments. We believe that through this, we could illustrate the strengths of our algorithms in real-world settings. We will try to clarify this further in the abstract, but we are also happy to receive any further suggestions from the reviewer on this matter.\n\n**Intuition of uncertainty-based exploration**: Uncertainty-based exploration is a well-established exploration strategy (see Lewis & Catlett, 1994). Intuitively, by rewarding the agent when going to areas with high uncertainty, we encourage the agent to collect data where it has the most information to gain, i.e., reduce its uncertainty about the system the most. \nTherefore, as the agent continues learning, it explores all relevant regions where the uncertainty is high and reduces its epistemic uncertainty of the dynamics.  Here, relevance is determined by the extrinsic reward of the state action space. Under the well calibration assumption, this implies that our learned model converges to the true dynamics. In contrast, greedy algorithms do not ensure the necessary coverage of the state-action space, i.e., exploration, to ensure convergence, and random exploration strategies are not directed. We actively visit points where we have less knowledge and learn, whereas greedy and random exploration approaches often tend to either visit the same points, under-explore, and get stuck in a local optimum (greedy) or over-explore the areas that don't carry much information about the unknown function (random exploration).\n\n**Effects of uncertainty model accuracy**:  In our experiments, we found the use of ensembles as suggested in HUCRL, Plan2Explore, and MaxInfoRL to work quite robustly across tasks, i.e., we did not notice any negative effects. However, these methods are also designed to be robust towards uncertainty estimation, e.g., Plan2Explore only quantifies uncertainty in a lower-dimensional latent space, and MaxInfoRL tunes how much exploration via uncertainty to inject into the system.  Nonetheless, theoretically, if the uncertainty estimates are off, e.g., too high, then the agent could end up overexploring, or, in the case of too low, underexploring. Hence, this can have an effect on the performance of the agent, and how to mitigate this, e.g., via better uncertainty quantification or auto-tuning mechanisms such as from MaxInfoRL, is an interesting research question.\n\n\nSOMBRL is a unique algorithm since it offers both simplicity and theoretical grounding for many common RL settings. Across our experiments, we show that SOMBRL performs strongly in real-world settings. We hope our response addresses your concerns, and we would appreciate it if you would increase our score.\n\n\n\n**References**:\n\n1. https://proceedings.mlr.press/v119/sekar20a.html\n\n2. https://proceedings.neurips.cc/paper_files/paper/2023/hash/77b5aaf2826c95c98e5eb4ab830073de-Abstract-Conference.html\n\n3. https://proceedings.mlr.press/v97/pathak19a.html\n\n4. https://proceedings.neurips.cc/paper/2020/hash/a36b598abb934e4528412e5a2127b931-Abstract.html"}, {"review_id": "kOwM9AHgBH", "reviewer": "Reviewer_ZcLd", "summary": "The authors proposed SOMBRL (Scalable an Optimistic MBRL) approach based on the pricipl of optimism in the face of uncertainty. SOMBRL learns an uncerrtainty-aware model of the true dynamics and use it to find a policy maximizig te extrinsic reward and moddel uncertainty to stimulate the exploration. Furthermore, the authors provide thoretical sublinear regreet results,", "questions": "1. To avoid potential misunderstandings, I would like to clarify the following: In Equation (8), the formulation appears different from MBPO. When applying your method to MBPO, do you mean that the synthetic data generation still uses only the mean dynamics (unlike the original MBPO), and that uncertainty is incorporated in the rewards of both real and synthetic data used for training the Q-networks and the actor network?\n\n2. In model-based RL, I think that the predictive performance of the model is crucial in practice. Introducing additional rewards after rollout might lead to overestimation in the synthetic data. I am concerned this could actually degrade performance rather than improve it. I wonder what your thoughts are on this point, or if you have already addressed this issue in the paper, please let me know.\n\n3. The figures seem to have inconsistent line widths, which affects visual consistency. In Figure 3, for example, if this is indeed model-based policy optimization, the label on the left should be \"Dreamer.\" Personally, I feel consistency across figures would improve the presentation. Additionally, I do not quite understand the purpose of Figure 1.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength\n1. Theoretical Justification\n2. Simple and Scalable Methods\n3. High Compatibillity\n\nWeaknesss\n1. The figures and descriptions of the experiments are overly cluttered and lack clarity.\n2. There are some model-base RL method considering uncertainty based on the model for dynamics additionally.", "was_revised": true, "final_justification": "While existing uncertainty-based exploration methods generally lack theoretical guarantees, **this work presents sublinear regret bounds for a practical exploration strategy.** Moreover, the validity of the algorithm has been demonstrated across various settings, including both continuous and discrete actions. However, there are **some mathematical notation** that need to be revised, and **it is regrettable that no comparison was made with existing exploration methods.**\n\nTherefore, I recommend **a weak accept.**", "rebuttal": "We thank the reviewer for their feedback and for acknowledging the simplicity, theory, scalability, and flexibility of our method.\n\n**Figures**:  In Figure 3, we have 2 panels. The left panel shows our experiments with MBPO, and the right with Dreamer.  We will clarify the different algorithms in our figure further and also use consistent line widths across the figures. Figure 1 is used as a teaser to showcase the diverse settings we consider in our experiments. Would the reviewer like us to remove the figure? If yes, we are happy to do so.\n\n**Other works on model uncertainty-based exploration**: We agree that other works also propose using the model uncertainty as a reward for exploration. In fact, we cite several works in Section A of the paper. However, to the best of our knowledge, we are the first to show that this simple approach enjoys theoretical grounding for most common RL settings under the same assumptions as prior principled exploration works (that are much more complex and not scalable), such as (1-5). We think this is an important result since it provides theoretical grounding for a much simpler and intuitive exploration strategy. To establish this result, we required a novel analysis as opposed to the prior works, which we tried to highlight in our proof sketch. \n\n**Planning in MBPO and how model uncertainty improves performance**: We use the mean model to use model-generated rollouts. We add the epistemic uncertainty of our model to the extrinsic reward (both for real and synthetic data) and use it for training the policy and critic. We find this to work quite robustly across our experiments. Moreover, by adding the uncertainty as a reward, we encourage the policy to explore areas of the state space where our model has high uncertainty. This typically corresponds to areas where we have less data/high model error. Thereby, by collecting data in high uncertainty regions, we improve our model. Note that uncertainty-based sampling is a well-established exploration technique (see the seminal work from Lewis & Catlett, 1994), even for RL (6-8). \nIntuitively, by adding the model uncertainty to the reward, we improve our model, which then leads to better policy optimization. Therefore, we believe that adding the reward does not lead to overestimation; instead, it improves our model and thereby results in better planning performance. The extreme case of our method is studied by works such as (6, 7), where only the model uncertainty is used as reward, and they show that this results in strong planning with a learned model for multiple downstream tasks.\n\n\nWe hope our response clarifies the reviewers' concerns. In summary, we provide theoretical grounding for a practical RL algorithm. Even though other works have also proposed using the model uncertainty as an intrinsic reward, to the best of our knowledge, we are the first to show that this has no regret for several common RL settings. Next, we also evaluate the exploration strategy across different tasks; state-based (including the high-dimensional humanoid tasks), visual control, and on real-world hardware.\n We think this makes our paper a strong contribution and would appreciate it if the reviewer would increase their score.\n\n**References**:\n1. https://proceedings.neurips.cc/paper/2020/hash/aee5620fa0432e528275b8668581d9a8-Abstract.html\n\n2. https://proceedings.neurips.cc/paper/2020/hash/a36b598abb934e4528412e5a2127b931-Abstract.html\n\n3. https://www.jmlr.org/papers/v23/20-807.html\n\n4. https://proceedings.neurips.cc/paper_files/paper/2023/hash/31e018f43ab9c7065c058cc2c5848128-Abstract-Conference.html\n\n5. https://proceedings.neurips.cc/paper_files/paper/2023/hash/836012122f3de08aeeae67369b087964-Abstract-Conference.html\n\n6. https://proceedings.mlr.press/v119/sekar20a.html\n\n7. https://proceedings.neurips.cc/paper_files/paper/2023/hash/77b5aaf2826c95c98e5eb4ab830073de-Abstract-Conference.html\n\n8. https://proceedings.mlr.press/v97/pathak19a.html"}, {"review_id": "toXF31vPQU", "reviewer": "Reviewer_dt6r", "summary": "This paper proposes a simple optimism-in-the-face of uncertainty algorithm for model based RL. The algorithm works with any given uncertainty estimate; in theory, kernel estimates suffice whereas in practice, popular ensemble based methods suffice. Experiments are compelling on relatively simple environments.", "questions": "(1) How well do the authors suspect the method to perform in more complex tasks?\n(2) Have the authors found the algorithm to be highly sensitive to the use of uncertainty estimates? What happens if the uncertainty estimate becomes unreliable?\n(3) What would the authors regard as the most original (note: the distinction from originality and novelty) and most surprising aspects of their submission?", "limitations": "Experimental domains are limited in their difficulty and sophistication. Consider, e.g. the RoboMimic, Libero, Calvin Kitchen control suites.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: the paper is very nicely written, theory is correct, well exposed, and relations to prior work are made clear. I apologize for not saying more here, but overall the execution is quite flawless and I do not have much more to add. \n\nWeaknesses: though the strengths are apparent, I would say that there are two key weaknesses:\n\n(1) Originality. Here, I would like to distinguish between originality and novelty. I believe the authors that their paper may be \"novel\" in the sense that no one has pursued this direction before. However, it does not feel particular original in the sense that anyone who is familiar with the ideas of Eluder dimension and OFU can believe that such an algorithm can be done and implemented. The salient difference between SOMBRL and the Kakade 2020 algorithm is that the uncertainty term is now aapplied on a per-time step basis, rather than aggregated across timesteps. This simplifies the optimism computation, but at the expense of some small statistical suboptimality. I don't think I find this particularly surprising, at least theoretically, nor would I be surprised if the authors of Kakade would have recommended this approach had someone tried to implement their algorithm theoretically. Ultimately, maybe the best conclusion here is \"well, I guess somebody had to do it\" and the authors did. And it was well executed. But still, I am not at all surprised (and in my own research, have been implementing something similar as a baseline). Again, I am not claiming this algorithm exists in the literature, but it feels more like \"folklore principles\" instantiated finally executed. \n\n(2) I think the authors are taking liberties with the description \"scalable.\" By the standards of modern RL, the settings are incredibly toy. The reason this is salient is that one need to be convinced that todays uncertainty-based methods are indeed robust enough for more dextrous, high-dimensional, or visually complex tasks. And because this method relies on explicit ensembling as a form of uncertainty estimation, the major research question is \"to what extent to todays uncertainty estimates suffice to implement optimism.\" So far, I see the limited experimental domains as providing limited (though perhaps encouraging!) evidence.", "was_revised": true, "final_justification": "The authors pointed out a salient difference in between their result and Kakade et al, motivating the increased efficiency of their algorithm. This warranted me to raise my score.", "rebuttal": "We thank the reviewer for their feedback and for acknowledging our writing, the simplicity of our algorithm, and the theoretical results.\n\n**Originality**: We agree that the algorithm is quite intuitive, and we find it to be one of its strengths. However, we think our theoretical analysis is original and not a straightforward extension of the results of prior work, such as Kakade et al 2020. Our analysis is based on a different question: how much exploration with the reward should I inject if I greedily plan with the mean dynamics? This underlying question is very different to what prior works like optimistic exploration methods consider. They pick the dynamics in the plausible set of learned models that are the most favourable. Hence, our approach only requires maximizing over the action/policy space, whereas the latter methods require optimizing over both the action space and the space of plausible dynamics. This is a much harder problem, which is practically less tractable and also does not scale with high input dimensions, e.g., visual control tasks. We show that through injecting exploration rewards, you can be optimistic. Which may sound intuitive, but we believe is an original theoretical insight. Our analysis requires us to study both the regret of the extrinsic reward and also bound the pure exploration objective (model uncertainty). Prior works have only studied them separately, i.e., Kakade et al 2020 only study the former, whereas OPAX (sukhija et al 2024) studies the latter.\nMoreover, regarding the reviewer’s comment: ``authors of Kakade would have recommended this approach had someone tried to implement their algorithm''. In fact, Kakade did recommend an empirical approach to their algorithm (see Section 3.1 on page 6) and their practical approach is based on a sampling-based approximation of the optimization over the dynamics. Moreover, they highlight the hardness of the optimization for their algorithm, and the method we chose is never discussed as a possible alternative. Furthermore, Curi et al (2020) propose the reparameterization trick instead to optimize over the dynamics. Compared to both these works, our much simpler and, compared to Curi et al (2020), our upper bound is exponentially better (see Theorem B.18). \nIn addition, several works study uncertainty-based exploration empirically (Plan2Explore, Disagreement, Buisson-Fenet et al (2020)) and none of which provide any theoretical guarantee. Furthermore, other methods with theoretical guarantees, such as Mania et al. (2020); Wagenmaker et al. (2023); Treven et al. (2024) (see Section A for more), also propose a much more complex optimization problem instead of our simple objective. Moreover, since the seminal work from Abbasi-Yadkori et al 2011, most works have focused on optimistically selecting the dynamics model among a set of plausible dynamics. Having studied these prior works and worked on similar topics ourselves,  we believe our method is based on an original perspective and regret analysis that is different from prior works (as described above).\nIn addition to the points above, most of these works focus on the finite horizon setting; we also provide our regret bound for the infinite horizon and the non-episodic case. \n\n**Scalability and Performance on Complex Tasks**: We evaluate our algorithm on well-established deep RL benchmarks, including high-dimensional tasks such as the humanoid (67-dimensional state and 21-dimensional action space), visual control problems, and also on a real-world hardware system. Across all our experiments, we found uncertainty-based exploration to either perform on par or better. We particularly build up on intrinsic exploration methods such as Plan2Explore and MaxInfoRL to scale our work for these settings. Hence, we believe this is encouraging evidence for testing our approach on more complex domains, and we find this a promising direction for future work.\n\n**Sensitivity to Uncertainty**: In our experiments, we found the use of ensembles as suggested in HUCRL, Plan2Explore, and MaxInfoRL to work quite robustly across tasks, i.e., we did not notice any sensitivity. However, these methods are also designed to be robust towards uncertainty estimation, e.g., Plan2Explore only quantifies uncertainty in a lower-dimensional latent space, and MaxInfoRL tunes how much exploration via uncertainty to inject into the system.  Nonetheless, theoretically, if the uncertainty estimates are off, e.g., too high, then the agent could end up overexploring, or, in the case of too low, underexploring. Hence, this can have an effect on the performance of the agent, and how to mitigate this, e.g., via better uncertainty quantification or auto-tuning mechanisms such as from MaxInfoRL, is an interesting research question.\n\n**Originality and Surprise**: As highlighted in our response above, we believe our perspective on regret and our theoretical analysis are not only novel but also original. Several prior works proposed no-regret model-based RL algorithms that require a much more complex and often intractable optimization over the set of dynamics models. These works start with the theoretical algorithm and then propose practical modifications, such as sampling-based optimization or the reparameterization trick, to solve the underlying optimization problem.\nWe took a different approach by starting with a practical algorithm and showing that it has no regret. This required a different theoretical analysis from the prior work, as we highlight in our response above and the proof sketches in the paper. Furthermore, perhaps not surprising, but what we really like about our approach is that it provides a unified approach for MBRL, our algorithm can be used for pure (unsupervised) exploration by setting the extrinsic reward to 0, Curi et al 2020 and Kakade et al 2020 do not work for this setting and the OPAX algorithm which is used for pure exploration does not work in the setting with rewards. Furthermore, our approach also enjoys guarantees in the nonepisodic settings, which the other algorithms do not provide (in fact, Kakade et al, 2020, leave that as an open problem in Section 5).\n\n\nWe hope that with our response, we could convince the reviewer of the strengths of our work, and we would appreciate it if they would consider increasing their score."}]

Author rebuttal:

Meta-review: This paper presents a model-based RL algorithm that uses an estimate of the epistemic uncertainty of the dynamics model to perform explorative planning during learning. Theoretical analysis studies the performance of the algorithm and shows it has sublinear-in-N regret in continuous state and action spaces. Experiments demonstrate the efficacy of the approach when applied to existing MBRL approaches in state-based and visual settings.

Reviewers mentioned concerns about novelty, specifically prior theoretical and empirical work has proposed using epistemic uncertainties in model-based RL. Quoting from a reviewer: "Ultimately, maybe the best conclusion here is "well, I guess somebody had to do it" and the authors did. And it was well executed. But still, I am not at all surprised (and in my own research, have been implementing something similar as a baseline). Again, I am not claiming this algorithm exists in the literature, but it feels more like "folklore principles" instantiated finally executed. "

While it does appear there's a disconnect between the theoretical results and the algorithm tested, and that the algorithm tested bears similarity to prior theoretical work, the simplicity of the method combined with the clear empirical gains suggest to me sufficient novelty warranting publication. The characterization of the approach being an instance of "folklore principles finally executed" is another compelling argument for acceptance, in the interest of turning folklore into a more well-known result that future work can more easily build off of.
Decision: Accept (poster)