Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling

## Abstract

Abstract Learning complex robot behavior through interactions with the environment necessitates principled exploration. Effective strategies should prioritize exploring regions of the state-action space that maximize rewards, with optimistic exploration emerging as a promising direction aligned with this idea and enabling sample-efficient reinforcement learning. However, existing methods overlook a crucial aspect: the need for optimism to be informed by a belief connecting the reward and state. To address this, we propose a practical, theoretically grounded approach to optimistic exploration based on Thompson sampling. Our model structure is the first that allows for reasoning about joint uncertainty over transitions and rewards. We apply our method on a set of MuJoCo and VMAS continuous control tasks. Our experiments demonstrate that optimistic exploration significantly accelerates learning in environments with sparse rewards, action penalties, and difficult-to-explore regions. Furthermore, we provide insights into when optimism is beneficial and emphasize the critical role of model uncertainty in guiding exploration.

## 1 Introduction

Reinforcement Learning (RL) is recognized as a promising approach to solving complex sequential decision-making tasks in robotics (Levine et al., ). However, many popular RL algorithms require millions of interactions with the real world to train effective policies (Schulman et al., ; Mnih et al., ). This sample inefficiency is primarily due to the central challenge of balancing exploration, gathering information about the world, with exploitation, maximizing rewards given current knowledge (Sutton & Barto, ; Szepesvári, ). Sample inefficiency is especially prohibitive for RL applied to robotics due to the high cost and potential wear on physical systems.

Efficient RL requires a careful balance between exploration and exploitation, and some approaches are better suited for this than others. Our work centers on model-based RL, a class of algorithms where agents build a predictive model of the environment dynamics from which they derive a policy. Many works have demonstrated impressive sample efficiency using model-based RL by leveraging the learned dynamics model to simulate future outcomes, thereby allowing agents to plan strategically and optimize the controller using fewer real-world trials (Ibarz et al., ; Chua et al., ; Janner et al., ; Yang et al., ; Sekar et al., ; Hansen et al., ). However, learning a model of the environment is a double-edged sword: a faithful model enhances efficiency by reducing the number of interactions needed in the real world while a biased model severely hampers performance by misguiding the policy (Gu et al., ). Planning with uncertainty-aware dynamics reduces the effects of model errors (Deisenroth & Rasmussen, ; Deisenroth et al., ; Gal et al., ) and allows model-based RL algorithms to reach state-of-the-art asymptotic performance on benchmark control tasks (Chua et al., ; Janner et al., ). Furthermore, by learning a probabilistic dynamics model, model-based RL enables directing exploration in a principled manner towards unseen parts of the state-action space.

In this work, we focus on optimistic exploration using model-based RL with an uncertainty-aware model. In a subtle departure from classical exploration, which prioritizes transitions with high uncertainty under the model, we define optimistic exploration as favoring transitions likely to yield high rewards. This approach aligns well with the RL objective of maximizing cumulative rewards (Sutton & Barto, ). Notable works unite optimistic exploration with uncertainty-aware model-based RL to hallucinate plausible, optimistic training experiences for superior robustness and sample efficiency (Curi et al., ; Sessa et al., ), however these formulations ignore joint uncertainty between the reward and dynamics. They also assume knowledge of a reward function, which can be impractical in real-world applications due to the difficulty of specifying dynamic, variable, and subjective reward functions accurately a priori. While learning the reward function is a straightforward remedy, there is a dearth of efficient and principled approaches to integrating reward learning into optimistic exploration for model-based RL.

Contributions. Our primary contribution is the first practical model-based RL algorithm, called Hallucination-based Optimistic Thompson sampling with Gaussian Processes (HOT-GP), that can be used with state-of-the-art off-policy RL algorithms for principled optimistic exploration. Inspired by the premise that efficient exploration should be informed by joint uncertainty over state and reward distributions, we propose a GP model to maintain this joint belief in order to simulate transitions that are simultaneously plausible under the learned dynamics and optimistic with respect to the estimated reward. We evaluate this approach on a set of MuJoCo (Todorov et al., ) and VMAS (Bettini et al., ) continuous control tasks. The results reveal that HOT-GP matches or improves sample efficiency on standard benchmark tasks and substantially accelerates learning in challenging settings involving sparse rewards, action penalties, and difficult-to-explore regions. As these findings validate our hypothesis, our secondary contribution is an empirically-supported argument for maintaining a belief over both the reward and state distributions, as this is key for effective optimistic exploration. Moreover, we study the factors that influence the utility of optimism and underscore the important role of model uncertainties in shaping exploration.

## 2 Related Work

Model-based RL methods are promising for complex real-world decision problems due to their potential for data efficiency (Kaelbling et al., ). Selecting the appropriate model is critical, as it must facilitate effective learning in both low-data regimes (during early stages of training) and high-data regimes (later in training). This requirement naturally aligns with the advantages of Bayesian models. Non-parametric models like Gaussian processes (GPs) provide excellent performance in settings with limited, low-dimensional data (Deisenroth et al., ; Deisenroth & Rasmussen, ; Kamthe & Deisenroth, ). Neural network predictive models have also been effectively used for tasks with non-smooth dynamics (Nagabandi et al., ) and high dimensions like the Atari suite (Oh et al., ; Kaiser et al., ). As deterministic neural networks tend to overfit to data in the early stages of training, uncertainty-aware neural network models have been shown to achieve better performance with algorithms such as PETS (Chua et al., ) and MBPO (Janner et al., ). However, designing scalable Bayesian neural networks remains an open problem (Roy et al., ; Guo et al., ; Osband, ) and popular approximations using dropout (Gal et al., ) and ensembles (Lakshminarayanan et al., ; Chua et al., ) fall short of the precise uncertainty quantification that GPs provide. In order to benefit from both the high-capacity function approximation of neural networks and probabilistic inference of GPs, in this work we model the joint dynamics and reward functions using a GP with a neural network mean function (Iwata & Ghahramani, ).

Thompson sampling is a provably efficient exploration algorithm in RL (Thompson, ). This approach implicitly balances exploration and exploitation by sampling statistically plausible outcomes from the posterior distribution over dynamics models and selecting actions based on the sampled outcomes (Russo & Van Roy, ; Russo et al., ). Thompson sampling has been studied on tabular MDPs (Osband et al., ) and extended in theory to continuous state-action spaces with sublinear regret under certain conditions (Chowdhury & Gopalan, ).

Optimism in the face of uncertainty is another theoretically grounded approach to guide exploration that selects actions that optimize for optimistic outcomes. The motivation is to promote exploration guided by beliefs about future value to facilitate efficient learning. While optimism has been extensively studied in tabular MDPs (Brafman & Tennenholtz, ; Jaksch et al., ) and linear systems (Jin et al., ; Abbasi-Yadkori & Szepesvári, ; Neu & Pike-Burke, ), optimism in continuous state-action MDPs with non-linear dynamics remains less explored. GP-UCRL (Chowdhury & Gopalan, ) is one such theoretical approach that targets dynamics models that lie in a Reproducing Kernel Hilbert Space. Another is LOVE (Seyde et al., ), which uses an ensemble of networks to direct exploration towards regions with high predicted potential for long-term improvement. TIP (Mehta et al., ) also employs optimism by selecting actions according to an optimistic acquisition function. Curi et al. () proposes a reparameterization of the dynamics function space in order to reduce optimistic exploration to greedy exploitation with the H-UCRL algorithm. H-UCRL has sublinear regret bounds under certain conditions and demonstrates efficient exploration on deep RL benchmark tasks with action penalties. The H-MARL algorithm extends this to the multi-agent setting for general-sum Markov games and introduces a practical approximation for generating optimistic states (Sessa et al., ). While these methods all derive useful optimistic exploration strategies for RL, they ignore the relationship between uncertainty around dynamics and uncertainty around the reward. Moreover, TIP, H-UCRL, H-MARL use the ground-truth reward function throughout training. Our approach, inspired by H-UCRL (Curi et al., ), improves optimistic hallucination for model-based RL by learning and leveraging the joint uncertainty over state transitions and associated rewards.

Reward learning is commonly applied in RL for reward shaping (Hu et al., ). This is closely related to exploration, as intrinsic rewards learned through curiosity-driven methods help guide exploration by reducing uncertainty over the agent’s knowledge of the environment (Pathak et al., ). Reward learning also plays an important role in model-based RL. Receding-horizon planning approaches like PETS (Chua et al., ), Dreamer (Hafner et al., ; ; ), and TD-MPC (Hansen et al., ; ) use estimated rewards to evaluate and optimize sequences of actions, while model-based policy learning approaches like MBPO (Janner et al., ) use estimated rewards to inform policy updates. Some of these approaches explicitly account for uncertainty in the learned reward function (Chua et al., ; Janner et al., ; Chowdhury & Gopalan, ), however they treat it as independent from the uncertainty in the dynamics. We argue that learning an uncertainty-aware reward-dynamics function and strategically leveraging the joint uncertainty leads to the most principled and effective optimistic exploration.

## 3 Problem Statement

We consider a stochastic environment with states $s\in\mathcal{S}\subseteq\mathbb{R}^{p}$ and actions $a\in\mathcal{A}\subseteq\mathbb{R}^{q}$. Given the current state $s_{t}$ and action $a_{t}$, the agent transitions to the next state $s_{t+1}$ and incurs reward $r_{t}$ with

$$ $\begin{pmatrix}s_{t+1}\\ r_{t}\end{pmatrix}=f(s_{t},a_{t})=\begin{pmatrix}f_{t}(s_{t},a_{t})\\ f_{r}(s_{t},a_{t})\end{pmatrix}$ (1) $$

for transition dynamics $f_{t}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$ and reward function $f_{r}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$. Our objective is to learn optimal control for this system within episodes of finite time horizon $T$. To control the agent, we learn a deterministic or stochastic policy $\pi$ that selects actions given the current state. For ease of notation, we write the policy deterministically as $\pi:\mathcal{S}\rightarrow\mathcal{A}$ so that $a_{t}=\pi(s_{t})$. We consider a specific transition dynamic $\hat{f}_{t}$ and reward function $\hat{f}_{r}$, denoted with a hat to indicate realizations that are drawn from random functions. The performance of the policy is the sum of accumulated rewards during an episode,

$$ $J(\pi,\hat{f}_{t},\hat{f}_{r})=\left[\sum_{t=0}^{T}\hat{f}_{r}(s_{t},a_{t}) \right],\quad\text{s.t.}\quad a_{t}=\pi(s_{t}),s_{t+1}=\hat{f}_{t}(s_{t},a_{t}).$ (2) $$

The optimal policy $\pi^{*}$ maximizes performance over the true dynamics and reward function with,

$$ $\pi^{*}=\operatorname*{arg\,max}_{\pi\in\Pi}J(\pi,f_{t},f_{r}).$ (3) $$

In model-based RL, the true dynamic $f_{t}$ is unknown and must be learned by interactions with the environment. In our setting, the ground-truth reward function $f_{r}$ is also unknown.

## 4 Background

In this section, we introduce relevant background on model-based reinforcement learning and existing exploration strategies.

### 4.1 Model-based Reinforcement Learning

When systems have unknown dynamics, we consider the dynamics $f_{t}$ to initially be random. Therefore, learning a model of the environment dynamics entails fitting an approximation $p(f_{t})$ over the space of dynamics functions, given observations from the real system. The same approach can be used to estimate the true reward function $f_{r}$. We consider model-based RL algorithms that, on each iteration of training, roll out the current policy $\pi$ on the real system for an episode and update $p(f)$ using the newly observed transition data. Standard approaches use the model to simulate sequential transitions and use this data to update the policy (Sutton, ). We opt to update the policy based on short $k$-length model-generated rollouts branched from the state distribution of a previous policy under the true environment, as recommended by Janner et al. () to reduce compounding model errors over extended rollouts. Algorithm 1 gives an overview of the process.

Figure: Algorithm 1 Model-based Policy Optimization

Model Design. We use an uncertainty-aware model to represent the prior distribution of dynamics models $p(f_{t})$ aligned with the 1-step transition data in $\mathcal{D}_{\text{env}}$. Probabilistic models allow for exploiting correlations in observed data by generalizing to unseen states and actions. Popular approaches include probabilistic ensembles of neural networks, which provide mean $\mu(s,a)$ and confidence estimates $\Sigma(s,a)$ by averaging outputs over multiple trained models (Lakshminarayanan et al., ; Chua et al., ). Bayesian inference with neural networks is computationally expensive and it is an open question if parameter uncertainties lead to relevant posterior function estimates (Sharma et al., ; Roy et al., ).
Alternatively, GPs provide distributions directly over the function space, enabling direct estimation of the posterior distribution $p(f_{t}\mid\mathcal{D}_{\text{env}})$ over dynamics models. Since GPs incorporate strong and interpretable prior beliefs, they naturally capture uncertainty due to limited data (epistemic uncertainty) and variability in the data (aleatoric uncertainty) (Rasmussen & Williams, ), which contributes to GPs’ effectiveness in low-data regimes. GP priors, defined by a mean and covariance function, describe how the function varies around the mean. With flexible mean functions, GPs offer principled uncertainty estimates for complex functions (Iwata & Ghahramani, ; Fortuin et al., ; Fortuin, ).

### 4.2 Exploration Strategies

It is challenging to learn the reward-dynamics model while solving for optimal control. Performance depends on the utility of model-generated data for updating the policy and likewise the value of real experiences gathered by the policy for updating the model. Solving this problem efficiently requires balancing exploitation of current knowledge with exploration of the state-action space. The mechanism for selecting the next model-generated state plays an important role, with different strategies incorporating varying degrees of uncertainty to encourage exploration. In this section, we provide an overview of the relevant exploration strategies, as categorized in Curi et al. ().

Greedy Exploitation. Most model-based RL approaches approximate $p(f_{t}\mid\mathcal{D}_{\text{env}})$ and maximize the policy greedily over the uncertainty in the dynamics model. They use this greedy policy:

$$ $\pi^{\text{Greedy}}=\operatorname*{arg\,max}_{\pi\in\Pi}\mathbb{E}_{p(f_{t} \mid\mathcal{D}_{\text{env}})}J(\pi,\hat{f}_{t},\hat{f}_{r}).$ (4) $$

For instance, GP-MPC (Kamthe & Deisenroth, ) and PETS (Chua et al., ) are MPC-based methods that use greedy exploitation and represent $p(f_{t}\mid\mathcal{D}_{\text{env}})$ with a GP and ensemble of neural networks respectively. Similarly, PILCO (Deisenroth & Rasmussen, ) uses a GP with moment matching to estimate $p(f_{t}\mid\mathcal{D}_{\text{env}})$ and greedy exploitation to optimize the policy. While such algorithms can converge to optimal policies under certain reward and dynamics structures (Mania et al., ), greedy exploitation is generally inefficient for exploration.

Thompson Sampling. This is a theoretically-grounded approach that offers a principled balance between exploration and exploitation (Russo et al., ; Chapelle & Li, ). Under Thompson sampling (Thompson, ), the policy is optimized with respect to a single model sample $\hat{f}$ on each episode,

$$ $\hat{f}\sim p(f\mid\mathcal{D}_{\text{env}}),\quad\pi^{\text{TS}}= \operatorname*{arg\,max}_{\pi\in\Pi}J(\pi,\hat{f}_{t},\hat{f}_{r}).$ (5) $$

This optimization problem is equivalent to greedy exploitation (equation 4) after sampling $\hat{f}\sim p(f\mid\mathcal{D}_{\text{env}})$. Under certain assumptions, Thompson sampling satisfies strong Bayesian regret bounds (Osband & Van Roy, ; ), theoretically positioning it as the optimal strategy.

Hallucinated Upper-Confidence Reinforcement Learning (H-UCRL). This is a tractable version of the UCRL algorithm (Jaksch et al., ), which optimizes an optimistic policy over the set of statistically plausible dynamics models. Curi et al. () propose reparameterizing plausible dynamics models in order to optimize over a smaller class of variables. Specifically, they write dynamics functions as $\hat{f}_{t}(s,a)=\mu(s,a)+\beta\ \Sigma(s,a)\eta(s)$ for some function $\eta:\mathbb{R}^{p}\rightarrow[-1,1]^{p}$ and optimize over $\eta(\cdot)$ instead. As this problem is still challenging to optimize, Sessa et al. () propose sampling $n^{j}\sim\text{Uniform}([-1,1]^{p})$ for $j=1,\dots,Z$ and updating the policy with

$$ $\pi^{\text{H-UCRL}}=\operatorname*{arg\,max}_{\pi\in\Pi}\max_{\eta^{j},j\in 1, \dots,Z}J(\pi,\hat{f}_{t},\hat{f}_{r}),\ \text{s.t.}\ \hat{f}_{t}(s,a)=\mu(s,a )+\beta\ \Sigma(s,a)\eta^{j}.$ (6) $$

Unlike greedy exploitation, this approach effectively optimizes an optimistic policy over the modified dynamics $\hat{f}_{t}$ in equation 6, although the reward function is assumed to be known. H-UCRL also does not model or utilize the joint uncertainty over transitions and rewards.

## 5 The HOT-GP Algorithm

We introduce a new exploration strategy based on the insight that principled optimism should reason about uncertainty over transitions and rewards jointly. Importantly, this requires a joint model of the transition dynamics and reward function so that optimism about the reward can be connected to a distribution over states. Given such a model, there are many possible acquisition strategies. A natural approach uses Thompson sampling, which we adapt for optimistic exploration with our method, Hallucination-based Optimistic Thompson sampling with Gaussian Processes (HOT-GP).

### 5.1 Model Structure

Optimistic exploration implies biased exploration towards regions of the state-action space believed to yield high rewards. Thus, optimistic learning requires a reward-dynamics model that describes the correlation of state and reward. In this work, we use a Gaussian process (GP) prior to describe our belief over the product space of reward and state,

$$ $\hat{f}\sim\mathcal{GP}\left(\begin{pmatrix}\mu_{s}(\cdot)\\ \mu_{r}(\cdot)\end{pmatrix},\begin{pmatrix}K_{ss}(\cdot,\cdot)&K_{sr}(\cdot, \cdot)\\ K_{sr}^{\textrm{T}}(\cdot,\cdot)&K_{rr}(\cdot,\cdot)\end{pmatrix}\right),$ (7) $$

where $\mu_{s}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$ and $\mu_{r}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ are the dynamics and reward mean functions and $K_{ss},K_{sr},K_{rr}$ represent covariance functions $\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$. Traditional GPs describe outputs as conditionally independent given the input and hence factorize across the state dimensions and the reward. However, we hypothesize that this assumption is unfounded in most RL tasks and furthermore antithetical to principled optimistic exploration. Therefore, we want to explicitly model the correlation structure between the dimensions of the predicted state, and more importantly, between the predicted state and its associated reward. While this correlation can naturally be included in the GP by parameterizing the full covariance structure across examples and output dimensions, this is computationally intractable due to cubic scaling with the size of the covariance matrix (Rasmussen & Williams, ).
Instead, we adopt a linear model of coregionalization (Grzebyk & Wackernagel, ; Wackernagel, ) where the covariance between output dimensions is restricted to be linear combinations of the input covariance. This induces a Kronecker structured covariance function that allows for efficient factorization while still meaningfully directing optimistic exploration.

In most GP regression use cases, the mean function is a constant function. While this is a valid modeling assumption for stationary data, the dynamics and reward structures we encounter in RL tasks vary substantially across different regions of the input space. A stationary assumption would lead to significant model mismatch and transfer the responsibility of the mean to the covariance function, causing the model to badly represent the uncertainty, resulting in inefficient exploration and misguided optimism. To circumvent this problem, we use a multi-layer MLP as a mean function as in Iwata & Ghahramani (); Fortuin et al. (). This allows learning a flexible mean function for non-stationary transitions while providing uncertainty that can be used in an optimistic fashion. To learn the covariance and mean functions, we perform approximate inference by optimizing a variational lower bound on the marginal likelihood, as done by Titsias (). This updates the posterior distribution $p(\hat{f}\mid\mathcal{D}_{\text{env}})$, which we use to obtain the posterior predictive distribution $p(s_{t+1},r_{t}\mid s_{t},a_{t})$. Given the current state $s_{t}$ and action $a_{t}$, this multivariate Gaussian distribution reflects the model’s predictions and uncertainty over the next state and reward.

### 5.2 Using Thompson Sampling in the GP Acquisition Function

Thompson sampling proscribes optimizing the policy using a single reward-dynamics model from the distribution of predictive models. A naive approach would sample from $p(s_{t+1},r_{t}\mid s_{t},a_{t})$ until discovering a good model as a function of the predicted reward $r_{t}$. Instead, we observe that we can condition the reward value $r_{t}$ to be high, specifically greater than some minimum percentile threshold $r_{\text{min}}$ over the reward distribution, and sample once from this distribution. Crucially, this induces optimism into the sampling procedure because we only consider transitions where the next state is plausible under the dynamics model and whose associated reward we believe to be large.

To do this, we sample independent realizations from one non-parametric reward-dynamics model for each episode of length $T$, denoted as

$$ $\hat{f}^{(t:0\rightarrow T)}\sim\prod_{t=0}^{T}p(s_{t+1},r_{t}\mid s_{t},a_{t} ,r_{t}>r_{\text{min}})$ (8) $$

for $s_{t}$ sampled uniformly from $\mathcal{D}_{\text{env}}$ and $a_{t}=\pi(s_{t})$ under the current policy. Then the optimistic Thompson sampling algorithm is

$$ $\hat{f}^{(0\rightarrow T)}\sim p(s_{t+1},r_{t}\mid s_{t},a_{t},r_{t}>r_{\text{ min}}),\quad\pi^{\text{HOT-GP}}=\operatorname*{arg\,max}_{\pi\in\Pi}J(\pi,\hat {f}_{t}^{(0\rightarrow T)},\hat{f}_{r}^{(0\rightarrow T)}).$ (9) $$

Figure: Figure 1: Learning curves on MuJoCo tasks averaged over 5 seeds. HOT-GP demonstrates equivalent or superior sample efficiency and performance for all tasks considered. The dashed line denotes SAC performance at convergence within 1,000,000 environment steps (3,000,000 for Half-Cheetah).
Refer to caption: extracted/5907130/figures/halfcheetah.png

Performing Thompson sampling following the joint distribution in equation 9 predicts both the next state and its associated reward in an optimistic manner. We argue that this would lead to a suboptimal exploration strategy as we are only interested in optimism over the state to the extent that it relates to optimism over the reward. To address this, we instead sample reward-dynamics model realizations using a two-step approach where only the reward is explored in an optimistic manner:

- 1.
Sample an optimistic reward $\hat{r}_{t}$ from $p(r_{t}\mid s_{t},a_{t},r_{t}>r_{\text{min}})$, a truncated normal distribution, using inverse transform sampling or another sampling technique.
- 2.
Compute the expected transition $\hat{s}_{t+1}=\mathbb{E}_{p(s_{t+1}\mid s_{t},a_{t},r_{t}=\hat{r}_{t})}[s_{t+1}]$ conditioned on the sampled optimistic reward $\hat{r}_{t}$.

This approach deviates from traditional Thompson sampling, and hence allows us to disentangle optimism over the state from optimism over the reward so that we can concentrate solely on optimism over the reward from the states. This aligns well with our goal of sample-efficient reinforcement learning as the objective is to maximize the reward in as few environment steps as possible.

## 6 Experiments

We evaluate the performance of our proposed algorithm, HOT-GP, on continuous state-action control tasks. Since inaccurate reward estimates are likely initially, we increase $r_{\text{min}}$ linearly throughout training to become more optimistic as the model becomes more reliable. In our experiments, we consider three variations of HOT-GP where $r_{\text{min}}$ begins at 0.1 and increases to either 0.3, 0.5, or 0.7. We also consider the alternative Thompson sampling and greedy exploration strategies. Our Thompson sampling implementation follows the HOT-GP procedure but draws samples from the reward-conditioned distribution $p(s_{t+1}\mid s_{t},a_{t},r_{t}=\hat{r}_{t})$ rather than taking the expectation. For greedy exploitation, we predict the next state and reward as the mean from the model.

First, we compare these approaches on standard MuJoCo benchmark tasks (Todorov et al., ) against additional model-free and model-based methods. Then we evaluate these approaches on a practical robotics task using the VMAS simulator (Bettini et al., ) and provide an analysis of the individual components. In-depth algorithmic descriptions of HOT-GP, along with the model-based methods we use for comparison, are given in Appendix A. Implementation details about our training procedures are discussed in Appendix B and specific tasks are described in Appendix C.

### 6.1 MuJoCo Benchmarks

Comparison Methods. In these experiments, we consider HOT-GP with varying levels of optimism using SAC (Haarnoja et al., ) as the underlying PolicySearch algorithm, as done in Janner et al. () with MBPO. We compare performance against SAC, Thompson sampling, greedy exploitation, and MBPO. SAC is a model-free actor-critic algorithm that is known to achieve better sample efficiency than DDPG (Lillicrap et al., ) on MuJoCo tasks (Chua et al., ). MBPO is a model-based approach that uses an ensemble of probabilistic neural networks to model uncertainty as done in Chua et al. (). Predictions are drawn from a Gaussian distribution with diagonal covariance over the reward and dynamics: $p(s_{t+1},r_{t}\mid s_{t},a_{t})=\mathcal{N}(\mu(s_{t},a_{t}),\Sigma(s_{t},a_{
t}))$, where $\mu(s_{t},a_{t})$ and $\Sigma(s_{t},a_{t})$ are bootstrapped from the ensemble. For an optimistic baseline, we introduce an optimistic version of MBPO where the reward is sampled from $p(r_{t}\mid s_{t},a_{t},r_{t}>r_{\text{min}})$ and expected next state $\mathbb{E}_{p(s_{t+1}\mid s_{t},a_{t})}[s_{t+1}]$ is selected. This is the same procedure as HOT-GP, however, due to MBPO lacking a joint model over the state and reward distributions, this implies a naive optimistic exploration as there is no explicit relationship between the predicted next state and reward.

Standard Benchmarks. The learning curves for all methods on Half-Cheetah, Reacher, and Pusher are given in the top row of Figure 1. In each case, the HOT-GP variants attain SAC-best performance with comparable or superior sample efficiency to other approaches. For instance, on Half-Cheetah, HOT-GP with $r_{\text{min}}$ from 0.1 to 0.3 achieves the same final performance as SAC in 250,000 environment steps rather than 3 million. We observe that the deterministic approach used in greedy exploitation leads to slower learning on Half-Cheetah compared to uncertainty-aware methods, while showing no impact on performance on Pusher. This indicates that model uncertainty is key for efficient learning on Half-Cheetah, likely because the dynamics are more challenging to model, which makes model biases more detrimental to learning (Chua et al., ). Interestingly, uncertainty-aware learning alone is insufficient on Pusher as optimistic exploration methods enhance sample efficiency compared to MBPO and greedy exploitation. This is likely due to the task dynamics as the agent must navigate a landscape of negative rewards and sparse feedback. Optimistic exploration allows the agent to discover valuable actions more effectively by pursuing riskier options with potential to help reach the goal state. Next, we examine the impact of optimistic exploration in sparse reward settings more closely.

Sparse Rewards. This task provides reward $r_{\text{dist}}(s)$ that is nonzero only when the agent is within a small threshold of the target location, thus demanding clever exploration strategies to reach the target as there is no dense feedback. As in Curi et al. (), we introduce an action penalty of the form $r(s,a)=r_{\text{dist}}(s)-\rho\cdot\text{cost}(a)$ with the aim of penalizing random and encouraging local exploration.
This is described in greater detail in Appendix C. We evaluate HOT-GP and the other comparison methods on Sparse Reacher using three different action penalty weights: $\rho=0$, $\rho=0.1$, and $\rho=0.5$, and present the results in the bottom row of Figure 1. HOT-GP demonstrates significantly superior sample efficiency over MBPO and moderately improved sample efficiency over optimistic MBPO in each case. The findings on Sparse Reacher with $\rho=0$ indicate that all the HOT-GP variants perform robust optimistic exploration and successfully uncover the signal despite sparse rewards. When using the action penalty $\rho=0.1$, HOT-GP with $r_{\text{min}}$ from 0.1 to 0.3 performs poorly relative to higher levels of optimism. This suggests that higher levels of optimism are necessary to overcome the prior against taking large actions and explore the reward signal $r_{\text{dist}}$. HOT-GP is able to do this by relating the action-based penalty to the state to the overall reward through the joint model. The less efficient learning of optimistic MBPO highlights the critical importance of having a joint model of the reward and state distributions to facilitate efficient exploration. For $\rho=0.5$, we observe that the cost term typically accounts for nearly half of the overall reward. As a result, the HOT-GP variants explore much of the cost signal rather than $r_{\text{dist}}$, leading to their similar performance in this setting.

### 6.2 Robot Coverage

Coverage Description. We consider a practically motivated continuous control robotics problem where an agent must visit as many unique, high valued cells as possible within a finite episode. This mirrors real-world target-tracking tasks where an agent monitors multiple regions of interest under time constraints (Robin & Lacroix, ; Xin et al., ).
The reward for visiting a new 2D cell is distributed as the probability density function of a mixture of three randomly placed Gaussians over the $x$ and $y$ axes, so there are cluster rewards that the agent must find and exploit. To incentivize thorough coverage, the agent receives no reward for revisiting cells in an episode.

Comparison Methods. As in the MuJoCo tasks, we evaluate HOT-GP with three levels of optimism and compare against Thompson sampling and greedy exploitation. A distinction is that, since DDPG outperforms SAC in this task, we use DDPG with a probabilistic actor for the PolicySearch algorithm. Learning a stochastic policy allows for a baseline level of exploration, which we found to be helpful when also incorporating uncertainty induced by the reward-dynamics model. For model-free baselines, we use DDPG, SAC, and the on-policy PPO algorithm (Schulman et al., ). For an optimistic baseline, we use two variations of H-UCRL: one has privileged access to the ground truth reward function $f_{r}$ while the other adapts to our unknown reward setting by learning the reward function $\hat{f_{r}}$ and applying it as in equation 6. In both cases, we use the same dynamics-reward model with H-UCRL as in HOT-GP for a fair comparison of the optimistic exploration strategies.

Performance Analysis. The first two subfigures in Figure 2 show the training curves for these approaches with and without access to the ground-truth reward function. Our method outperforms the model-based approaches in both settings and achieves DDPG-best performance marginally ahead of DDPG. The poorer performance of greedy exploitation without access to the true reward function suggests that strategic exploration is key to efficient learning for this task. Indeed, the agent must explore the environment sufficiently well enough to learn the relationship between its observations and the locations of the Gaussian centers, make informed decisions about which Gaussian center to prioritize, and realize the importance of avoiding previously seen cells. HOT-GP also demonstrates performance superior to H-UCRL, which does not converge to the optimal performance level within 200,000 environment interactions, with and without access to the ground truth reward. This may be due to infeasible optimism where the rewards associated with perturbed states $\mu(s,a)+\beta\ \Sigma(s,a)\eta$ are high but the states are not reachable in practice. HOT-GP avoids this problem by selecting states conditioned on optimistic rewards from the joint distribution, thus ensuring the states are plausible under the learned model and reward value. These findings reinforce the benefits of principled optimistic exploration for improving sample efficiency in tasks where exploration is key.

Figure: Figure 2: Learning curves in the coverage environment averaged over 10 seeds. HOT-GP achieves strong performance within 200,000 environment steps while other methods exhibit poorer sample efficiency or asymptotic performance. The dashed line denotes DDPG performance at convergence within 500,000 environment steps.
Refer to caption: extracted/5907130/figures/coverage_all_performance_comparison.png

Ablation Study. Finally, we conduct an ablation study over the main components of HOT-GP to investigate their relative importance. The ablation study in Figure 2 presents the findings. We observe that scaling $r_{\text{min}}$ from $0.1$ to $0.5$ yields the best results, indicating that this task benefits from a medium level of optimistic exploration. Since HOT-GP using other ranges of $r_{\text{min}}$ values still performs fairly well, our approach does not appear to be overly sensitive to $r_{\text{min}}$ settings. Indeed, $r_{\text{min}}$ is an interpretable parameter that can be informed by the hypothesized level of exploration required. The ablation study in Figure 2 shows that HOT-GP without $k$-branching results in consistently poor performance. We also observe that $k$-branching with $k=5$ leads to less optimal performance than with $k=1$. This aligns with prior findings that accumulating errors in the dynamics model can be detrimental to learning (Janner et al., ; Gu et al., ). Furthermore, this highlights the importance of learning a faithful model to benefit from optimistic exploration, as optimism guided by an inaccurate belief harms learning. The ablation study also reveals the importance of modeling the correlation between the predicted state and reward, as this is also essential to be able to harness optimistic exploration. Lastly, sampling the next state from an optimistic distribution with Thompson sampling leads to a larger variance in performance than selecting the expected next state, thus validating our two-step sampling approach. Overall, this ablation study confirms that each component of HOT-GP is necessary to learn effectively.

## 7 Conclusion

We have introduced a principled approach to optimistic exploration based on leveraging a joint belief over the reward and state distribution. Our proposed algorithm, HOT-GP, uses a Gaussian Process to approximate the reward-dynamics and allows us to simulate plausible transitions associated with optimistic rewards, in a practical adaptation of Thompson sampling. Our experiments showed that HOT-GP achieves comparable or superior sample efficiency relative to other model-based methods and exploration strategies. Notably, we found that joint model uncertainty over outputs is crucial for effective exploration, particularly in challenging settings with sparse rewards, action penalties, and difficult-to-explore regions. Ultimately, this work establishes the importance of joint uncertainty modeling for optimistic exploration and makes a meaningful advancement towards more sample-efficient reinforcement learning.

Future Work. Future work should study how the schedule on $r_{\text{min}}$ affects optimism and principled ways for setting this value throughout training, as we only considered a linear schedule. Another promising avenue would extend the planning horizon. We concentrated on learning 1-step lookahead reward-dynamics as HOT-GP can implicitly propagate uncertainty across the horizon by using pointwise uncertainty estimates from the model. Nevertheless, there may be benefits in explicitly propagating uncertainty to facilitate optimism over longer-term reward predictions (Seyde et al., ; Mehta et al., ; Hansen et al., ). Furthermore, our investigation indicates that the utility of optimistic exploration depends on the complexity of the task, although our experiments were limited to five tasks. Future research should explore the role of optimism across a broader range of problems, including in real-world scenarios with on-robot learning.

#### Acknowledgments

J. Bayrooti is supported by a DeepMind scholarship. A. Prorok is supported in part by European Research Council (ERC) Project 949940 (gAIa). We also thank Aidan Scannell for insightful conversations and useful feedback on this work.

## Appendix A Algorithm Details

In this section, we provide more detailed information about the algorithms we use for evaluation.

### A.1 Model-Free Baselines

We compare model-based RL approaches with the following popular model-free algorithms:

SAC. We use a standard version of SAC as introduced in Haarnoja et al. (). Our implementation utilizes a Gaussian policy and automatic entropy tuning.

PPO. We also use a standard version of PPO as introduced in Schulman et al. ().

DDPG. We use a variant of DDPG (Lillicrap et al., ) with a probabilistic actor network.
In standard DDPG, the deterministic nature of the policy can lead to insufficient exploration of the action space. We mitigate this tendency by using a probabilistic actor, which incorporates exploration into the policy by sampling actions from a probability distribution. Specifically, the probabilistic actor network outputs a mean $\mu_{\theta}(s)$ and standard deviation $\sigma_{\theta}(s)$ for the current state $s$ and we select the action by sampling from the Gaussian distribution $a\sim\mathcal{N}(\mu_{\theta}(s),\sigma_{\theta}(s)^{2})$. In our experiments, we compare with this version of DDPG using natural exploration under the label \sayDDPG w/o explore. With a deterministic policy, it is also common to facilitate exploration by adding noise to the actions. Therefore, we additionally evaluate our DDPG baseline with explicit exploration under the label \sayDDPG w explore. Specifically, we add Gaussian noise to the sampled action with $a=a+\zeta$ for $\zeta\sim\mathcal{N}(0,\sigma_{\text{explore}}^{2})$.

### A.2 Greedy Exploitation

This exploration strategy follows the optimization rule given in equation 4. In particular, we hallucinate the next state as the mean output from the model and disregard the uncertainty around the prediction. In one variation, we assume access to the ground-truth reward function $f_{r}$ (referred to as \sayGreedy w known reward) for a comparison with H-UCRL. In another variation, we also learn the reward function (referred to as \sayGreedy w/o known reward in the coverage experiment and \sayGreedy in the MuJoCo experiments) and similarly hallucinate rewards as the mean output from the model. In pseudocode, this approach follows Algorithm 1 exactly with the following algorithm-specific routine replacing line 10 to select the next state and reward:

- •
Compute the next expected state as $\hat{s}_{k+1}=\mu_{s}(\hat{s}_{k},\hat{a}_{k})$
- •
Observe the expected reward $\hat{r}_{k}=\mu_{r}(\hat{s}_{k},\hat{a}_{k})$ or true reward $\hat{r}_{k}=f_{r}(\hat{s}_{k},\hat{a}_{k})$

### A.3 MBPO

We implement MBPO as described in Janner et al. () and summarized in Section 6.1. Specifically, the model is an ensemble of probabilistic neural networks that each predict a mean output and variance (exponentiated log variance in practice) $\mu^{i}(s_{k},a_{k})$ and $\Sigma^{i}(s_{k},a_{k})$ and parameterize a multivariate Gaussian distribution with diagonal covariance $p^{i}(s_{k+1},r_{k}\mid s_{k},a_{k})=\mathcal{N}(\mu^{i}(s_{k},a_{k}),\Sigma^{
i}(s_{k},a_{k}))$. To predict an output, the parameters are bootstrapped using expectation or randomly selecting a member and the outcome is drawn from the Gaussian distribution. This is described with the following steps replacing line 10 in Algorithm 1:

- •
Compute the predictions for each ensemble member
$\mu^{i}(\hat{s}_{k},\hat{a}_{k}),\Sigma^{i}(\hat{s}_{k},\hat{a}_{k})\leftarrow
\hat{f}^{i}(\hat{s}_{k},\hat{a}_{k})$
- •
Bootstrap predictions to get $\bar{\mu}(\hat{s}_{k},\hat{a}_{k}),\bar{\Sigma}(\hat{s}_{k},\hat{a}_{k})$
- •
Sample outcome from the Gaussian distribution $\hat{s}_{k+1},\hat{r}_{k}\sim\mathcal{N}(\bar{\mu}(\hat{s}_{k},\hat{a}_{k}),
\bar{\Sigma}(\hat{s}_{k},\hat{a}_{k}))$

For the optimistic MBPO baseline, we maintain the ensemble model structure but perform a procedure similar to HOT-GP, following these steps to generate next states and rewards:

- •
Compute the predictions for each ensemble member
$\mu^{i}(\hat{s}_{k},\hat{a}_{k}),\Sigma^{i}(\hat{s}_{k},\hat{a}_{k})\leftarrow
\hat{f}^{i}(\hat{s}_{k},\hat{a}_{k})$
- •
Bootstrap predictions to get $\bar{\mu}(\hat{s}_{k},\hat{a}_{k}),\bar{\Sigma}(\hat{s}_{k},\hat{a}_{k})$
- •
Separate the reward distribution by taking its component
$p(s_{k+1},r_{k}\mid\hat{s}_{k},\hat{a}_{k})_{[r]}=\mathcal{N}(\bar{\mu}(\hat{s
}_{k},\hat{a}_{k}),\bar{\Sigma}(\hat{s}_{k},\hat{a}_{k}))_{[r]}$
- •
Sample reward from an optimistic reward distribution
$\hat{r}_{k}\sim p(s_{k+1},r_{k}\mid\hat{s}_{k},\hat{a}_{k},\hat{r}_{k}>r_{
\text{min}})_{[r]}$
- •
Select expected next state from state distribution $\hat{s}_{k+1}=\mathbb{E}_{p(s_{k+1}\mid\hat{s}_{k},\hat{a}_{k})_{[:r]}}[s_{k+1}]$

### A.4 H-UCRL

We implement H-UCRL (Curi et al., ) using the sampling procedure introduced in (Sessa et al., ). On each model-generated rollout step, this approach hallucinates $Z$ candidate next states that are plausible under the learned dynamics model and selects the one leading to the highest reward under $f_{r}$. Therefore, we assume access to the ground-truth reward function $f_{r}$ and only learn the dynamics model $f_{t}$ in our implementation of H-UCRL. In pseudocode, this approach follows Algorithm 1 exactly with the following steps replacing line 10 where we refer to $\sigma_{s}(s,a)$ as the standard deviation of an output from the GP dynamics model:

- •
Draw random perturbations $\eta^{j}\sim\text{Uniform}([-1,1]^{p})$ for $j=1,\dots,Z$
- •
Compute plausible next states $\hat{s}^{j}_{k+1}=\mu_{s}(\hat{s}_{k},\hat{a}_{k})+\beta\ \sigma_{s}(\hat{s}_{
k},\hat{a}_{k})\eta^{j}$
- •
Select next state $\hat{s}_{k+1}=\operatorname*{arg\,max}_{\hat{s}_{k+1}^{j}}f_{r}(\hat{s}^{j}_{k
+1},\hat{a}^{j}_{k+1})$ for associated actions $\hat{a}_{k+1}^{j}$
- •
Observe the true reward $\hat{r}_{k}=f_{r}(\hat{s}_{k},\hat{a}_{k})$

We also adapt H-UCRL to our setting where the reward is unknown with a variation we call \sayH-UCRL Approx. We do this by learning both the dynamics and reward models and selecting the candiate state leading to the highest predicted reward, i.e., under $\hat{f}_{r}$ rather than $f_{r}$.

### A.5 Thompson Sampling

This approach follows the optimization rule given in equation 5. Our Thompson Sampling implementation only differs from HOT-GP in that we sample both the reward and next state from the optimistic reward-dynamics distribution. In practice, we implement this as described in Algorithm 1 with the following algorithm-specific routine replacing line 10:

- •
Compute predictive posterior distribution $p(s_{k+1},r_{k}\mid\hat{s}_{k},\hat{a}_{k})$ using $p(f\mid\mathcal{D}_{\text{env}})$
- •
Sample an optimistic reward $\hat{r}_{k}\sim p(r_{k}\mid\hat{s}_{k},\hat{a}_{k},r_{k}>r_{\text{min}})$
- •
Sample a plausible next state $\hat{s}_{k+1}\sim p(s_{k+1}\mid\hat{s}_{k},\hat{a}_{k},r_{k}=\hat{r}_{k})$

### A.6 HOT-GP

In HOT-GP, we sample an optimistic reward and select the most likely next state associated with the sampled reward on each step of the model-generated rollout, as described in equation 9. The steps are detailed in Algorithm 1 with the following routine replacing line 10:

- •
Compute predictive posterior distribution $p(s_{k+1},r_{k}\mid\hat{s}_{k},\hat{a}_{k})$ using $p(f\mid\mathcal{D}_{\text{env}})$
- •
Sample an optimistic reward $\hat{r}_{k}\sim p(r_{k}\mid\hat{s}_{k},\hat{a}_{k},r_{k}>r_{\text{min}})$
- •
Select expected next state $\hat{s}_{k+1}=\mathbb{E}_{p(s_{k+1}\mid\hat{s}_{k},\hat{a}_{k},r_{k}=\hat{r}_{
k})}[s_{k+1}]$

## Appendix B Training

Learning Framework. We implement our learning framework with MBRL-Lib (Pineda et al., ) for the MuJoCo tasks and TorchRL (Bou et al., ) for the coverage task. Additionally, we use GPyTorch (Gardner et al., ) to build the GP reward-dynamics model.

GP Training. Our approach requires access to the full posterior distribution of the Gaussian process. To facilitate this in an efficient manner, we train the GP using a variational bound based on inducing points (Hensman et al., ) following the approach given in Iwata & Ghahramani (). This allows for efficient training using stochastic inference. To further reduce the computational cost of learning the model, we do not train on the whole dataset on each iteration and instead we draw $1,000$ randomly sampled transitions from $\mathcal{D}_{\text{env}}$. This approach reduces the training cost significantly and provides an additional source of stochasticity that helps training. Using a Gaussian likelihood allows us to compute the predictive posterior in closed form, allowing us direct access to its mean and variance.

Hyperparameters. We train all algorithms using a set of hyperparameters that we found to be optimal through trial-and-error or were recommended. Table 1 describes the hyperparameter values specific to the MuJoCo and VMAS tasks we considered. We also report additional algorithm-specific hyperparameters used. In H-UCRL, we use the exploration-exploitation coefficient $\beta=0.01$ and number of samples $Z=5$. In MBPO and optimistic MBPO, we use an ensemble of size 7. For DDPG, we use an exploration noise $\sigma_{\text{explore}}$ from 1 to 0.1. For GP-based approaches, we use the Matern kernel as the covariance function and learn from 100 inducing points. For the MuJoCo tasks, we use a 4-layer MLP with 200 hidden units per layer and SiLU activation function. For the VMAS coverage task we use a 2-layer MLP with 200 hidden units per layer and Mish activation function. We use the Adam optimizer in all cases.

**Table 1: Task-specific hyperparameter values**
| Hyperparameter | Half-Cheetah | Reacher | Pusher | Sparse Reacher | Coverage |
| --- | --- | --- | --- | --- | --- |
| Environment steps $N$ | 250,000 | 20,000 | 20,000 | 37,500 | 200,000 |
| Steps per rollout $T$ | 1000 | 150 | 150 | 150 | 150 |
| Model rollouts $M$ | adaptive | adaptive | adaptive | adaptive | 150 |
| Steps per model rollout $K$ | 1 | 1 | 1 | 1 | 1 |
| Batch size $B$ | 256 | 256 | 256 | 256 | 150 |
| Discount factor $\gamma$ | 0.99 | 0.99 | 0.99 | 0.99 | 0.9 |
| Learning rate $\alpha$ | 0.001 | 0.001 | 0.001 | 0.001 | 0.00005 |
| Mixing factor $\tau$ | 0.005 | 0.005 | 0.005 | 0.005 | 0.005 |
| Replay buffer size | unlimited | unlimited | unlimited | unlimited | 20,000 |

## Appendix C Evaluation Tasks

Figure: Figure 3: Frames from a rollout of a HOT-GP trained policy in the coverage environment. The agent (purple circle) drives around to cover target areas which are brightly colored. The target locations change on each episode.
Refer to caption: extracted/5907130/figures/coverage.png

In this work, we evaluate HOT-GP on five continuous control tasks: four using the MuJoCo physics engine and one using the VMAS simulator. In this section, we outline the setup for each task.

Half-Cheetah. The Half-Cheetah is a MuJoCo continuous control task where a 2D bipedal agent, resembling a simplified cheetah, must learn to run as quickly as possible. The observations are 17-dimensional vectors that include the position and velocity of the agent’s torso and 6 joints. Actions are 6-dimensional vectors, where each component corresponds to the torque applied to one of the agent’s joints. The reward is primarily based on the forward velocity of the agent’s torso while also penalizing excessive torques on the joints. Let $x_{0}$ be the agent’s original position, $x_{1}$ be the position after a simulation step, and $dt$ be the time between. Then the reward is given by $r(s,a)=\frac{x_{1}-x_{0}}{dt}-0.1\sum_{i=1}^{6}a_{i}^{2}$.

Pusher. The Pusher task in MuJoCo is a continuous control task where a robotic arm must learn to push an object to a target location on a 2D plane. The agent controls a three-link planar arm and must manipulate an object to move it towards a fixed goal position. The observation space is a 23-dimensional vector, which includes the position and velocity of the robot arm’s joints, position and velocity of the object being pushed, and the position of the target. The action space is a 7-dimensional vector, where each component represents the torque applied to one of the joints in the robotic arm. The reward function is based on the Euclidean distance between the object and the target location, incentivizing the agent to move the object closer to the target while also penalizing large torques. Let $p_{\text{obj}}$ be the position of the object, $p_{\text{goal}}$ be the position of the goal, $p_{\text{arm}}$ be the position of the arm’s tip. Then the reward is given by $r(s,a)=-\|p_{\text{obj}}-p_{\text{goal}}\|_{2}+0.1\cdot\left(-\sum_{i=1}^{7}a_
{i}^{2}\right)+0.5\cdot\left(-\|p_{\text{obj}}-p_{\text{arm}}\|_{2}\right)$.

Reacher. The Reacher task in MuJoCo is a continuous control environment where a robotic arm must learn to position its end effector at a goal location in 3D space. The observations are 19-dimensional vectors that include the joint angles and velocities of the robotic arm. Actions are 6-dimensional vectors, where each component corresponds to the torque applied at one of the arm’s joints. The reward incentivizes the agent to reach the goal while also penalizing excessive control actions. Letting $EE_{\text{pos}}(s)$ represent the position of the end effector and $\rho$ be the action penalty, the reward function is $r(s,a)=-\sum_{i=1}^{3}(EE_{\text{pos}}(s)_{i}-\text{goal}_{i})^{2}-\rho\cdot\|
a\|^{2}$.

Sparse Reacher. The Sparse Reacher is the same as the Reacher task, except with a modified reward structure.
Let $EE(s)$ represent the position of the end effector, $\epsilon$ be a small threshold distance, and $\rho$ be the action penalty. The reward function is

$$ $r(s,a)=-\rho\left(\exp\left(-\sum_{i=1}^{6}a_{i}^{2}\right)-1\right)+\begin{ cases}\exp\left(-\left\|\text{EE}(s)-\text{goal}\right\|^{2}\right)&\text{if } \left\|\text{EE}(s)-\text{goal}\right\|<\epsilon\\ 0&\text{otherwise}\end{cases}.$ $$

The first term rewards the agent based on the negative squared distance between the end effector’s position and the target goal if the distance is less than $\epsilon$ and gives no reward otherwise. The second term penalizes the agent for applying excessive torques to the joints. This design creates a challenge for exploration as the agent receives minimal feedback about its performance until it is sufficiently close to the target. We take inspiration for this setup from Curi et al. () although our rewards are more sparse and we use $\epsilon=0.2$ in our experiments.

Coverage. In this VMAS task, the agent must discover and visit unique, discretized cells with high value in a continuous state-action space. The reward for visiting a new cell at coordinates $(x,y)$ is distributed as the probability density function of a mixture of three Gaussians over the $x$ and $y$ axes,

$$ $p(x,y)=\frac{1}{3}\sum_{i=1}^{3}\mathcal{N}((x,y)\mid(\mu_{i},\Sigma)).$ (10) $$

To incentivize thorough coverage of the regions around each Gaussian center, the agent receives no reward for revisiting cells in an episode.
The agent observes its position $(x,y)$, velocity, current value $p(x,y)$, values of neighboring cells $p(x_{n},y_{n})$ for $(x_{n},y_{n})\in\mathcal{N}(x,y)$, and the target locations $\{\mu_{i}\}_{i=1}^{3}$. The target locations are randomly generated in each episode so that the agent must learn the high-level pattern of reward distribution as a function of $(x,y)$ and $\{\mu_{i}\}_{i=1}^{3}$, and $\Sigma$. Throughout our experiments, we use $\Sigma=0.05$. We provide snapshots from sample rollouts on the coverage task in Figure 3.

## Appendix D Extended Experiments

In this section, we extend our experiments by evaluating H-UCRL approx and MBPO without uncertainty clamping on the Half-Cheetah task, with the results given in Figure 4.

Figure: Figure 4: Learning curves for select methods on Half-Cheetah averaged over 5 seeds. The dashed line denotes SAC performance at convergence within 3,000,000 environment steps for Half-Cheetah.
Refer to caption: extracted/5907130/figures/halfcheetah_extra.png

As described in Section A, H-UCRL approx differs from the algorithm presented in Curi et al. () in a few ways: (1) we use our GP model rather than a probabilistic ensemble, (2) we use the sampling-based optimistic hallucination method in equation 6 to select states, and (3) we use predicted rewards to guide optimism rather than the ground truth rewards. This version of H-UCRL does not perform well on Half-Cheetah relative to MBPO and HOT-GP, with the best seeds achieving mean episode rewards as high as 6000 after training with 250,000 environment steps. As we observed when discussing H-UCRL performance on the coverage task, this performance gap may stem from infeasible optimism, where the rewards associated with perturbed states $\mu(s,a)+\beta\ \Sigma(s,a)\eta$ are high but unreachable in practice. Additionally, a suboptimal choice of $\beta$ could be a contributing factor, as we did not fine tune this parameter on Half-Cheetah due to its extended runtime and instead used the optimal $\beta$ finetuned on the coverage task. HOT-GP also has a hyperparameter $r_{\text{min}}$, which controls the level of optimism during training. While the domain dependence of $r_{\text{min}}$ means some adaptation is required for new tasks, we found that setting $r_{\text{min}}$ is generally intuitive and, moreover, provides interesting insights into the nature of the task itself. For instance, if the dynamics in the domain are simple, we can be more optimistic about exploration (i.e., use a higher value of $r_{\text{min}}$), whereas if they are complex, we need to spend more time learning the dynamics before becoming more optimistic.

We also experimented with removing uncertainty clamping from MBPO, which resulted in a complete failure to learn as shown in Figure 4. Standard MBPO uses an ensemble of probabilistic neural networks that predicts both the mean and log variance for a given input, with log variances clamped to prevent excessive growth. Removing this clamping significantly degrades performance, indicating that the model uncertainties are poorly calibrated and heavily dependent on clamping to maintain stable learning. This finding reinforces our choice of model, as GPs provide well-calibrated uncertainties (Chowdhury & Gopalan, ).

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 8
presentation: 3
questions: Could you provide any results on computational cost or computation speed for the proposed method?
soundness: 2
strengths: - principled approach for exploration in continuous state-action space
- the first method to predict joint uncertainty over next state and reward
summary: This paper proposed Hallucination-based Optimistic Thompson Sampling with Gaussian Processes (HOT-GP),  a method for principled optimistic exploration. It uses a GP with Thompson Sampling as the acquisition function to maintain a joint belief over state and reward distributions, enabling simulating optimistic transitions wrt the estimated reward. Experiments on MuJoCo and VMAS continuous control tasks demonstrate that the proposed approach matches or outperforms baselines, showing substantial improvements in challenging exploration settings. The paper also investigates factors influencing optimism through ablation studies, highlighting the importance of joint uncertainty modelling in guiding effective exploration.
weaknesses: - The description for Fig.2 in Section 6.2, references several methods that are not shown in the plots. For instance,
    - SAC and PPO as described in Line 430, “we use … SAC, and the on-policy PPO”
    - HOT-GP with k-branching (k=5) as described in Line 470.
- Learning curves seem to be from the training episodes. It would be better to report the performance in evaluation episodes instead, as done in SAC.
- The method introduces a crucial hyperparameter reward threshold r_{min}, which controls optimism during exploration. While the results do not seem to be overly sensitive to it, its effect in other tasks remains unclear. Although future work is mentioned to address this, its current implementation could limit its applicability.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why do the authors aim to relieve the stationarity assumption in line 285. Since this GP models the transition dynamics of an MDP, the transition kernel should indeed be stationary. Aiming to address non-stationary MDPs, I believe, would entail significant complications for the entire learning algorithm.

2. In most experiments, the asymptotic performance of the shown algorithm seems to be lower than the base-SAC implementation in all of the environments. Is this due to the cutoff in plotting or is the asymptotic performance indeed lower in most environments? Inserting a dashed line as with SAC into the figure might clear this up. 

3. Given that the authors perform k-branching rollouts, the value estimates bootstrapped at the end of model-generated rollouts should play an important role in quantifying uncertainty. Would omitting this uncertainty skew the ability of the algorithm to explore efficiently? It seems unlikely to me, that Thompson sampling maintains it's theoretical guarantees if one only samples single-step transitions around known trajectories. 

My suggestion would be to analyze the implications of the proposed algorithmic components carefully. 
1. What is the implication of performing Thompson sampling with a sampling probability dependent on the expected return of some policy?
2. What is the implication of performing Thompson sampling with a sampling probability dependent on the return and reward distribution of some policy (as suggested here).
3. What is the implication of performing Thompson sampling with limited branched rollouts?

Addressing these questions in my opinion would provide a compelling motivation for the proposed algorithm.
soundness: 2
strengths: The paper is written clearly and it is easy to follow. The method adresses an important issue and from what I can see is applicable to a variety of algorithms. The experiments suggest that the proposed algorithm may aid sample efficiency, especially in early stages of training.
summary: This paper introduces a novel model-based reinforcement learning algorithm to explore effectively and improve sample efficiency. The introduced method uses a Gaussian processes to represent a posterior over transition functions and reward functions and models the (linearized) covariance of reward and successor states (the mean functions are later modeled by MLPs instead). The authors introduce an exploration method by modifying Thomposon sampling to be more optimistic: rewards are sampled with the condition of a minimum reward and successive states are taken to be the expected successor state, conditional on the same minimum reward. The authors conducted experiments that analyze the introduced algorithms empirically and show that their agents learns faster in early stages of training.
weaknesses: The main weakness of this paper to me is of conceptual nature. I have no good intuition for why the proposed algorithm performs efficient exploration. This is mainly due to two reasons:

1. I am not convinced the reward-uncertainty is significantly more useful than the dynamics uncertainty. As the authors describe, the de-facto standard way of formulating GPs is that output dimensions are considered independent predictions. The uncertainty of each dimension (the variance of the GP) of a GP conditioned on $X$ in a query-point $x$ is then given by $K(x,x) - K(x,X)K^-1(X,X)K(X,x)$ and depends only on the kernel choice and the data distribution. Unless one has specific knowledge about an appropriate reward-kernel, I have no intution  why the reward offers a better (or different) information gain signal than the dynamics.

2. It is unclear to me why the proposed sampling distribution is meaningful. The sampling distribution described in Eq. 8 seems to imply a risk-sensitive sampling strategy without motivating this. For example, one may easily construct MDPs, where optimal trajectories are guaranteed to contain a very small reward. Such trajectories could not be sampled under this strategy, despite being optimal. Mathematically speaking, it is unclear to me what the implications of conditioning on an impossible event is here (i.e., what if $r_t$ can't be greater than $r_{min}$). I can unfortunately not see why one would expect this algorithm to explore reliably.


Minor points: 

- The dimensions of the GP definition in line 269 don't seem to add up to me. The covariance matrix, as written here, has dimensions $2 \times 2$, while the mean functions are of dimension $|\mathcal{S}| + 1$. I moreover think the input dimensions of covariance functions $K$ defined in line 271 should operate on pairs of $S\times A$, not simply $S \times A$.

- Expected successor states (as described in 5.2, line 323) don't seem sensible to me. The expectation of a high-dimensional state is not necessarily a meaningful or even existant state. It was moreover not clear to me, how to the model predictions are conditioned on a certain reward level in practice.

- Given that the authors use a GP, I suspect many readers would appreciate a table (or similar) comparing the computational burden of this approach with alternative methods. Furthermore I would be interested to know the effect of subsampling smaller batches (as described in line 850) on the quality of uncertainty estimates with a GP.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: None
soundness: 4
strengths: 1) The use of the joint distribution of dynamics and rewards in the learning process is a clever and impactful approach, offering new perspectives in model-based reinforcement learning.
2) The implementation is robust and well-executed, showcasing attention to both theoretical and practical details.
3) The manuscript is excellently presented—clear, concise, and engaging. It's a pleasure to read and straightforward to follow.
4) The experiments are adequately designed, with thoughtful discussions that provide meaningful insights into the method's strengths and limitations.
5) Providing the code for replicating experiments is greatly appreciated, allowing for easy validation and further exploration by the community.
6) The appendix is comprehensive and well-detailed, giving me confidence that I could implement the method from scratch!
summary: This manuscript introduces HOT-GP, a novel and efficient approach to model-based reinforcement learning that enhances performance through a unique combination of techniques. The core innovation of HOT-GP lies in its ability to learn the joint distribution of dynamics and rewards, paired with an optimistic sampling strategy for "in-model" policy optimization. By leveraging the latest model-based reinforcement learning techniques, HOT-GP selects models for policy optimization based on optimistic sampling, which promotes exploration in high-potential regions of the policy space. Evaluated across several continuous-action benchmarks, HOT-GP demonstrates superior performance compared to both baseline methods and other state-of-the-art algorithms.
weaknesses: 1) The evaluation would benefit from more complex scenarios, such as object manipulation or long-horizon tasks, which could further demonstrate the method’s robustness and adaptability.

2) While the integration of a neural network to learn the GP mean is discussed in the main text, I feel like a small paragraph or additional description in the appendix is required. I understand the general approach and can infer the authors' method, but a concise explanation would be a helpful addition for readers.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: According to the paper, the HOT-GP algorithm is inspired by H-UCRL. Why, then, was H-UCRL not used as a comparative algorithm in the MuJoCo tasks?
soundness: 2
strengths: The HOT-GP proposed in the paper performs joint uncertainty modeling between rewards and state dynamics. This method innovatively combines the uncertainties in rewards and state transitions, achieving more efficient exploration in environments with sparse rewards and high penalties.
summary: The paper proposes a model-based reinforcement learning algorithm called HOT-GP (Hallucination-based Optimistic Thompson sampling with Gaussian Processes). This approach leverages Gaussian Processes to jointly model the reward and transition dynamics, addressing both state and reward uncertainties. The main goal is to optimize exploration by focusing on high-reward regions in the state-action space. HOT-GP introduces a new optimism-driven Thompson sampling strategy to balance exploration and exploitation, which is particularly useful in environments with sparse rewards or penalties for actions.
weaknesses: 1. The experiments are insufficient. For environments with sparse rewards and difficult exploration, tasks like Maze and Navigation, such as Sparse-Point-Robot, might better illustrate the effectiveness of the approach.
2. In some environments, the training curves presented in the paper do not appear to have converged, leaving the convergence of the algorithm unclear and making it difficult to judge whether an optimal policy has been achieved.
3. The paper seems to lack ablation studies on different modules and tasks, which fails to adequately demonstrate the independent contributions of various components to overall performance. Without detailed ablation studies, it becomes challenging to evaluate the individual effects of joint uncertainty modeling and the optimistic strategy across different tasks.

Meta-review: The paper proposes HOT-GP, a model-based Thompson sampling-based RL method that takes into account the *joint* uncertainty in dynamics and reward, as modeled by a joint Gaussian Process, when making exploration decisions. The empirical evaluation shows it to be generally more sample-efficient than several baselines, including vanilla SAC, on MuJoCo and maze tasks.

Strengths:
- Novelty of the sampling distribution for exploration
- Good practical implementation
- Clarity

Weaknesses:
- Not entirely clear why it works better than other exploration types, since the uncertainty is estimated over instantaneous reward, not cumulative return
- The empirical evaluation isn't very strong, done on simple benchmarks, although it has definitely improved after the discussion with the reviewers

The metareviewer recommends this work for acceptance due to the novelty of the method, the empirical results showing its promise, and further analysis/discussion it can spark in the community. The metareviewer partly agrees with reviewer *QgWe* that it's not obvious why exploring based on uncertainty in both reward and dynamics gives better results than based on uncertainty in the dynamics alone. One possible explanation is that this simply allows RL to have a more complete picture of uncertainty and hence resolve the uncertainty about the model more quickly, more even if the uncertainty is over instantaneous reward rather than cumulative return. In any case, the metareviewer doesn't view the lack of complete clarity in this aspect as a blocker for publication, but rather as something that will give the research community food for further thought.

Decision: Accept (Poster)