Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
Authors: Tim Seyde
Wilko Schwarting
Sertac Karaman
Daniela Rus
Venue: CoRL2021 Poster (2021)
Keywords: Learning Control
Sample Efficiency
Exploration

Abstract: Learning complex robot behaviors through interaction requires structured exploration. Planning should target interactions with the potential to optimize long-term performance, while only reducing uncertainty where conducive to this objective.  This paper presents Latent Optimistic Value Exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards.  We combine latent world models with value function estimation to predict infinite-horizon returns and recover associated uncertainty via ensembling. The policy is then trained on an upper confidence bound (UCB) objective to identify and select the interactions most promising to improve long-term performance. We apply LOVE to visual robot control tasks in continuous action spaces and demonstrate on average more than 20% improved sample efficiency in comparison to state-of-the-art and other exploration objectives. In sparse and hard to explore environments we achieve an average improvement of over 30%.

Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
Tim Seyde* MIT CSAIL
Wilko Schwarting MIT CSAIL
Sertac Karaman MIT LIDS
Daniela Rus MIT CSAIL
Abstract: Learning complex robot behaviors through interaction requires struc- tured exploration. Planning should target interactions with the potential to optimize long-term performance, while only reducing uncertainty where conducive to this objective. This paper presents Latent Optimistic Value Exploration (LOVE),, a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards. We combine latent world models with value function esti- mation to predict infinite-horizor returns and recover associated uncertainty via ensembling. The policy is then trained on an upper confidence bound (UCB) objec- tive to identify and select the interactions most promising to improve long-term performance. We apply LOVE to visual robot control tasks continuous action spaces and demonstrate average more than 20% improved sample efficiency in comparison to state-of afettrrrrt and other exploration objectives. In sparse and hard to explore environments we achieve average improvement of over 30%.
Keywords: Learning Control, Sample Efficiency, Exploration
Introduction
Wa:
dle)
HDD
IONN ub)
The ability learn complex behaviors through interaction will enable the autonomous deploy- ment of various robotic systems in the real world. Reinforcement learning (RL) provides key framework for realizing these capabilities, but ef- ficiency of the learning process remains a preva- lent concern. Real-life applications yield com- plex planning problems due to nigh-dimensional environment states, which are further exacer- bated by the agent's 's continuous action space. For RL to enable real- sol-w autonomy, there- fore becomes crucial to determine efficient rep- resentations of the underlying planning problem, while formulating interaction strategies capable of exploring these representations efficiently.
volue reluce uncertainty
LOW value ihore uncestalnty
Figure 1: Top Uncertainty over returns can guide exploration to regions with high potential for im- provement (blue dot) during sampling-base based value learning (red line). Bottom uncertainty reduction should focus on expected high-reward behaviors.
In traditional controls, planning problems are commonly formulated based on the underlying state- space representation. This may inhibit efficient learning when the environment states are high- dimensional or their dynamics are susceptible to non-smooth events such as singularities and dis- continuities [1, 2, 3]. It may then be desirable for the agent abstract latent representation that facilitates efficient learning [4, 5, 6). The latent representation may then be leveraged either in model-free or model- based setting. Model-free techniques estimate ftttevvalues directly from observed data to distill policy mapping. Model- -ased techniques learn an explicit representation of the environment that is leveraged in generating fictitious interactions and enable policy learning in imagination [7]. While the former reduces potential sources of bias, the latter offers structured representation encoding deeper insights into underlying environment behavior.
"gqual contribution. Correspondence (tseyde, wilkos] @mit. edu
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

The agent should leverage the chosen representation to efficiently identify and explore informative interactions. For example, an engineer would ideally only provide sparse high-level feedback to robot and the system should learn without getting distracted by environment behavior irrelevant to task completion. This requires models that can disentangle behavioral surprise from expected performance and motivate the agent to investigate the latter. Figure provides a one-dimensional example of a value mapping. The true function and its samples are visualized in red with the true maximum denoted by the green dot. Relying only on the predicted mean can bias policy learning towards local optima (orange dot; Sutton and Barto [8]), while added stochasticity can waste samples on uninformative interactions. Auxiliary information gain objectives integrate predicted uncertainty, however, uncertain environment behavior does not equate to potential for improvement (pink dot). It is desirable to focus exploration on interactions that harbor potential for improving overall performance. Combining mean performance estimates with their uncertainty into an upper confidence hound (UCB) objective provides a concise method for this (blue dot; Auer al. [9], Krause and Ong [10]). The uncertainty can be explicitly represented by an ensemble of hypothesis on environment behavior [11, 12]. Figure demonstrates this selective uncertainty reduction by providing forward predictions of model ensemble on two motion patterns for Walker. The expected high-reward walking behavior has been sufficiently explored and the models strongly agree, while little effort has been extended to reduce uncertainty and learn the specific details of the expected low-reward falling behavior.
This paper demonstrates that exploring interactions through imagined positive futures yield information-dense sampling and data-efficient learning. We present latent optimistic value exploration (LOVE), an algorithm that leverages optimism in the face of uncertain long-tern rewards in guiding exploration. Potential futures are imagined by an ensemble of latent variable models and their predicted aefinite-horizon performance obtained in combination with associated value function estimates. Training on a UCB objective over imagined futures yields a policy that behaves inherently optimistic and focuses the robot learner on interactions with the potential to improve performance under its current world model. This provides a concise, differentiable framework for driving deep exploration while not relying on stochasticity. LOVE therefore provides an agent with
an uncertainty-awar model for diverse hypotheses on trajectory evolution and performance, intrinsic exploration without reward feedback by active querying of uncertain long-term returns,
targeted exploration guided by improvement potential, ignoring uncertainty tangential to the task.
We present empirical results on challenging visual robot control tasks that highlight the necessity for deep exploration in scenarios with sparse rewards, where we achieve on average more than 30% improvement over of- and an exploration baseline. We further demonstrate 15% average improvement on benchmarking environments from the DeepMind Control Suite [13]. We compare to the recent agents Dreamer [7], a curiosity-bame agent inspired by Sekar et al. [14], and DrQ [15].
2 Related work
Problem representation Model-free approaches learn a policy by directly estimating performance from interaction data. While their asymptotic performance previously came the cost of sample complexity [16, 17, 18), recent advances in representation learning through contrastive methods and data augmentation have improved their efficiency [19, 20, 15). However, their implicit representation of the world can make generalization of learned behaviors under changing task specifications difficult. Model- techniques leverage a structured representation of their environment that enables them to imagine potential interactions. The nature of the problem hereby dictates model complexity, ranging from linear [21, 22], over Gaussian process models [23, 24] to deep neural networks [25, 26]. In sigh-dimensional environments, latent variable models can provide concise representations that improve efficiency of the learning process [27, 4, 6, 7].
Planning interactions Model-based approaches leverage their representation of the world in pre- dicting the performance of action sequences. The agent may then either solve a model predictive control-style optimization [28, 25, 29] or train policy in simulation [30, 26]. The resulting finite- horizon formulations can be extended by value function estimates to approximate an infinite-horizon planning problem [31, 7, 32]. When considering learned models, ensembling the model predictions may further be leveraged in debiasing the actor during training [30, 25, 26, 32). Both explicit and implicit model rollouts together with value estimation can accelerate model-fre ffee learning [33, 34, 35].

---

Directed exploration Directed exploration can improve over random exploration by focusing on information-denss interactions [36]. These methods are commonly driven by uncertainty estimates. Information gain techniques define an auxiliary objective that encourages exploration of unexpected environment behavior or model disagreement and have been applied in discrete [37, 38, 39] and continuous actions spaces [40, 41, 42]. Improving knowledge of the dynamics yields general purpose models, while the agent may explore uncertain interactions tangential to specific task objective. Alternatively, exploration can be driven by uncertainty over performance as encoded by value functions [11, 43, 44, 45, 46], multi-step imagined returns [47, 48] or their combination [31, 1, 32].
Model-ensemble agents Related work on ensemble agents has demonstrated impressive results. We note key differences to our approach. ME-TRPO [30] leverages a dynamics ensemble to debias policy optimization on anite-horizon returns under known reward function and random exploration. MAX [49] and Amos et al. [50] explore via finite-horizon uncertainty in a dynamics ensemble. Plan2Explore [14] combines this idea with Dreamer [7], learning a model that enables adaptation to multiple downstream tasks. RPI [51] explores in reward space via finite-horizon returns, but assumes access to the nominal reward function and full proprioceptive feedback. Seyde et al. [32] leverage full proprioception and embed optimism into the value function, which prohibits adjustment of the exploration trade-off during learning and limits transferability. Concurrent work by Rafailov et al. [52] penalizes uncertainty over latent states for learning robust visual control policies via offline RL. Exploring uncertain dynamics samples interactions orthogonal to task completion and finite-horizon objectives limit exploration locally, while full-observabilityy and access to the reward function are strong assumptions. We learn latent dynamics, reward and value functions under partial observability explore uncertain infinite-horizon returns. This enables backpropagation through imagined trajectories to recover analytic policy gradients, as well as guided deep exploration based on expected potential for long-term improvement.
Preliminaries
3.1 Problem formulation
We formulate the underlying optimization problem partially observable Markov decision process (POMDP) defined by the tuple {X, A, 2, O,)) where X, A, denote the state, action and observation space, respectively, T: X' x A X signifies the transition mapping, R: X' A + R the reward mapping, Ω2: X - O the observation mapping, and y E [0, 1) is the discount factor. We define xı and a to be the state and action at time respectively, and (e) denote a policy parameterized by Φ and define the discounted infinite horizon return G, = G=>> y 7- R(x,, a7), where 1 T( | ax) and de ~ (a (oz). The goal is then to learn the optimal policy maximizing G, under unknown nominal dynamics and reward mappings.
the notation r R(x,, az).
Let (T(agoor)
3.2 Planning from pixels
We build on the world model introduced in Hafner et al. [29] and refined in Hafner et al. [7]. High- dimensional image observations first embedded into a low-dimensional latent space using neural network encoder. A recurrent state space model (RSSM) then provides probabilistic transitions and defines the model state Together, the encoder and RSSM define the representation model. The agent therefore abstracts observation Og of environment state 2; into model state 8t, which is leveraged for planning. Consistency of the learned representations is enforeed by minimizing the reconstruction error of decoder network in the observation model and the ability to predict rewards of the reward model. For details, we refer the reader Hafner al. [7], and provide their definitions
Representation model:
Transition model:
go g0(s((s(-1,0)-1) (o.())))
Observation model:
Reward model:
ge(r)(st),
where p and y denote distributions in latent space, with 0 as their joint parameterization The action model (at (s)) is then trained to optimize the predicted return of imagined world model rollouts. The world model is only rolled-out over finite horizon H. but complemented by predictions from
3

---

the value model By (S))) at the terminal state St-H to approximate the infinite horizon return. The policy and value function are trained jointly using policy iteration on the objective functions
L+
++ Σ
Σ VV((+=))
- ( VA((+>))))
max Eigosgo &
min Eigusgo
respectively. Here, (*())) represents an exponentially recency-weighter average of the k-step value estimates V along the trajectory stabilize the learning [8] further specified in Appendix K.
4 Uncertainty-guidee latent exploration
The world model introduced in Section 3.2 can be leveraged in generating fictitious interactions for the policy train on. However, the learned model will exhibit bias in uncertain regions where insufficient samples are available. Training on imagined model rollouts then propagates simulation bias into the policy. Here. we address model bias by ensembling our belief over environment behavior. We can leverage the underlying epistemic uncertainty in formulating a UCB objective for policy learning that focuses exploration on regions with high predicted potential for improvement.
4.1 Model learning with uncertainty estimation
The model parameters are only weakly constrained in regions where interaction data is scarce and random influences strongly impact prediction performance. In order prevent the agent from learning to exploit these model mismatches, we consider predictions from an ensemble. Individual predictions will align in regions of high data support and diverge in regions of low support. The ensemble mean then serves as a debiased estimator of environment behavior, while the epistemic uncertainty is approximated via model disagreement [12]. We consider an ensemble of M latent-space particles. Each particle is represented by unique pairing of a transition, reward and value model to yield,
Particles: ((go, (#+| 8t-1 at-1); 4o; (relse), Vey (s)))
The encoder and decoder remain shared to ensure that all particles leverage the same latent space, while the internal transition dynamics retain the ability expressing distinct hypothesis over environ- ment behavior. For particle i, we define the predicted infinite-horizar trajectory return
Gr, Σ Vx.(s+),,
(4)
where V*r((7) is computed via Eq. (8) with the particle's individual transition, reward and value models. Distinctness of the particles encouraged by varying the initial network weights between ensemble members and shuffling the batch order during training. Predicted trajectory returns with corresponding uncertainty estimates are then obtained by considering the ensemble mean and variance,
M
(G vh +)) =
Σ
(0,(Or,th,
(5)
M (G,
00 10, = M
Σ
mo(0, 1,,00,,,))) 2))²
(6)
1=1
4.2 Policy learning with directed exploration
The policy learning objective in Eq. (2) could be replaced by the ensemble mean in Eq. (6). This would reduce model bias in the policy, but require an auxiliary objective to ensure sufficient exploration. We consider exploration to be desirable when reduces uncertainty over realizable task performance. The trajectory return variance in Eq. (6) encodes uncertainty over long-term performance of actions. In combination with the expected mean, we recover estimated performance bounds. During data acquisition, we explicitly leverage the epistemic uncertainty in identifying interactions with potential for improvement and define the acquisition policy objective via an upper confidence bound (UCB) as
Gag (0, 1b, Φ) = (a(0,, 1h, ②)) + Bag 1/, Φ))

---

where the scalar variable 3 quantifies the -etpptoiioneepooiattitooooo trade-off. For 3 < O we recover safe-interaction objective, while 3 translates an inherent optimism that uncertainty harbors potential for improvement. Here, we learn an optimistic policy Té., that is intrinsically capable of deep exploration and focuses interactions on regions with high information-density. Furthermore, in the absence of dense reward signals, the acquisition policy can leverage prediction uncertainty in driving exploration. This behavior not limited to the preview window, as the value function ensemble projects long-term uncertainty into the finite-horizon model rollouts. While training in imagination, we leverage the optimistic policy to update our belief in regions that the acquisition policy intends to visit. In parallel, we train evaluation policy Te... that aims select the optimal actions under our current belief. The evaluation policy optimizes the expected mean return (3 = 0).
4.3 Latent optimistic value exploration (LOVE)
In the following, we provide a high-level description of the proposed algorithm, LOVE, together with implementation details of the overall training process and the associated pseudo-code in Algorithm 1.
Summary LOVE leverages an ensemble of latent variable models in combination with value function estimates to predict infinite- horizon trajectory performance and associ- ated uncertainty. The acquisition policy is trained on UCB objective to imagine pos- itive futures and focus exploration on interac- tions with high predicted potential for long- term improvement. The ensemble members constrained to operate over the same la- tent space to encourage learning of abstract representations conducive the objective, while ensuring prediction consistency.
Algorithm 1: LOVE
Initialize random parameters (0., thi, Rog: Dev) for episode in episodes do for timestep 1 to T do
at ~ ( St ~ pe, (-1s:-1, a1-1, 0t)
Observe environment transition and add to ②> for trainstep
to S
for particle i = 1 to M do Sample sequence hatch
{ at 1 Compute states 8; ~ po, (se (se|sc-1,, a:-;; ot)
D
Estimate values Vx,+(*-) +- rollout(s;, ())
Representation learning of 0; on
{(re: 0
Regression of 9/4 on (x,) targets in (2)
Implementation The algorithm proceeds in two alternating phases. In the online phase, the agent leverages its acquisition policy to explore interactions optimistically and result- ing transitions are appended to memory D. In the offline phase, the agent first updates its belief about environment behavior and then adjusts its policy accordingly. The represen-
Sample sequences {(o1, az, rt) D)
for particle 1 M do
Compute states 8; ~ pa: (st [s--1 11 01)
Estimate values V(,,(57) + rollout (st, i)
Compute ensemble statistics μcc, og via (6)
Update Dag by optimizing UCB objective in (7)
Updatc Φeu by optimizing mean returns in (5)
tation learning step extends the procedure introduced in Hafner et al. [7] to ensemble learning and groups each model with unique value function estimator into a particle. The batch order is varied between particles during training to ensure variability in the gradient updates and to prevent collapse into a single mode. The policy learning step combines particle predictions to generate value targets in Eq. (4) by simulating ensemble rollouts from the same initial conditions. The trajectory return statistics of Eq. (6) and Eq. (5) are combined into the UCB objective of Eq. (7), which the acquisition policy optimizes, while the evaluation policy optimizes the mean return in Eq. (5).
5 Experiments
We evaluate LOVE on continuous visual control tasks. First, we showcase intrinsic exploration in the absence of reward signals. Then, illustrate interpretable uncertainty estimation of long-term returns with sparse reward signals. Finally, we benchmark performance on tasks from the DecpMind Control Suite [13]. We use single set of hyperparameters throughout, as detailed in Appendix A.
5.1 Intrinsic exploration without reward feedback
In the absence of reward feedback an agent relies on intrinsic motivation to explore. LOVE enables intrinsic exploration by considering uncertainty over infinite-horizon returns. Without informative mean performance estimates, optimism acts as a proxy to guide exploration (see Eq. 7).
5

---

Scerario 1
Scenario
Escapes over Episedes
Velue
Value C
Retums over Foisodes
Epicode
Epappe
Eo
Figure 2: Left the agent starts inside bug trap (red dot) and does not receive reward feedback. We provide occupancy maps for LOVE, LVE (no optimism), and Dreamer with escape counts on seeds. LOVE's ability consider uncertain long-term performance enables consistent exploration in absence reward signals yielding the highest escape rate and area coverage. Right Pendulum initialized at the bottom with sparse rewards at the top. Predicted ensemble mean (left) and standard deviation (right) the value function. The agent first learns to quantify model uncertainty (Ep. 10), and then discovers high reward potential at the top (Ep. 50). Once the agent learned to reach the top, remaining uncertainty with low potential for improvement is not further explored (Ep. 100, circle).
We consider reward-free bug trap enviromment. The agent starts inside the trap and can exit through narrow passage. There is reward feedback to guide planning and escapes need arise implicitly through exploration. The agent is represented as pointmass under continuous actuation actuation-limited acceleration control and observes a top-down view the environment. The ratio of agent diameter to passage width is 0.9 and collisions are modelled as inelastic with the coefficient of restitution set to 0.3. The relative size constrains the set of policies allowing for entry into the passage, while inelastic collisions reduce random motions.
We compare performance of LOVE to both Dreamer and LVE, a variation that ablates on LOVE's optimism (3 = 0). Two variations of the environment are run on random seeds for 100k timesteps. Exemplary occupancy maps are provided in Figure 2 (left). We note that LOVE escapes the bug trap in both scenarios to explore much larger fraction of the state space than either LVE or Dreamer. This can be attributed LOVE's ability to envision uncertain rewards beyond the preview horizon, which optimistically drives deep exploration in the absence of reward signals (row 1). Removing long-term optimism by only guiding interactions through mean performance estimates can lead to prediction collapse and the agent assuming the absence of reward without explicitly querying the environment for confirmation (row 2). We observe similar behavior for the Dreamer agent, which employs single latent model and leverages random noise for exploration (row 3). The total escape count confirms this insight (column 3), where the remaining occupancy maps are provided in Appendix
LOVE
LVE 0.0)
We further highlight the differences in intrinsic exploration between LOVE and LVE on reward-frr maze explo- ration task. The environment follows the same physics as the bug trap and the ratio of agent diameter to corridor width is 0.33. The occupancy maps in Figure 3 exemplify how optimism over returns can set its own exploration goals in domains without reward feedback. These ob- servations further underline the potential of combining model ensembling for policy debiasing with optimism over Siiiite--horizn returns in guiding deep exploration.
ك
Occupancy
Figure 3: Reward-free maze exploration.

---

5.2 Targeted exploration with sparse rewards
Efficient planning under sparse reward feedback requires long-term coordination. LOVE focuses on interactions with high potential for improving unfiiie-horizon performance by trading-off predicted returns with their uncertainty. The agent then only reduces uncertainty when it aligns with successful task completion. We consider a sparse reward pendulum task, its functional representations readily interpretable. The pendulum is always initialized in a downward configuration and only receives reward at the top. The agent can therefore not rely random initialization observe initial reward signals and needs to actively explore the upright configuration.
Figure 2 (right) visualizes the mean and variance of the value ensemble during training. We generate predictions in state-space by providing contextual images for velocity estimation. Initially, the agent builds a world model within which it learns to quantify uncertainty (Ep. 10). The agent then leverages the model to explore the top. This yields a spinning behavior and reward propagates into the value mean, while uncertainty is allocated for trajectories that could yield stabilization at the top (Ep. 50). The agent queries these promising motions and refines its behavior to reach the upright, reducing associated uncertainty (Ep. 100). The UCB objective allows the agent to ignore remaining uncertainty tangential to successful task completion (circle). LOVE therefore guides exploration via potential for long-term improvement while offering interpretable uncertainty estimates.
5.3 Performance with mixed sparse and dense rewards
The previous sections highlighted LOVE's intrinsic motivation to explore potential for long-term im- provement based on its uncertainty estimation over returns. the following, we compare performance on a variety of visual control tasks from the DeepMind Control Suite and sparsified variations.
The environments feature (64, 64) image observations, continuous actions, and dense or sparse rewards. The sparse Cheetah Run, Walker Run and Walker Walk tasks are generated by zeroing rewards below a threshold (0.25, 0.25 and 0.7, respectively; rescaled [O, IJ). We use the same set of parameters throughout all experiments mirroring those of Dreamer. However, we do not use random exploration noise. We furthermore use an ensemble of M 5 latent models, an initial UCB trade-off' of 3 = 0.0, an episodic growth of 8 = 10- and alter the policy learning rate and training steps to account for the more complex UCB objective. The changes to the policy learning rate and training steps are also applied to Dreamer to yield ADreamer, as we found this to improve performance (see Appendix G). We furthermore introduce an exploration baseline, Drreamer+ Curious, inspired by the curiosity bonus in Sekar et al. [14] over an ensemble of RSSM dynamics.
Table 1 provides benchmarking performance on 5 seeds over 300 episodes (see also Appendix F). On the sparsified environments, LOVE achieves 40% average improvement over the state-of-the-art ADreamer agent and 30% over both the exploration baseline A baseline LVE. On the other environments, LOVE improves by 26%, 37%, and 9%. respectively. This suggests that LOVE's combination of latent model ensembling with directed exploration can aid in identifying interactions conducive to task completion. Ensembling reduces propagation of model-specific biases into the policy. Optimistic exploration of uncertain long-term returns focuses sampling on regions with promising performance estimates while ignoring uncertain regions that are tangential to task completion. This is an advantage of formulating the exploration objective in terms of uncertain rewards and not uncertain dynamics, shown by the comparison to ADreamer+Curious
and ensembling
We ablate the performance of LOVE on the UCB trade-off parameter 33 and compare against LVE (3 0). We note that while LOVE outperforms LVE. on the majority of tasks, both reach similar terminal performance in several instances. However, LOVE provides clear improvements on the fully-sparse Cartpole Swingup and the partially-spars. Hopper Hop tasks, as well as the sparse locomotion tasks (Table top). These environments initialize the agent in configurations that provide no reward feedback, thereby forcing the agent to actively explore. LOVE leverages uncertainty -guided exploration and gains an advantage under these conditions. Similarly, this can explain performance on the Pendulum Swingup task. While the task only provides sparse reward feedback, random environment initializations offer sufficient visitation to the non-zero reward states, removing the need for active exploration. LOVE also improves performance on the dense Walker tasks. These environments require learning of stable locomotion patterns under complex dynamics, where directed exploration efficiently identifies tweaks to the gait. Similar to the bug trap environment of Section 5.1,
7

---

300k steps
LOVE
LVE
ADreamer
+Curious
DrQ
A3C
D4PG
Cheetah Sparse (R)
61
2122
606=117
259
514
Walker Sparse (R)
129
164
78:60
106:1220
Walker Sparse (W)
935+24
164
5841315
6394277
Hopper Sparse (S)
831+72
778+142
649
530+
Avg. Ditf. to LOVE
+0%
-11%
.46%
-35%
Cartpole Dense
688
6664 78
4
781:1000
558
862
Cartpole Sparse
631 2339
192=132
388+245
219+199
231
180#6
482+37
Cheetah Run
771
753=120
655=23
760+73
533:148
214
52447
Finger Spin
584
605
348
380
138
129
986
Hopper Hop
203::104
1651196
118:95
87
151:51
24212
Pendulum
641#229
639
320
177::208
3994
49
681++42
Walker Run
528
5064:67
438=85
451:119
338
192
567:19
Walker Walk
9-1436
914+52
902=04
873#122
815+184
311#2
968+2
Avg. Diff. to LOVE
+0%
-12%
-29%
-35%
-18%
-70%
49%
Table 1: DeepMind Control Suite. Mean and standard deviation on 9 seeds after 3 x 109 timesteps. LOVE improves sample efficiency particularly under sparse reward feedback. Temporally-extende optimism helps LOVE in actively exploring uncertain returns, providing an advantage over LVE. Formulating intrinsic motivation in reward-space enables LOVE to identify uncertain interactions conducive to solving the task, providing an advantage over the curiosity baseline Arreamer CCurious. *D4PG, A3C: converged results 10 timesteps for reference.
observe that optimistic exploration is especially favoured by objectives that provide sparse reward feedback, while enabling efficient discovery tweaks to motion patterns under complex dynamics.
We additionally compare to Data Regularized Q (DrQ), a concurrent model-free approach. DrQ updates its actor-critic models online giving an advantage over LOVE and Dreamer. LOVE performs favourably the majority of environments with significant differences the sparse tasks (Pendulum, Cartpole Sparse) and the locomotion tasks (Hopper, Walker). DrQ outperforms LOVE on Finger Spin and Cartpole with dense rewards. On these tasks, learning explicit world model may actually be detrimental to attaining performance quickly. The former task features high-frequency behaviors that may induce aliasing, while the latter task allows the agent leave the frame yielding transitions with no visual feedback. We additionally provide converged performance results for pixel- based D4PG [53] and proprioception based A3C [54] 10 environment steps to put the results into perspective.
6 Conclusion
We propose Latent Optimistic Value Exploration (LOVE), a model-based reinforcement learning algorithm that leverages long-term optimism guide exploration for continuous visual control. LOVE leverages finite-horizon rollouts of a latent model ensemble in combination with value function estimates predict long-term performance of candidate action sequences. The ensemble combined predictions are then combined into upper confidence bound objective for policy learning. Training
on this objective yields policy that optimistically explores interactions that have the potential of improving task performance while ignoring uncertain interactions tangential to task completion. These aspects are particularly important for robot control complex environments, where the engineer aims to provide sparse high-level feedback without intensive reward engineering and the system should efficiently learn without getting distracted by irrelevant environment behavior. To this end, we evaluate LOVE regarding its exploration capabilities and performance on variety of tasks. In the absence of reward signals, LOVE demonstrates an intrinsic motivation to explore interactions based on their information density. Empirical results DeepMind Control Suite tasks showcase LOVE's competitive performance and ability to focus exploration on interactions conducive to task completion, particularly when reward signals are sparse. Lifting intrinsic motivation into reward-space is then preferred over dynamics-space as shown by comparison to a curiosity baseline. Furthermore, LOVE demonstrates improved sample efficiency over the recent model-based Dreamer agent and competitiveness with the model-free DrQ agent.

---

Acknowledgements
This work was supported in part by the Office of Naval Research (ONR) Grant N00014-18 1--1- 2830, Qualcomm and Toyota Research Institute (TRI). This article solely reflects the opinions and conclusions of its authors and not TRI, Toyota, or any other entity. We thank them for their support. The authors further would like to thank Lucas Liebenwein for assistance with cluster deployment, and acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources. We would also like to thank the reviewers and program chairs for their helpful feedback and suggestions for improvement.
References
[1] J. Schritwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. arXiv preprint 1rxiv:9991.0826. 2019.
[2] J. Hwangbo. J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26), 2019.
[3] Y. Yang, K. Caluwaerts, A. Iscen, T. Zhang, J. Tan, and V. Sindhwani. Data efficient reinforce- ment learning for legged robots. arXiv preprint arXiv:1907.0853 2019.
[4] D. Ha and J. Schmidhuber. World models. arXiv preprint arXiv: 1803. 2018.
[5] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: Deep struc- tured representations for model-based reinforcement learning. In International Conference on Machine Learning, pages 7444-7453, 2019.
[6] A. X. Lee, A. Nagabandi, P. Abbeel, and Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv: 190700099, 2019.
[7] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.0160) 2019.
[8] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[9] Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.
[10] A. Krause and C. S. Ong. Contextual gaussian process bandit optimization. In Advances in neural information processing systems, pages 2447-2455, 2011.
[11] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. In Advances neural information processing systems, pages 4026 4034, 2016.
[12] B. Lakshminarayanann A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pages 6402-6413, 2017.
[13] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq. T. Lillicrap, and M. Riedmiller. DeepMind Control Suite. Technical report, DeepMind, 2018.
[14] R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak. Planning to explore via self-supervised world models. arXiv preprini arXiv:2005.0594 005660 2020.
[15] I. Kostrikov, D. Yarats, and Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint MrXiv:2004.13644 2020.
[16] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 00..5509992277 2015.
[17] Fujimoto, Hoof, and D. Meger. Addressing Function Approximation Error in Actor-Critic Methods. In International Conference on Machine Learning, pages 1582-1591, 2018.

---

[18] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1856-1865, 2018.
[19] A. Srinivas, M. Laskin, and Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint X0iv:20040041366 2020.
[20] M. Laskin, K. Lee, A. Stooke, Pinto, Abbeel, and A. Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. 14990, 2020.
[21] Levine and P. Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pages 1071-1079, 2014.
[22] V. Kumar, E. Todorov, and S. Levine. Optimal control with learned local models: Application to dexterous manipulation. In 2016 JEEE International Conference on Robotics and Automation (ICRA), pages 378-383 IEEE, 2016.
[23] M. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-cfficient approach to policy search. In International Conference Machine Learning, pages 465-472, 2011.
[24] S. Kamthh and M. Deisenroth. Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control. In International Conference Artificial Intelligence and Statistics, pages 1701-1710, 2018.
[25] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pages 4754 4765, 2018.
[26] I. Clavera, J. Rothfuss, J. Schulman, Y. Fujita, T. Asfour, and P. Abbeel. Model-based reinforce- ment learning via meta-policy optimization. In Conference Robot Learning, pages 617-629, 2018.
[27] M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pages 2746-2754, 2015.
[28] A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559-7566. IEEE, 2018.
[29] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning Latent Dynamics for Planning from Pixels. In International Conference on Machine Learning, pages 2555-2565, 2019.
[30] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
[31] K. Lowrey, A. Rajeswaran, Efficient Learning and Exploration via Model-Based Control. International Conference on Learning Representations, 2019.
S. Kakade, E. Todorov, and I. Mordatch. Plan Online, Learn Offline:
[32] Seyde, W. Schwarting. S. Karaman, and D. Rus. Learning to plan via deep optimistic value exploration. In Proceedings of the 2nd Annual Conference on Learning for Dynamics and Control, 2020.
[33] J. Oh, S. Singh, and H. Lee. Value prediction network. In Advances in Neural Information Processing Systems, pages 61-6188 2017.
[34] V. Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv: 1803.00101, 2018.
10

---

[35] J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficiet reinforcement learn- ing with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, pages 8224-8234, 2018.
[36] J. Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010) IEEE Transactions on Autonomous Mental Development, 2(3):230 247, 2010.
[37] B. C. Stadie, Levine, and Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint /507.000 2015.
[38] G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos. Count- exploration with neural density models. In International Conference on Machine Learning, pages 2721-2730, 2017.
[39] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. IEEE Conferencee on Computer Vision and Puttern Recognition Workshops, pages 16-17, 2017.
[40] S. Still and D. Precup. An information-theoreiic approach to suriosity-driver reinforcement learning. Theory in Biosciences, 131(3):19-148,, 2012.
[41] R. Houthooft, Chen, Y. Duan, Schulman, F. De Turck, and Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 1109-1117, 2016.
[42] M. Henaff. Explicit Explore-Exploit Algorithms in Continuous State Spaces. In Advances in Neural Information Processing Systems, pages 9372-9382, 2019.
[43] I. Osband, B. Van Roy, D. Russo, and Z. Wen. Deep exploration via randomized value functions. Journal of Machine Leurning Research, 2017.
[44] R. Y. Chen, S. Sidor, P. Abbeel, and Schulman. UCB exploration via Q-ensembles. arXiv preprint 17xvv:1706.0150, 2017.
[45] B. O'Donoghue, I. Osband, R. Munos, and V. Mnih. The Uncertainty Bellman Equation and Exploration. In International Conference on Machine Learning, pages 3836-3845, 2018.
[46] K. Lee, M. Laskin, A. Srinivas, and Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. arXiv preprint 2007v:2007.049388 2020.
[47] S. Depeweg, J.-M. Hermandez-Lobat,, F. Doshi- and S. Udluft. Decomposition of Uncer- tainty in Bayesian Deep Learning for Efficient and disk-sensitive Learning. In International Conference Machine Learning, pages 1184-1193, 2018.
[48] M. Henaff, Y. LeCun, and A. Canziani. Model-predictive policy learning with uncertainty regularization for driving in dense traffic. In 7th International Conference on Learning Repre- sentations, ICLR 2019, 2019.
[49] P. Shyam, W. Jaskowski, and F. Gomez. Model-based active exploration. In International Conference on Machine Learning, pages 5779-5788. PMIR, 2019.
[50] B. Amos, L. Dinh, S. Cabi, Rothörl, S. G. Colmenarejo, A. Muldal, T. Erez, Y. Tassa, N. de Freitas, and M. Denil. Learning awareness models. urXiv preprint arXiv: (0di::w800003111 2018.
[51] Ball, J. Holder, A. Pacchiano, K. Choromanski, and Roberts. Ready policy one: World building through active learning, arXiv preprint arXiv:2002.0 02693, 2020.
[52] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pages 1154-1168. PMLR, 2021.
[53] G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, and T. Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint
2018.

---

[54] V. Mnnh,, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap. T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning, In International conference on machine learning, pages 1928-1937, 2016.
[55] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint MrXiv:1412.6980, 2014.
[56] T. Pearce, F. Leibfried, and A. Brintrup. Uncertainty in neural networks: Approximately bayesian ensembling. In International conference artificial intelligence and statistics, pages 234-244. PMLR, 2020.
[57] V. Campos, A. Trott, C. Xiong, R. Socher, X. Giró-i Nieto, and J. Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
12


Reviews:
[{"review_id": "vnJb0YFpzNH", "reviewer": "Reviewer_KfAh", "summary": "Learn a value function with model based reinforcement learning, augmented with an ensemble of dynamics models which are used to imagine rollouts and provide a variance estimate on the future return prediction. This estimate directs exploration by learning a aggregation policy which acquires rollouts based on the sum of the estimated return, and a beta-regularized variance of the return. This focuses the agent on regions of high return complexity, based on the magnitude of beta. The ensemble of models are trained to reconstruct the (distribution over) current state from observation and last state, action and current observation, predict the next state from last state and action, and predict reward. The ensemble of models is trained by shuffling input batch order, and network weight initialization.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe experimental ablation over beta values provided a much clearer understanding of how much the imagined model rollouts provided benefit as opposed to the exploration method\n\nThe algorithm performs well with relatively simple augmentations to the classic model based reinforcement learning framework.\n\nDreamer is a state of the art baseline in the same area that performs many similar computations, making it a decently good experimental comparison\n\nThe paper describes much of the core concepts clearly, so that it is easy to envision how the system is created. One note is that for how important it is to the core algorithm, how the diversity in the world models is actually injected is somewhat terse.\n\n\nWeaknesses:\n\nThe use of ensemble of agents in reinforcement learning for exploration should probably be better charactized with comparison with: Planning to Explore via Self-Supervised World Models (Sekar et. al.). This is refernced in the paper, but the similarities between this work and the given work appear to be closer than that of the compared baseline, Dreamer. A longer discussion of why this work, which shares much of the same formalism, and same core method (explore where the ensemble disagrees), would be useful, because this work appears to be doing almost the same thing as Sekar et. al. \n\nSince diversity in the model ensembles is only enforced at initialization, this seems like it would have deleterious effects, especially in low-return environments. For example, one can imagine a scenario where the nature of the environment makes it so that the networks are biased towards treating much of the unknown regions with the same output, thus making unseen states provide no intrinsic reward. Perhaps a method that incorporates ensemble diversity explicitly would be more effective in these cases. Additionally, because it does not appear that the forward models capture the variance of the target outputs, this might cause natural stochasticity in the environment to be interpreted as high exploration regions, since the means might as a result vary greatly based on how the values are optimized.\n\nModel ensembles can be highly hyperparameter dependent, especially because network initialization and batch order randomization are the primary modes of injecting variance into the model ensemble. While a single set of hyperparameters seemed to work well in this case, ablations over the network initialization schemes could be useful to determine exactly how sensitive it is.\n\nEven though exploration is supposed to be primarily directed by the variance in the dynamics models, in reality it appears that much of the results are from having an optimistic agent. By comparison, the compared method Dreamer does not appear to have this, using random noise instead. However, optimism should not be difficult to add to a method like Dreamer, and it would be useful to know exactly how much optimism adds, and how much exploration is actually encouraged through the variance reward\n\nIn the navigation task, because the objective, at least when comparing with the baseline performance, is to get good coverage, a comparison with an exploration algorithm which has been shown to get good coverage in these kinds of environments, such as Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (Campos et. al.) would have been preferred, and more informative as the domain is not one that is commonly used.\n\nWhile the deepmind control suite provides some interesting kinematics tasks, experiments on domains which require more exploration, such as a robot arm interaction with multiple object in an environment, or even monetzuma's revenge, might have been more informative. This is because the deepmind control tasks tend to be dense reward (forcing a hacked method for creating reward sparcity), and the exploration generally requires a more dense search of the environment, rather than a directed search towards reward-generating regions of the state space. This would help clarify how much of the benefit in exploration comes from the proposed ensemble of experts.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I propose a weak accept/borderline, primarily because of the similarity with existing work, the limitations of the algorithm in how variance is added and maintained in the model ensemble, and the limitations of the experimental component in properly clarifying the level of improvement based on optimism, and based on the proposed variance based intrinsic reward. An evaluation task that better highlighted this would have been desirable.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Primarily, an increased discussion of how this method significantly differs from Planning to Explore via Self-Supervised World Models (Sekar et. al.)\n\nAdditional experiments ablating on the network initialization which is used for model variance"}}, {"review_id": "I82n4GTOWq8", "reviewer": "Reviewer_5s1s", "summary": "This paper extends latent-space model-based RL methods by incorporating ensembles for the latent model - modeling uncertainty in the latent transition model, reward function, and value function. These ensembles are factored into the overall policy iteration approach by taking an optimistic viewpoint on latent model uncertainty. To compute the policy learning objective, rollouts are performed for each ensemble member and used to estimate the infinite horizon return of the trajectory. These estimates are aggregated to form a mean and standard deviation, which are used to create an upper confidence bound. By training the policy on this upper confidence bound, the policy in inherently incentivized to visit states with high uncertainty if those states hold promise towards the desired task.\n\nThe authors demonstrate the approach on several simulated examples in the DeepMind control suite, comparing against several ablations and baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this paper presents a clean approach to implementing optimism under uncertainty for exploration in latent model-based RL. The idea is clean, and the implementation through ensembles is simple, but effective. The paper is well written and easy to follow. The experiments were thorough, and I appreciated the insightful qualitative results showing how the approach leads to deep exploration in the bug-trap environments, and that the exploration is targeted, through the snapshots of uncertainty in the cart pole experiment. I also appreciated the visualizations of the dynamics model predictions, showing the uncertainty reduction around high-reward behavior while no uncertainty reduction on low-reward behaviors. The quantitative results were also convincing, and I appreciated that the authors included model-free baselines, even if these baselines sometimes performed better than the latent model-based approach.\n\nThe paper could be improved with more details on the implementation of the ensembles, and discussion of hyperparameter optimization. While the policy optimization objective G was clearly defined, it was not clear how the policy was optimized. Furthermore, the success of LOVE depends significantly on the ensemble’s uncertainty being high enough to encourage the model to explore, rather than get stuck in a suboptimal solution. The authors handle this in the paper by increasing the UCB coefficient at every episode in an ad-hoc manner, as opposed to using a calibrated confidence bound as suggested in the original UCB work in the bandits literature. More discussion on the impact of hyperparameters like the growth rate \\delta would strengthen this paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, while the idea in this paper is relatively simple, the paper has strong experimental results and is very well written, leading to my vote for acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The authors should add a discussion of how hyperparameters were optimized and how sensitive the approach is to different hyperparameters such as the UCB coefficient growth rate.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "E6KlZzuiSt", "reviewer": "Reviewer_BJGj", "summary": "This paper presents an exploration mechanism that optimistically explores the state space in search of interactions that increase long-term performance. This is done by designing a UCB objective that maximises the sum of the expected infinite horizon return and a second term that captures the weighted uncertainty of these returns. Setting this weight (\\beta) greater than zero leads to an optimistic objective that explores to reduce uncertainty around high-performant regions and avoids reducing uncertainty around low-reward regions that might not be task relevant. This objective is used together with Dreamer, a popular model-based RL algorithm to explore efficiently in high-dimensional continuous control tasks with pixel observations. The approach builds on top of Dreamer by introducing an ensemble of latent dynamics models, reward and value predictors through which the expected value and its uncertainty are computed as the mean and variance of the ensemble predictions (which are in turn computed via latent imagined rollouts). This exploration objective is used to train a policy for exploration; this policy is executed in the environment and optimistically explores the space to find high-performant transitions which are stored in a replay buffer. These transitions are used to train the dynamics, reward and value models as well as an “exploitation” policy via the standard Dreamer objective that uses imagined value gradients. \n\nSeveral experiments are conducted to test different parts of the approach. First, a control experiment is done to show the exploration capabilities of the proposed agent; this is done in a task-agnostic manner without use of rewards. The agent is able to cover significant parts of the state space in this setting and does better than Dreamer and a baseline that optimise just the expected return for exploration. Next, the method is applied on continuous control tasks with sparse and dense rewards from DMControl (and a toy pendulum example). The method performs just as well or slightly better than baselines including Dreamer (with additional modifications including better hyper parameters and a dynamics-dissimilarity based objective for exploration), DrQ and other model-free methods on the dense reward tasks and beats baselines on sparse reward settings. Several visualisations of the learned exploration objective and ablations to different parameters (including the tradeoff \\beta, imagination horizon and ensemble size) are provided in the supplement.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper proposes an objective for optimistic exploration, implements it on top of Dreamer, a MBRL algorithm, and tests it on several continuous control tasks in simulation. The objective focuses on uncertain returns and pushes the agent to explore high-value regions with high-uncertainty; any uncertainty is seen as a positive, thereby making the approach risk-taking. The method shows good performance on the tasks it is tested on and can lead to improvements in data efficiency compared to model-based and model-free baselines, especially for sparse reward tasks. A few comments:\n1. The idea of using a UCB baseline that leverages uncertainty in values rather than uncertain dynamics is appealing as it naturally focuses exploration on task-relevant regions. There are pros and cons to this: on a given downstream task, task-relevant exploration methods like the one proposed can lead to faster learning, but if the goal is to generate data that works well across tasks then task agnostic exploration methods (such as with uncertain dynamics) might be needed. Additionally, the latter can also generate interesting contact-rich transitions across domains (as such transitions can be hard to predict); it is not clear if the proposed approach also has the capability to do this.\n2. While there is a control experiment showing the capability of this method to operate without any rewards, i.e. in a task-agnostic manner, this intuitively seems to depend heavily on initialisation of the value function as there are no returns for the value function learning to bootstrap from, solely relying on initialisation variance to account for the uncertain value predictions on unvisited states. This is in contrast with methods that use uncertain dynamics or dynamics prediction error. It would be useful if more intuition is provided on the behaviour of this task-agnostic exploration setting and it’s relationship to other exploration methods such as those using dynamics models or approaches such as Random Network Distillation (RND). Additionally, it would be useful to see videos of the task-agnostic exploratory behaviour when applied to the DMControl tasks presented in the paper. This will add further insight.\n3. The proposed approach needs an ensemble of value estimators to compute the exploration objective, and as such can be applied on top of any standard actor-critic approach with either a value or Q-function ensemble. Instead, the proposed implementation uses Dreamer, a model-based method for this value estimation. Is there any reason why this method was chosen as using an actor-critic approach could have made implementation significantly easier and would have had far less moving parts thereby allowing better understanding of the parts that matter.\n4. In a similar vein, the proposed approach already has an ensemble of latent dynamics models. Is it possible to combine the proposed objective with dynamics uncertainty (through some form of weighting for example) to capture complementary behaviour?\n5. A potential weakness of the proposed approach, especially when concerning real world applications, is the optimistic point of view. Risk-sensitive or Risk-aware methods can be a better fit for real world robotics problems as they can be conservative with respect to their action choices. Can the proposed approach be applied on real robots? What issues might arise with such an application and how can they be mitigated? A discussion on this would be enlightening.\n6. While the approach shows good performance on the sparse tasks in DMControl, especially showing improvements in data efficiency early on, it would also have been useful to compare the asymptotic performance of these methods, especially at 500k env steps as has been recently shown in several related papers such as DrQ, CURL, etc. Does the approach lead to similar asymptotic performance compared to baselines? Does the exploration need to be turned off for this to happen? This needs to be discussed further. It would also be helpful to see additional results on harder DMControl tasks and/or tasks from a different benchmark such as OpenAI Gym or even Atari; this would add significantly more strength to the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents an exploration objective based on return uncertainty and applies it to several continuous control tasks showing improvements in data efficiency. The approach is simple and appealing as it encourages exploration in regions where returns can be potentially high, thereby encouraging task relevant exploration. Overall the experiments show good promise but more analysis needs to be done to clearly tease out the strengths and weaknesses of the proposed approach, especially in comparison with other exploration methods. Additionally, it would have been easier for the approach to have been applied on top of a standard actor-critic setup, it is not clear why this was not done. Lastly, the experiments on the continuous control tasks seem half-finished (w.r.t num env steps) and further experimentation, also on other tasks if possible, can provide more strength to the paper. I would consider it a borderline reject as it is currently.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Some comments:\n1. Add details on how the evaluation policy is trained (at least in appendix). I presume this is trained with the standard Dreamer objective.\n2. Is \\beta bounded? If not, why would it not look at uncertain parts later down the line once \\beta becomes sufficiently large? Do you also have to decay it later in the exploration phase to get the agent to do well asymptotically?\n3. Why are results cut off at 300k steps? As mentioned above, prior papers (DrQ, CURL etc.) provide 500k (and Dreamer provides 1e6). How does the asymptotic performance look like? Does it drop in performance later?\n4. On simple point mass task it did well without rewards, what would it do on complex tasks? Some tests in control suite would be quite illuminating — would also be nice to see some videos of exploration behaviour on these tasks.  \n5. Instead of executing the exploration policy only on the actor is it possible to interleave the execution of the exploration and exploitation policy on the actor? Say with a fixed probability sample from the exploration policy and otherwise run the exploitation policy. This may improve data efficiency further.\n6. The thresholds for converting the dense reward to a sparse reward for the DMControl tasks are quite low, were these tuned for LOVE to get good results?\n7. The different particles in the ensemble share the latent transition model, reward and value model. Is there a single decoder that is shared across the particles? Please mention this in the paper.\n8. NIT: The title of the paper says “learning to plan”, but there is no planning (in the traditional sense) in the presented approach. I would prefer if the title were changed to something else that reflects the presented approach better.\n9. NIT: Fig.1, the orange dot is hard to distinguish, please change the color."}}, {"review_id": "BnIaInxsQqX", "reviewer": "Reviewer_Z9mA", "summary": "This paper presents Latent Optimistic Value Exploration (LOVE), a learning method for visual robot control that leverages optimistic exploration. At its core, the method uses an ensemble of latent-space models that share a common encoder but different models for transitions, rewards and values. The different returns predicted by the ensemble members are then used to quantify uncertainty over the long term performance of actions, which in turn is used to define an optimistic learning objective based on UCB that encourages the policy to explore uncertain regions in the latent state space that likely yield good task performance. The method is compared against baselines on a set of scenarios from the DeepMind Control Suite where it shows encouraging results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First, the paper is in general well written an easy to follow. The method mostly builds upon previous work on recurrent state space models (Haffner et al. '19) but from what I can tell, using an ensemble of latent-space models to formulate a UCB-type learning objective is new. I believe that the idea is sound, since using objectives for optimistic exploration that use some form of UCB have been very successful in tangential fields, including Bayesian RL or online POMDP planning.\n\nOnce concern I have regarding the proposed method is the relatively small (5) number of ensemble members. While I suppose that this number was kept small to reduce the number of parameters that have to be trained, it is not clear to me how estimates from such a small number yield accurate estimates for the mean and -- in particular -- the variance in eq.(5) and eq.(6). As a consequence, I'm not too convinced that the policy is actually encouraged to explore regions with uncertain optimistic returns. This also raises the question whether the reported performance benefit of LOVE compared to Dreamer really stems from a more efficient exploration of the environments or if there's something else in the architecture of LOVE that explains the results. While the ablation study in Appendix J suggests that an ensemble member size of 5 has an advantage over a size of 2, I would have liked to see a more extensive study with varying numbers of members (for instance 1 to, say 20). This would not only show how important more accurate uncertainty estimates are (if at all), it would also shed some light on the tradeoff between accuracy and parameter size. \n\n\nOne additional comment: In what sense is LOVE a \"deep exploration\" method?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper provides an interesting extension for on recurrent state space models and I believe the paper could have some impact in case some additional experiments are provided.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "- Please provide a study with a varying number of ensemble members (instead of just 2 and 5) to give the reader an intuition how important accurate uncertainty estimates are.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends latent-space model-based RL methods by using the uncertainty of the latent model for exploration. The uncertainty estimates are obtained using an ensemble method. 

**Pre-rebuttal:** The largest issue with the paper is the low number of seeds. The variance of the results is high and the significance of the results is questionable. Run at least 10 seeds for these and perform a significance test. Only make items bold in the table that are actually significant.

**Post-rebuttal:** 
The proposed method is rather incremental but nevertheless interesting. The reviewers pointed out missing comparisons that were made during the rebuttal. Also, the number of repetitions was increased. There are still a few minor things that need to be updated for the camera-ready version: Make sure the numbers in the abstract match the updated results after all 10 seeds are run. Second, perform a significance test and highlight significant entries in the main table.  

Other details can be found in the reviews. All reviewers suggest acceptance and since many of the issues were fixed, I recommend acceptance of the paper.

Decision: Accept (Poster)