Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
Authors: Tim Seyde
Wilko Schwarting
Sertac Karaman
Daniela Rus
Venue:  (2021)
Keywords: Model-Based Reinforcement Learning
Deep Exploration
Continuous Visual Control
UCB
Latent Space
Ensembling

Abstract: Learning complex behaviors through interaction requires coordinated long-term planning. Random exploration and novelty search lack task-centric guidance and waste effort on non-informative interactions. Instead, decision making should target samples with the potential to optimize performance far into the future, while only reducing uncertainty where conducive to this objective. This paper presents latent optimistic value exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards. We combine finite-horizon rollouts from a latent model with value function estimates to predict infinite-horizon returns and recover associated uncertainty through ensembling. Policy training then proceeds on an upper confidence bound (UCB) objective to identify and select the interactions most promising to improve long-term performance. We apply LOVE to continuous visual control tasks and demonstrate improved sample complexity on a selection of benchmarking tasks. 

LEARNING TO PLAN OPTIMISTICALLY: UNCERTAINTY-GUDDE DEEP EXPLORATION VIA LATENT MODEL ENSEMBLES
Anonymous authors Paper under doublc-blind review
ABSTRACT
Learning complex behaviors through interaction requires coordinated long-term planning. Random exploration and novelty search lack task-centric guidance and waste effort on non-informative interactions. Instead, decision making should target samples with the potential to optimize performance far into the future, while only reducing uncertainty where conducive this objective. This paper presents latent optimistic value exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards. We combine finite- horizon rollouts from a latent model with value function estimates to predict infinite- horizon returns and recover associated uncertainty through ensembling. Policy training then proceeds and select the interactions most promising to improve long-term performance. We apply LOVE to visual control tasks in continuous state-action spaces and demonstrate improved sample complexity on a selection of benchmarking tasks.
upper confidence bound (UCB) objective to identify
INTRODUCTION
The ability to learn complex behaviors through interaction will enable the autonomous deployment of various robotic systems in the real world. Reinforcement learning (RL) provides key framework for realizing these capabilities, but efficiency of the learning process remains a prevalent concern. Real- life applications yield complex planning problems due to high-dimensional environment states, which are further exacerbated by the agent's continuous actions space. For RL to enable real-world autonomy, it therefore becomes crucial to determine efficient representations of the underlying planning problem, while formulating interaction strategies capable of exploring the resulting representation efficiently.
In traditional controls, planning problems are commonly formulated based on the underlying state- space representation. This may inhibit efficient learning when the environment states are high- dimensional their dynamics are susceptible to non-smooth events such singularities and discon- tinuitics (Schrittwieser et al.) 2019: Hwangbo et al.] 2019 Yang et al.) 2019). It may then be desirable for the agent to abstract latent representation that facilitates efficient learning (Ha & Schmidhuber] 2018, Zhang et al.) 2019 Lee et al.) 2019). The latent representation may then be leveraged either model-free model- based setting. Model-free techniques estimate state-values directly from observed data to distill policy mapping. Model- techniques learn an explicit representation of the environment that is leveraged in generating fictitious interactions and enable policy learning in imagination (Hafner et al. 2019a). While the former reduces potential sources of bias, the latter offers structured representation encoding deeper insights into underlying environment behavior.
The agent should leverage the chosen representation to efficiently identify and explore informative interactions. We provide a motivational one-dimensional example of a potential action-value mapping in Figure [ (left). The true function and its samples are visualized in red with the true maximum denoted by the green dot. Relying only on the predicted mean can bias policy learning towards local optima (orange dot; Suttonn& & Barto (2018)), while added stochasticity can waste samples on un-informative interactions. Auxiliary information-gal objectives integrate predicted uncertainty, however, uncertain environment behavior does not equate potential for improvement (pink dot). It is desirable to focus exploration on interactions that harbor potential for improving overall performance. Combining mean performance estimates with the associated uncertainty into upper confidence

---

Under review as a conference paper at ICLR 2021
Ground Trush
Prediction
Ground Iruch
Prediction
p(e) max o(e)
Via) mex p(a) o(4)
5
Samples
Confidence
Tirr estess
Figure I: Left illustrative example of an action-value mapping (red line) and associated samples (red dots). The agent aims to maximize obtained value (green dot) and builds a model through interaction. Exploration based on maximization of the predicted mean can exploit local optima (orange dot), while information-gal bonuses may focus on uncertain regions with little potential of improvement (pink dot). Explicitly considering uncertainty over expected performance can help focus exploration on regions with high potential for improvement (blue dot). Right reducing uncertainty over expected high-reward behaviors (walking, top); ignoring expected low-reward behaviors (falling, bottom).
bound (UCB) objective provides a concise method of doing so (blue dot: Auer et al. (0022) Krause &
Ong (201T)) The underlying uncertainty can be explicitly represented by maintaining ensemble of hypothesis on environment behavior (Osband et af. [016)] Lakshminarayanal et al. (017). Figure
(right) demonstrates this selective uncertainty reduction by showcasing forward predictions of ensembled model on two motion patterns of a walker agent. The expected high-reward walking behavior has been sufficiently explored and model hypotheses strongly agree (top), while little effort has been extended to reduce uncertainty over the expected low-reward falling behavior (bottom).
This paper demonstrates that exploring interactions through imagined positive futures can yield information-dense sampling and data-efficient learning. We present latent optimistic value exploration (LOVE), an algorithm that leverages optimism in the face of uncertain long-term rewards guiding exploration. Potential futures are imagined by ensemble of latent variable models and their predicted infinite-horizon performance obtained in combination with associated value function estimates. Training on UCB objective over imagined futures yields a policy that behaves inherently optimistic and focuses on interactions with the potential to improve performance under the current world model. This provides concise, differentiable framework for driving deep exploration while not relying on stochasticity. We present empirical results on challenging visual control tasks that highlight the necessity for deep exploration in scenarios with sparse reward signals, and demonstrate improved sample-efficiency on a selection of benchmarking environments from the DeepMind Control Suite (Tassa al., 2018). We compare both Dreamer (Hafner al. 2019a), the current state-of-the- model-based agent, and DrQ (Kostrikov et al., 2020), concurrent model- ffeee-free approach.
2 RELATED WORK
Problem representation Model-free approaches learn a policy by directly estimating performance from interaction data. While their asymptotic performance previously came the cost of sample complexity (Lillicrap et al., 2015) Fujimoto et al.) 2018 Haarnoja et al. 2018), recent advances representation learning through contrastive methods and data augmentation have improved their efficiency (Srinivas et al. 2020 Laskin et al., [2020] Kostrikov et al. 2020). However, their implicit representation of the world can make generalization of learned behaviors under changing task specifi- cations difficult. Model- based techniques leverage structured representation of their environment that enables them to imagine potential interactions. The nature of the problem hereby dictates model complexity, ranging from linear (Levine & Abbeel| 2014 Kumar al. 2016), over Gaussian process models (Deisenroth & Rasmussen 2001 Kamthe & Deisenroth] 2018 to deep neural networks (Chua et al. 2018: Clavera al. 2018). In high-dimensional environments, latent variable models can provide concise representations that improve efficiency of the learning process (Watter et al. 2015
Ha & Schmidhuber 2018 Lee et al., 2019 Hafner et al. 2019a).
2

---

Under review as a conference paper at ICLR 2021
Planning interactions Model-based approaches leverage their representation of the world in pre- dicting the performance of action sequences. The agent may then either solve a model predictive control-style optimization (Nagabandi et al., 2018 Chua et al.) 2018: Hafner et al. 2019b) train a policy in simulation (Kurutach et al. 2018; Clavera et al. 2018). The resulting finite-horizon formulations be extended by value function estimates approximate
anfinite-horizon planning problem (owwryy et al.) 2019; Hafner et al., 2019a Seyde et al. (020). When considering learned models, ensembling the model predictions may further be leveraged in debiasing the actor strategy during training (Kurutach et al., 2018 Chua et al. 20118] Clavera et al., 2018 Seyde et al. 2020).
Both explicit and implicit model rollouts in combination with value estimates can also be utilized to accelerate model-free learning (Oh et al. 2017 Feinberg et al. 2018 Buckman et al. 2018))
Directed exploration Directed exploration can improve over random exploration by focusing on information-dense interactions (Schmidhuber, 2010). These methods are commonly driven by uncertainty estimates. Information gain techniques define an auxiliary objective that encourages exploration of unexpected environment behavior model disagreement and have been applied in discrete (Stadie et al., 2015;
Ostrovski et al.) 2017; Pathak et al. 2017) and continuous actions
spaces (Still& & Precup 2012 Houthooft et al. 2016 Henaff (219)) When driving interactions to improve knowledge of the environment dynamics, effort may be wasted on uncertain interactions that tangential to the objective. Alternatively, exploration can be driven by uncertainty over expected performance as encoded by value functions (Osband et al. 2016 2017, Chen et al. 2017]
O'Donoghue et al. 2018 Lee et al. 2020), multi-step imagined returns (Depeweg al. 2018 Henaff ct al. 2019) or their combination (Lowrey et al.) 2019 Schrittwieser et al. 2019 Seyde et al. (2220).
Model-ensemble agents Related work on ensemble agents has demonstrated impressive results. We note key differences to our approach. ME-TRPO (Kurutach et al. 2018) leverages a dynamics ensemble debias policy optimization on finite-horizon returns under a known reward function and random exploration. MAX (Shyam et al. [219)) and Amos et al. (0088) explore via finite-horizon uncertainty in a state and latent dynamics ensemble, respectively. RPI (Ball al., 2020) explores in reward space via finite-horizon returns, but assumes access to the nominal reward function and full proprioceptive feedback. (Seyde et al. (2020) also leverage full proprioception and embed optimism into the value function, which prohibits adjustment of the exploration trade-off during policy learning and limits transferability. Exploring uncertain dynamics samples interactions orthogonal to task completion and to the reward function are strong assumptions. We learn latent dynamics, reward and value functions in partially observable settings explore uncertainty over infinite-horizon returns. This enables backpropagation through imagined trajectories to recover analytic policy gradients, while offering a flexible framework to guide exploration based on expected potential for long-term improvement.
objectives limit exploration locally, while full- full-observability and access
PRELIMINARIES
In the following, we first introduce the general problem definition and the associated nomenclature. We then provide overview of the representation learning framework used abstract environment behavior, which leverages the recurrent state space model (RSSM) proposed by Hafner et al. (2019a).
3.1 PROBLEM FORMULATION
We formulate the underlying optimization problem as a partially observable Markov decision process (POMDP) defined by the tuple {X' A, T, R. Ω, 7, where X', A, O denote the state, action and observation space, respectively, T: X A + X signifies the transition mapping, R: X' x A + R the reward mapping, Ω: X + O the observation mapping, and E [0, 1) is the discount factor. We define Ie and aj to be the state and action at time t, respectively, and use the notation r; = R(x,, a.). Let (aglo)) denote policy parameterized by o and define the discounted infinite horizon return G. = -m - 7 R(z,,, a,), where Φ,41 ~ T( 1 a,) and a, ~ T (a lo.). ). The goal then to lear the optimal policy maximizing G2 under unknown nominal dynamics and reward mappings.
3

---

Under review as a conference paper at ICLR 2021
3.2 PLANNING FROM PIXELS
We build on the world model introduced in Hafner et al. (2019b) and refined in Hafner et al. (2019a). High-dimensional image observations first embedded into a oowdimensional latent space using neural network encoder. An RSSM then serves as probabilistic transition model and defines the model state s. Together, the encoder and RSSM define the representation model. The agent therefore abstracts observation of environment state z into model state 8t, which is leveraged for planning. Consistency of the learned representations is enforced by minimizing the reconstruction error of decoder network in the observation model and the ability to predict rewards of the reward model. For details, we refer the reader to Hafner et al. (2019a), and provide their definitions of the models as
Representation model:
po(st (1-11 go g0(s{)5(-1, (t-1) go(o)(s;)
Transition model:
Observation model:
Reward model:
go(re(se),
where p and q denote distributions latent space, with 9 as their joint parameterization. The action model (a (8t) is then trained to optimize the predicted return of imagined world model rollouts. The world model only rolled-out over a finite horizon H, but complemented by predictions from the value model uy (st) at the terminal state 8 to approximate the infinite horizon return. The policy and value function are trained jointly using policy iteration on the objective functions
max Ego.9e E Vi(s))).
miny Eqa.go EE 1 (ll)) (x()))1)))
respectively, Here, (s())) represents an exponentially recency-weightet average of the k-step value estimates (s+) along the trajectory to stabilize the learning (Sutton & Barto 2018 such that
Vs(++) =1 =1 Σ
x" X"= + -V (s+).
(3)
(*+))) = Equ.gg
Σ
2n rn T
with h min(T k, H).
UNCERTAINTY-GUIDER LATENT EXPLORATION
The world model introduced in Section 3.2 can be leveraged in generating fictitious interactions for the policy to train on. However, the learned model will exhibit bias in uncertain regions where insufficient samples are available. Training imagined model rollouts then propagates simulation bias into the policy. Here, we address model bias by ensembling our belief over environment behavior. We can leverage the underlying epistemic uncertainty in formulating UCB objective for policy learning that focuses exploration on regions with high predicted potential for improvement.
4.1 MODEL LEARNING WITH UNCERTAINTY ESTIMATION
The model parameters are only weakly constrained in regions where interaction data is scarce and random influences have a detrimental effect prediction performance. In order to prevent the agent from learning to exploit these model mismatches, we consider predictions from an ensemble. Individual predictions will align in regions of high data support and diverge in regions of low support. The ensemble mean then serves as a debiased estimator of environment behavior, while the underlying epistemic uncertainty is approximated via model disagreement (Lakssminarayanan al. 2017). We consider an ensemble of M latent-space particles. Each particle is represented by a unique pairing of transition model, reward model and value model to yield,
Particle ensemble:
{go, (sx|s;- ar-1), go, (relse), up, Ugg (s
(4)
The encoder remains shared between the individual representation models. This ensures that all particles operate over the compact latent space, while the internal transition dynamics retain
4

---

Under review as a conference paper at ICLR 2021
the ability of expressing distinct hypothesis over environment behavior. For each particle we then define the predicted unfnite-horizon trajectory return as
H+
Gr th o) Σ Vs,(s,),,
(5)
where (5,) is computed via Eq. (3) with the particle's individual transition, reward and value models. Distinctness of the particles is encouraged by varying the initial network weights between ensemble members and shuffling the batch order during training. Predicted trajectory returns with corresponding uncertainty estimates are then obtained by considering the ensemble mean and variance,
pG(0, 0)
1 M
Σ
Grs vh, 6). 2g ()
M Σ (G,, v/s x) 1o 1/, ())
4.2 POLICY LEARNING WITH DIRECTED EXPLORATION
The policy learning objective in Eq. could be replaced by the ensemble mean in Eq. (6) This would reduce model bias in the policy, but require auxiliary objective to ensure sufficient exploration. We consider exploration to be desirable when it reduces uncertainty over realizable task performance. The trajectory return variance in Eq. (6) encodes uncertainty over long-term performance of actions. In combination with the expected mean, we recover estimated performance bounds. During data acquisition, we explicitly leverage the epistemic uncertainty in identifying interactions with potential for improvement and define the acquisition policy objective via an upper confidence bound (UCB) as (0, th, 6) = (G vi o) + (0, th, o),
(7)
where the scalar variable 3 quantifies the safe-interaction objective, while 3 >> 0 translates an inherent optimism that uncertainty harbors potential for improvement. Here, we learn an optimistic policy Té., that is intrinsically capable of deep exploration and focuses interactions in the absence of dense reward signals, the acquisition policy can leverage prediction uncertainty in driving exploration. This behavior is not limited to within the preview window, the value function ensemble projects long-term uncertainty into the finite-horizon model rollouts. While training in imagination, we leverage the optimistic policy to update our belief in regions that the acquisition policy intends to visit. In parallel, we train an evaluation policy Te.. that aims to select the optimal actions under our current belief. The evaluation policy optimizes the expected mean return (3 0).
trade-off. For 3 < we recover a
regions with high information-densee
Furthermore,
4.3 LATENT OPTIMISTIC VALUE EXPLORATION (LOVE)
In the following, we provide a high-level description of the proposed algorithm. LOVE, together with implementation details of the overall training process and the associated pseudo-code in Algorithm
Summary The proposed algorithm leverages an ensemble of latent variable models in combination with value function estimates predict infinite-horizon trajectory performance and associated uncertainty. The acquisition policy is then trained on UCB objective to imagine positive futures and focus exploration on interactions with high predicted potential for long-term improvement. The ensemble members are constrained to operate over the same latent space to encourage learning of abstract representations conducive to the objective, while ensuring consistency between predictions.
Implementation The algorithm proceeds in two alternating phases. In the online phase, the agent leverages its acquisition policy explore interactions optimistically and resulting transitions are appended to memory D. In the offline phase, the agent first updates its belief about environment behavior and then adjusts its policy accordingly. The representation learning step extends the procedure introduced in |Hafner et al. (2019a) to ensemble learning and groups each model with a unique value function estimator into particle. The batch order is varied between particles during training to ensure variability in the gradient updates and to prevent collapse into a single mode. The policy learning step combines particle predictions to generate the value targets of Eq. (5) by simulating ensemble rollouts from the same initial conditions. The trajectory return statisties of Eq. (6) are combined into the UCB objective of Eq. Z.. which the acquisition policy optimizes. We provide overview in Algorithm I where evaluation policy training has been omitted for brevity.
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Latent optimistic value exploration (LOVE)
Initialize: parameters {0,, thi, Dag. &ev) randomly; memory with 5 random episodes
for episode k +- to K do
Sample representation model to be used online to do Generate 0g ~ Tuo (dg]st) based on 5t Po, (5e1)t111 (c-,, Interact with the environment and add transition to D
for timestep
Online
for trainstep 8 - 1 to $ do
for particle 1 to M do
Sample transition sequence batch {(o), ae, rt) b+L D
Compute model states 8t Po, (t (1)
Model
Compute value estimates (8+=(8+) < rollout (8t,
Update
12
Update 0, via representation learning on ((Te, Oc+1) J Update eb via regression the Vs,)(s>) targets in Eq. (
Offline
Sample transition sequence batch {(or, 0t, b for particle i t- 1 to M do Compute model states 8; ~ Pa, (3:(81-14 (s 1,-1,
14
16
Policy Update
Compute value estimates (8,) +- rollout (8t,
Compute ensemble statistics Pc: dg based on Eq. 6
18
Update Dag via optimization of the UCB objective in Eq. 7
19
EXPERIMENTS
We provide results for training agents using LOVE on selection of visual control tasks. First, we illustrate intrinsic exploration in the absence of reward signals on a classic continuous control task. We then benchmark performance
selection of environments from the DeepMind Control Suite (Tassa et al. 2018). We use a single set of parameter values throughout, as detailed in Appendix
5.1 CASE STUDY: EXPLORATION IN THE ABSENCE OF REWARDS
We consider a planar bug trap environment in which the agent starts inside an enclosure and needs exit through narrow passage. The environment does not provide any reward signals to guide planning and an escape needs to arise implicitly through exploration. The agent is represented as pointmass under continuous aetuation-limites acceleration control. The ratio agent diameter to passage width is 0.9 and collisions modelled as inelastic with the coefficient of restitution set to 0.3. The relative size strictly limits the range of policies allowing for entry into the passage, while the low coefficient of restitution encourages directed exploration reducing the presence of random motion patterns. The agent always continues from the terminal configuration of the previous episode.
We consider variations of the same environment distinct in the location of the passage and the relative initial position of the agent. The two initial configurations are provided in the first column of Figure The agent observes top-down view and interacts with the environment for 100 episodes. We compare performance of LOVE with both Dreamer and LVE, a variation of LOVE that does not leverage optimism (3 0). The resulting occupancy maps are provided the remaining panels of Figure 2 We note that LOVE escapes the bug trap in both scenarios to explore a much larger fraction the state space than either LVE Dreamer. This can be attributed to LOVE's ability to explicitly consider reward uncertainty beyond the preview horizon, which optimistically drives deep exploration in the absence of reward signals (column 2). Removing this long-term optimism by only guiding interactions through mean performance estimates can lead to prediction collapse and the agent assuming the absence of reward without explicitly querying the environment for confirmation (column 3). We observe similar behavior for the Dreamer agent, which employs single latent model and leverages random noise for exploration (column 4). Repeating both scenarios on random seeds confirms this trend (column 5), where the associated occupancy maps are provided in Appendix

---

Under review as a conference paper at ICLR 2021
Initial
LOVE
LVE (8
Dreamer
Escapes Episodes
LVE (8-00
Dreamer
Scenario
Scenarto
70
Bug trap environment. The agent starts inside the bug trap and explores for 100 episodes in the absence of reward signals. Top row, left right: initial configuration and occupancy traces for agents trained using LOVE (B > 0), LOVE without deep optimism (3 0), and Dreamer. Bottom row: results for a variation of the environment. LOVE's ability consider uncertain long-term performance in driving exploration enables the agent to escape in both scenarios. We consider results on random seeds and highlight the number escapes over episodes in the right panel. LOVE provides the highest escape rate and the best area coverage (see Appendix for occupancy maps).
Figure 2:
5.2 BENCHMARKING PERFORMANCE ON DEEPMIND CONTROL SUITE
The previous section highlighted the ability of LOVE to guide exploration even in the absence of proper reward feedback. In the following, we benchmark performance on a variety of visual control tasks from the DeepMind Control Suite. Each environment features (64, 64) RGB image observations and a continuous action space, while reward signals vary from dense to sparse. Each episode initializes the agent in a random configuration and runs for 1000 environment steps with an action repeat of 2. We use the same set of parameters throughout all our experiments, mirroring the base parameters of Dreamer to ensure fair comparison. However, noise and set the corresponding magnitude to O. We furthermore use an ensemble of M latent variable models, an initial UCB trade-off parameter of 0.0, a per episode UCB growth rate of 5 0.001, and alter the policy learning rate and training steps to account for the more complex UCB objective, requiring propagation of both updated mean and uncertainty estimates into the policy, The changes to the policy learning rate and training steps are also applied to the Dreamer agent to yield Drremmer, as we found this modification to generally affect performance favourably (see Appendix D}. We furthermore introduce exploration baseline, ADreamer Curious, inspired by the curiosity bonus in Sekar et al. (2020) that combines Arreamer with an ensemble of RSSM dynamics.
do not use random exploration
Figure [ results for benchmarking performance over 300 episodes. Performance is evaluated on 5 seeds, where solid lines indicate the mean and shaded areas correspond to one standard deviation. The numeric data is provided in Table We that LOVE improves performance over both ADreamer and DDreamer+ Curious on the majority of experiments. This suggests that LOVE's combination of latent model ensembling with directed exploration can aid in identifying interactions conducive to solving the task. Ensembling reduces propagation of model-specific biases into the policy, Optimistic exploration of uncertain long-term returns focuses sampling on regions with promising performance estimates while ignoring uncertain regions that are tangential to task completion.
We ablate the performance of LOVE on the UCB trade-off parameter 3 and compare against LVE ( = 0). We note that while LOVE outperforms LVE on the majority of tasks, both reach similar terminal performance several instances. However, LOVE provides clear improvements on the fully- sparse Cartpole Swingup and the partially-sparse Hopper Hop tasks. Both environments initialize the agent in configurations that provide no reward feedback, thereby forcing the agent to actively explore. exploration and gains an advantage under these conditions.
LOVE leverages uncertainty-guiin Similarly, this can explain the performance parity on the Pendulum Swingup task. While the task only provides sparse reward feedback, random environment initializations offer sufficient visitation

---

Under review as a conference paper at ICLR 2021
Cartpole Swingup
Cartpole Swingup Sparse
Cheetah Run
Finger Spin
1000
1000
1000
1000
Episode Return
750
500
250
250
250
250
0
0-
0
Hopper Hop
Pendulum Swingup
Walker Run
Walker Walk
1000
600
1000
Episode Return
200
500
500
100
200
250
250
0
0-
0
2
0
Timesteps
1e5
Timesteps
1e5
Timesteps
1e5
Timesteps
1e5
LOVE (Ours)
LVE (Ours)
ADreamer
ADreamer+ Curious
DrQ D4PG (pixel)
A3C (praprio)
Figure 3: DeepMind Control Suite. We evaluate performance over the first 300 episodes on seeds. Solid lines indicate mean performance and shaded areas indicate one standard deviation. We note that LOVE's ss method of deep optimistic exploration can help in identifying interactions conducive to solving the task and in turn may improve sample efficiency. especially for sparse reward feedback. *D4PG, A3C: converged results at 10 environment steps provided for reference.
to the non-zero reward states and therefore remove the necessity for active exploration. LOVE also improves performance on the Walker Walk and Walker Run tasks. These environments require learning of stable locomotion patterns under complex system dynamics, where directed exploration helps to efficiently identify tweaks to the gait, Similar to the bug trap environment of Section 5.] we observe that optimistic exploration is especially favoured by objectives that provide sparse reward feedback, while enabling efficient discovery of tweaks to motion patterns under complex dynamics.
We additionally compare to Data Regularized Q (DrQ). a concurrent model-free approach. DrQ updates its actor-critic models online giving it an advantage over both LOVE and Dreamer. We note that LOVE performs favourably on the majority of environments with significant differences on the sparse tasks (Pendulum, Cartpole Sparse) and the locomotion tasks (Hopper, Walker). DrQ outperforms LOVE on Finger Spin and Cartpole with dense rewards. On these tasks, learning an explicit world model may actually be detrimental to attaining performance quickly. The former task features high-frequency behaviors that may induce aliasing, while the latter task allows the agent to leave the frame yielding transitions with visual feedback. We additionally provide converged performance results for pixel-based D4PG BBarh-Maaron et al.) 2018 and proprioception-ben based A3C (Mnih et al. 2016) 10 environment steps put the results into perspective.
5.3 BENCHMARKING FURTHER EXPLORING SPARSE REWARDS
To further analyze exploration in sparse reward settings, we introduce five additional tasks. The Cheetah Run, Walker Run and Walker Walk tasks are modified with a reward threshold below which rewards are zeroed (0.25, 0.25 and 0.7, respectively; rescaled to [0. 1]). Whereas termi- nal performance on the dense task versions was similar, sparse reward feedback favors LOVE in Figure (left). Particularly on the modified Cheetah Run and Walker Run tasks a strong separation can be observed. In the maze envi- ronment, a planar pointmass is controlled via continuous forces to navigate maze in the absence of reward feedback. Motions need to arise from an intrinsic motivation to explore and test for this property by letting the agent interact with the
LOVE
LVE (8 0.0)
FCC 2S
Initial Position
Maze Wall
Occupancy (Low-High)
Figure 4: Reward-free maze exploration task.
8

---

Under review as a conference paper at ICLR 2021
Sparse Cheetah Run
Sparse Walker Run
Sparse Walker Walk
Sparse Hopper Stand
1000
300
1000
1000
Episoce Return
750
750
750
200
500-
500
500
100
250
250
250
0
Timesteps
les
Timesteps
2
1e5
0
Timesteps
2
1e5
Timesteps
105
LOVE (Ours)
LVVE (Ours)
ADreamer
4Dreamer+Curiaus
Cheetah Run, Walker Run and Walker Walk provide re-scaled rewards only above a threshold (0.25, 0.25 and 0.7, respectively). LOVE achieves the highest performance, displaying significant improvement over LVE on Cheetah Run and Walker Run.
Figure 5: Additional sparse reward tasks:
300k steps
LOVE
LVE
ADreamer
Curious
DiQ
A3C
D4PG
Caripole Dense
676=100
64477
133=141
550=109
781
558
862=1
Cartpole Sparse
694
245+118
3894282
242=220
231
180
482
Cheetah Run
699=113
620+123
658+82
715+64 +:64
5334
214+2
524-7
Finger Spin
569
440
335
299
898 131
129
986=1
Hopper Hop
223
139
124+
89+61
151
1#a
242=2
Pendulum
635=305
865
3762277
168=217
399
4925
681 1:42
Walker Run
568
511::4
431
426::136
338
192:2
5671:9
Walker Walk
955
927
871+116
830
815+184
311#2
968=2
Cheetah Sparse (R)
815
587:147
224
490=287
Walker Sparse (R)
218:1102
24130
118:57
163
Walker Sparse (W)
918#21
795+161
757
736496
Hopper Sparse (S)
847164
817+121
6332132
492-268
Table 1: DeepMind Control Suite. Mean and standard deviation after 10th timesteps. Best performance bold. Cartpole and Pendulum refer to the Swingup tasks, while (R, W.S) denote Run, Walk, and Stand. Sparse Cheetah and Walker tasks were created by introducing reward threshold.
environment for 100k timesteps. The occupancy maps in Figure B (right) suggest that optimism over returns set its own exploration goals in domains without reward feedback. These observations further underline the potential combining model ensembling for policy debiasing with optimism over fefiite-horizon returns for guided deep exploration.
CONCLUSION
We propose latent optimistic value exploration (LOVE), model-based deep reinforcement learning algorithm that leverages long-term optimism in directing exploration for continuous visual control. LOVE leverages finite-horizon rollouts of a latent model ensemble in combination with value function estimates to predict long-term performance of candidate action sequences. The ensemble predictions then combined into upper confidence bound objective for policy training. Training on this objective yields a policy that optimistically explores interactions that have the potential of improving task performance while ignoring interactions that are uncertain but tangential to task completion. We evaluate LOVE experimentally regarding its exploration capabilities and performance on a variety of tasks. In the absence of reward signals, LOVE demonstrates an intrinsic motivation to explore interactions based on their information density. Empirical results on various tasks from the DeepMind Control Suite demonstrate LOVE's competitive performance and ability to focus exploration on interactions that are conducive to the task. LOVE demonstrates improved sample efficiency over the current tfate-of-the-ar model-bee based Dreamer agent and compared favourably to the concurrent model-free DrQ agent. Future work will consider learning more efficient representations without assigning model capacity to image reconstruction. Application of concurrent methods contrastive learning and data augmentation for reinforcement learning will likely further improve performance.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Brandon Amos, Laurent Dinh, Serkan Cabi, Thomas Rothörl, Sergio Gómez Colmenarejo, Alistair Muldal, Tom Erez, Yuval Tassa, Nando de Freitas, and Misha Denil. Learning awareness models. arXiv preprint arXiv: 1xXi::8844003188 2018.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-255 2002.
Philip Ball, Jack Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts. Ready policy one: World building through active learning. arXiv preprint wrXiv:2002.02693 2020.
Gabriel Sarth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.086117 2018.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, pp. 8224-8234, 2018.
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. UCB exploration via Q- ensembles. arXiv preprint vrXiiv1706.000 502, 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754-4765, 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. In Conference on Robot Learning, pp. 617-629, 2018.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model- and data-efficient approach policy search. In International Conference on Machine Learning, pp. 465-472, 2011.
Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Veles and Steffen Udluft. Decom- position of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning. In International Conference Machine Learning, pp- 1 1184-1192 1193, 2018.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.001011 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In International Conference on Machine Learning, pp. 1582-1591, 2018.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1803 10122200..01021..
2018. 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1856-1865, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXi::1912.01603 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning Latent Dynamics for Planning from Pixels. In International Conference on Machine Learning, pp. 2555-2565, 2019b.
Mikael Henaff. Explicit Explore-Explot Algorithms in Continuous State Spaces. In Advances in Neural Information Processing Systems, pp. 9372-9382, 2019.
Mikael Henaff, Yann LeCun, and Alfredo Canziani. Model-predicte policy learning with uncer- tainty regularization for driving in dense traffic. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
10

---

Under review as a conference paper at ICLR 2021
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1117, 2016.
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robolics, 4(26), 2019.
Sanket Kamthe and Mare Deisenroth. Data-Efficien Reinforcement Learning with Probabilistic Model Predictive Control. In International Conference on Artificial Intelligence and Statistics, pp. 1701-1710, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprini arXiv:1412.6980, 2014.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
Andreas Krause and Cheng S Ong. Contextual gaussian process bandit optimization. In Advances in neural information processing systems, pp. 2447-2455, 2011.
Vikash Kumar, Emanuel Todorov, and Sergey Levine. Optimal control with learned local models: Application to dexterous manipulation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 378-383. IEEE, 2016.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. 14990, 2020.
Alex X Lee, Anusha Nagabandi, Picter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv:1907.0095: 2019.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. arXiv preprint arXiv:2007.0499. 2020.
Sergey Levine and Picter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 1071-1079, 2014.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: /509: 000009999991.... 2015.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan On- line, Learn Offline: Efficient Learning and Exploration via Model-Base Control. In International Conference on Learning Representations, 2019.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynam- ics for bodel-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018
Junhyuk Oh, Satinder Singh, and Honglak Lee, Value prediction network. In Advances in Neural Information Processing Systems, pp. 61 18-8128,881111111111111111811111 2017.

---

Under review as a conference paper at ICLR 2021
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances in neural information processing systems, pp. 4026-4034, 2016.
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 2017.
Georg Ostrovski, Marc G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In International Conference on Machine Learning, pp. 2721-2730,
2017.
Brendan O' "Donoghue, Ian Osband, Remi Munos, and Volodymyr Mnih. The Uncertainty Bellman Equation and Exploration. In International Conference Machine Learning, pp. 1836-3845, 2018.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity exploration by self-supervised prediction. IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):230-247 2010.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv: /91 1.08265, 2019.
Ramanan Sekar, Oleh Rybkin, Kostas Danilidiss Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via sele-super world models. arXiv preprint arXiv:2005.005 0059000 2020.
Tim Seyde, Wilko Schwarting, Sertac Karaman, and Daniela Rus. Learning to plan via deep optimistic value exploration. In Proceedings of the 2nd Annual Conference on Learning for Dynamics and Control, 2020.
Pranay Shyam, Wojciech Jaškowski, and Faustino Gomez. Model-based active exploration. In International Conference Machine Learning, pp. 5779-5788. PMLR, 2019.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.04136 2020.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprini arXiv: /507.500/1111 2015.
Susanne Still and Doina Precup. An information-thherede approach to curiosity-driven reinforcement learning. Theory in Biosciences, 131(3):139-143 2012.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind Control Suite. Technical report, DeepMind, 2018.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pp. 2746-2754, 2015.
Yuxiang Yang, Ken Caluwaerts. Atil Iscen, Tingnan Zhang, Jie Tan, and Vikas Sindhwani. Data efficient reinforcement learning for legged robots. arXiv preprint arXiv:1907.03613, 2019.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, pp. 7444-7453, 2019.
12

---

Under review as a conference paper at ICLR 2021
A PARAMETERS AND IMPLEMENTATION
We use as single set of parameters throughout all experimental evaluations. The general model architecture follows Hainer et al. (019a), where the variational autoencoder from Ha & Schmidhuber (2018) combined with the RSSM from Hafner et (019b)). We extend their default parameters by the ensemble size M 5, the initial UCB trade-off parameter Pans 0.0, and the per-episode linear UCB growth rate 5 = 0.001. The learning rates for the model, the value function and the policy are 6 x 10-4 8 10 5 10-1, respectively, and updates are computed with the Adam optimizer Kingmaa & Ba 2014). Throughout all experiments, the online phase consists of 1000 environment interactions with an action repeat of 2, while the offline phase consists of 200 learning updates.
An overview of the range of hyper-parameter values that were investigated is provided in Table 2 Not all possible pairings were considered and suitable combinations were determined by inspection, while the best pairing was selected empirically. Our implementation builds on Dreamer (ht / //github. cwm dani jar rr and the remaining parameters are set their default values. Experiments were conducted on 4 CPU cores in combination with 1 GPU (NVIDIA V100).
Parameter
Values
Policy learning
[8 10 6. 2 10
Training steps
[100, 200]
UCB initial trade-off (Binx)
[-0.1, 0.0, 0.1, 0.3, 0.5]
UCB growth rate (6)
[- 0-0001 +0.000, +0.001, +0.002, x1.010, x1.015]
Table Hyper-parameters consideeed during training.
B NETWORK ARCHITECTURES
The base network architectures employed throughout this paper are provided in Table 3 Each particle is assigned a distinct instance of its associated models. In the following, we briefly comment on how the parts of the transition model interact and provide further insights into the remaining models.
Transition model The transition model follows the recurrent state space model (RSSM) archi- tecture presented in Hafner al. (2019a]b). The RSSM propagates model states consisting of a deterninistic and a stochastic component, respectively denoted by 8t, and 8t,s at time 4. The stochastic component St,s is represented as a diagonal Gaussian distribution. The transition model then leverages the imagine 1-step method to predict priors for the associated mean and standard deviation, (urrior _irror) ) based on the previous model state and applied action. In the presence of observations. the observe 1-step method can be leveraged to convert prior estimates into posterior estimates, (utte .o The transition model may then propagate posteriors based on a context sequence using both the imagine 1-step and observe I-step methods, from which interactions can be imagined by propagating prior estimates based on the imagine Isstep method. Each particle uses a transition model that follows the presented network architecture, but possesses distinct parameters.
Encoder model The encoder parameterization follows the architectural choices presented Ha & Schmidhuber (2018). The encoder generates embeddings based on 64x64 RGB image observations.
Observation model The observation model follows the decoder architecture presented in Ha & Schmidhuber (001)) The image observations are reconstructed from the associated model states 5,.
Reward and value model Rewards and values are both predicted as scalar values from fully- connected networks that operate on the associated model states sy, similar to Hafner et al. (2019a). Each particle uses a pairing of a reward model and a value model with distinct sets of parameters.
Action model passed through a tanh to allow action saturation. It is combined with a softplus standard deviation based on aa and the resulting Normal distribution is squashed via tanh (see Haarnoja et al. (2018).
The action model follows Hafner al. 2019a), where the mean Ma is rescaled and
13

---

Under review as a conference paper at ICLR 2021
Layer Type
Input (dimensions)
Oulput (dimensions)
Additional Parameters
Transition model (imagine 1-xtep)
Dense
5,-1,a (30), d (Tt.)
Tet, (200)
a=ELU
GRU
fet,2 (200), 8r-1,d (200)
I8; (200), 8r.d (200)
a=tanh
Dense
r8+ (200)
feî,, (200)
==RLU
Dense
fef,4 (200)
MF.s (30),
orri
ofs
prior
(30)
N=None
Transition model (observe I-step)
Dense
8r,d (200), 2+ (1024)
fol.o (200)
a=ELU
Dense
fc (200)
p4. (30),
post (30)
a=None
Encoder model
Conv2D
obs (64, 64, 3)
ev1 (31, 31, 32)
a=ReLU, s=2, k=(4.4)
Conv21)
cv1 (31, 31, 32)
cv2 (14, 14, 64)
a=ReILU, U1, s=2, k=(4,4
Conv2D
(14, 14, 64)
cv3 (6, 6, 128)
R= s=2, k=(4,4)
Conv2D
cv3 (6, 6, 128)
cv4 (2, 2, 256)
R=RLLU, s=2, k=(4.4)
Reshape
cv4 (2, 2, 256)
2; (1, 1, 1024)
Observation model
Dense
8r,d (200), 87,* (30)
fc] (1, 1, 1024)
a=None
Dcconv2D
fo: (1, 1, 1024)
de1 (5, 5, 128)
R=LLL,, s=2, k=(5,5
Deconv2D
de1 (5, 5, 128)
dc2 (13, 13, 64)
R=RLLU, s=2, k=(5,)
Deconv2D
de2 (13, 13, 64)
de3 (30, 30, 32)
a=ReLU, s=2. k=
Deconv2D
dc3 (30, 30, 32)
de4 (64, 64.
a=ReLU, s=2,
Reward model
Dense
8r, (200), 87,a (30)
fcl (400)
==
Dense
fef (400)
!e!2 (400)
a=ELU
Dense
foß (400)
fe?
a=ELU
Value model
Dense
8r, (200), 8.,* (30)
fel (400)
==FLUU
Dense
(1.2) (400)
(2.3) (400)
a=ELU
Dense
fc (400)
fe. (1)
a=ELU
Action model
Dense
St,d (200), 8+,. (30)
[c], (400)
a=ELU
Dense x
fo (1.2.3 (400)
(2.3.4) (400)
==
Dense
fe.: (400)
(4 (ma). da (na)
R=LLU
Table General network architectures of the underlying models. note that repeated layers have been condensed with Dense x i referring application of the same dense layer architecture i times. Parameter abbreviations: a=activation, k=kernel. and s=stride. Adapted from Hainer et al. (2019a).
14

---

Under review as a conference paper at ICLR 2021
C PREDICTION UNCERTAINTY
We provide an illustrative visualization of how the prediction uncertainty in the ensemble evolves during model training. The ensemble is provided with context from a sequence of 5 consecutive images and then predicts forward an open loop fashion for 15 steps (preview horizon). The ground truth sequence is compared to ensemble predictions after 10, 150, and 300 episodes of agent training.
Figures 6 and 7 show two different motion patterns for the Walker Walk task. The motion Figure be described regular walking pattern. At the beginning of model training, the agent will have mostly observed itself falling to the ground and, in combination with poorly trained policy, the ensemble predictions place the agent the ground in variety of configurations. After 150 episodes, short-term uncertainty has been significantly reduced, while considerable uncertainty remains at the end of the preview window. After 300 episodes, the ensemble predictions align with the ground truth sequence. The agent therefore focused on reducing uncertainty over this desirable motion pattern. This can be contrasted with the results of Figure 7 where uncertainty over irregular falling pattern remains even after 300 episodes. The falling motion is undesirable, and while the ensemble predictions agree fall being imminent, no significant amount of effort was spent on identifying exactly how the agent would fall. We can observe similar results on the Cheetah Run task for running motion pattern Figure and a falling motion pattern in Figure 9 However, the lower complexity Cheetah dynamics seem to allow for more precise predictions than the Walker task.
Context
Prediction
-4
-2
-1
0
3
5
7
11
13
15
1
Ground Truth
Episode 10
Episode 150
Episode 300
Ground Truth
Figure 6: Motion pattern of the Walker with low predictive uncertainty. The agent is provided with 5 contextual images and predicts forward for 15 steps (preview horizon), at different stages of training. The regular walking pattern is well-explored and only induces little deviation in the ensemble. This motion is desirable and the agent should focus on reducing its uncertainty over environment behavior.
Context -2
Prediction
-3
11
0
3
5
7
11
13
15
Ground Truth
Episode 10
Episode
Episode 300
Ground Truth
Figure Motion pattern of the Walker with high predictive uncertainty. The agent provided with contextual images and predicts forward for 15 steps (preview horizon), different stages of training. The irregular falling pattern has not been extensively explored and high uncertainty remains in the ensemble. This motion is undesirable and the agent should not focus reducing its uncertainty.
15

---

Under review as a conference paper at ICLR 2021
Context
Prediction
4
0
3
5
11
13
15
Ground Truth
T
K
K
Y
T
V
R
FL
Episode 10
T
Episode 150
Episode
Ground Truth
Figure 8: Motion pattern of the Cheetah with low predictive uncertainty. The agent is provided with contextual images and predicts forward for 15 steps (preview horizon), different stages of training. The regular running pattern is well-explored and only induces little deviation in the ensemble. This motion is desirable and the agent should focus on reducing its uncertainty over environment behavior.
Context
Prediction
0
3
5
7
11
13
15
Ground Truth
7
L
Episode 10
7
R
Episode 150
Episode 300
Ground Truth
T
Figure 9: Motion pattern of the Cheetah with high predictive uncertainty. The agent is provided with contextual images and predicts forward for steps (preview horizon), at different stages of training. The irregular falling pattern has not been extensively explored and uncertainty remains in the ensemble. This motion undesirable and the agent should not focus on reducing its uncertainty.
D BASELINES
The baseline performance data for DrQ was taken from Kostrikov et al. (2020), the ones for D4PG and A3C from Tassa et al. (2018), while the data for Dreamer was generated by running the official TensorFlow 2 implementation of Hafner et al. (2019a). It should be noted that both DrQ and D4PG use 84 image observations, whereas LOVE and Dreamer use 64 64 image observations. Larger resolution provides more fine-grained information, which potentially translates to improved planning. Furthermore, DrQ continuously refines its policy online, while the other algorithms only do so offline.
E BUGTRAP EXTENDED
We provide additional occupancy maps for the bug trap environment in Figure 10 The environment provides reward feedback and assesses the agent's 's ability to actively search for informative feedback through intrinsic motivation. Furthermore, the environment geometry makes exploration of the outside area difficult. In the absence of useful mean performance estimates, LOVE leverages uncertainty-guii guided exploration to query interactions. This allows for escaping in out of 6 trials and achieving the largest only escapes during 3 trials (column 3), while displaying a highly reduced area coverage (rows and 3). Similarly, random exploration allows the Dreamer agent only escape in 2 instances (column 4).
coverage (column 2). LVE does not leverage uncertainty estimates and
16

---

Under review as a conference paper at ICLR 2021
Initial
LOVE
LVE (B 0.0)
Dreamer
Scenario
Scenario 2
Scenario
C
Scenario
Scenario
t
t
Scenario
Figure 10: Occupancy maps of the bug trap environment for scenarios and three random seeds. In the absence of reward feedback, the amcertainty-guidee exploration allows LOVE to escape during 5 out of runs while achieving the highest area coverage in search of non-zero reward feedback. LVE removes optimistic exploration and as result only escapes during runs, while significantly reducing area coverage. similar pattern be observed for the randomly exploring Dreamer agent.
17

---

Under review as a conference paper at ICLR 2021
F ABLATION STUDY: DREAMER
We compare performance to ADreamer, a variation that uses our changes to the default parameters. Figure II indicates that performance improves several tasks, while deteriorating Finger Spin. LOVE outperforms ADreamer on the majority of tasks. can thus be concluded that increased information propagation generally affects performance favourably. However, relying single model can propagate simulation bias into the policy and turn impede efficient learning. This could as explanation for the unchanged performance the not fully observable Cartpole Swingup tasks, as well as the deteriorating performance on the high-frequency Finger Spin task.
Cartpole Swingup
Cartpole Swingup Sparse
Cheetah
Finger Spin
1000
1000
1000
1000
Egisode Return
750
750
250
0
0
Hopper Hop
Pendulum Swingup
Walker Run
Walker Walk
1000
1000
Episode Return
200-
750
400
750
500
500
100
200
250-
250-
0
1
2
0
0
Timesteps
le5
Timesteps
le5
Timesteps
le5
Timesteps
1e5
LOVE (Ours)
ADreamer
Dreamer
Figure 11: Comparison to Dreamer with adapted policy learning rate and training steps (ADreamer). The changes improve performance Dreamer on some environments, while significantly decreasing performance on the Finger Spin task. LOVE still outperforms Arreamr the majority of tasks.
G ABLATION STUDY: PLANNING HORIZON
We investigate increasing the planning horizon used during latent imagination. Longer horizons shift performance estimation from values towards rewards, which can be advantageous when the value function has not been sufficiently learned. Prediction quality over long horizons relies on accurate dynamics rollouts. Figure 12 indicates that intermediate horizon is good trade-off. We note that the two Walker tasks had not completed at the time of submission, but provide a visible trend.
Cartpole Swingup
Cartpole Swingup Sparse
Cheetah
Finger Spin
1000
1000
1000
1000
Epicode Return
750
750
750
750
500
250-
250-
250
0
0
Hopper Hop
Pendulum Swingup
Walker Run
Walker Walk
1000
1000
Episode Raturn
200-
750
400
750
500
500
100
200
250
250
0
Timestops
1e5
Timesteps
le5
Timestops
1c5
Timestops
1c5
LOVE (H=15)
LOVE (Hf=10)
LOVE (H=20)
Figure 12: LOVE under variation of the planning horizon.
18

---

Under review as a conference paper at ICLR 2021
H ABLATION STUDY: 3 SCHEDULE
We investigate variations of the beta schedule, initializing cither with a negative (initially pessimistic) positive value (initially optimistic). The former variation penalizes uncertainty in the beginning and then transitions to become optimistic, while the latter seeks out uncertainty from the start. Based on Figure 13 notice that terminal performance is mostly similar. The initially pessimistic agent exhibits reduced performance on the sparse pendulum, where it only explores well after transitions the challenging Hopper task, where
optimism (Episodee 100), and improved performance initial pessimism potentially guards against local optima. Our choice of parameters tries to mitigate unfounded optimism during initialization (initial value 0), while encouraging exploration throughout the course of training (linear increase). Here, we chose the same values for all tasks, but one could imagine task-specific choices (negative beta for safe-RL,, positive beta for optimistic exploration).
Cartpole Swingup
Cartpole Swingup Sparse
Cheetah Run
Finger Spin
1000
Episade Retumn
750-
750-
500
500
500
500
250
250
250
250
01
0-
0
Hopper Hop
Pendulum Swingup
Walker
1000
600
1000
Episode Retum
200-
750
400
750
500
100-
Timesteps
Timesteps
Timesteps
Timesteps
LOVE
0.0,48 +.0001)
(Ro
0.001)
LOVE +0.1, 0.0001)
Figure 13: LOVE under variation of the beta schedule.
I ABLATION STUDY: ENSEMBLE SIZE
We investigate variation of the ensemble size. A smaller ensemble generates uncertainty estimates that are more susceptible to bias in the individual ensemble members and may even generate misleading estimates. Figure demonstrates that smaller ensemble (M=2) impacts performance unfavorably.
Cartpole Swingup
Cartpole Swingup Sparse
Cheetah Run
Finger Spin
1000
1000
1000
Episade Retur
750-
750
750
750
500-
500-
500
500
250
250
250
250
0
0
0
Hopper Hap
Pendulum Swingup
1000
600
1000
Episode neturn
200
750
750
100-
200
0
0
0
Timesteps
Timesteos
1e5
Timesteps
1c5
Timesteps
1c5
LOVE (M=5)
LOVE (M=2)
Figure 14: LOVE under variation of the ensemble size.
19


Reviews:
[{"review_id": "BwZgeBkJAsU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed latent optimistic value exploration (LOVE) as a mechanism to leverage optimistic exploration for continuous visual control. The main idea is to use a small (~5) ensemble of latent models with shared encoders (and therefore shared learned latent space) but different transition, reward and value models. The variance of predictions from this ensemble can be used as uncertainty estimates of each action sequence while the mean is the typical policy learning objective. Then LOVE puts more weighs on the states with high variance during exploration to enforce the agent to visit (optimistically) uncertain states. This is inherently optimistic because there is a positive bias (beta > 0 and var > 0) addition to the expected reward. \n\nThe idea of the paper is not novel and it has been explored before such as (T. Kurutach et al ICLR 2018) and unfortunately the authors did not provide enough context in the related works to distinguish their work with previous research. However, in its proposed form,  LOVE  is new to the best of my knowledge. Using the variance of predictions of an ensemble of agents is an interesting way of approximating uncertainty and the experiments demonstrate how this bias can improve the results. However, I find the experiments not to be convincing enough that the added complexity is necessary to achieve the improved performance:\n\nFirst, the main claim of the paper is that the proposed method achieves better score by exploring better. However, there are many changes that can cause this improvement. For example, using an ensemble of the models with different transition, reward and value models is essentially using a bigger (i.e. with more parameters) model. There is an ablation study that demonstrates only using an ensemble doesn't work (LVE alternate), however, this is not convincing that the improvements are coming from better *exploration*. This can be done in multiple ways such as demonstrating that a base method (e.g. Dreamer) works better if trained on the data collected by LOVE. An ablation study with a negative beta can be also super helpful (although I believe the results of that is kinda trivial).\n\nSecond, there is no study that visualizes the importance of the ensemble. The authors used 5 particles throughout the experiments but why 5? I'm not suggesting optimizing this number as a hyper-parameter or anything like that but what I'm looking for is a study that clearly demonstrates that having a more accurate approximation of uncertainty is important. This can be achieved by studying the effect of number of particles on the performance of the agent. Visualizing the actual variance of the predictions and demonstrating that the predictions actually vary is also important. This tightly related to various values for beta which also requires another ablation study. There are also many other way of enforcing optimism that the proposed method can be compared to.\n\nOverall, the authors are on the right track. The paper (except for related works) is very well written and easy to read. However, more experiments are required to clearly demonstrate why the method is working and how. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "8bf_fPmfkue", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--------------\n\nThis paper proposes LOVE (Latent Optimistic Value Exploration), a model-based exploration algorithm for POMDP or pixel-based control systems. The method builds upon Dreamer (Hafner et al. 2019) for learning latent models, and the variance of value estimates (one transition/reward/value model per particle) estimates the epistemic uncertainty of the world, which can be used as a UCB-style exploration reward. LOVE can achieve a comparable or better performance on a simple pointmass environment with a trap and standard DeepMind continuous control suites.\n\n\nStrength:\n- This paper studies a high-significance problem of learning representation for visual-based control and deep exploration. Self-supervised learning of latent representation and model-based exploration is an important and timely research problem to study.\n- Overall, the paper is written well, with a clear motivation and a good organization of the method. Presentation in terms of pseudocode, plot, table, and hyperparameters look great.\n- The method is straightforward and the idea behind the algorithm is very reasonable.\n\nWeakness\n- The paper has limited novelty; the core idea of model-based planning and disagreement-based exploration are taken from existing works, though combining them and making them work in a challenging pixel-based continuous control tasks would be a non-trivial work.\n- Crucially, there are a few of *very* similar works in model-based exploration, which also builds upon Dreamer (please see the detailed comments below). There might be some technical difference, but a comprehensive comparison with existing works would be critical.\n- It is not clearly discussed or mentioned in the paper why the proposed method can be called as a “deep” exploration, or how beneficial it would be compared to “shallow” explorations.\n\nDetailed comments:\n-----------------\n\nThere are some similar works in the model-based exploration literature, which this paper did not cite or compare with.\n\nPlan2Explore (Sekar et al., ICML 2020): this paper learns a Dreamer-like model, where an ensemble of $q(h | s, a)$ is learned and the variance of prediction (of the latent variable $h$) is used as the uncertainty, or the “disagreement” intrinsic reward. The difference to LOVE would be, the uncertainty is measured by estimation of value function or latent representations. It will be interesting to see how these approaches can be compared.\n\nValue estimation might be much harder than the transition model, especially when the task reward is sparse, and would not be directly applicable in a reward-free/unsupervised setup. In such scenarios, how can LOVE learn to explore the world?\n\nMore related works to consider:\n- Ready Policy One (RP1; Ball et al., arXiv 2020)\n- Model-based Active Exploration (MAX; Shyam et al., ICML 2019)\n- Learning Awareness Models (Amos et al., ICLR 2018)\n\nIn terms of experiments/empirical evaluation, there is no baseline exploration algorithm presented that is based on Dreamer or pixel-based control, which makes the effectiveness of the proposed approach a bit difficult to be assessed. For example, what if we do a straightforward exploration (such as Curiosity [Pathak et al. 2017], RND [Bruda et al., 2018]) on Dreamer?\n\n\n**Additional Comments**\n\n- It would be great to provide more clarification/explanation on how LVE (LOVE with $\\beta=0$) and Dreamer are different.\n- Additional analysis and experiments would also have strengthened the paper, such as: how does the algorithm performance differ under different values of the planning horizon? How does the performance of the algorithm vary under different number $M$ of ensembles?\n- As in Hafner et al., 2019a, it would be great to clarify the difference between $p(\\cdot)$ and $q(\\cdot)$ in their meaning, i.e. $p$ for distributions that samples from real environments, and $q$ for approximations.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6WshcSaufjL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed method seems like a simple and effective combination of existing ideas and approaches. Overall while I like the approach but am leaning towards a reject since I think there are aspects in the experiment section that can be clarified and improved. However I am open to changing my decision if my concerns are addressed during the rebuttal period.\n\nStrengths:\n- The ideas presented are simple extensions of existing methods and are quite easily digested.\n- The proposed approach does seem to learn faster on some of the tasks considered although some aspects of this need clarification.\n- The experimental details presented in the Appendix are quite thorough and did help me understand some aspects of the work in more detail.\n\nIssues and points worth clarifying:\n\n- Apart from a number of points of clarification that are listed later, my main concern is with the experimental section of the paper. This work compares performance of the proposed method on 8 of the DeepMind Control Suite domains and one toy domain with very little ablation. In contrast, the original DrQ and DREAMER work which are used as baseline here show results on 15 and 20 control suite tasks on top of some results on Atari. \n- The number of domains becomes somewhat more important in light of the fact that on 2 of the 8 domains considered DrQ outperforms the proposed approach. The explanation regarding these 2 tasks having dense rewards does make sense but more data would help substantiate the claim.\n- Another concern I have is with regard to the presentation of the results in Figure 3. Most of the curves are cut off before they have converged. This makes it hard to  sanity check against the existing results from the DREAMER and DrQ paper and it remains unclear to me if LOVE performs asymptotically as well as the baselines on some of the tasks. Perhaps this information could be included instead in Table 1 which as far as I can tell does not currently present any new information that is not present in Figure 3.\n- Finally the paper could also include more ablations - especially regarding the effect of the ensemble size and how important the step increase schedule is for the beta parameter of the UCB objective.\n\nFinally there are some points which I don’t fully understand based on my reading of the text and for which clarification would greatly help.\n- The main text mentions that LOVE has exploration noise turned off since exploration happens through the UCB noise. Was this the same setting used for the LVE where beta is set to 0? My understanding is that LVE does have exploration noise but it would be good for the authors to confirm this.\n\n- The parameter in the UCB objective Beta is progressively increased from 0 in delta steps of 0.001. Where does this number come from? Was this a hyperparameter?\n\n- Learning multiple DREAMER models is an interesting idea but the obvious issue with this is that it could bloat wall clock learning time. How much slower is the proposed approach to vanilla DREAMER  in terms of wall clock time? While understandably the focus of this work is data efficiency, I think this is an important number to mention perhaps even in the Appendix to paint a more complete picture.\n\n--Update (Nov 25)--\n\nI am happy that the authors improved the paper with reviewer feedback. In particular I think the new ablations and comparison and mention of previous work makes the work more complete. The results on new sparse tasks are also interesting. I still think more can be done in terms of experimental validation (in particular my original note regarding early cutting of the curves has not been addressed). However overall I think the paper does meet the acceptance threshold as things stand.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "31hZ1qoeM-H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---- Summary ----\n\nThe paper proposes LOVE, an adaptation of DOVE (Seyde’20) to latent variable predictive models (Seyde’20 only condsidered predictive models without latent variables). Seyde’20 proposes to use a generalization of Upper Confidence Bound to deep model-based RL, by training an ensemble of models and value functions, and training a policy to maximize mean + variance of this ensemble (similarly to Lowrey’19). The submission empirically demonstrates that tuning the learning rate and number of training steps per environment steps of Dreamer (Hafner’20) improves sample efficiency, using an ensemble of predictive models further improves data efficiency slightly (on cartpole and walker tasks), while on top of that the proposed exploration method slightly improves sample efficiency on the Hopper and sparse Cartpole tasks. \n\n---- Decision ----\n\nThe submission contains little technical novelty over prior work of Seyde (2020). The experimental results are weak, but somewhat justify the claims as there is a slight but consistent improvement on some tasks. However, the paper suffers from a major flaw in the empirical evaluation. Figure 3 and the relevant discussion describe LOVE as significantly outperforming the Dreamer baseline. This difference is largely due to the fact LOVE uses a different learning rate and number of epochs, which improves sample efficiency. The paper graciously provides the comparison to the fairly tuned baseline in the appendix as Figure 9, confirming this. The fairly tuned baseline needs to be moved from the appendix to the main paper, and the contribution section and the discussion of the experiments need to be rewritten accordingly. If this is provided, I will reevaluate the paper. In the current state of the paper, I am unable to consider its merits on the basis of this flaw.\n\n---- Strengths ----\n\nThe paper is technically correct (except for the flaw explained above), and proposes a promising approach to a relevant problem of exploration in RL from images. The experimental results indicate that the proposed method could be effective.\n\n---- Weaknesses ----\n\nThe major flaw of the paper is described earlier. In addition, there are two other major issues with experimental evaluation.\n\nThe experimental evaluation of the paper is rather weak. The proposed exploration method only improves performance in 2 out of 8 environments. This might be because the other environments do not require sophisticated exploration, in which case the method needs to be tested on more than 2 relevant environments. Sparse-reward versions of the evaluated environments can be easily designed and would be suitable for evaluating the method.\n\nThe second major issue is that the method is not evaluated against any competing exploration baselines, even though the paper cites multiple prior works on this. For instance, the paper claims that methods based on information gain or improving knowledge of the dynamics will not explore as efficiently as the proposed method. Both of these claims need to be empirically evaluated or toned down.\n\n---- Additional comments ----\n\nThe related work section is missing the following papers: \n- Ball’20 is a model-based RL method that uses UCB for exploration\n- Sekar’20 is a model-based RL method that uses latent ensemble variance for task-agnostic exploration\n\nBall’20, Ready Policy One: World Building Through Active Learning\n\nSekar’20, Planning to Explore via Self-Supervised World Models\n\n## Update \n\nThe new sparse tasks and comparison to Dreamer + Curious improve the paper and address some of my concerns. Specifically, a sizable improvement due to exploration is now seen on 3 tasks, Hopper, Cartpole Sparse, and Cheetah Sparse. The new maze task is also more challenging than the bug trap task.  \n\n--- Final Decision ---\n\nAfter the significant improvements in the experimental evaluation, I believe the paper provides a reasonable case for the proposed latent UCB method. It also provides an interesting discussion on the advantages of UCB-style methods, and an interesting observation that optimistic reward-based exploration can be effectively used even in absence of (positive) rewards. Even though the experimental evaluation of prior work on exploration is still rather lacking, I believe that these contributions are enough for the paper to be interesting to the ICLR audience. I raise my score to 6.\n\n--- Remaining weaknesses ---\n\nThe experimental evaluation in the paper is still quite lacking in terms of baselines, making it impossible to judge whether the paper actually works better than prior work.\n\nFirst, the proposed method contains two improvements, model ensembling, and optimistic exploration, but doesn't go much in-depth analyzing either of these improvements, instead trying to focus on both at the same time. This makes comparison to prior exploration methods hard because the proposed method receives an additional boost due to an ensemble of dynamics (the paper conveniently quantifies this boost in the LVE method, and it is shown to be rather large). For a more fair comparison, the ensemble of dynamics might be ablated (leaving only the ensemble of value functions), or the competing baselines could also be built on top of LVE.\n\nSecond, the paper only compares against one competing exploration method, Dreamer + Curious. There has been a large amount of proposed exploration methods, and it would be appropriate to evaluate the proposed UCB method against at least a few of them. For instance, the paper could compare against similar value function ensemble techniques (Osband'16, Lowrey'18, Seyde'20), or other cited work (Ostrovski'17, Pathak'17). Burda'18 is not cited, but perhaps should be compared against. All these methods can be relatively easily implemented on top of LVE for a fair comparison.\n\nBurda'18, Exploration by random network distillation.\n\n--- Additional comments ---\n\nWould be great to clarify what is the observation space for the bug trap and maze tasks. For instance, you could add observations and predictions for these tasks to the appendix.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission is acknowledged as having potential value in terms of proposing a new approach for exploration based on ensembles and value functions. However, there are lingering concerns about the discussion of what this paper brings to the table vis-a-vis prior work, together with a lack of clear demonstration of the explicit gains from the exploration mechanism and more experimental studies. The author(s) would do well to revise as per the feedback given and resubmit a version with a more compelling argument. 
Decision: Reject