ReaPER: Improving Sample Efficiency in Model-Based Latent Imagination
Authors: Martin A Bertran
Guillermo Sapiro
Mariano Phielipp
Venue:  (2021)
Keywords: model-based reinforcement learning
visual control
sample efficiency

Abstract: Deep Reinforcement Learning (DRL) can distill behavioural policies from sensory input that solve complex tasks, however, the policies tend to be task-specific and sample inefficient, requiring a large number of interactions with the environment that may be costly or impractical for many real world applications. Model-based DRL (MBRL) can allow learned behaviours and dynamics from one task to be translated to a new task in a related environment, but still suffer from low sample efficiency. In this work we introduce ReaPER, an algorithm that addresses the sample efficiency challenge in model-based DRL, we illustrate the power of the proposed solution on the DeepMind Control benchmark. Our improvements are driven by sparse , self-supervised, contrastive model representations and efficient use of past experience. We empirically analyze each novel  component of ReaPER and analyze how they contribute to sample efficiency. We also illustrate how other standard alternatives fail to improve upon previous methods. Code will be made available.

Under review as a conference paper at ICLR 2021
REAPER: IMPROVING SAMPLE EFFICIENCY IN MODEL-BASED LATENT IMAGINATION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Reinforcement Learning (DRL) can distill behavioural policies from sen- sory input to solve complex tasks, however, the policies tend to be task-specific and sample inefficient, requiring large number of interactions with the environ- ment that may be costly or impractical for many real world applications. Model- based DRL (MBRL) can allow learned behaviours and dynamics from one task to be translated to a new task in a related environment, but still suffer from low sample efficiency. In this work we introduce ReaPER, algorithm that addresses the sample efficiency challenge in model-based DRI, and illustrate the power of the proposed solution on the DeepMind Control benchmark. Our improvements are driven by incorporating sparse, self-supervised, contrastive model represen- tations and efficient use of past experience. We empirically analyze each novel component of ReaPER and analyze how they contribute to overall sample effi- ciency. We also illustrate how other standard alternatives fail to improve upon previous methods Code for the plug-and houg-and-play -lay tools introduced here will be made available.
INTRODUCTION
Real world agents are able to efficiently achieve goals in complex, partially-controllblee environ- ments even situations the agent has never experienced before. This capacity can be attributed to robust model of how the agent's actions affect their surroundings, model that is distilled from past experience, allowing the agent estimate the effect their actions in the pursuit of goal in novel situation. In practice, learning an explicit world model as in Hafner et al. (2019a); Sekar et al. (2020) is challenging in terms of sample efficiency (number of environment interactions), since the agent needs to simultaneously learn good reward-seeking behaviours and refine a generalizeable model of the environment.
This sample-efficiency challenge has been empirically observed to be more pronounced on agents that act based on sensory input (e.g., raw pixels) instead of proprioception or latent states Lake et (2017); Kaiser et al. (2019); Tassa et al. (2018). Since the latter is often inferrable based pixels, this indicates that pixel-basce pipelines have room for improvement. This an important issue since many real-world applications are more naturally solved and articulated in terms of pixels or sensory input.
raw
In this work investigate several potential sources of sample-inefficienccy in model-based re- inforcement learning (MBRL) methods, addressing each source separately using simple tech- niques. We propose ReaPER (Regularized Contrastive Model-Based DRL with Prioritized Episodic Replay), an MBRL algorithm built upon Dreamer Hafner et al. (2019a) that addresses the identified sources of sample-inefficiency and improves results over the DMControl suite Tassa al. (2020). We provide ablation experiments for all the introduced components, interestingly, we show that tech- niques that individually improve sample efficiency sometimes show detrimental results when combined, but can be made to work together by the introduction of prioritized replay. Although cur- rent state of the art methodologies for single-task agents in the context of, for example, DMControl suite are geared towards model free learning, we specifically choose to address MBRL as a stepping stone towards continual learning RL. An efficient Pytorch implementation of this code will be made available.

---

We use the letters p. q to denote true and estimated distributions respectively. To interact with the environment, observations the current model state is sampled from the representation po(st $t-1; Q1-1, J4), and the action is Under review as a conference paper at ICLR 2021
Main Contributions. We introduce ReaPER, an MBRL agent for control tasks in visual envi- ronments that outperforms the previous state-of-the-art MBRL agent (Hafner et al. (2019a)) the DMControl benchmark Tassa et al. (2020). We empirically study key sources of sample inefficiency in MBRL methods, and address each individually using simple approaches, the combination of all successful approaches is integrated onto ReaPER. We experimentally show that individually promis- ing approaches can fail to compose when combined simply, but that the introduction of prioritized replay can improve the combined approach beyond what each technique could individually achieve. This provides self-contained ablation study of the effects of cach component of the proposed ReaPER architecture.
We empirically show how ReaPER constructs coarser but more robust world model, and how the sample efficiency improvement holds throughout model training. Ideas to improve sample efficiency that were ultimately unsuccessful such as exploration via latent disagreement Sckar al. (2020) and bisimulation metrics Ferns et al. (2011); Ferns & Precup (2014); van der Pol et al. (2020) are also shown and discussed due to their value improve the understanding of MBRL. Code will be made available.
2 AGENT DESCRIPTION
We model the environment as partially observable Markov decision process (POMDP) with dis- crete time steps E [1 T). high-dimensional observations and rewards produced by the en- vironment Or, p(o,, Tz 01.r-1,F1et- d1:t), and real-valued actions selected by the agent a, ~ plae 01 T1:t, (1;t:)) The goal of the agent is to maximize the expected discounted reward ( (r)), with discount factor y E (0, 1].
We introdice Regularized Contrastive Model Based DRL with Prioritized Episodic Replay (ReaPER), latent dynamics arquitecture extending the work of Dreamer Hafner et al. (2019a), with the following key components, which will detailed and exploited in the coming sections:
Sequence Augmentation
{oî k T4L = oo )+)) It=
Image encoders
h = folot), hi fo (o))
Representation
po St-1, a1-1, h():
Transition
( $t-1, a1-1),
(1)
Image Decoder
qo (o st),
Reward predictor
(r: 8t);
Policy
Te Ta(at (t 8t):
Value
Up (sr).
perturbed and then passed through the image encoder h fo.).
selected from policy a, (a, (a, sy). The full algorithm is presented below, an explanation of each component and the reasoning behind its inclusion is provided in the following sections.
2.1 MODEL LEARNING
The model is trained batches of sequential, observable past experience ((,)) where t indicates time from start of episode, is sequence length, b a batch indicator, and B is the total number of batches. The model's parameters, denoted by 0, are trained plementary objectives, some of these objectives are computed independently for each time sequence (i.e., one per batch element), while others include comparisons against elements in different batches. To simplify the notation, we denote each model loss by a super-index or t, b to indicate if these losses are computed across time across time and batch respectively. The losses are described next.
variety of com-
Reconstruction loss. We follow the derivation in Hafner al. (2019a) increase the variational lower bound of the model (ELBO, Jordan et al. (1999)). The resulting reconstruction loss is

---

Under review as a conference paper at ICLR 2021
Algorithm ReaPER
Require:
Hyper-parameters: Seed Episodes S, Collect interval Batch size Sequence length L, Imagi- nation horizon H. learning rate Contrastive and Sparsity bonuses Ac, Ara. Contrastive momentum &
Initialize Priority Episodic replay buffer D with S random episodes Initialize network parameters 0, Φ, while rraining do step = I.....
Model Learning
Draw B sequential samples {(or, Tt, at) ~ D
Do sequence-consister contrastive image augmentation {(oî, of)) SA((oi)(t)) Compute model states st ~ p( of (t-1)
Compute model component CREC, Ltg, Le in equations 2 4. Compute model loss Lss Lasc λe, Lit AcLe
Update buffer sample priorities according to LM
Update model parameters on model loss <- lrVo(LM)
Update key encoder parameters 0' <- a''' + (1 a)0 Policy Learning
Imagine latent trajectories {Sr, 0r) |==/ for each state Compute value targets (x(8-) as in Eq
Compute policy and value losses L Lv as Eq 7 Update á < IrV oC=); & +- 1b (Lv);
end for
Data Collection
01 < env.reset() 1,...; do
Sample $t, ae ~ (s(s: os, al-1; 81 st)
Add exploration noise 4- clamp + ezi; 1, 1), 21 10)
Observe o,, ri env.step(as)
end for
Add to buffer + D ((0,, T1) end while
C In go 8t)) LsR = In Sc), 8: 45-1, hi) ll go(s: (11)),
G KL (st
REC Co Ls CKL
The reconstruction loss LREC is intuitively interpretable, the state needs to encode sufficient information about the environment reconstruct the current observation and reward or, Fr (Co, Lr,,, it also needs to be compact enough to be predictable given the past state and action (C The state then should capture the minimum amount of information of the past trajectory needed to reconstruct current and future observations and rewards.
State-space compression. The state-space representation of the model may attempt to capture irrelevant or non non-generalizeable details of the environment. These details may still be consistent across the training dataset, and thus are not eliminated by the state prediction loss Lk We therefore add a sparsity prior to the model state to alleviate this issue,
C =|s Le = E, EE C.
(3)
This prior is directly applied to the state-space representations, and not to the model weights in standard & regularization.
Robustness to visual perturbations. Contrastive learning has proven an effective technique to learn robust representations from visual input Oord et al. (2018); He al. (2020); Chen al. (2020a;b); Srinivas et al. (2020). We follow Srinivas et al. (2020), where a batch of sequential observations {fo)et 1 Jb-1 drawn from the dataset are augmented independently into query and

---

Under review as a conference paper at ICLR 2021
key observations {{(o), of )0+))+++++ )-1 (denoted by the superscript q¹ and l: respectively), these aug- mented pairs are fed through their corresponding encoders fo(+), , for (.) to produce paired encodings The objective of the contrastive loss is match the encoded query n2,, (where index the query both by time and sequence indicator) to its corresponding encoded key through
{((h), ) )==1-
bilinear log-loss classifter,
exp ((hg,)) (WT,) c6 log Ev.o v exp ((hi,b) Whh h/. )
(4)
Here W E RB*H is learned bi-linear matrix, (.)T denotes transposition, and key parameters 0' are updated via exponential moving average 0' <- al' + (1 a)).
The augmentations used generate the key-query pairs define the invariance class we induce into our image encodings. Since ReaPER is a model-based architecture that predicts future states based on current model state, it is imperative for these augmentations erwise the augmentation procedure is in direct conflict to the observation reconstruction loss S and the prediction loss I, since introduces an unpredictable perturbation in successive obser- vations that makes pixel-wise prediction impossible. We follow Srinivas et al. (2020) and chose conesconsittent random cropping our augmentation class, where all observations of the same sequence are cropped using the same mask. Randomized temporally-consister added benefit of having low computational budget, since can be computed as a tensorized view of the observation matrix, and requires no extra post-processing. An example is shown on Figure 1.
be temporally consistent, oth-
cropping has the
Figure 1: Random crop augmentation. Top row shows original observations from a single episode on the walker run environment, middle and bottom row show corresponding key and query observa- tions. Note that the crop consistent across time on both key and query. Original image 80 x 80, cropped images are 64 x 64.
Per-sample model loss. ous objectives,
The resulting per-sample model loss is a linear combination of the previ-
C/ C'REo de, Ct, + Ac
(5)
Since this model loss contains both state sparsity term and contrastive robustness term, we expect the model to produce coarser and more robust reconstructions of its environment, this simpli- fied and robust reconstruction yields faster policy improvement than the baseline, improving sample efficiency. Figure 2 shows a comparison between Dreamer and ReaPER reconstructions on 4 envi- ronments. In this figure on non-critical components of the image, such as the background splatter pattern, but still accurately reconstruct both the agent and the floor, the latter of which be used as a proxy for relative motion in these environments.
observe that ReaPER input reconstructions have diminished fidelity
'To be consistent with the literature, is used both for the estimated distributions and for the query, the context making the distinction clear.

---

Under review as a conference paper at ICLR 2021
(a) Dreamer
(b) ReaPER
Figure 2: Comparison of observation versus reconstructed observation in Dreamer vs ReaPER for the walker run, hopper hop, finger spin, and cartpole two poles environments. ReaPER builds coarse and robust reconstructions compared to Dreamer, where irrelevant background details are blurred but important elements in the environment are well modeled.
2.2 PRIORITIZED EPISODIC REPLAY
Prioritized experience replay Schaul al. (2015) has been extensively used in off-policy, model- free learning (Schaul et al. (2015); Wang et al. (2016); Hessel et al. (2017), as way to enable the learning agent to focus on high error samples from the replay buffer, and thus converge faster good quality agent. We extend this notion to episodic (femporally-sequenttta) sampling, where sequence {(o,, at, rt) fott is sampled from the buffer with probability
ta+L
Pto X
MM
(6)
Since policy and value learning are performed exclusively on imagined trajectories, we chose model loss base priority measure, since experience is primarily used to train model components. Standard prioritized replay utilizes a sum-tree for efficient sampling. We extend this algorithm to cemporally-sequenttal sampling via tuple-valued sum-tree, where each leaf in the tree (sample) contains both its own sample loss Cs and the episodic sample loss (Er= Cx);; whenever sample is fed through the model, we register 5,, the change in the sample loss, and add this sample to an update queue. Before sampling from the prioritized replay buffer, all samples in the update queue update their own Cú, value and the cumulative ΣrE, C values of all leaves up to distance M with change 8, the non-leaf nodes are then updated based the cumulative loss usual. This update is computationally efficient and only introduces a small ((L) lincar overhead w.r.t the standard sum- tree algorithm. To our understanding, this is the first application of prioritized replay in MBRL.
2.3 POLICY LEARNING
Imagined trajectories. The latent dynamic components (e(s; St-1, a-1, (o st) define fully observable Markov decision process (MDP, Sutton & Barto (2018)); we denote the time indices drawn from these imaginary trajectories by Imaginary trajectories start at true model states s, and then follow the MDP defindd below, the policy and value functions are trained jointly on imagined trajectories on the objectives
max E me
EE [s(s,,,, min Ego 1 (sle - (1)
where (s,) is an exponentially-woighhed average of the bootstrapped value target as defined in Hafner al. (2019a),
(x()) (x))
Eggarre Σh -1 - + T~ h= min(r + k,t II)), プmー
(8)
11 1| A 1 LV (s,)
-

---

Under review as a conference paper at ICLR 2021
Note that we can differentiate through model transitions and through the reward function, so the policy can propagate gradients throughout the transitions and learn directly via gradient descent, rather than relying on policy gradients. Likewise for the value function.
2.4 ARCHITECTURE
We emulate the architecture of Hafner et al. (201 our observation encoder and decoder fo and (o(0; | 8t) are implemented with CNN and transposed CNN respectively, the representation and transition functions Po(St 8t-1, az-1. hg), (s 8t-1, 0+-1) are jointly implemented with an RSSM Hafner et al. (2019b), which splits the state &; into a deterministic component and stochas- tic component, the deterministic component is shared for the representation and transition function, and does depend on the current image embedding h4, All other components implemented as MLPs. We note that only the contrastive component introduces an additional architcctural com- ponent compared to bascline Dreamer, and thus ReaPER incurs very limited additional processing overhead compared to the former method. The experiments with each ReaPER component were de- signed so that minimal or no changes to the architecture were made whenever possible. For details on architecture and hyperparameters refer to section 6.1.
3 RELATED WORK
In this work we combine and extend ideas from diverse sources to address sample efficiency in MBRL. The works of Sekar et al. (2020); Hafner et al. (2019a) propose the use of analytic gradients through latent dynamics train the policy, making efficient use of the world model to reduce the number of environment interactions needed to recover good behaviours from the available environ- ment interactions.
Schaul et al. (2015) proposed the use of prioritized memory replay to enable the agent to focus on high loss samples in the experience buffer. This ensures that examples from rare or novel transitions are adequately explored by the model, reducing the need to collect a large number of transitions before these peculiarities of the environment be learnt. While their work focused on Q-learning, here we extend their work to episodic sampling.
The use of sparse priors and data augmentation to improve performance in machine learning al- gorithms is well documented. Recent works in contrastive learning in particular He et al. (2020); Hénaff et al. (2019); Chen et al. (2020a) have yielded impressive results in other areas of machine learning, and recently Srinivas et al. (2020) applied these notions to augment SAC-like agent with contrastive learning. The use of data augmentation in RL has also proven effective in Laskin et al. (2020), who also identify temporally-consise data augmentation for image-based pipelines in RL.
random cropping as a particularly effective type of
The use of alternative metrics to improve representation learning has been extensively studied Oord et al. (2018); Lee et (2019); Hafner et al. (2019b); one approach to avoid pixel reconstruction relies on the use of bisimulation distance Ferns et al. 12011); Ferns & Precup (2014); van der Pol et (2020), where two states are taken to be similar if for any action sequence effected on these states, the rewards are similar. The work of Zhang et al. (2020) addressed the use of bisimulation in DRL, and their experiments show that their agent robust large nuisance perturbations in the visual input. In section 6.2 we adapt this idea to latent bisimulation distance, but we observe no improvement in terms of sample-effiiencyy
The work of Sekar et al. (2020) utilizes an ensemble of embedding predictors to estimate model uncertainty, this uncertainty is used as an unsupervised reward learn robust world model without extrinsic rewards. In section 6.3 we utilize latent disagreement to train an auxiliary exploration policy for data collection. Our results indicate that this did not improve sample efficiency the low sample regime, further motivating our approach.
4 EXPERIMENTS AND RESULTS
We experimentally show the improvement in sample efficiency over a range of tasks from the Deep- mind control suite. All enviromments feature an articulated robot that the agent controls, with contin-

---

Under review as a conference paper at ICLR 2021
uous actions in the range [-1, 1]. instantaneous rewards line in the range r2 E [0, 1], the maximum episode length is 1000, all environments can accrue most 1000 reward per episode. To iso- late the contribution of each novel component of ReaPER, we first conduct an ablation study over smaller subset of tasks. avoid architecture changes between contrastive and non-contrastive implementations, we render observations on non-contrastive experiments at 64 x 64 pixels, and ex- periments using contrastive loss and random cropping are rendered 80 80 pixels then cropped to ht original 64 x 64 resolution. The results, shown in Table I for the best hyperparmeters, indicate that contrastive augmentation and sparse priors can both improve sample efficiency over the benchmark, achieving higher rewards for the same number of environment interactions. Surprisingly, prioritized episodic replay by itself seemed reduce the performance when compared against the baseline. Prioritized experience replay was, however, effective at improving sample efficiency when paired with the contrastive and sparsity auxiliary losses
Episodic reward average for Dreamer, ReaPER, and the individual components ReaPER (Con- trastive loss, LI and Prioritized Episodic replay, noted Contrast, LI and PER respetively). as function of environment steps, the conjunction of LI regularization and Contrastive loss is also shown (LiContrast). Re- wards are averaged across the caripole balance, cartpole swingup, reacher easy, cup catch, finger spin, walker walk, walker run, and cheetah run environments DMControl. ReaPER consistently outperforms the other options. Prioritized episodic replay by itself does not improve upon the Dreamer baseline, but is effective in conjunction with the rest of the ReaPER pipeline. Conversely, LIContrast under- sedderperforms both L and Contrast methods over all tested hyperparameters, showing that these methods do not compose well unless paired with prioritized episodic replay.
Table 1:
Steps
Dreamer
Contrast
L1
LIContrast
PER
ReaPER
100K
309
349
358
262
263
200
549
663
563
619
452
670
300
636
714
693
691
562
756
400K
682
732
752
758
595
780
500K
707
760
774
741
611
787
We further validate these observations on significantly larger set of experiments. We compare results across 18 environments and observe that, in general, ReaPER is able to obtain the same or greater rewards than Dreamer asymptotically, but it achieves these results in smaller number of environment interactions. Figure 3a shows how the average reward over the benchmark increases as function of environment interactions, and how this metric is improved by the addition the regu- larization proposed in ReaPER. Figure 3b highlights these results by showing the average number of interactions required to achieve or surpass a given reward target for the first time over the bascline. Table shows the number of steps required to reach reward target for Dreamer and ReaPER over the same environments. We observe that ReaPER achieves better results on the benchmark across the training procedure, it also requires less samples achieve these targets; this is especially pro- nounced for higher values of target rewards, since a good model is required for consistently high rewards.
Figure 4 shows the reward vs environment step curves for the best 8 environments. These environ- ments show marked improvement by the proposed ReaPER from the Dreamer baseline, having overall better results at each timestep. Some the environments can continue improving with ad- ditional training time, but the results here highlight that the auxiliary objectives in ReaPER can provide a marked improvement on environments of varying complexity, making model based RL to be more effective learning the task with less environmental samples. We also highlight that this performance increase is also marked by significantly decreased inter-seed variance.

---

Under review as a conference paper at ICLR 2021
Episode Rewards
Dreamer ReaPER
Environment steps
400K
300K
200K
300
100K
100
Environment Steps
100K 200K
300K 400K 500K
200
Episode Rewards
400
600
800
1000
(a) Average reward environment steps
(b) Average time-to-reward
Performance comparison across 18 environments of the DMControl benchmark. Left Figure shows average agent reward function of environment steps. Across all environments, ReaPER achieves consistently better episode rewards for the same number of environment steps. Right figure compares the average number of steps needed first meet exceed a reward target on the benchmark; in this metric Reaper also outperforms Dreamer across the benchmark.
Figure 3:
ReaFtk
304
Figure 4: Performance comparison between Dreamer and the proposed ReaPER across of the best performing environments in the benchmark. mean and standard deviations are computed across 3 seeds. These environments show marked contrast in sample efficiency between the two methods, highlighting the benefit of selecting the appropriate auxiliary objectives during training.
Table 2: Time-to-reward comparison between Dreamer and ReaPER on environments DMCon- trol benchmark. Values indicate the ratio between average number of steps required to reach reward target (expressed percentage of maximum reward achieved by the ReaPER agent). Average num- ber of steps computed over the seeds that reached the target, so the estimate is optimistic. Larger numbers indicate longer times for Dreamer to reach the target reward, numbers preceeded by indicate that Dreamer did not achieve the target before 500k timesteps.
Cartpole Two-poles
Reacher casy
Finger spin
Walker walk
Walker run
Pendulum swingup
Quadruped walk
Hopper hop
Reward%
10%
.70
2.21
.87
2.69
.83
1.10
1.5
30%
1.11
.76
1.68
.90
1.26
1.69
1.13
1.98
50%
2.47
1.15
1.35
.95
1.49
1.75
1.55
1.37
70%
2.15
> 2.46
1.42
1.01
1.43
1.84
1.77
1.64
90%
> 1.01
1.64
1.62
1.49
1.03
1.76
1.09
1.81

---

Under review as a conference paper at ICLR 2021
5 DISCUSION
We study sample efficiency in the context of model based RL and propose ReaPER, agent that learns behaviours on latent imagination and improves upon Dreamer by explicitly introducing aux- iliary objectives that enable the agent only encode and learn the parts of the environment which are relevant for behaviour learning. We ablate each novel component of ReaPER to show how each contributes to the overall agent, and experimentally validate our findings on visual control tasks in the DMControl environment suite.
Our experiments indicate that methods that have individually been shown be effective outside of model-based RL do not necessarily translate to MBRL. Such was the case of latent disagreement for exploration and bisimulation, which have been successfully used for other purposes in RL, but proved to be ineffective at addressing sample efficiency in this context. Furthermore, components that individually do improve sample efficiency can, counterintutively, reduce performance when combined; in the case of ReaPER, this apparent anti-synergy was resolved with the addition and adaptation of prioritized episodic replay.
This paper focuses heavily model-based architectures since we believe MBRL is better suited to handle curriculum learning and continual learning, and environments where reward sparsity in itself a major hurdle to overcome; a potential future direction of our research. We wish to extend this approach for other visual-based tasks that may rely on discrete control.
REFERENCES
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint trXiv:2002.05709 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.06 04297, 2020b.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 15/1 2015.
Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM Journal on Computing, 4066;:662--1114, 2011.
Norman Ferns and Doina Precup. Bisimulation metrics are optimal value functions. UAI, pp. 210-219. Citeseer, 2014.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 10122, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01e 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference Machine Learning, pp. 2555-2565. PMLR, 2019b
Kaiming Hc. Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervised visual representation lcarning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729 9738, 2020.
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint /905. 2019.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv: 1710.0228.. 2017.
Zhimin Hou, Kuangen Zhang, Yi Wan, Dongyu Li, Chenglong Fu, and Haoyong Yu. Off-policy maximum entropy reinforcement learning: Soft actor-critic with advantage weighted mixture pol- icy (sac-awmp). arXiv preprint arXiv:2002.028820 2020.

---

Under review as a conference paper at ICLR 2021
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction variational methods for graphical models. Machine learning, 37(2))183 233, 1999.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv: 00374, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.694 2014.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. arXiv preprint arXiv:2004. 14990, 2020.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critie: Deep reinforcement learning with a latent variable model. arXiv preprint #rXiv:1907.0095. 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: w1Xiv:1807.03748 2018.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay, arXiv preprint arXiv:1511.05990 2015.
Ramanan Sekar, Olch Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning explore via self-supervised world models. arXiv preprint arXiv:2005.055 05900, 2020.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. arXiv preprint arXiv:2004.04 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv: 100.00090, 2018.
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm_control: Software and tasks for continuous control, 2020.
Elise van der Pol, Thomas Kipf, Frans A Olichoek, and Max Welling. Plannable approximations to mdp homomorphisms: Equivariance under actions. arXiv preprint arXiv: /963, 2020.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2000 2016.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.


Reviews:
[{"review_id": "weSWHbrRV4z", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends and ablates several modifications to Dreamer, the current SOTA in MBRL for control tasks. More specifically, they introduce a L1 sparsity prior on the representations, a contrastive loss with random crop data augmentation and prioritized experience replay. They also test additional changes in the Appendix, like Bisimulation distances (from Zhang et al, 2020), and exploration via latent disagreement (from Sekar et al, 2020.).\n\nOverall, this is a well executed model exploration, and if the code gets open-sourced will be very valuable for other researchers to build upon. The results aren’t extremely strong, but do appear significant, so might still be valuable to share more broadly.\n\nComments/questions:\n  1. Appendix 6.2 and 6.3 could use some contextualization, as they seem to come out of the blue, even though they present some valuable extra baselines. It’d be valuable to point to them from the main text as well (it is only mentioned in the Related work quite quickly).\n  2. Figure 3b) should have its y-axis flipped (i..e. 0 at the bottom). It was pretty confusing to see the orange line flattening and having to interpret this as being better than the blue curve.\n  3. For bisimulation, the work of van der Pol, 2020 might deserve a citation as well.\n\n* [van der Pol, 2020], https://arxiv.org/abs/2002.11963", "extra_scores": {}, "extra_text": {}}, {"review_id": "uU4h5wuHIHp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a model-based RL method which is built upon a baseline DREAMER [1] with additional components which empirically improve performance:\n* Prioritised experience replay\n* Temporally-consistent data augmentation with a contrastive loss\n* L1 regularisation\n\n# High-level comment:\n* The contribution of authors is a combination of all the proposed ideas into one framework. The ideas are not novel and were considered in other works, so the contribution is really is a combination of these. From the experiments proposed by the authors, it's not easy to infer the impact of each of these ideas on the performance, the ablations do not provide a clear consistent signal and lack of explanation on why it provides an improvement. On top of that, the paper lacks experimental details, such as how the hyperparameters were optimised and what ranges were considered. It makes it hard to reproduce the experiments. More importantly, the proposed method underperforms with respect to the model-free variants which use one of these ideas, therefore it is not clear how useful the method could be for the community.\n\n# Strengths of the paper:\n* Paper is relatively clearly written and high-level ideas are easy to follow\n* Equations (1) are very helpful to be able to compare the differences of the model to any other baseline\n\n# Weaknesses:\n* It is hard to judge whether the proposed combination would be a method to use due to multiple reasons. First of all, the proposed ideas are not novel and were considered in [2, 3, 4] for data augmentation, in [5] for prioritised replay. More importantly, the proposed method underperforms with respect to the model-free variants such as CURL [2] and others (cited). Why would we want at all to use a complex model-based method if what we could do is to simply use the data augmentation and get much better performance? Not clear.\n* It is very good that the authors provided the ablations, but I think they do not completely answer the question of impact of each of the component. For example, for data augmentation + contrastive loss, is the most of the positive impact coming from the random crops or with both crops plus contrastive loss ? In some cases, adding L1 + Contrast underperforms with respect to Dreamer, why is it the case ? Is there any other difference besides these 3 components, such as hyperprameters, architecture or/and implementation ? If yes, the authors should report the results of their method without these 3 tricks to convince reader that it matches the performance of Dreamer.\n* The impact of prioritised replay is negative but when combined with other tricks, is positive. Why is it the case ? There is no explanation of this phenomenon in the paper. Maybe it would only work for control suite and not for other problems ?\n* Figure 3 shows that the difference with respect to Dreamer is quite weak.\n* Figure 4 shows performance on \"best\" environments. It would be important to see the performance on \"worst\" environments. How do the results look like?\n\n# Conclusion\nGiven quite weak performance of the method with respect to much simpler model-free baselines and not a clear effect of added ideas without a clear explanation on why it is the case, I would recommend to reject the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "d-0wWMZOnhO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to improve sample-efficiency in model-based reinforcement learning (MBRL). The approach termed ReaPER is based on Dreamer (Hafner et al. 2019) and the paper integrates several ideas from prior works that are known to improve sample efficiency. Specifically, the paper borrows the contrastive learning idea from CURL (Srinivas et al. 2020) improving learning from pixels, and extends the prioritized episodic replay (PER, Schaul et al. 2015) to a more efficient version. Experiments on DeepMind control suite show that the proposed approach outperforms dreamer across 8 environments. The paper also conducts ablation study to validate the effectiveness of each design choice.\n\n\n**Pros:**  \n++ The paper combined ideas from recent papers to improve sample efficiency of MBRL. The ablation studies shed some light on how each design helps with sample efficiency.  \n++ Additional experiments to show the ineffectiveness of latent disagreement for exploration and bisumulation for improving sample efficiency.  \n++ The overall model outperforms prior work, Dreamer.  \n++ The paper is relatively easy to read.  \n\n**Concerns:**  \n-- My biggest concern is that the paper lacks technical novelty. The overall learning framework is based on Dreamer (Hafner et al. 2019). The proposed components, contrastive learning (Srinivas et al. 2020) and prioritized episodic replay (PER, Schaul et al. 2015), are proposed by prior works and have already shown their effectiveness for improving sample efficiency in reinforcement learning. I don’t think applying these techniques to MBRL makes a compelling case.  \n-- The ablation studies in Table 1 seem to raise questions about the efficacy of the proposed components. For example, L1+Contrast seems to perform worse than Dreamer, L1, and Contrast. Similarly, PER alone performs the worst. It’s unclear why ReaPER, i.e., combining L1+Contrast and PER, suddenly brings the performance from the worst to the best. The magic combination that suddenly boosts the performance prevents an intuitive interpretation of the results and obscures the true effectiveness of each component.  \n-- I would recommend plotting the performance of each environment in the ablation study just like Fig. 7, 9, 10 in (Hafner et al. 2019). Instead of providing an average reward across different environments, providing these plots will help better understand the convergence properties of each method.  \n\n\n**Rating Justification:**  \nI commend the authors for their attempts to the important problem of sample efficiency in MBRL. However, the paper’s main issue is the lack of novelty as it is mainly combining existing techniques in very similar settings. Thus, I feel it has not met the bar of ICLR.\n\n\n**Additional Comments:**  \n-- “Complimentary objectives” -> “Complementary objectives”  \n-- Missing right bracket in Eq. 8  \n-- I don’t think Algo 1 should occupy an entire page. The space could be saved for more detailed and self-contained explanations for Dreamer.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SDX_gEuuOya", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " ##########################################################################\n\nSummary:\n \nThis paper investigates several potential sources of sample-inefficiency and proposes an integrated algorithm to improve the sample-efficiency of model-based reinforcement learning. This paper builds upon Dreamer and incorporates sparse, self-supervised, contrastive model representations and efficient use of past experience. The experiments on DeepMind control suit show that the proposed method outperforms dreamer and the ablation study further verifies the contribution of each component of the proposed algorithm.\n\n##########################################################################\n\nPros: \n \n1. This paper tackles a valuable problem of improving the sample efficiency of model-based RL. \n\n2. The idea of incorporating sparse, perturbation-invariant, contrastive model representations and Prioritized experience replay is interesting and promising.\n \n3. The paper is well written and the results section is well structured. They outperform baseline methods on a popular benchmark and conduct an ablation study.\n \n##########################################################################\n\nCons: \n \n1. Novelty is limited. They use a series of techniques that have already been respectively proved to be useful. \n\n2. This paper has a much more complex architecture than Dreamer but the performance improvement is not very significant as shown in Figure 3 but\n3. The authors claim the contrastive and sparse representations are useful for sample-efficiency but they do not visualize and check the learned representations. They should first show their proposed loss functions can truly lead to more sparse and contrastive representations and then show such representations can contribute to improve the sample efficiency.\n\n##########################################################################\n\nQuestions and suggestions:\n1. In Section 2.2, why do the authors choose the model error for the sampling probability. Can you use the policy error or value error？\n\n2. As shown in Table 1, contrastive learning and L1 regularization can individually improve the performance, but why contrastive + L1 has a negative result? Can you give some insights?\n\n3. For Table 2, plots of learning rate will be more intuitive.\n\n4. How to determine the weighting factor in Equation 5?\n\n\n ##########################################################################\n\nMinor comments: \n1. There are too much white space on page 3.\n2. In Figure 2, it is better to compare ReaPER with Dreamer on the same state.\n3. In Table 1, what does “PER” mean?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores losses and other training details to produce a model-based agent for pixel-input continuous control problems.  The authors present a rainbow-like approach that combines various separate innovations into a single system.  They show an improvement over a previous baseline on this class of problem, and break down the contributions of the various components.

Though the paper was seen as clearly written, fundamentally, the reviewers did not feel they gained insight through the presentation of the experiments.  For example, one quirk brought up by multiple reviewers is that some combinations of methods show worse performance, but then adding yet another method makes things improved relative to baseline (the authors clarified that this was with the same hyperparameters).  Reviewers found this a bit confusing and insufficiently explored (i.e. was this just hyperparameter tuning or does just the right selection tricks actually need to be combined).  This confusion around method combinations is perhaps relatively minor by itself but indicative of how this paper did not build intuition for the reviewers.  Moreover, none of the reviewers were impressed by the magnitude of improvement over the baseline dreamer agent.  While it was acknowledged the the set of methods improved things, the reviewers felt that each innovation had already been independently validated as likely to improve sample efficiency, so the fact that they did so together was not especially insightful.    

I'd like to clarify for the authors that I believe this work was, in many respects, technically well executed.  Ultimately, based on the reviews and my own assessment, I don't think the scope was sufficiently ambitious considering the competitiveness of this conference.  While it is useful to occasionally produce summary works which pool a set of separate innovations, such papers must be insightful to readers, aggregate a sufficiently large number of innovations, and/or show striking performance gains.  The final reviewer scores are 4, 5, 6, 4. 



Decision: Reject