On Trade-offs of Image Prediction in Visual Model-Based Reinforcement Learning
Authors: Mohammad Babaeizadeh
Mohammad Taghi Saffar
Danijar Hafner
Dumitru Erhan
Harini Kannan
Chelsea Finn
Sergey Levine
Venue:  (2021)
Keywords: world models
model based reinforcement learning
latent planning
model-based reinforcement learning
model predictive control
video prediction

Abstract: Model-based reinforcement learning (MBRL) methods have shown strong sample efficiency and performance across a variety of tasks, including when faced with high-dimensional visual observations. These methods learn to predict the environment dynamics and expected reward from interaction and use this predictive model to plan and perform the task. However, MBRL methods vary in their fundamental design choices, and it there is no strong consensus in the literature on how these design decisions affect performance. In this paper, we study a number of design decisions for the predictive model in visual MBRL algorithms, focusing specifically on methods that use a predictive model for planning. We find that a range of design decisions that are often considered crucial, such as the use of latent spaces, have little effect on task performance. A big exception to this finding is that predicting future observations (i.e., images) leads to significant task performance improvement compared to only predicting rewards. We also empirically find that image prediction accuracy, somewhat surprisingly, correlates more strongly with downstream task performance than reward prediction accuracy. We show how this phenomenon is related to exploration and how some of the lower-scoring models on standard benchmarks (that require exploration) will perform the same as the best-performing models when trained on the same training data. Simultaneously, in the absence of exploration, models that fit the data better usually perform better on the down-stream task as well, but surprisingly, these are often not the same models that perform the best when learning and exploring from scratch. These findings suggest that performance and exploration place important and potentially contradictory requirements on the model.

Under review as a conference paper at ICLR 2021
ON TRADE-OFFS OF IMAGE PREDICTION IN VISUAL MODEL-BASED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Mldell-ased reinforcement learning (MBRL) methods have shown strong sample efficiency and performance across variety of tasks, including when faced with high-dimensional visual observations. These methods learn to predict the envi- ronment dynamics and expected reward from interaction and use this predictive model to plan and perform the task. However, MBRL methods vary in their funda- mental design choices, and there is no strong consensus in the literature on how these design decisions affect performance. In this paper, we study number of design decisions for the predictive model in visual MBRL algorithms, focusing specifically methods that use predictive model for planning. We find that a range of design decisions that are often considered crucial, such as the use of latent spaces, have little effect on task performance. A big exception to this finding is that predicting future observations (i.c., images) leads to significant task performance improvement compared to only predicting rewards. We also empirically find that image prediction accuracy, somewhat surprisingly. correlates more strongly with downstream task performance than reward prediction accuracy. We show how this phenomenon is related to exploration and how some of the lower-scoring models on standard benchmarks (that require exploration) will perform the same the best-pent pest--erforming models when trained on the same training data. Simultaneously. in the absence of exploration, models that the data better usually perform better on the downstream task as well, but surprisingly. these are often not the same models that perform the best when learning and exploring from scratch. These findings suggest that performance and exploration place important and potentially contradictory requirements on the model.
INTRODUCTION
The key component of any model-based reinforcement learning (MBRL) methods is the predictive model. In visual MBRI., this model predicts the future observations (i.e., images) that will result from taking different actions, enabling the agent to select the actions that will lead to the most desirable outcomes, These features enable MBRL agents to perform successfully with high data- efficiency (Deisenroth & Rasmussen 2011) in many tasks ranging from healthcare (Steyerberg et al., 2019). to robotics (Ebert et al. 2018) and playing board games (Schrittwieser et 2019).
More recently. MBRL methods have been extended to settings with high dimensional observations images), where these methods have demonstrated good performance while requiring substantially less data than model-free methods without explicit representation learning (Watter et al., 2015 Finn
Levine, [0017] Zhang et al.) 2018 Hafner et al.) 2018: Kaiser al. 2020). However, the models used by these methods, also commonly known as World Models (Ha & Schmidhuber] 2018). vary in their fundamental design. For example, some recent works only predict the expected reward (Oh) et al. 2017) or other low low-dimensional task-relevant signals (Kahn et al., 2018), while others predict the images as well (Hainer et al.) 2019). Along a different axis, some methods model the dynamics of the environment in the latent space (Hafner et al.) 2018), while some other approaches model autoregressive dynamics in the observation space (Kaiser et al. 2020).
Unfortunately, there is little comparative analysis of how these design decisions affect performance and efficiency, making difficult to understand the relative importance the design decisions that have been put forward in prior work. The goal of this paper is understand the trade-off's between the design choices of model- based agents. One basic question that we ask is: does predicting images actually provide benefit for MBRL methods? A tempting alternative to predicting observations is to simply predict future rewards, which, in principle, gives a sufficient signal to infer all task-relevant information. However, will see, predicting images has clear and quantifiable benefits in fact,

---

Under review as a conference paper at ICLR 2021
observe that accuracy in predicting observations correlates more strongly with control performance than accuracy of predicting rewards.
Our goal is to specifically analyze the design trade-off's in the models themselves, decoupling this as much as possible from the confounding differences in the algorithm. While a wide range different restrict our analysis arguably the simplest
algorithms have been put forward in the literature, class of MBRL methods, which train a model and then use it for planning without any explicit policy. While this limits the scope of our conclusions, it allows us draw substantially clearer comparisons.
The main contributions of this work are two-fold. First, we provide a coherent conceptual framework for high-level design decisions in creating models. Second, investigate how each one of these choices and their variations can affect the performance across multiple tasks. We find that:
1. Predicting future observations (i.e. images) leads to significant task performance improvement compared to only predicting rewards. And, somewhat suprisingly, image prediction accuracy correlates more strongly with downstream task performance than reward prediction accuracy.
2. We show how this phenomenon is related to exploration:
Some of the lower-scoring models standard benchmarks that require exploration will perform the same as the best-performing models when trained on the same training data, In the absence of exploration, models that fit the data better usually perform better on the downstream task, but surprisingly. these are often not the same models that perform the best when learning and exploring from scratch.
3. A range of design decisions that are often considered crucial, such as the use of latent spaces, have little effect on task performance.
These findings suggest that performance and exploration place important and potentially contradictory requirements on the model. We will open-source our implementation that be used reproduce the experiments and can be further extended to other environments and models.
RELATED WORK
MBRL is commonly used for applications where sample efficiency is essential, such as real physical
systems (Deisenroth & Rasmussen 2011 Deisenroth et al., 2013 Levine et al.] 2016) or health-
care (Raghu al. [018 Yu al.] 2019 Steyerberg et al.) 2019). In this work. our focus is on settings
with high-dimensional observation space (i.e., images). Scaling MBRL to this setting has proven to
be challenging (Zhang et al. 2018), J) but has shown recent successes (Hafner et al. 2019
Watter et al.
2015
Levine et al. [216 Finn et al. 2016b, |Banjjamali et al. 2017 Oh et al. 2017
Zhang et al.
2018
Ebert et al. 2018 8 Hafner et al. 2018 Dasari et al. 2019
Kaiser et al. 2020).
Besides these methods, number of works have studied MBRL methods that do not predict pixels,
and instead directly predict future rewards (Oh et al. 2017 Liu et al., 2017] /Schritwwieser et al. 2019
Sckar et al. 2020), other reward-basee quantities (Gelada al. [219)) features that correlate with the reward or task (Dosovitskiy & Koltun| 2016 Kahn et al. (2188). It might appear that predicting rewards is sufficient perform the task, and reasonable question to ask is whether image prediction accuracy actually correlates with better task performance or not. One of our key findings is that predicting images improves the performance of the agent, suggesting way to improve the task performance of these methods.
Visual MBRL methods must make a number of architecture choices in structuring the predictive model. Some methods investigate how to make the sequential high-dimensional prediction problem
casier by transforming pixels (Finn et [0166a] De Brabandere et al. 2016
Liu et al.) 2017) or
decomposing motion and content uuuyakov et al. Dentoon
Denton et al. 2017
Hsich et al. 2018)
Wichers et al. 2018 Amiranashvili et al. 2019). Other methods investigate how to incorporate
stochasticity through latent variables (uue et al.) 2016 Babaeizadeh et al. 2018: Denton & Fergus
2018 Lee et al. 2018 Villegas et al. 2019), autoregressive models (Kalchbrenner et al. 2017] Reed
al. 2017 | Weissenborn et al. 2019), flow-based approaches (Kumar et al. 2019) and adversarial
methods (Lee et al. 2018). However, whether prediction accuracy actually contributes to MBRL performance has not been verified in detail on image-basee based tasks. We find a strong correlation between image prediction accuracy and downstream task performance, suggesting video prediction is likely fruitful area of research for improving visual MBRL.
2

---

Under review as a conference paper at ICLR 2021
a
a
ar+1
h
hED
h
h++
h
h
O
O 11
O
O++11
11
Ol
7t-1
Latert space Transition Latert-based Reward
Lakant speice Transilion Observation-Be bassdd Reward
observation predictin Reward prediction only
LR
L
R
a
hi
h
Input
Prediction
Transition
O
0
71
Of
O
Observatio space Transition Obsenvaiin based Reward
Observatio space Transiion Latent Reward
OrOR
OTLR
Figure I: The possible model designs for visual MBRL, based on whether or not to predict images. The rightmost model (R. only predicts the expected rewards conditioned on future actions and previous observations, while other designs predict the images well. These designs can model the transitions (dynamics) either in the latent space (CT) or observation space (OT). The input source for the reward prediction model can be the predicted images (OR) or the latent state of the model (Cx). Note how these models differ in their training losses (in the case of R) and how the errors are back propaappp,,,,,, either directly through the latent space or via predicted targets.
3 A FRAMEWORK FOR VISUAL MODEL-BASER RL
MBRL methods must model the dynamics of the environment conditioned the future actions of the agent, as well as current and preceding observations. The data for this model comes from trials in the environment. In an online setting, collecting more data and modeling the environment happens iteratively, and in an offline setting, data collection happens once using a predefined policy. We are interested in cases where the observation state is pixels, which usually not enough reveal the exact state of the environment. Since our goal is specifically to analyze the design trade-offs in the models themselves, we focus our analysis only on MBRL methods which train model and use it for planning, rather than MBRL methods that learn policies or value functions. We defer discussion and analysis of policies and value functions to future work.
Therefore, we consider the problem as a partially observable Markov decision process (POMDP) with a discrete time step t E [L, 71, hidden state S,, observed image o,, continuous action a, and scalar immediate reward rr. The dynamics are defined as a transition function S, p(s, 8;-1, ar- 1). observation function o, ~ p(o, sz) and reward function r ~ p(r, s,). In RL, the goal is to find policy p(a//ocs T',,, agi) that maximizes the expected discounted sum of future rewards Ep [Σin 7r2).. where is the discount factor, T is the task horizon and the expectation is over actions sampled from the policy. In MBRL, approximate the expected reward by predicting its distribution conditioned on the previous observations and the future actions p(r: /ost, azz). and then search for high-reward action sequences via a policy optimization method such as proximal policy optimization (PPO) (Schulman et al. 2017) or planning algorithm like the cross-entropy method (CEM) (Rubinstein) 1997 Chua al. 2018). In this paper we focus on the latter.
Given the assumption that the enviromment is partially observable and the ground-truth states are not accessible, the models in visual MBRL can be divided into five main categories, depending on training signals and learned representations (Figure[T). The first category approximates the expected rewards conditioned on future actions and previous observations p(r; O a>+) without explicitly predicting
images (Oh et al. 2017 Dosovitskiy & Koltun 2016 Racanière et al. 2017 Liu al. 2017 Kahn ctt al. 2018 Schrittwieser et al. 2019). In contrast, the next four categories learn to predict the next observations 0 in addition to the reward, which results in learned latent representation.
Given this latent space, it is possible to: (1) Model the transition function of the environment in the observation space: θ.+1 p(o, P<t,az) directly in the latent space: h, p(h. h a,) where h, is the hidden state of the model at time step t (2) Predict the future reward using the learned latent space: rt p(r: he) or from the predicted future observation: rt p(r: @
3

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS AND ANALYSIS
Our goal in this paper is to study how each axis of variation in the generalized MBRL framework, described in Section 3 impacts the performance of the agent. To this end, we will not only study end- to-end performance of each model on benchmark tasks, but also analyze how these models compare when there is no exploration (i.e. trained from the same static dataset), evaluate the importance of predicting images, and analyze a variety of other model design choices.
4.1 EXPERIMENT SETUP
Planning Method Studying the effects of each design decision independently is difficult due to the complex interactions between the components of MBRL (Sutton & Barto (2018). Each training run collects different data, entangling exploration and modeling. The added complexity of policy learning further complicates analysis. To isolate the design decisions pertaining to only the predictive model, we focus on MBRL methods that only learn a model, and then plan through that model to select actions (Chua al. 2018 Zhang et al. 2018 Hafner et al. 2018). ). We use the cross-entropy method (CEM) (Rubinstein 1997 to optimize over the actions under a given model, following prior work. We also include Oracle model in our comparisons, which uses the true simulator predict the future rewards. Since we are using the same planning algorithm for all the models, the Oracle approximates the maximum possible performance with our planner. Note, however, that given the limited planning horizon of CEM, the Oracle model is not necessarily optimal, particularly in tasks with sparse and far reaching rewards.
Environments We seven image-based continuous control tasks from the DeepMind Con- trol Suite (Tassa et al., 2018). These environments provide qualitatively different challenges. cheatah_run and wa walker_wwk exhibit larger state and action spaces, including collisions with the ground that are hard to predict. finger-spin includes similar contact dynamics between the finger and the object. cartpole.balanee require long-term memory because the cart can move out of the frame while ball in, cup_euppppcooo catch also have sparse reward signal, makes them hard to plan for given a short planning horizon. In all tasks, the only observations are 64 x 64 x 3 third-person camera images (visualized Figure 3 in appendix).
-wsinguup requires long-term planning. reacher.easy and
Implementation and Hyper-Parameter tuning Given the empirical nature of this work, our observations are only as good as our implementations. To minimize the effect of hyper- parameters tuning, we chose not to design any new models, and instead used existing high performance well- known implementations. The only exception to this was R., for which we tested multiple architectures and used the best one based on asymptotic performance (see appendix for details). While there are going to be numerous shortcomings for ANY implementation, in close collaboration with the original authors of these implementations, we took great care in making these choices so as to minimize the performance effects of implementation details. Moreover, we conducted comprehensive hyper-pane parameter optimization for cheetah. run and relied on the common practice of using the same set of hyper--arameters across other tasks (Hafner et al. 2018). Given all this, are confident that the performance of our implementations cannot be improved significantly on these tasks and for these same methods by further tyyer-parameee tuning. Details of these implementations, the planning method, model variations, and perea-paaressers can be found in the appendix.
4.2 ONLINE PERFORMANCE
First, we compare the overall asymptotic performance, the stability of training, and the sample efficiency of various model designs across all of the tasks in the online setting. this setting, after every ten training trajectories, collected with the latest version of the model, we evaluate the agent on one episode. For training episodes, we add random planning noise for better exploration, while the evaluation episodes there is no additional noise. We train each model for total of 1000 episodes where each episode is 1000 steps long with an action repeat of four. Therefore, each model observes 250K training samples of the environment during its entire training (1M environment steps), while evaluated on a total number of 25K test samples across 1000 evaluation episodes. The possible reward range per environment step is (0, 1]: therefore, the maximum possible score each task is 1000.
4

---

Under review as a conference paper at ICLR 2021
cheetah run
TRacher
Epolo swingup
Cal Cartpole balance
630
103
332
4003
232
230
233
600
137
200X
1026
1036
200K
291
2006
D006
134
2036
430x
6.0c
6006
2006
600K
ROCK
Enwironment Step
Environment Stop
Environment
ormment
Finger spin
cup. caacch
walkeer walk
630
1000
3317
592
0,0
330
LxCR
2
Oracle
4026
5006.
6006
Erviromment
Environment Step
Environment
Figure 2: The performance of different models the online setting. The æ-axis is the number of environment steps and the y-axis is the achieved score by the agent in evaluation episode. Each curve represents the average score across three independent runs while the shadow the standard deviation. The model designs are from Figure From this graph, models which predict images can perform substantially better than the model which only predicts the expected reward (R). In some cases, these models perform even better than the Oracle which suggests that there not a clear relation between prediction accuracy and performance, given the fact that the Oracle is the perfect predictor. The curves smoothed by averaging in moving window of size ten.
be clearly
that the
The results are shown in Figure There are number of patterns that we note in these results. First, consistently under-performs compared the rest of the models, attaining lower scores and substantially worse reward distributions in Figure 3 Second, some models clearly perform better than others on each given task e.g. LTCR on ball
From this, we might suppose that these models are better able fit the data, and therefore lead to better rollouts. However, as we will see the next section, this may not be the case. Third, some rare cases, some models outperform the Oracle model c.g. OfOR cartpole.balaa planning horizon of CEM, the Oracle model is not necessarily optimal, particularly in tasks with sparse and far reaching rewards such rea cher easy. This also suggests that there is not clear relation between prediction accuracy and performance, given the fact that the Oracle the perfect predictor.
Note, that given the limited
EyCr
0,0g
325
2.0
2.50
3.6
1.5
Prbbaiityy
2.5
322
0.5
6.5
20 8.00 05
2. 25
0.00 60 200
Figure 3: This figure illustrates the distribution of the rewards in trajectories collected by best performing models and R in the online setting for cheetah. run. Since the rewards each frame is [0, 1] and the action repeat is set to four, the observed reward is always in [0, 4] (æ-axis)) This graph demonstrates how L LR and O,Or managed to explore different subset state space with higher rewards (compared to R) which directly affected their performance (compare at Figure 2 This shows how entangled exploration and model accuracy are which makes it hard to analyse the effect of each design decision in the online settings.
5

---

Under review as a conference paper at ICLR 2021
cheetah
reacher easy
cartpole
cartpole finger ball.in.cup walker
run
swingup
balance
spin
catch
walk
R 216.8
996.0
258.3
415.2
171.1
985.1
299.6
OTOR
502.6
981.1
684.7
978.5
148.0
910.3
567.5
OrLR 489.9
999.0
406,0
923.5
244.1
949.1
730.6
LT 502.2
994.1
413.4
543.5
373.1
996.0
552.0
281.7
993.0
281.0
411.0
259.3
932.0
383.3
Table 1: The asymptotic performance of various model designs in the offline setting. In this setting there is no exploration and the training dataset is pre-collected and fixed for all the models. The reported numbers are the 90th percentile out of 100 evaluation episodes, averaged across three different runs. This table indicates significant performance improvement by predicting images almost across all the tasks. Moreover, there is a meaningful difference between the numbers in this table and Figure 2 some of best-performing models in this table performed poorly in Figure
signifying the importance of exploration in the online settings. Please note how
cheetah
reacher easy
cartpole
cartpole balance
finger ball in cup walker spin
run
swi swingup
catch
wa
R
0.2274
0.0226
0.4152
0.1690
0.0049
0,0055
0.2258
OrOr
0.0432
0.0612
0.2447
0.2175
0.0068
0.0010
0.1779
OrLR
0.0472
0.0280
0.2169
0,2310
0.0099
0,0009
0.1814
LT 0.0781
0.0045
0.1183
0.1409
0.0025
0.0019
0.1367
LTOR 0.2022
0.0601
0.4096
0.1649
0.0033
0.0018
0.1984
S) -0.98
-0.62
-0.53
0.86
-0.47
0.58
-0.61
Table 2: Median reward prediction error (Ln) of each model across all of the trajectories in evaluation partition of the offline dataset. This table demonstrates generally better task performance for more accurate models in the offline setting, when compared with Table The last row reports the Pearson correlation coefficient between the reward prediction error and the asymptotic performance for each task across models. This row demonstrates the strong correlation between reward prediction error (Lx) and task performance (S) in the absence of exploration. In cases which all models are close to the maximum possible score of 1000 (such as ball wiiiiinnnpppp. cwp.cctcch the correlation can be misleading because a better prediction does not help the model anymore.
is important to emphasize that, in the online setting, agents may explore entirely different regions of the state space, which means they will be trained different data, visualized in Figure Because the models are all trained on different data, these results mix the consequences of prediction accuracy with the effects of exploration. We hypothesize that these two capabilities are distinct, and isolate modeling accuracy from exploration in the following section by training all of the models on the same data, and then measuring both their accuracy and task performance. To further understand this complicated relationship between model accuracy, exploration, and task performance, we next compare the different model types when they are all trained on the same data.
4.3 OFFLINE PERFORMANCE
In the offline setting, all of the training trajectories are pre-collected, and therefore all of the models will be trained the same data. We create this controlled dataset by aggregating all of the trajectories from all of the model variations trained in the online setting (Section 4.2). This dataset contains 1.25M training and 125K testing samples per task. We train each model on this dataset, and then evaluate its performance on the actual task. Table [ contains the results of this experiment while Figure illustrates the prediction error of each model for cheetah.run.
These results show very different trend from that seen in Section For example, although OrLR was a poor performer in the online setting on walkor.walk, it performs substantially better on the same task when all the models are trained from the same data. As another observation, almost all of the models managed to achieve very high scores in the offline setting on reacher reacher.easy and ball bal..in.eppccater that result in good performance on given dataset are not necessarily the models that lead to best performance during exploration. We speculate that some degree of model emror can result in optimistic
while many struggled in the online setting. This suggests that models

---

Under review as a conference paper at ICLR 2021
cheutah run
FOAC THE eaiyy
cartadle swingup
caripole balance
330
2000
200
420
500
400
350
230
50
200
2006
KOOK
2008
400e
2008
saa dape
Stop
finger
ball_ingccpp
2003
100%
6%
77%
2%
402
Loss Mask
(bout
100
200C
BEEK
2006
SOOK
KOOK
REDC
2006
COOK
893C TM
100%
25% 0%
Enviromment Stop
Environment Stop
Environment Step
Figure 4: The effect of limiting the image prediction capacity on the performance of the agent. The graphs follow the same format as Figure 2 Each model is a variant of L. LR. which predicts the entire image, or part of it (center cropped) no image at all. This graph shows that predicting more pixels results in higher performance as well as more stable training and better sample efficiency. Note that in this experiment, the mode still observes the entire image, however, predicts different number of pixels, forced by a loss mask. The labels indicated what percentage of the image was being predicted by the model.
exploration strategies that make easier to obtain better data for training subsequent models, even such models do not perform as well on given dataset as their lower-error counterparts. Please check the appendix for more on this speculation. However, there a strong correlation between prediction errors in Table 2 and the scores in Table I suggesting that in absence of exploration and given the same training data, any model that fits the data better performs the given task better as well.
We would clarify however that analysis does necessarily imply that models with worse prediction accuracy result in better exploration, though that is one possible interpretation of the results. Instead, we observing that simply modeling the current data accurately as possible does not by itself guarantee good exploration, and models that are worse at prediction might still explore better during planning. We speculate that this can be explained in terms of (implicit) optimism the face of uncertainty (Auer 2002) or stochasticity (Osband & Van Roy 2017 Levine et al. 2020
4.4 THE IMPORTANCE OF IMAGE PREDICTION
The results in Section 4.2 and Section 4.3 differ to great extent, but there is a common pattern between the two: R achieves lower scores compared to other variations in both offline and online settings. To provide further empirical support for this observation, we explore the effect of predicting fewer pixels LR. In this experiment, we limit how many pixels L LR predicts by center- cropping (Figure and resizing (Figure Glin appendix) the target image. It is important to note that the model still observes the entire image; however, predicts different number of pixels.
These results suggest that prediction of images and its reconstruction actually plays an important role in visual MBRL, and simply substituting reward prediction not sufficient. Methods that predict images, rather than just reward, perform better in both the online (Section 4.2) and offline (Section (.3) settings and, as shown in this section, the performance of these models scales with the amount of pixel supervision that they receive. This question raises an immediate question: does the accuracy of image prediction actually corresponds to better task performance? This question particularly important because predicting the images is computationally expensive, given the fact that observation space is typically digh-dimensional.
To explore this correlation, scale down O7OR and LT multiple levels to limit their modeling capacity, and thereby increase their prediction error (details in appendix). Then, we calculate the Pearson correlation coefficient p between reward prediction error and observation prediction error to the asymptotic performance of the agent in the online setting. The results of these experiments summarized in Table This table shows strong correlation between image prediction error and
7

---

Under review as a conference paper at ICLR 2021
OTOR
LTCR
Task
(Lo,8)
((
Ø8
(Lo,
[LCR,
0S
cheetah.run
-0.96
-0.09
338
111
-0.92
0.17
566
60
ca cartpole_swinge
-0.94
-0.69
572
112
-0.74
0.72
503
51
cartpole_balance
-0.93
-0.85
877
108
-0.81
0.93
647
93
walker walk
-0.72
0.94
308
89
-0.26
-0.84
489
149
reacher.easy -0.36
0.63
644
277
-0.88
0.25
992
10
finger_spin -0.08
0.93
185
87
-0.03
0.35
417
25
ball.in.cup.catcti
0.08
0.33
925
32
-0.03
0.75
997
3
Table 3: Pearson correlation coefficient between image prediction emor Co, reward prediction enror LR and asymptotic score S. To calculate the correlation, we scaled down OJOR and CT LR. at mul- tiple levels to limit their modeling capacity and thereby potentially increase their prediction error. js and 0g are the average and the standard deviation of asymptotic performances across different scales of each model. In cases with low standard deviation of the scores (such as ball in cup cat eh), meaning all version of the models did more or less the same, the correlation can be misleading. This table demonstrates the strong correlation between image prediction error and task performance.
asymptotic performance. This evidence suggests that MBRL methods can substantially perform better using a better next frame predictive model, especially for more visually complex tasks.
Surprisingly, Table also suggests that reward prediction error does not have strong correlation to asymptotic performance. Even in some cases, there is positive correlation between the reward prediction error and the asymptotic performance prediction error is high which is counter-intuitive One potential explanation is the same relation between model accuracy and exploration Section 4.2 and Section [.3 In some sense, one should the
meaning the agent performs better when the
expect that the reward accuracy correlates negatively with performance, since overfitting observed states, combined with pessimism, can harm exploration by guiding the planning algorithm towards familiar states that have high predicted reward, rather than visiting new states that may have higher reward. More related experiments can be found the appendix.
5 CONCLUSIONS
In this paper. we take the first steps towards analyzing the importance of model quality in visual MBRL and how various design decisions affect the overall performance of the agent. We provide empirical evidence that predicting images can substantially improve task performance over only predicting the expected reward. Moreover, we demonstrate how the accuracy of image prediction strongly correlates with the final task performance of these models. We also find models that result in higher rewards from a static dataset may not perform as well when learning and exploring from scratch, and some of the best-performing models on standard benchmarks, which require exploration, do not perform well sower-scoring models when both trained on the same data. These findings suggest that performance and exploration place important and potentially contradictory requirements the model.
Our results with offline datasets present a complex picture of the interplay between exploration and prediction accuracy, On the one hand, they suggest that simply building the most powerful and accurate models may not necessarily be the right choice for attaining good exploration when learning from scratch. On the other hand, this result can be turned around to imply that models that perform best on benchmark tasks which require exploration may not in fact be the most accurate models. This has considerable implication on MBRL methods that learn from previously collected offline data (Fujimoto et al.) 2019 Wu et al. 2019 Agarwal et al. 2020) a setting that is particularly common for
applications e.g. Roboties (Finn & Levine 2017).
We would like to emphasize that our findings are limited to the scope of our experiments and our observations may vary under different experimental setup. First, we only trained the models with a fixed number of samples. It would be interesting to investigate whether the benefits of predicting images gets diminished or amplified in lower and higher sample regimes, affecting sample efficiency of these models. Second, using policy optimization method is natural next step for expanding these experiments. Finally, is interesting to investigate whether the findings of this paper can be generalized more (or less) visually complex tasks in other domains.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. In International Conference on Machine Learning. 2020.
Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, and Thomas Brox. Motion perception in reinforcement learning with dynamic objects. arXiv preprint arXiv:1901.0312,, 2019.
Peter Auer. Using confidence bounds for exploitation-eexplraiion trade-offs. Journal of Machine Learning Research, 3(Nov):397-422 2002.
Mohammad Babacizadch, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. International Conference on Learning Representations (ICLR), 2018.
Ershad Banijamali, Rui Shu, Mohammad Ghavamzadeh, Hung Bui, and Ali Ghodsi. Robust locally- linear controllable embedding. arXiv preprint arXiv:17100053373 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754-4765. 2018.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv: htth:1910.1121111 2019.
Bert De Brabandere, Xu Jia, Tinne Tuytelaars, and Lue Van Gool. Dynamic filter networks. Neural Information Processing Systems (NIPS), 2016.
Mare Deisenroth and Carl Rasmussen. Pilco: A model-based and data- efficieff approach to policy search. In Proceedings of the 28th International Conference machine learning (ICML-11), pp. 465-472, 2011.
Mare Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data- efficient learning in robotics and control. IEEE transactions on pattern analysis and machine intelligence, 37(2):408-42) 2013.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. arXiv preprint arXiv: arXiv:1802.0700 07687, 2018.
Emily L Denton et al. Unsupervised learning of disentangled representations from video. In Advances in neural information processing systems, pp. 4414-4423, 2017.
Alexey Dasovitskiy and Vladlen Koltun. Learning to act by predicting the future. arXiv preprint 16X1v:1611.01779/ 2016.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine, Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint 18/2v 2018.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2786-2793. IEFE, 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pp. 64-72, 201 6a.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512-519. IEEE, 2016b.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062. 2062, 2019.

---

Under review as a conference paper at ICLR 2021
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. arXiv preprint (rXii::906.02736. 2019.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.101222 2018.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: (00::88/1.044 1.04551, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint
2019.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles, Learning to decompose and disentangle representations for video prediction. In Advances in Neural Information Processing Systems, pp. 517-526, 2018.
Gregory Kahn, Adam Villaflor, Bosen Ding, Pieter Abbeel, and Sergey Levine. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8. IEEE, 2018
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari. International Conference on Learning Representations (ICLR), 2020.
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. International Conference on Machine Learning (ICML), 2017.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. arXiv preprint arXiv:1903.01433 2019.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:/804.01522, 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):133-1137, 2016. 2016.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.0 2020.
Ziwei Liu, Raymond Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. International Conference Computer Vision (ICCV), 2017.
Junhyuk Oh, Satinder Singh, and Honglak Lee, Value prediction network. In Advances in Neural Information Processing Systems, pp. 6118-6128 2017.
Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In International Conference Machine Leurning, pp. 2701-2710, 2017.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imagination- augmented agents for deep reinforcement learning. In Advances in neural information processing systems, pp. 5690-5701, 2017.
Aniruddh Raghu, Matthieu Komorowski, and Sumectpal Singh. Model-base reinforcement learning for sepsis treatment. arXiv preprint arXiv: /81 1.09602, 2018.
10

---

Under review as a conference paper at ICLR 2021
Scott E. Reed, Aäron den Oord, Nal Kalchbrenner, Sergio Gomez Colmenarejo, Ziyu Wang, Yu- tian Chen, Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density estimation. International Conference Machine Learning (ICML), 2017.
Reuven Y Rubinstein. Optimization of computer simulation models with rare events. European
Journal of Operational Research, 99(1):89-112, 1997.
Julian Schrittweser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. arXiv preprint arXiv:1911.08263 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Ramanan Sekar. Oleh Rybkin, Kostas Danillidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. arXiv preprint arXiv:2005.005 005660 2020.
Ewout W Steyerberg et al. Clinical prediction models. Springer, 2019.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Frez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 2018.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang. and Jan Kautz. Mocogan: Decomposing motion and content for video generation. arXiv preprint arXiv: DrxXiv:/07.000 1707.0499, 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 81-91, 2019.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from images. In Advances in neural information processing systems, pp. 2746-2754, 2015.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprint arXiv: 1906. 190..0240,,,, 2019.
Nevan Wichers, Ruben Villegas, Dumitru Erhan, and Honglak Lee. Hierarchical long-term video prediction without supervision. arXiv preprint arxiv:1806.047766 2018.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint
2019.
Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. Advances in neural information processing systems, pp. 91-99, 2016.
Chao Yu, Jiming Liu, and Shamim Nemati. Reinforeement learning in healthcare: survey, arXiv preprint arXiv: 08796, 2019.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine Solar: deep structured representations for model-based reinforcement learning. arXiv preprint 10X.001808.09105. 2018

---

Under review as a conference paper at ICLR 2021
(a) cheetah
(b) reacher
(c) cartpole
finger
(c) ball in cup
walker
Figure 5: The environments from DeepMind Control Suite (Tassa et al. 2018 used in our experiments. The images show agent observations before downscaling. (a) The cheetah run includes large state and action space. (b) The reacher _eaay has only a sparse reward. (c) The ca cartpol e_ba a has fixed camera so the cart can move out of sight while cartpol lessi ngup requires long term planning. (d) The finger-spin includes contacts between the finger and the object. (e) The bawl.i.ccup.c. www.ccccpccccccccoccccccccc ch. has sparse reward that only given once the ball is caught the cup. (1) The walker.wal' x has difficult to predict interactions with the ground when the robot is lying down.
A ADDITIONAL EXPERIMENTS
A. EFFECT OF RESIZING THE TARGET IMAGE
We repeat the same experiment in Section 4.4 this time by resizing the target instead of cropping The results of this experiment, visualized in Figure 6 supports the same observation as Figure the performance of models scales with the amount pixel supervision that they receive. is important to note that the model still observes the entire image; however, it predicts different number of pixels. However, we observe a sharper transition with more pixels when compared with Figure This is expected because resizing the observation preserves more information vs cropping.
A.2 ON EFFECT OF OPTIMISM
We speculated that some degree of model eгrоr can result in optimistic exploration strategies that make it easier to obtain better data for training subsequent models, even if such models do not perform as well a given dataset as their lower-error counterparts. To expand on this, we compare the models on another carefully designed dataset, which includes only low-reward states in its training set but has trajectories with high reward for evaluation. The rationale is that models that perform well on this evaluation are likely to generalize well to higher-reward states, and thus be more suitable for control. Figure shows heat-map of ground-truth reward predicted reward for each model on cheetab run. The best performing models on this task in Table [ OrOR and L do not actually exhibit the best generalization. Indeed, they are both pessimistic (very few points in the upper-left half of the plot in the bottom row). In contrast, LrOr appears to extrapolate well, but performs poorly in Table [T] likely due to the excessively optimistic prediction. At the same time, R has high error and performed poorly at the same time. One possible implication of this conclusion is that future work should more carefully disentangle and study the different roles of exploration vs. prediction accuracy.
A.3 EFFECT OF SHARING THE LEARNT LATENT SPACE
In this experiment, we investigate the effect of sharing the learned representation when model is predicting both reward and images (Figure 7). is common belief (Kaiser et al. 2020) that learning shared representation, typically done by mack-propagating from both losses into the same hidden layers, can improve the performance of the agent. However, our results indicate that doing so does not necessarily improve the results, and, particularly in the case of models with observation space dynamics, can result a large performance drop. This an interesting observation since most the prior work, including recent state-of-tee tiee-o--theerrr models such as Hafner et al. (2018) and Kaiser et al. (2020), simply back mack-propagate all of the losses into shared learned representation, assuming it will improve the performance.
12

---

Under review as a conference paper at ICLR 2021
1000
500
Sritee
2.3
330
200
2005
400K
200K 3E0K
101
2206
4006
5O0K oook
2006
5006
6006
0335
3M
2005
4205
600K DOOK
Environmen Step
Ensiromment Step
centronmmntt
Ervitenment Step
finger_ppe
ball Jn catch
walker walk
100%
6%
25%
Resized Target + Loss Mask stways
100
320
100
200C
600K
400IC
GOO
200C
400E
Environment Step
Environment Step
Enstronment Step
Figure 6: The effect of limiting the observation prediction capacity on the performance of the agent. The setup is exactly the same as Figure 4 except we resized the target image instead of cropping. Similarly. this graph shows that predicting more pixels results in higher performance as well more stable training and better sample efficiency. Note that in this experiment, all the agents still observe the entire image, however, they predict different number of pixels.
cheelab run
reacher ewy
caripoll winguu
caripole balance
200
-400
200K
4226
628
2006
4008
600K
20.8
2006
a008
2006
620K
628
Environment Stop
Environment Stop
Envirormant Step
Environment Stap
Anger stin
ball in cup. catch
walker we
CyLa
200
200
220
RnOR
OrOR
OrLa
-420
-0.0
4000
2205.
4205.
620K
BOOK
2000
420K
400K.
B03K
2005
4001
GEOK
S05K
Ervinonment Stop
Envirromment Stop
Envilorman
Step
Figure 7: Effect of sharing the learned latent space. The graphs follow the same format Figure except the y-axis is the difference between achieved scores by each model with and without sharing the learned latent space. In other words, a positive delta score on y-axis means that not sharing the learned latent space is improving the results for each particular model. This figure demonstrates how sharing the learned latent space can degrade the performance, particularly for models with observation space dynamics.
13

---

Under review as a conference paper at ICLR 2021
CyLR
2,02
O,0g
OrL
R
Prodicted Reward on oiffine train dotoset
Preditted Rowerd on offline eval dataset
True Reward
Trua Reward
True Roward
True Reward
True Roward
Figure Comparison of training and evaluation accuracy of models with and without image prediction. Each graph demonstrates the heat map of the predicted vs. ground-truth reward cheetah,run. The training and evaluation dataset is fixed for all of the models with fix train/evaluation partitions. Since the rewards for each frame is [0, 1] and the action repeat is set to four, the observed reward always in [0, 4J. The green dash line is the perfect prediction. This figure demonstrates better task performance for more accurate models when compared with Table [
CyLR
L,Or
OfOr
OfL
R
Predicted Roward on offtine train detaset
Prodiced Reward or offiine evel detaset
True Rewerd
True Reward
True Roward
True Roward
True Reward
Figure 9: Comparison of training and evaluation accuracy of models with and without image prediction. Each graph demonstrates the heat map of the predicted vs. ground-truth reward for cheetah_run. The training and evaluation dataset is fixed for all of the models and designed specifically to make sure that there are unseen high reward states in the evaluation (separated by the dotted black line). Since the rewards for each frame is [0. 1] and the action repeat is set to four, the observed reward is always in [0, 4]. The green dash line the perfect prediction. This figure illustrate better generalization to unseen data for models which predict images. It also demonstrates the entanglement of exploration, task performance and prediction accuracy, when comparison with Figure and Table
14

---

Under review as a conference paper at ICLR 2021
run
easy
swingup
225
5.5
375
203
138
300
175
192
123
182
200
230
225
300c
anoc Environment Step
2008:
2016
200
Environment Step
Environment Stop
Envronment Stop
Finger spin
cup catch
walker malk
4013
188
1.6
Rrecurent
142
Reone
Score
48
Rec
180
2006
10.0
50.00
201
2006
4DOK
2006
1H
2006
430c 630c
600c
Environment Stop
Environment Stop
Figure 10: Comparison between different types of reward predictor models described in Section B R A simple recurrent convolutional network to predict the future rewards given the last observation (Table 8). This model has its own recurrent latent space which can be unrolled to predict future states. Reony: simple convolutional network to predict the future rewards given the last observation (Table p.. This model predicts the rewards for the entire planning horizon in one shot. Rec: The exact same model as CR without image prediction. We finally used Reecureet as R since it had the best overall performance, can be in this figure.
LTLR on cheetah run
OjOr on cheetah run
275
250
225
500
200
Median Score
175
450
150
125
100
350
75
300
50
250
1.6
2.0
2.2
2.4
4.0
1e-3
1e-4
Median Observation Prediction Loss
Median Observation Prediction Loss
Figure 11: Correlation between accuracy of observation prediction and task performance, in the online setting. For each graph, we scaled down the indicated model (left: O,OR, right: Cx by multiple levels and report their median observation prediction error during training (æ-axis) and median achieved score (y-axis). The annotation on each data point is the scale multiplier: higher means a bigger model (the detailed numbers can be found in appendix.) This graph clearly demonstrates the strong correlation between observation prediction accuracy and asymptotic performance.
15

---

Under review as a conference paper at ICLR 2021
B MODELS ARCHITECTURE AND DETAILS
To minimize the required hyper-parameter search, we reuse the existing stable implementations for the model designs in Figure I For other variations, we modified these existing models to match the new design. These models are inherently different in the tasks they do which means these models vary substantially their training and inference time. Table summarized the characteristics of these models.
LT Lx: Latent transition Latent reward
This variant corresponds several prior methods such PlaNet (Hafner et al., 2018). Therefore, we directly use PlaNet as a representative model of this class (a description of this model can be found in Section B.2 By default, this model does paok-prop from both the reward loss and observation prediction loss into its learned latent space. Given the computational cost and for consistency, we had use the same number of trajectories between models. This means used different set of planning lyper-paramesn 50 and lower number of trajectory proposals of 128 vs 1000). We had to make this change since other models are much slower than PlaNet to train and training these models with the original numbers from PlaNet would take over 2 months using eight v100s.
for PlaNet (particularly shorter training and prediction horizon of 12 vs
LrOR: Latent transition Observation reward Here, we again use the PlaNet architecture to model the dynamics in the latent space; however, we remove its reward predictor and instead utilize the reference reward model predict the future reward given the predicted next observation. Following the design of PlaNet, we again back propagate gradients from both the reward and observation.
OrORt Observation transition Observation reward
For this design, we used SV2P (Babaeizadeh et al. 2018) for modeling the dynamics in the observation space (details in Section B.I. Then, we use the reference reward model to predict the expected reward given the prediction observation. The main reason for using SV2P the video model is its stability compared to other next frame predictors. For this model, we do not back-propagatt propagate gradients from the reward prediction into the latent space learned by observation prediction. We also did not use 3-stages of training and only performed the last stage of training (identical to PlaNet). According to the authors of SV2P, the three stage training only required for highly stochastic and visually complex tasks. We verified the same performance with and without three stages of training on cheetah.run. To be clear, this means that all our models follow the exact same training regime.
Ex- Observation transition Latent reward Again, use SV2P to model the dynamics in the observation space, however, the input to the reward predictor is the internal states of SV2P (i.c., the aggregated states of all Conv-LSTMs of the main tower deseribed in (Bahaeizadeh al. 2018 Finn et al. 2016a)). This is the design behind some of the most recent works in visual model- based RL such as SimPLe Kaiser et al. (2020). However, we only back-propagate gradients from the observation prediction error into the learned representation.
R: Reward prediction without observation prediction
We tried multiple reward predictors from images and chose the best one as R:
1. R A simple recurrent convolutional network predict the future rewards given the last observation (T)ble(8). This model has its own recurrent latent space which can be unrolled to predict future states.
2. Reons A simple convolutional network to predict the future rewards given the last obser- vation (Table 9 This model predicts the rewards for the entire planning horizon in one shot.
3. Rec: The exact same model as CrC without image prediction.
Figure 10 compares the performance of each of these models all the tasks. We finally used Reecumee as R since had the best overall performance.
Oracle: We the environment itself oracle model. Since using the same planning algorithm across all of the models above, the rewards from the environment approximate the maximum possible performance with this planner.
16

---

Under review as a conference paper at ICLR 2021
Model
Training Signal
Transition
Reward Input
R
Reward Error
OTOR
Reward Error + Observation Error
Observation Space
Predicted Observation
OTLR
Reward Error Observation Error
Observation Space
Latent States
LT
Reward Error + Observation Error
Latent Space
Latent States
LTOR
Reward Error Observation Error
Latent Space
Predicted Observation
Table 4: Summary of possible model designs based on whether or not to predict the future observations. All of the models predict the expected reward conditioned on previous observations, rewards and future actions. Moreover, the top four methods predict the future observations as well. These methods can model the transition function and reward function either the latent space or in the observation space. Another design choice for these models is to whether not share the learned latent space between reward and observation prediction.
3x0
363
-N(p.or))
The Dooas
ploS4e3
Souds
Bxb
Sus
Images
6605463
Figure 12: Architecture SV2P. At training time, the inference network (top) estimates the posterior (2( = VV (M(xx:r)) or (x. The latent value 2o (2) (z(xor)) is passed to the generative network along with the (optional) action. The generative network (from Finn et al. (2016a)) predicts the next frame given the previous frames, latent values, and actions, At test time, z sampled from the assumed prior. I).
B.1 SV2P ARCHITECTURE
Introduced in Babaeizadeh et 2018). SV2P is a conditional variational model for multi-frame future frame prediction. Its architecture, illustrated in Figure 12 consists of ConvLSTM based predictor p( [x=:r|X0.6-11 z) where z sampled from a convolutional posterior approximator assuming Guassian distribution qs (z) xu:r) N (u(Xx:)) ) or (xp:r)) The predictor tower is conditioned on the previous frame(s) and future proposed actions while the posterior is approximated given the future frame itself during the training. At inference, z sampled from A A((0, I). This network trained using the reparameterization trick (Kingma & Welling 2014), and optimizing the variational lower bound, as in the variational autoencoder (VAE) (Kingma & Welling 2014):
((x) = -E
Ey Exg( log Pe (x:r) Xo:t-1, z)] + DE
where DEL is the Kullback-Leibler divergence between the approximated posterior and assumed prior p(z) which in this case is the standard Gaussian (0, I). Here, 0 and 6 are the parameters of the generative model and inference network, respectively. SV2P is trained using 3-stage training regime which is only required for visually complex tasks and not utilized in this paper. The list of hyper-parameter used can be found Table
B.2 PLANET ARCHITECTURE
Introduced by Hainer et al. (2018), PlaNet a model-basee agent that learns latent dynamics model from image observations and chooses actions by fast planning in the latent space. To enable accurate
17

---

Under review as a conference paper at ICLR 2021
+
0
Figure 13: Architecture of PlaNet (Hafner et al. 2018 Learned Latent Dynamics Model: In a latent dynamics model, the information of the input images is integrated into the hidden states (green) using the encoder network (grey trapezoids). The hidden state is then projected forward in time to predict future images (blue trapezoids) and rewards (blue rectangle).
Long-term predictions, this model uses stochastic transition functions while utilizing convolutional encoder and decoder as well as image predictors. The main advantage of PlaNet is its fast unrolling in the latent space since it only needs to predict future latent states and rewards, and not images, to evaluate action sequence. PlaNet also optimizes variational bound on the data log-likelihood:
T
((x)
Σ E[lnp(orls)] (s a(a, uca-1"
act) p(s a--])
reconstruction
complexity
We used the same training procedure and hyper-parametts optimization described in Hafner al. (2018). The list of prper-parameter used can be found Table 12
worth mentioning that although we used PlaNet as LTCR with no changes, our results are different from Hafner et al. (2018). This is because we used 128 trajectories vs 1K in CEM (Table I0 as well as training horizon of 12 vs in Hafner et al. (2018) (Table [2). We made these changes to keep planning consistent across our models. Other models are slower than Ly LR to train and explore (Table 7) which made it infeasible to use them with original planning and training horizon of Hafner et al. 2018). Please check Section B.4 for our approximate compute cost.
B.3 HYPER-PARAMETEL SEARCH PROCEDURE
To get reasonable performance from each one of these models, we follow the same hyper-parameter search as Hafner et al. (2018); we grid search the key hyper-parametert the best performing one, and then using the same set of parameters across all of the tasks. This means that these models will perform their best cheetah.rie hyper-parameter search computationally feasible. The final used hyper-parameter can be found in Table II and Table 12 Contrary to Hafner al. (2018). we did optimize the action repeat per task and instead used the fixed value of four.
che selecting
We are mainly doing this to keep the
B.4 APPROXIMATED COMPUTATION COST
In this paper we have three main experiments using eight models across seven tasks in two online and offline settings, where each run can take up to seven days on eight V100s (please check Table B.4 a model break-down). That is 100K V100-hours (or $70K at the time of writing of this paper) per random seed (we did three). Note that the cost is not only about training but also unrolling hundreds of random proposals per step. This both time and computation expensive particularly for models with observation space transitions.
18

---

Under review as a conference paper at ICLR 2021
L
0
OyLx
R
1.5
175
135
575
175
1.2
135
125
820-
830
039
000
200
140
1.9
1100
10
1.00
100
109
1.0
1:00
090
0.0
8:0
000
059
090
0.00
8.0
009
2.00
2.00
1.00
2.0
111
1.00
1.00
130
020
0:25
025
1.9
1.7
131
115
322
87%
0:5
025
8:25
0.5
025
131-
121
100
330
030
059
0100
099
2.03
2.00
2.00
1.00
208
150
159
150
1.0
1.0
100
1.0
150
030
8.0
0:00
050
090
8.00
0.00
0.00
2701
718
1.0
130
1.5
125
8.2
0.5
9.h%
502
Sowurd
Roverd
Taward
Figure 14: This figure illustrates the distribution of the rewards in trajectories collected by model variations in the online setting. Since the rewards for each frame is [0, 1] and the action repeat is set to four, the observed reward is always in [0, 4] (x-axis).
19

---

Under review as a conference paper at ICLR 2021
cheetah reacher cartpole cartpole finger ball_in.cup walker
run
easy
swingup
balance
spin
catch
walk
Online
R
102 23
588 + 95
215
317 31
37
205 + 75
156
O7OR R
447 + 24
480 106
381 - 30
849 111
135
37
373 - 21
Or
240 + 40
801
398
17
575 10
98
71
251 - 23
LT CR
426 + 10
768 + 135
341
18
445 16
355
711 + 177
645 = 20
Lr
233 + 12
757 + 91
261 + 8
343 + 41
308 + 25
421 + 144
541 + 12
Oracle
671
85
394
595
556
957
926
Offline
R
217 8
996 +
258 + 6
415 =
5
985 + 20
300 = 7
07OR R OTLR
503 12
981 + 16
685
+ 11
978 + 11
148 6
910 +
568 +
490 11
9 +
406
+
923 + 12
244
949 18
731 +
CTC
502 7
994 10
413
9
543 + 8
373 5
5
996
- 19
552 +
LT
282 7
993 + 15
281
7
411 7
259
= 6
932
+ 19
383
Difference
R
115
408
43
98
94
780
144
OTOR
56
501
304
129
13
867
195
07OR R
250
198
348
146
745
480
OTOR 76
226
72
98
18
285
93
0, 49
236
20
68
49
511
158
Table 5: The asymptotic performance of various model designs in the online and offline settings and their differences. For the online setting the reported numbers are the average (and standard deviation) across three runs after the training. For the offline setting, the reported numbers the same as Table[Trounde up + their standard deviation across three runs. This table demonstrates significant performance improvement by predicting images almost across all the tasks. Moreover, there is meaningful difference between the results for the online and the offline settings sigmifying the importance exploration. Please note how some of best-performing models in the offline setting perform poorly in the online setting and vire-versa. This is clear from the bottom section of the table which includes the absolute difference of offline and online scores.
# Parameters
Training Step
Inference Step
#GPUs
R OTOR
81.6K
3
5
1 1 V100
8.34M
40
310
8 V100
OrLR
8.38M
44
361
8 X V100
LTLR 5.51M
32
10
V100
LT
5.29M
56
101
8 x V100
Table 6: Wall clock time of training and inference step of each model. The numbers are in seconds. Model R which only predicts the reward is the fastest model while the models with pixel space dynamics are the slowest. Please note that R is much smaller models compare to the other ones since it does not have any image decoder.
cheetah run
reacher cartpole cartpole finger ball-in.cup walker swingup
easy
balance
spin
catch
walk
R
20.4
117.6
43.0
63.4
15.4
41.0
31.2
OrOr
1.4
1.5
1.2
2.7
0.4
0.1
1.2
OTLR
0.7
2.2
1.1
1.6
0.3
0.6
0.7
Lr
42.6
76.8
34.1
44.5
35.5
71.1
64.5
L,OR
2.3
7.5
2.6
3.4
3.0
4.2
5.4
Table 7: Cost-normalized scores foe each model. These numbers the online score achieved by each model divided by the inference cost (time) of the corresponding model. As expected, faster models -.e.. i.e. R and LR which do predict the images inference time get big advantage. This table clearly shows that L- Li is generally good design choice if there is no specific reason for modeling the dynamic or reward function the pixel space.
20

---

Under review as a conference paper at ICLR 2021
Layer Type
Filters
Size
Strides
Activation
Last Observation Encoder
Convolutional
32
3x3
LeakyReLU
Convolutional
64
3x3
LeakyReLU
Convolutional
16
3x3
LeakyReLU
Convolutional
8
3x3
LeakyReI.U
Action Encoder
Dense
32
LeakyReLU
Dense
16
LeakyReLU
Dense
8
LeakyReLU
Last Reward Encoder
Dense
32
LeakyReIU
Dense
16
LeakyReLU
Dense
8
LeakyReLU
Next State Predictor
LSTM
64
LeakyReLU
Reward Predictor
Dense
32
relu
Dense
2
relu None
Dense
1
Table 8: The architecture of Reecurrent-
Layer Type
Filters
Size
Strides
Activation
Last Observation Encoder
Convolutional
32
3x3
LeakyReLU
Convolutional
64
3x3
LeakyReLU
Convolutional
16
3x3
LeakyReLU
Convolutional
3x3
2
LeakyReLU
Action Encoder
Dense
32
LeakyReLU
Dense
16
LeakyReL ReLU
Dense
8
LeakyReLU
Last Reward Encoder
Dense
32
LeakyReLU
Dense
16
LeakyReLU
Dense
8
LeakyReLU
Reward Predictor
Dense
32
LeakyReLU
Dense
16
LeakyReLU
planning horizon
Dense
None
Table The architecture of R
Parameter
Value
Planning Horizon
12
Optimization Iterations
10
Number of Candidate Trajectories
128
Number of Selected Trajectories
12
Table 10: The hyper-parameterss used for CEM. We used the same planning algorithm across all models and tasks.
21

---

Under review as a conference paper at ICLR 2021
Hlyper-parameter
Value
Input frame size
x3
Number of imput frames
Number of predicted frames
12
Beta /
1e-3
Latent channels
Latent minimum log variance
-5.0
Number of masks
10
ReLU shift
le-12
DNA kernel size
Optimizer
Adam
Learning Rate
le-3
Epsilon
le-3
Batch Size
64
Training Steps per Iteration
100
Reward Loss Multiplier
1.0
Table 11: The hyper-parameter values used for SV2P [Babaeizadeh et al. 2018 Finn et al. 201 6a model (used in OROR and CR) The number of ConvLSTM filters can be found Table The rest of parameters are the same as Babaeizadeh et al. (2018).
Hyper-parameter
Value
Input frame size
64x64x3
Number of predicted frames
Units in Deterministic Path
200
Units in Stochastic Path
Units in Fully Connected Layers
300
Number of Encoder Dense Layers
Number of Reward Predictor Layers
Free Entropy nats
3.0
Minimum Standard Deviation
0.1
Optimizer
Adam
Learning Rate
le-3
Epsilon
le-3
Batch Size
Training Steps per Iteration
100
Reward Loss Multiplier
10.0
Table 12: The hyper-pprametter values used for PlaNet (Hafner et al. 2018) model (used in Or
and LT LR). The rest of parameters the same as Hafner et al. (018)).
Hyper-paree parameer
Value
Input frame size
64x64x3
Number of predicted rewards
12
Number of Reward Predictor Layers
Number of Reward Predictor Layers
3
Minimum Standard Deviation
0.1
Optimizer
Adam
Learning Rate
le-3
Epsilon
le-3
Batch Size
32
Training Steps per Iteration
100
Table 13: The paper-parameter values used for Reecument (used R). The hyper-parameters for the planner is the same as other models (Table 10).
22

---

Under review as a conference paper at ICLR 2021
ConvLSTM
ConvLSTM
ConVLSTM
ConvLSTM
ConvLSTM
ConvLSTM
ConvLSTM
Layer
Model IT)
2
2
2
4
2
16
8
4
16
16
32
16
8
5
16
16
32
32
64
32
16
32
32
64
128
64
Table 14: The downsized version of OR We down-scaled the model by reducing the number of ConvLSTM filters, limiting the modeling capacity and thereby potentially increasing their prediction error. The detailed architecture of the model and layers can found Finn et (2016a);
Babaeizadeh et al. (2018).
Model
Units in Fully Connected Layers
Units in
ID
Deterministic Path
100
200
2
100
100
3
100
50
4
50
200
5
50
100
6
50
50
7
200
200
8
200
100
9
200
50
Table 15: The downsized version of LR We down-scaled the model by reducing the number of units in fully connected paths, limiting the modeling capacity and thereby potentially increasing their prediction error. The detailed architecture of the model and layers can be found in Hafner et al. (2018).
23


Reviews:
[{"review_id": "szKRH3smRlr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors compare several kinds of MBRL models, including observation prediction, to show which approach is better. In conclusion, The models that contain prediction on both a reward and an observation outperform the ones with only reward prediction. \n\n[Quality]\n\nThis paper requires a revision for enhancing clarity. For example, Section 4 seems exceedingly long, and it contains too many redundant details, which should not have been emphasized.\n\n[Originality & Significance]\n\nThis paper provides neither a new method nor a different perspective.\n\n[Strengths]\n+ The authors showed the effect of using visual observation prediction for MBRL models. It might show the way for further MBRL models.\n+ It was mentioned on the paper that prediction accuracy of observations and rewards and exploration performance is on the trade-off relationship.\n\n[Weaknesses]\n- The number of parameters on the model that predicts only rewards is much (about a hundred times) smaller than other models. Additionally, the authors used SV2P and PlaNet to implement each experiment; it is doubtable that comparisons are fair enough.\n- There are no clear and consistent differences between models with an observation prediction, which reduces this work's novelty. It is better to use four different SOTA models or four modified models from one backbone model instead of modifying two models.\n- The trade-off relationship is mentioned, but there is not enough quantitative and theoretical analysis to show this point. \n\n[Comment]\n- There is no bolded item in Table 2; it seems to be omitted, or the result is not strong enough.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "U5OTDfM45Qq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper empirically investigates the effect of different reward-prediction models for model-based reinforcement learning (MBRL) in the particular context of decision-time planning with the cross-entropy method and visual tasks. To that end, five models are evaluated: four models that predict rewards and observations jointly, while one model only learns to predict rewards but without observation prediction. The authors claim that predicting observations in addition to rewards improves performance of MBRL in their setting (in terms of cumulative reward maximization). This hypothesis is tested on a bunch of tasks from the DeepMind Control Suite, both in an online and offline setting. The authors conclude, based on their experiments, that their hypothesis can be confirmed.\n\n\nQuality and Details\n\nMy main problem is that I don't entirely understand the particular constellation of models to be compared with each other to begin with---see Figure 1. To my understanding, Figure 1 just depicts two different classes of models: the four on the left and the one on the right. Note however that this becomes clear only after studying the appendix (Appendix B) in more detail, since the models are not described properly in the main text. The four models on the left can be subdivided into 2 further subclasses depending on whether transitions are modelled in latent or observed space: latent-variable models refer to the PlaNet architecture and observation-space models to the SV2P architecture (each can be trained with and without additional reward prediction). The stand-alone model on the right is a specific model not related to PlaNet or SV2P.\n\nI believe the question of whether additional observation prediction helps can be investigated for each of the 2 top-level classes separately. For example, the left four models could be amended to only predict rewards---running PlaNet or SV2P but stripping away the observation prediction component? Looking at appendix Figure 13 that describes PlaNet, the observation component could be ignored in the course of training such as to only predict rewards. Similarly in Appendix Figure 12 that describes SV2P, the autoregressiveness could happen at the hidden layer without observation input/output. In a same way could the stand-alone model for reward prediction be amended to enable observation prediction.\n\nAfter studying the appendix in more detail, I came to realize that the authors actually followed one of the ideas above and trained PlaNet with only reward prediction but not observation prediction. They found that this performs worse compared to another reward-only-prediction baseline (which they finally report in the main paper). Concluding, the presentation is quite convoluted and it is still missing whether SV2P can be amended to predict rewards only, or whether the best-performing reward-prediction baseline can be extended to predict observations.\n\nA clear execution and presentation of the setting described above would allow to draw conclusions about whether predicting observations actually helps *across* different architectures, but not the setting studied by the authors that uses separate architectures for joint reward-observation prediction and reward-only prediction.\n\nAdditionally, assuming the experimental setup makes sense (which I feel it does not entirely), I am still concerned about the statistical significance of the experimental results from Section 4.2 on online learning. There are just 3 seeds per experiment---so it is not clear which joint reward-observation model is best, although it seems indicative that the reward-only prediction model is worse across tasks compared to the joint models. Statistical significance is only getting worse in Section 4.3 which studies the offline setting: because here only one performance value is reported *at the end of training* as opposed to *in the course of training* (making it even more difficult to judge the statistical significance, since RL algorithms are known to produce rapidly performance-changing policies in the course of training).\n\n\nClarity\n\nThe clarity of the paper is low. While multiple different prediction models are studied, none of them are presented in detail in the main paper, neither regarding architecture nor learning objective. The appendix provides more information, revealing how complex the models are. These details do matter in an empirical study and a missing description in the main paper makes it even harder to judge the conclusions drawn from the experiments.\n\n\nOriginality and Significance\n\nInvestigating the necessity of observation prediction (in addition to reward prediction) on RL performance is interesting. However, the paper does not entirely convince me regarding both the experimental setup and statistical significance.\n\n\nPros\n\nA lot of environments are tested.\n\n\nCons\n\nThe experimental setup is questionable and so is the statistical significance of the results.\n\n\nMinor\n\nI believe the caption of Figure 3 draws conclusions about which regions of the state space have been visited. However, only empirical reward distributions are presented, i.e. I don't see how one can draw conclusions about state-space visitation from there. \n\nAlso, the main online results from Figure 2 show in some environments that the oracle baseline (that knows the true environment dynamics) is worse compared to the MBRL algorithms. The authors mention in the text that this is due to a limited planning horizon---I still do not understand why the MBRL algorithms work better given that they also need to cope with a limited planning horizon.", "extra_scores": {}, "extra_text": {}}, {"review_id": "L64Zh48zoBj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a simple taxonomy of objectives for training predictive models for planning in visual model-based RL, and evaluates them in a set of consistent experimental tasks and using a largely consistent set of models. It claims that in the context of the models investigated, backpropagating losses based on future observations yields better performance than using only the reward as training signal, and that prediction accuracy of future observations is also more predictive of task performance than reward prediction errors. It additionally empirically demonstrates the magnitude of effect exploration policies can have in this setting, showing how a lot of the models that perform worse online simply explore less well. \n\nI think the paper is largely easy to read and understand, and systematic/structured exploration of modeling choices is important especially given the steady trickle of results showing that specific modeling choices can be less important than how well-tuned the implementations are. The additional insight about exploration vs modeling efficiency is well-supported and the additional experiment that separates exploration efficiency is well-thought out. I also think it's great that less-conclusive or less-succesfful experiments (e.g. on optimism) are still included in the appendix rather than file-drawered, though I'm puzzled that the claim regarding implicit optimism is made at the end of section 4.3. considering what appendix A2 shows. \n\nThat said, I have some additional comments about ways in which the paper could be improved: \n\n1. I think the paper overstates the generality of its results, considering the usage of only one specific choice of each architecture and loss (with the exception of the R-model) and one planner. The choices seem to be reasonable and consistent with the SOTA, but they're also only one way of setting up each kind of loss in the taxonomy, and the paper doesn't make a strong argument for why they should be general. Especially given how the paper shows that there's no clear uniform winner across tasks, one could reasonably expect that the variability across other variants not explored is similarly large. Similarly, the underperforming oracle highlights the contribution of the fixed-horizon planner to the overall results, and calls into question the claim in section 4.1 about how the oracle approximates optimal performance. I think it's good that the architectures were kept consistent across variants when possible, but convergent evidence (even from scaled-down toy examples) would help make the point, in combination with more caveats about the generality. \n\n2. A related concern is whether the reward prediction models are genuine competitors or straw man models, since the remaining architectures / models come from previously published results and I'm not sure if the reward prediction models do (I think $\\mathcal{R}_{LL}$ is similar to what was used by Havens et al. 2019, though that was a non-archival NeurIPS workshop paper). It's surprising that the paper does not take any of the prior work it cites as the exemplar model for reward prediction, similarly to how it does with pixel prediction. On the other hand, if these models are credible competitors from past work, that work should be cited. \n\n3. The focus in the writing on what the model predicts is a bit misleading in that unless I'm missing something, the critical difference is which loss is being backpropagated. In order to plan, I need to simulate rollouts, so even some of the R-models have a next state predictor (except if they predict the full horizon at once, as in $\\mathcal{R}_{conv}$), and the reward-loss models can have worse reward predictions, presumably by having less training signal (e.g. Tab 2). This could be clarified in the text and Figure 1. \n\n4. The overall results are ultimately \"messy\" in the sense that there's no clear winners or patterns. This in itself is worth knowing (and a point that the paper mostly embraces). That said, there are a few missed opportunities for further insights, including: \n    - Performance across different kinds of difficulties. The paper makes the point early on that its test environments focus on different difficulties (contact discontinuities, large state spaces, long-term memory, long-term planning, sparse rewards). This point isn't revisited -- some discussion along these dimensions might be useful, beyond \"different training signals win on different tasks, except reward prediction, which never does\". \n    - The paper makes the point that using reward only as training signal can be substantially cheaper (section 4.4), and that latent-dynamics models are faster than pixel-space dynamics models (table 5). Does this mean we might expect cost-normalized performance to show some interesting patterns? \n\n5. It would be useful if a visual comparison between the online results (Fig 2) and offline results (Tab 1) could be made, either by including Tab 1 results in panels on the figure, or moving the table to the appendix in favor of a figure that compares offline and online results. Similarly, the tables do not facilitate the comparison made in the caption to Tab. 2 regarding the relationship between reward prediction accuracy and task performance. \n\n[I think the paper provides a useful case study and I appreciate the caveats added to the conclusion, but based on additional discussion with the other reviewers, I think that the paper's more general claims remain unsupported and insufficiently moderated. I am reducing my rating accordingly.]", "extra_scores": {}, "extra_text": {}}, {"review_id": "KkftpJ5jQlr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary:\n    - This paper presents a study on the trade-offs of image prediction for model based RL\n    - They find that image prediction loss is important and, surprisingly, reward prediction accuracy can be negatively correlated performance while image prediction accuracy is considerably more well correlated.\n- Stengths\n    - Clear presentation of results and overall well written\n    - Comprehensive series of results that all build on one another\n    - A large amount of care given to the implementations. The sentence \"Given the empirical nature of this work, our observations are only as good as our implementations.\" is very apt.\n    - The authors do a good job at pointing out the caveats and shortcomings of their study.\n- Weaknessses\n    - While a fair number of environments are studied, they are all visually very similar (all 3rd person camera, same background, etc.).  Adding in a very different environment, i.e. Visdoom, would strength the analysis.  While I don't think adding a different environment is necessary, this should be pointed out in the paper.\n    - While the presentation is overall quite clear, I do have some suggestions (see bellow)\n    - Questions:\n        - How was training stopped in the off-line setting?\n        - Were the same hyper-parameters used for both online and offline?\n- Suggestions for improvement\n    - A version of Table 1 with the online results at convergence would be great as the delta between online and offline performance is interesting.\n    - A Pearson correlation coefficient and/or a Wilcoxon signed ranked test on the ranking induced by Table 1 vs. Table 2 for each environment would be a nice summary statistic.  Although the dynamic range of the values may not be large enough for this to be sensible.\n    - If possible, color code the variant names in the tables with their line colors in plots.  Also, order the table the same as the figure legends.\n- Overall\n    - This paper presents an interesting, well-motivated, and comprehensive study on the trade-offs in image prediction for model based RL that also points to future directions of research.  \n\n\n## Post Rebuttal\n\nI thank their authors for their response.  I have decided to maintain my rating, overall I still think this is a good paper that presents an interesting set of results.  The result that image prediction accuracy correlated better with asymptotic performance than reward prediction accuracy continues to intrigue me.  My fellow reviewers have some concerns that while I don't agree, I think they could be avoided with some changes in the presentation to the paper:\n\n* The models used.  I understand the decision to move the model description to the appendix (not everything is going to fit in 8 pages), but the main paper would benefit from paragraph or two describing why which models were chosen and then referring to the proper places in the appendix for the full details.\n\n* Reward-only model.  There was considerable concern about how much smaller that models is than the others.  Additional results showing that one with a comparable number of parameters does as poorly if not worse would be beneficial.  The 0% line in Fig 6 and Fig 8 is very similar to this hypothetical (if it exactly) so I believe this will pan out as expected.\n\n* Presentation of results.  In my read, the most interesting result, anti-correlation between reward prediction accuracy and asymptotic performance, comes at the end.  Without that result, there is a way to read the paper as \"yes image prediction helps, it increases the amount of supervision given to the model\".  That result and the difference between online and offline shows that there is something unexpected going on here so perhaps leading with those and/or highlighting them more would help.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors compare different model based R algorithms to see whether observation prediction is important. They show that, as expected, it is. On the other hand, they seem to show that latent space prediction is not very useful. The study is limited to domains with image data: Does this domain have something particularly special? Perhaps experiments with smaller-scale POMDP problems might actually have shown something different. It is very difficult to do a study of this type properly, and although the authors have tried, it's hard to see how this paper can be accepted. I agree with some the positive points some reviewers have raised, but I think that, at the end of the day, the paper is trying to draw too general conclusions from a handful of datapoints. Were I writing this paper, I would first try the simplest version of the hypothesis with very basic environments that are, however, more varied than the ones shown here. Would the hypothesis hold, I'd scale up to more complex environments and try to also run with more seeds to get a clearer signal.

Decision: Reject