Title: Recurrent Environment Simulators
Authors: Silvia Chiappa
Sébastien Racaniere
Daan Wierstra
Shakir Mohamed
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning
Applications

Abstract: Models that can simulate how environments change in response to actions can be used by agents to plan and act efficiently. We improve on previous environment simulators from high-dimensional pixel observations by introducing recurrent neural networks that are able to make temporally and spatially coherent predictions for hundreds of time-steps into the future. We present an in-depth analysis of the factors affecting performance, providing the most extensive attempt to advance the understanding of the properties of these models. We address the issue of computationally inefficiency with a model that does not need to generate a high-dimensional image at each time-step. We show that our approach can be used to improve exploration and is adaptable to many diverse environments, namely 10 Atari games, a 3D car racing environment, and complex 3D mazes.


Review discussion:
`jumpy predictions have been developed in low-dimensional observation spaces' - could you elaborate on the work that inspired you to develop jumpy predictions for higher-dimensional spaces? What is the difference in dimensionality between theirs and yours?
Jumpy predictions as we described can be performed in most existing probabilistic time-series models since, in most cases, these models do not have a link from the observation at time-step t-1 to the hidden state at time-step t-1. Such models include the popular linear Gaussian state-space model and its generalizations (see, e.g., planning-as-inference (Rawlik et al, 2010), KL control (Kappen et al. 2009), and linearly solvable MDPs (Todorov, 2006)). These approaches focus on low dimensional input-space applications (well under 100 dimensional input) and therefore there are no computational issues in predicting the observation at every time-step (the lack of link from the observation to the hidden state is motivated by modelling rather than computational reasons). We are not aware of any other work in the literature that uses the same order of dimensionality as in our case (around 100,000).
B. Kappen, V. Gomez, and M. Opper. Optimal control as a graphical model inference problem. 2010.
K. Rawlik, M. Toussaint, and S. Vijayakumar. Approximate inference and stochastic optimal control. 2009.
E. Todorov. Linearly-solvable Markov decision problems. 2006.
Thank you for this interesting work.
1. One interesting result is that 100%-Prediction (fully prediction-dependent training) method performs best.
This seems a bit contradictory to the result of Scheduled Sampling [Bengio et al.] paper.
Have you compared your 100%-prediction method to scheduled sampling method (with epsilon annealing)?
Showing this would be interesting.
2. Figure 9 shows that the wall configurations are same as previously generated walls after 360-degree spin.
Does this mean that generated scenes do not necessarily match with the ground-truth, but your model can remember and preserve generated topologies?
3. What's the difference between your model-based exploration and informed exploration in [Oh et al.]?
Can you give an intuition why your exploration method can be potentially better than [Oh et al.]?
Thank you for your words of appreciation.
1. Our results appear indeed in contradiction with the ones in the Scheduled Sampling paper. Our belief is that this is due to the different application but also potentially to the different model and initialization considered. Notice that, in some Atari games, the 100%-prediction method, which focuses on long-term prediction at the price of lower sharpness, can be problematic, and that, perhaps due to the initially high error fed back to the hidden state, the 100%-prediction method can get more often stuck into local optima at the beginning of training.
Therefore, it can be challenging to make the 100%-prediction method to work successfully.
But when possible, this is the method that consistently gives lowest prediction errors (this is also the case for the other environments considered in the paper in addition to Atari).
We did not think it was necessary to compare with Scheduled Sampling, as the different training schemes that we used clearly show that there is a smooth coherent progression of error when going from a 0%-prediction method to a 100%-prediction method, thus a Scheduled Sampling approach would perform worse than the 100% approach.
But we are running some experiments using Scheduled Sampling that we can add to the paper to show quantitatively the difference.
2. Yes that’s correct.
3. Our approach is similar to Oh et al. in that we are using the simulator to choose actions leading to unseen frames.
However, our algorithm selects a group of d actions rather than a single action at a time. More specifically, Oh et al.'s algorithm is "choose action a_1 such that the next frame is as different as possible from recently seen frames, whereas our algorithm is "choose a sequence of actions a_1, ..., a_d such that the frame in d time-steps from now is as different as possible from recently seen frames, then perform actions a_1, ..., a_d".
Rather than proposing a substantially different approach, our goal was to show that our simulator can be used in a more challenging task, namely a 3D partially observable environment, with randomly generated mazes rather than a fixed 2D layout. We also present a qualitative analysis, which unlike Oh et al., is possible in our case as we can exactly measure and plot the proportion of the maze visited over time.
1. The current formulation is claimed to overcome the limitation of traditional LSTM which makes accurate predictions for only a few time-steps. Why is the current formulation which is a variant of LSTM that fuses action (a_t-1) with hidden state (h_t-1) instead of (h_t) expected to overcome this limitation?
2.
Also, have you considered fusing (a_t-1) with (s_t-1) instead of h_t-1? How would the performance be affected if a_t-1 is treated as an additional input along with s_t-1 for the traditional LSTM? Is there a strong reason for fusing a_t-1 with h_t-1 to change the LSTM formulation?
3. At many places, it is claimed that jumpy simulators can make make predictions at any point in the future.
However, can't the prediction-dependent "single step" simulator also achieve this, since it does not require observed frames?
4. Is the color coding same for Fig. 2a and Fig. 2b?
5. The comparison of the current approach with Oh et al. in Fig. 5a is useful.
However, I would have liked to see the breakdown in performance improvement due to the (a) difference in training scheme and (b) difference in architecture. What happens if the same training scheme is used for Oh et al?
1.
When fusing the action a_t-1 with h_t-1, a_t-1 influences the transition dynamics from time-step t-1 to time-step t directly, i.e. h_t (and c_t) is generated using a_t-1. This means that a_t-1 can have a stronger influence and that only the results of taking action a_t-1 are forwarded from t-1 rather than all possibilities due to all possible actions a_t-1. Fig. 12 in the Appendix shows how fusing the action this way versus fusing the action with h_t is advantageous for Seaquest, but similar conclusions apply to the other games.
2. We have done experiments that fuse a_t-1 with s_t-1 in addition to h_t-1.
Overall, this gave worse or similar performance, probably as the action-independent part of the observation (such as the background or action-independent moving objects) can be more easily predicted with explicit absence of the action. This is confirmed in the jumpy model, where we obtained drastically better results having a transformation of h_t-1 that is action independent in addition to one that is action dependent (see L(s_t-1) in the equations describing the model). We are currently running the experiments that you suggested that fuse a_t-1 with s_t-1 instead of h_t-1. We will add a discussion and quantitative analysis in the Appendix to clarify these points.
3. The prediction-dependent "single step" simulator can indeed also make accurate predictions for many time-steps into the future (Fig. 5b shows that the two models achieve similar accuracy on three Atari games).
However, the prediction-dependent "single step" simulator requires many more computations than the jumpy simulator, namely 200 millions more flops for each time-step, this can be cumbersome if wanting to do many predictions for many time-steps and therefore a jumpy simulator would be preferable in model-based RL -- that’s what motivated us to develop jumpy simulators rather than wanting to improve performance.
4. Yes, the color coding is the same for Fig. 2a and Fig. 2b. We will add a note in the paper to clarify this.
5. We address this aspect of comparison using Fig. 2 and Fig. 12 in the Appendix. In Fig. 2, we use the same training scheme as used by Oh et al. (see the 0%-20%-33% predicted frames scheme) and show the benefits that our training scheme offers. In Fig. 12, we compare the same architecture used by Oh et al. with our architecture and show the benefits that our architecture offers. Other differences such as using randomized ReLU rather than ReLU, etc., give some improvement as well (we do not discuss these but we could add some more experiments in the Appendix highlighting these points).
When seen together, these two figures show that there is an advantage in using both a different architecture and a different training scheme. We will modify the paper to make it clearer that the highest improvements come from using a different training scheme.
The authors propose a recurrent neural network architecture that is able to output more accurate long-term predictions of several game environments than the current state-of-the-art.
The original network architecture was inspired by inability of previous methods to accurately predict many time-steps into the future,
and their inability to jump directly to a future prediction without iterating through all intermediate states.
The authors have provided an extensive experimental evaluation on several benchmarks with promising results.
In general the paper is well written and quite clear in its explanations.
Demonstrating that this kind of future state prediction is useful for 3D maze exploration is a plus.
# Minor comments:
`jumpy predictions have been developed in low-dimensional observation spaces' - cite relevant work in the paper.
# Typos
Section 3.1 - `this configuration is all experiments'
We would like to thank the reviewer.
We have cited the relevant work on the jumpy predictions and corrected the typo (see pages 2 and 5 in the latest revision).
We particularly appreciated that the reviewer recognized our effort to perform an extensive experimental evaluation --
this was particularly challenging and time consuming in this high-dimensional and complex scenario, taking over one year
to complete.
The paper presents an action-conditional recurrent network that can predict frames in video games hundreds of steps in the future. The paper claims three main contributions:
1. modification to model architecture (used in Oh et al.) by using action at time t-1 to directly predict hidden state at t
2. exploring the idea of jumpy predictions (predictions multiple frames in future without using intermediate frames)
3. exploring different training schemes (trade-off between observation and prediction frames for training LSTM)
1. modification to model architecture
+ The motivation seems good that in past work (Oh et al.) the action at t-1 influences x_t, but not the state h_t of the LSTM. This could be fixed by making the LSTM state h_t dependent on a_{t-1}
- However, this is of minor technical novelty.
Also, as pointed in reviewer questions, a similar effect could be achieved by adding a_t-1 as an input to the LSTM at time t. This could be done without modifying the LSTM architecture as stated in the paper.
While the authors claim that combining a_t-1 with h_t-1 and s_t-1 performs worse than the current method which combines a_t-1 only with h_t-1, I would have liked to see the empirical difference in combining a_t-1 only with s_t-1 or only with h_t-1.
Also, a stronger motivation is required to support the current formulation.
- Further, the benefits of this change in architecture is not well analyzed in experiments. Fig. 5(a) provides the difference between Oh et al. (with traditional LSTM) and current method.
However, the performance difference is composed of 2 components (difference in training scheme and architecture). This contribution of the architecture to the performance is not clear from this experiment. The authors did claim in the pre-review phase that Fig. 12 (a) shows the difference in performance only due to architecture for "Seaquest".
However, from this plot it appears that the gain at 100-steps (~15)  is only a small fraction of the overall gain in Fig. 5 (a) (~90). It is difficult to judge the significance of the architecture modification from this result for one game.
2. Exploring the idea of jumpy predictions:
+ As stated by the authors, omitting the intermediate frames while predicting future frames could significantly sppedup simulations.
+ The results in Fig. 5(b) present some interesting observations that omitting intermediate frames does not lead to significant error-increase for at least a few games.
- However, it is again not clear whether the modification in the current model leads to this effect or it could be achieved by previous models like Oh et al.
- While, the observations themselves are interesting, it would have been better to provide a more detailed analysis for more games.
Also, the novelty in dropping intermediate frames for speedup is marginal.
3. Exploring different training schemes
+ This is perhaps the most interesting observation presented in the paper. The authors present the difference in performance for different training schemes in Fig. 2(a). The training schemes are varied based on the fraction of training phase which only uses observation frames and the fraction that uses only prediction frames.
+ The results show that this change in training can significantly affect prediction results and is the biggest contributor to performance improvement compared to Oh et al.
- While this observation is interesting, this effect has been previously explored in detail in other works like schedule sampling (Bengio et al.) and to some extent in Oh et al.
Clarity of presentation:
- The exact experimental setup is not clearly stated for some of the results.
For instance, the paper does not say that Fig. 2(a) uses the same architecture as Oh et al.
However, this is stated in the response to reviewer questions.
- Fig. 4 is difficult to interpret. The qualitative difference between Oh et al. and current method could be highlighted explicitly.
- Minor: The qualitative analysis section requires the reader to navigate to various video-links in order to understand the section. This leads to a discontinuity in reading and is particularly difficult while reading a printed-copy.
Overall, the paper presents some interesting experimental observations.
However, the technical novelty and contribution of the proposed architecture and training scheme is not clear.
Thank you for your review, to which I will reply in full later. I wanted, however, to make a correction on my previous statement that mixed up Fig. 2 and Fig. 12, and apologize for the confusion caused (the problem was due to a quick late review of my statement from another author that ended up mixing the sentences). I will correct the original statement.
The wrong sentence is the following:
5. We address this aspect of comparison using Fig. 2 and Fig. 12 in the Appendix. In Fig. 2, we use the same architecture as used by Oh et al. and show the benefits that our training scheme offers. In Fig. 12, we use the same training scheme as used by Oh et al. (see the 0%-20%-33% predicted frames scheme). This graph shows the benefits that our architecture offers.
This should be corrected to:
5. We address this aspect of comparison using Fig. 2 and Fig. 12 in the Appendix. In Fig. 2, we use the same training scheme as used by Oh et al. (see the 0%-20%-33% predicted frames scheme) and show the benefits that our training scheme offers.  In Fig. 12, we compare the same architecture used by Oh et al. with our architecture and show the benefits that our architecture offers.
[UPDATE]
After going through the response from the author and the revision, I increased my review score for two reasons.
1. I thank the reviewers for further investigating the difference between yours and the other work (Scheduled sampling, Unsupervised learning using LSTM) and providing some insights about it.
This paper at least shows empirically that 100%-Pred scheme is better for high-dimensional video and for long-term predictions.
It would be good if the authors briefly discuss this in the final revision (either in the appendix or in the main text).
2. The revised paper contains more comprehensive results than before.
The presented result and discussion in this paper will be quite useful to the research community as high-dimensional video prediction involves large-scale experiments that are computationally expensive.
- Summary
This paper presents a new RNN architecture for action-conditional future prediction. The proposed architecture combines actions into the recurrent connection of the LSTM core, which performs better than the previous state-of-the-art architecture [Oh et al.]. The paper also explores and compares different architectures such as frame-dependent/independent mode and observation/prediction-dependent architectures. The experimental result shows that the proposed architecture with fully prediction-dependent training scheme achieves the state-of-the-art performance on several complex visual domains. It is also shown that the proposed prediction architecture can be used to improve exploration in a 3D environment.
- Novelty
The novelty of the proposed architecture is not strong. The difference between [Oh et al.] and this work is that actions are combined into the LSTM in this paper, while actions are combined after LSTM in [Oh et al.]. The jumpy prediction was already introduced by [Srivastava et al.] in the deep learning area.
- Experiment
The experiments are well-designed and thorough.
Specifically, the paper evaluates different training schemes and compares different architectures using several rich domains (Atari, 3D worlds).
Besides, the proposed method achieves the state-of-the-art results on many domains and presents an application for model-based exploration.
- Clarity
The paper is well-written and easy to follow.
- Overall
Although the proposed architecture is not much novel, it achieves promising results on Atari games and 3D environments.
In addition, the systematic evaluation of different architectures presented in the paper would be useful to the community.
[Reference]
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov. Unsupervised Learning with LSTMs. ICML 2016.
We thank the reviewer for the review update.
We are still doing experiments, including some (currently not in the paper) that will shed more light on the difference with Scheduled Sampling.
More generally, we are working on making the results and discussion as comprehensive and clear as possible.
We tried to address the reviewer's concerns by adding a substantial amount of new results and discussion, mainly to the Appendix (notice that some experiments are still running, thus many figures will need to be updated). We highlighted the modifications in red. We did not finish making all the modifications: we mostly still need to improve the main text.
However, we decided to send a first update to give the reviewers the time to digest the new material, and to make the remaining changes in the next few days.
Below we describe in some detail how we addressed the reviewer's comments.
1. Modification to model architecture.
To address the reviewer's concern about lack of comparison of different ways of incorporating the action, we added a section in the Appendix (Section B.1.3, Figures 19 and 20) that compares the architecture used in the main text (combining the action a_t-1 only with h_t-1 in a multiplicative way), with 8 alternatives, namely combining a_t-1 with the frame before encoding (ConvA), combining a_t-1 only with the encoded frame s_t-1, combining a_t-1 with both h_t-1 and s_t-1 in 4 possible ways, having a multiplicative/additive interaction of a_t-1 with h_t-1, and considering the action as an additional input.
From the figures and videos it looks like combining the action with the frame (dark green lines) is a bad idea, and that there is no much difference in performance among the remaining approaches, although combining a_t-1 with both h_t-1 and s_t-1 (W^h h_{t-1}xW^s s_{t-1}x W^{a} a_{t-1)) seems to be overall a bit better (notice that this architecture results in a quite different core than the traditional LSTM core).
The experiments are still running -- the experiments with considering the action as an additional input (light green lines) are behind the other experiments.
We also modified the section in the Appendix describing the difference between our direct action influence and the indirect action influence as in Oh et al. (now at the end of Section B.1.3). We changed Fig. 12 in the previous version with Figure 21 to show the difference in the other games in addition to Seaquest (we need to add Riverraid for which we want to use 2 subsequences of length T=15).
Direct action influence performs similarly to indirect action influence in games with lower prediction error, such as Bowling, Freeway and Pong, but better in games with higher prediction error. The most striking difference is with Seaquest, which is most evident when looking at the videos (is is difficult to judge the difference in performance by looking at the prediction error only, that's why we added randomly selected videos from which the types of errors in the prediction can be visually understood). These results are computed using the best training scheme for Seaquest. The videos show that, with indirect action influence, the simulator models the fish quite poorly, even though using a training scheme that more strongly encourages long-term accuracy enables the simulator to learn the appearance of the fish better than with the training scheme of Oh et al.
Therefore, these videos clearly show that the training scheme alone is not sufficient, our architecture plays a crucial role in substantially improving the prediction. Having indirect action influence also makes it more difficult to correctly update the score in some games such as Seaquest and Fishing Derby.
Therefore, whilst the training scheme is the major responsible for improving long-term prediction, it needs to be combined with an appropriate architecture in order to work.
All together, the experiments in Section B.1.3 suggest that having a direct (as opposed to Oh at al.) and global (as opposed to ConvA) action influence on the state dynamics is much preferable.
On the other hand, performance is less sensitive to different ways of imposing direct action influence. We need to improve the discussion in the paper about this point.
2. Exploring the idea of jumpy predictions.
To address the reviewer's concerns about lack of a detailed analysis, we added a new section in the Appendix (Section B.2, Figures 25 and 26), and modified Figure 5(b) to include all games (except Breakout for which the prediction error is uninformative -- see the discussion about this game in Section B.1.1). Section B.2 compares different architectures, sequence lengths and number of subsequences (we plan to add a few more experiments in the next few weeks).
It is clear from the results that the jumpy simulator is much more sensitive to changes of structure, and therefore fragile, than the single-step simulator.
However, we show that with the right training structure, the jumpy simulator can be accurate in many games. Having accurate and fast to use simulators is essential for model-based reinforcement learning.
Thus, whilst the novelty might seems marginal, it is of extreme interest for the community to advance in this type of approaches.
Furthermore, accurate jumpy approaches are very difficult to obtain -- we believe that we are the first ones showing some success for long-term prediction in large scale domains.
We do not understand the sentence 'However, it is again not clear whether the modification in the current model leads to this effect or it could be achieved by previous models like Oh et al.'. The model of Oh et al. cannot be modified to perform jumpy predictions as, when omitting intermediate frames, the corresponding actions do not have any influence (as they influence the hidden states indirectly through the frames). This is actually an important reason why a direct action influence is preferable.
3. Exploring different training schemes.
We would like to comment on the sentence “While this observation is interesting, this effect has been previously explored in detail in other works like schedule sampling (Bengio et al.) and to some extent in Oh et al.?”
The schedule sampling work of Bengio et al. focuses on different applications, with discrete data and on shorter-term prediction (for example, the length of sequences in the Image Captioning example (Section 4.1) is 11 on average). In this applications, the 100% Pred. Frames training scheme (called Always Sampling) actually performs poorly.
Thus, based on the results of Bengio et al., we would never use the 100% Pred. Frames. We show that, in simpler games, this is actually the scheme that performs best, and that, in more complex games, a scheme that is as close as possible to such a scheme is preferable. Our analysis (especially in the new material) shows in detail the characteristics of each scheme on both short-term and long-term prediction.
Thus, we make a significant contribution in advancing the understanding of different training schemes in frame prediction.
Oh et al. used a mixed scheme without any explanation.
Thus, it is totally unclear from their paper the importance of using a mixed scheme and what are the differences of different types of mixing. That's actually the reason that motivatesd us to perform a detailed analysis.
To improve the understanding of the results, we have added a section in the Appendix (Section B.1.1) that discusses the characteristics and effects of different training schemes on each game.
In conclusion, we addressed the reviewer's concerns on the analysis by adding more results and discussion.
Regarding the concern on little novelty, we strongly disagree that this should be a concern. We greatly improved on the performance of previous methods (it is enough to compare the videos in Oh at al. with ours to realize that there is an enormous difference). Even more importantly, we improved the understanding of these methods with a detailed analysis. We believe that papers like our paper are as important for the community as papers that provide completely novel architectures without much analysis.
Finally, the reviewer should understand that providing a detailed analysis and understanding is extremely challenging and time consuming, due to both the large scale and the complexity of the problem. Ours, whilst not fully comprehensive (as this is not possible), represents an enormous effort in trying to provide an in depth analysis and advance the understanding on the problem of high dimensional frame prediction.
We have thought more deeply about the fact that the results in Bengio et al. appear to be in contradiction with ours, in the sense that the Always Sampling training scheme (corresponding to our 100% Pred. Frames training scheme) seems to perform worse than mixed schemes. We also had a chat with one of the authors of the Bengio et al. paper to get a better understanding of the methods and experiments.
We have reached the conclusion that the difference might be due to the fact that Bengio et al. focus on discrete problems and to the fact that we were mostly reasoning about long-term prediction, whilst Bengio et al. focus on shorter-term prediction (e.g., in Image Captioning Section, the average prediction length is 11 (this is not mentioned in the paper, but has been pointed out to us by one of the authors)) -- if we look at our results for short-term prediction only, they do not look so much different anymore.
In Figures 12-16 of the latest version of our paper, in addition to the prediction error at time-step 100, we also plot the prediction error at time-steps 5 and 10 for most games. We can notice that the prediction error with the 100% and the 0%-100% Pred. Frames training schemes (called here Schemes I and II) (red and dark green lines) is almost never lower than the prediction error with the other mixed schemes for time-steps 5 and 10, and often higher (see for example Fishing Derby in Fig. 13). The situation is different at time-step 100, where Schemes I and II are always almost preferable to the other mixed schemes.
Therefore, by looking only at the error up to time-step 10, we would also prefer other schemes to Schemes I and II. The error is higher at lower time-steps with Schemes I and II as the objects are less sharply represented in the frames.
In other words, these two schemes capture the global dynamics (as this enables better long-term prediction) at the expense of not representing the details very accurately, as lack of the details at earlier predictions do not harm subsequent predictions. In Bengio et al., where the problem is discrete, one error at the beginning of the sequence might lead to drastic errors at subsequent time-steps.
We thank the reviewer for this question, as he made us thinking more carefully about this point. We need to modify the paper to include as summary of this discussion.
We also thank the reviewer for pointing out that in “Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov. Unsupervised Learning with LSTMs. ICML 2016”, the unconditional model can perform jumpy prediction. We briefly mention this paper in the introduction at the moment, but we need to add a discussion. Notice that, in this paper, prediction is shorter-term than in our case, namely 10-13 time-steps ahead. We are not sure whether, at training time, the real or the generated frames are fed in the conditioned model. From the sentence “At test time we feed in the generated frame from the previous step without adding any noise. At training time we feed in the ground truth.” in Section 2.3, it would look like the model is trained with feeding in the real frames, i.e. with the 0% Pred. Frames training scheme. This approach seems to work better than the unconditional model -- this is surprising to us, as it is in contradiction with our results.

Meta-review:
role: PC
data: comment: Quality, Clarity: 
  The paper is well written. Further revisions have been made upon the original.
 
 Originality, Significance:
  The paper presents an action-conditional recurrent network that can predict frames in video games hundreds of steps in the future. This is done using a mix of (a) architectural modifications; (b) jumpy predictions; and (c) particular training schemes. The experimental validation is extensive, now including additional comparisons suggested by reviewers.
 There is not complete consensus about the significance of the contributions, with one reviewer seeking additional technical novelty. Overall, the paper appears to provide interesting and very soundly-evaluated results, which likely promises to be the new standard for this type of prediction problem.

Decision: Accept (Poster)