Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors provide a deeper exploration of Imagination Agents, by looking more closely at a variety of state-space models.  They examine what happens as both the representation of state, update algorithm, as well as the concept of time, is changed.  As in the original I2A paper, they experiment with learning how best to take advantage of a learned dynamics model.

To me, this strain of work is very important.  Because no model is perfect, I think the idea of learning how best to use an approximate model will become increasingly important.  The original I2A of learning-to-interpret models is here extended with the idea of learning-to-query, and a variety of solid variants on a few basic themes are well worked out.

Overall, I think the strongest part of this paper is in its conceptual contributions - I find the work thought-provoking and inspiring.  On the negative side, I felt that the experiments were thin, and that the work was not well framed in terms of the literature of state-space identification and planning (there are a zillion ways to plan using a model; couldn't we have compared to at least one of them?  Or discussed a few popular ones, and why they aren't likely to work?  Since your model is fully differentiable, vanilla MPC would be a natural choice [In other words, instead of learning a rollout policy, do something simple like run MPC on the approximate model, and pass the resulting optimized action trajectory back to the agent as an input feature]).

Of course, while we can always demand more and more experiments, I felt that this paper did a good enough job to merit publication.

Minor quibble: I wasn't sure what to make of the sSSM ideas.  My understanding is that in any dynamical system model, the belief state update is always a deterministic function of previous belief state and observation; this suggests to me that the idea of "state" here differs from my definition of "state".  I don't think you should have to sample anything if you've represented your state cleanly.
review_title: Could be impactful

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a method for inferring dynamical models from partial observations, that can later be used in model-based RL algorithms such as I2A. The essence of the method is to perform variational inference of the latent state, representing its distribution as Gaussian, and to use it in an ELBO to infer the dynamics of state and observation.

While this is an interesting approach, many of the architectural choices involved seem arbitrary and unjustified. This wouldn't be so bad if they were justified by empirical success rather than principled design, but I'm also a bit skeptical of the strength of the results.

A few examples of such architectural choices:
1. What's the significance of separating the stochastic state transition into a stochastic choice of z and a transition g deterministic in z?
2. How is the representational power affected by having only the observations depend on z? What's the intuition behind calling this model VAE, when sSSM is also trained with variational inference?
3. What is gained by using pool-and-inject layers? By the way, is this a novel component? If so please elaborate, if not please cite.

As for the strength of the results, in Table 1 the proposed methods don't seem to outperform "RAR" (i.e., RNN) in expected value. They do seem to have lower variance, and the authors would do well to underline the importance of this.
In Figure 3, it's curious that the model-free baseline remains unnamed, as it also does in the text and appendix. This makes it hard to evaluate whether the significant wins are indicative of the strength of the proposed method.

Finally, a notational point that the authors should really get right, is that conditioning on future actions naively changes the distribution of the current state or observation in ways they didn't intend. The authors intended for actions to be used as "interventions", i.e. a-causally, and should denote this conditioning by some sort of "do" operator.
review_title: Unclear contribution

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper studies how to learn (hidden)-state-space models of environment dynamics, and integrate them with Imagination-Augmented Agents (I2A). The paper considers single-agent problems and tests on Ms Pacman etc.

There are several variations of the hidden-state space [ds]SSM model: using det/stochastic latent variables + using det/stochastic decoders. In the stochastic case, learning is done using variational methods. 

[ds]SSM is integrated with I2A, which generates rollouts of future states, based on the inferred hidden states from the d/sSSM-VAE model. The rollouts are then fed into the agent's policy / value function.

Main results seem to be:
1. Experiments on learning the forward model, show that latent forward models work better and faster than naive AR models on several Atari games, and better than fully model-free baselines. 
2. I2A agents with latent codes work better than model-free models or I2A from pixels. Deterministic latent models seem to work better than stochastic ones.

Pro:
- Relatively straightforward idea: learn the forward model on hidden states, rather than raw states.
- Writing is clear, although a bit dense in places.

Con:
- Paper only shows training curves for MS Pacman. What about the other games from Table 1?
- The paper lacks any visualization of the latent codes. What do they represent? Can we e.g. learn a raw-state predictor from the latent codes?
- Are the latent codes relevant in the stochastic model? See e.g. the discussion in "Variational Lossy Autoencoder" (Chen et al. 2016)
- Experiments are not complete (e.g. for AR, as noted in the paper).
- The games used are fairly reactive (i.e. do not require significant long-term planning), and so the sequential hidden-state-space model does not have to capture long-term dependencies. It would be nice to see how this technique fares on Montezuma's revenge, for instance.

Overall:
The paper proposes a simple idea that seems to work well on reactive 1-agent games. However, the paper could give more insights into *how* this works: e.g. a better qualitative inspection of the learned latent model, and how existing questions surrounding sequential stochastic model affect the proposed method. Also, not all baseline experiments are done, and the impact on training is only evaluated on 1 game. 

Detailed:
-

review_title: Simple idea that seems to work well on Ms Pacman, but paper needs more quantitative results/qualitative inspections

Meta-review: There was quite a bit of discussion about this paper but in the end the majority felt that, though the paper is interesting, the results are too limited and more needs to be done for publication.

PROS:
1. Good comparison of state space model variations
2. Good writing (perhaps a bit dense in places)
3. Promising results, especially concerning speedup

CONS:
1. The evaluation is quite limited


Decision: Reject