Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper provides a surprise-based intrinsic reward method for reinforcement learning, along with two practical algorithms for estimating those rewards. The ideas are similar to previous work in intrinsic motivation (including VIME and other work in intrinsic motivation). 
As a positive, the methods are simple to implement, and provide benefits on a number of tasks.
However, they are almost always outmatched by VIME, and not one of their proposed method is consistently the best of those proposed (perhaps the most consistent is the surprisal, which is unfortunately not asymptotically equal to the true reward). The authors claim massive speed up, but the numerical measurements show that VIME is slower to initialize but not significantly slower per iteration otherwise (perhaps a big O analysis would clarify the claims).
Overall it's a decent, simple technique, perhaps slightly incremental on previous state of the art.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present a novel approach to surprise-based intrinsic motivation in deep reinforcement learning. The authors clearly explain the difference from other recent approaches to intrinsic motivation and back up their method with results from a broad class of discrete and continuous action domains. They present two tractable approximations to their framework - one which ignores the stochasticity of the true environmental dynamics, and one which approximates the rate of information gain (somewhat similar to Schmidhuber's formal theory of creativity, fun and intrinsic motivation). The results of this exploration bonus when added to TRPO are generally better than standard TRPO. However, I would have appreciated a more thorough comparison against other recent work on intrinsic motivation. For instance, Bellemare et al 2016 recently achieved significant performance gains on challenging Atari games like Montezuma's Revenge by combining DQN with an exploration bonus, however Montezuma's Revenge is not presented as an experiment here. Such comparisons would significantly improve the strength of the paper.
review_title: Important extension of existing work on intrinsic motivation. Experimental results are less convincing.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper explores the topic of intrinsic motivation in the context of deep RL. It proposes a couple of variants derived from an auxiliary model-learning process (prediction error, surprise and learning progress), and shows that those can help exploration on a number of continuous control tasks (and the Atari game “venture”, maybe).

Novelty: none of the proposed types of intrinsic motivation are novel, and it’s arguable whether the application to deep RL is novel (see e.g. Kompella et al 2012).

Potential: the idea of seeking out states where a transition model is uncertain is sensible, but also limited -- I would encourage the authors to also discuss the limitations. For example in a game like Go the transition model is trivially learned, so this approach would revert to random exploration. So other forms of learning progress or surprise derived from the agent’s competence instead might be more promising in the long run? See also Srivastava et al 2012 for further thoughts.

Computation time: I find the paper’s claimed superiority over VIME to be overblown: the gain seems to stem almost exclusively from a faster initialization, but have very similar per-step cost? So given that VIME is also performing very competitively, what arguments can you advance for your own method(s)?
review_title: null

final_score: rating: null
review_content: I'm intrigued by "RAN", your random baseline with the surprise of a random, fixed, untrained model. Judging from Fig 5 (it should really be plotted with the rest), this is sometimes competitive with the best of your variants, and it got me thinking: maybe a random-but-fixed shaping reward is often enough to make exploration consistent enough to get off the ground. But of course it can get unlucky, so I'd be very curious to see another baseline: the same as RAN, but sampling a new random model after each couple of iterations if there was no progress in real reward. Maybe none of the (fancier) surprise/learning progress tricks are even necessary?
review_title: more random consistent baselines

Meta-review: The paper proposes an intuitive method for exploration, namely to build a model of the system dynamics and explore regions where this approximation differs from the observed data (i.e., how "surprised" the agent was by an observation). The idea is a nice one, and part of the benefit comes from the simplicity and wide applicability of the approach.
 
 The main drawback of this paper is simply that the resulting approach doesn't substantially outperform existing approaches, at least not to a degree where it seem like the paper should should be clearly accepted to ICLR. On the continuous control tasks, the advantage over VIME seems very unclear (at best the results are mixed, showing sometimes surprisal and sometime VIME do better), and on the Atari games no comparison is made against many of the methods tuned for this setting, such as Gorila (Nair, 2015) which achieves some of the best results we are aware of on the Venture game, which is definitely the strongest result in this current paper, but not as good as this previous work. We know the settings are different, but overall it seems like the approach is largely outperformed by existing approaches, and thus the advantage mainly comes from runtime. This is certainly an interesting take, but needs to be studied a lot more thoroughly before it would make a really compelling case. We would like to recomend this paper to the workshop track. The pros/cons are as follows:
 
 Pros:
 + Simple and intuitive method for exploration
 
 Cons:
 - Doesn't seem to substantially outperform existing methods
 - No comparison to many alternative approaches for some of the "better" results in the paper.
Decision: Invite to Workshop Track