Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This work proposes an RL approach SMiRL that is able to learn effective policies in unstable environments without the need for external reward. The idea at a high-level is almost the opposite of intrinsic motivation RL approaches, which encourage novelty-seeking behaviors. The proposed method instead aims to minimize surprise or state entropy. To train the agent, rewards come from state marginal estimates, but because this distribution is changing, the authors create an augmented MDP. Through experiments on game domains and robot control tasks, the authors show that SMiRL outperforms intrinsic motivation methods. The authors also show that SMiRL can be used to do imitation and can be combined with regular reward signals.

Pros:
- The problem formulation is interesting and novel. Intrinsic motivation is well studied, but this problem considers the setting where the environment is unstable rather than static, which requires new methods.
- The paper is written well and is clear. The motivation is described well.
- The authors evaluate on many domains, highlighting the diversity of settings in which the approach can be applied.

Cons:
- It seems like this approach is only applicable to unstable environments. Does the approach fail for regular, static environments? I’m assuming the agent might just end up staying still because it’s trying to seek stable states. It can be combined with the external reward signal but will the minimizing entropy objective hurt you?
- Without common sense knowledge, this approach would take many iterations to learn. So while this formulation might be more similar to the real world, the real world would only allow for a few interactions with the world.

Comments:
- How is SMiRL doing better than the oracle in Figure 3 center?
- In Figure 3 left, it might be a problem that with minimal episodes, SMiRL does worse. If SMiRL is useful for more real-world unstable environments, this would require a simulator good enough to model the real world.

Recommendation:
Overall, the paper is interesting and novel. The approach is reasonable and experiments show the value of the method in unstable environments. I recommend “accept”.

------------
Post-rebuttal response:
The authors addressed most of my concerns so I continue to recommend acceptance of the paper. Specifically, they answered my question about whether the approach will work in static environments and how prior data can be used to improve sample efficiency. They also conducted additional experiments to verify some of my questions.
review_title: Official Review #2

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors target the unsupervised reinforcement learning problem. An opposite idea from the existing approaches by maximizing state entropy is adopted to minimize state entropy. It is interesting that such an idea has achieved good performance in unstable environments. A state distribution is fitted during the interaction with an environment and the probability of the current state is used as a virtual reward. The parameters or sufficient statistics are also applied to the policy. The motivation is clear and verified. It is generally a good paper.

It is surprising that the exploration is achieved in the long term even minimizing state entropy. Is that possible the exploration events are from the 'unstable' environment? What if there are some patterns underlying the exploration events but only part of the 'unstable' environment? Is that OK to totally rely on unexpected events from the environment to explore the environment? Is that possible to add some exploration strategy in the developed model? 
review_title: The authors target the unsupervised reinforcement learning problem. An opposite idea from the existing approaches by maximizing state entropy is adopted to minimize state entropy. It is interesting that such an idea has achieved good performance in unstable environments.

final_score: rating: 7: Good paper, accept
review_content: ------------------------------------        
**Summary:**

This paper proposes a new intrinsic objective for RL agents: surprise minimization. This may come as a surprise, as other related works usually propose to maximize surprise, or to maximize novelty. The authors present motivations and conduct an empirical study on several environments to support their idea.

------------------------------------        
**Strong points:**

Overall, I think it is a good paper. Let me list some strong points:

* The idea is simple, novel and well motivated. The paper positions this new intrinsic objective with respect to variants of novelty maximization objectives and brings a new perspective.
* It’s well written and organized
* The algorithm is tested on a relevant selection of environments and against state-of-the-art algorithms using intrinsic objectives.
* The empirical evidence seems to support the claims.
* The related work is quite complete.
* I liked the discussion about stable vs unstable environments. It is the first time I see it discussed.
* The website brings visualizations of trained policies.
------------------------------------        
**Weak points:**

I will now list a few weak points of the paper.
* Some descriptions of the results are missing. In figures, what does the shaded area represent? (std, sem, confidence intervals, etc). In Table 1, what are the numbers? (what is the central tendency, what is the error, how many seeds?). Same for Appendix D.
* The paper does not provide all necessary information to reproduce the results. There is no detail about the RL algorithms (TRPO and DQN), no description of the architectures, and no hyperparameters. This is important and should be contained somewhere in the Appendix. 
* Will the code be released? If not, why so? Same questions for the environments, are they accessible somewhere? 
* It seems to me that this approach could potentially tackle harder problems, but the paper is limited to a simplification of the tetris game, planar humanoid variants and Doom. The x-axes of the figures also tell us that only a few episodes were needed to solve them. I am not saying that I need the hardest games solved to find an algorithm interesting, but I am wondering whether it would scale. Could you tell us whether you attempted to tackle harder environments, and if so, why do you think SMiRL failed? I think we can gain a lot of understanding by looking at negative results. 
For example, I feel like testing this algorithm on a 3D humanoid would better demonstrate the power of this approach. 
* It seems to me that there might be a confounding factor that could partially explain the success of the surprise reward. Indeed, it seems that all environments presented here can terminate when the agent dies. This is a guess, as I could not find this information in the paper (please add it). If so, then the expected cumulative rewards is an increasing function of the lifetime of the agent in the game. Because of this, maximizing the cumulative surprise might be a good idea because it goes in the same direction (by construction) as maximizing survival.
A counter-argument from the paper can be the performance of SMiRL + reward in Walk, that is superior to the performance of reward alone. However, this is unclear as I could not find the description of the reward function in the paper (please add it). 
Another way to disprove this hypothesis would be to compare the performance of SMiRL to the performance of an agent maximizing a reward function that gives +1 whenever the agent is alive (survival bonus). If it performs better, then surprise maximization brings something to the table, if it does not, then it might work because it is correlated to the survival time.
Another way could be to have episodes that do not include death-related resets (fixed length episodes).

------------------------------------        
**Recommendation and justification:**
This is overall a strong paper. However I’m concerned about the potential confounding factor of the survival time. I’ll give a score of 6, but I would happily increase that score if 
1. The authors convince me that the success of the surprise maximization is not due to the survival confounding factor.
2. The authors include all necessary details for reproducibility and/or release the code. 
3. Discuss the scalability to harder problems (e.g. 3D Humanoid).

------------------------------------        
**Feedback to improve the paper (not part of assessment)**

* I would move the discussion about how surprise minimization and novelty maximization can be complementary to the intro. These two approaches seem to go in opposite directions and, as a reader, I would be happy to read this discussion early.
* It would be interesting to discuss how it plays out in natural agents. The intuition is that minimizing surprise leads to finding a stable configuration and staying there. In practice, it is probably balanced with other driving needs like the need for food. I guess it is discussed in related papers like Friston 2009. In natural agents, surprise minimization must also be model-based. Indeed, animals do not need to jump out of cliffs several times to know that it’s bad. 
* Not sure I understand the inequality in Eq1. Maybe I missed something.
* Discuss the surprise maximization approach of Achiam et al and whether it differs from yours. 
* Table 1: the legend seems to disagree with the results. It seems to me that the entropy difference is as low in your environment as in the others, but the caption says “note the clear negative  entropy gap on our tasks, whereas this clear trend is absent on the Atari games”.
* Fig 4. Is it really training in 80 episodes? There are very few images to train a VAE, especially if the episode resets when the agent falls (does it?) How many steps per episode?
* How do you get demos for humanoid tasks? I guess there are not human demos but previously trained agents? 
* Results in Fig. 6 are not super satisfying, they are quite far from the target (although I guess it is a difficult task). I am not even sure the second example is achievable. In the traditional Tetris, wouldn’t cubes fall due to gravity?

**Typos:**                
* “our results are available online” → missing full stop.
* “In such environments, which we believe are more reflective of the real world” → previous sentences do not discuss environments but intrinsic objectives.
* “unexpected events don’t happen” → “do not”
* “deep DQN” → “Deep Q-Networks”, or “DQN”


**Update post-rebuttal**
The authors addressed most of my concerns, especially the one about the confounding factor. I am updating the score from 6 to 7.


review_title: This paper introduces suprise minimization as a intrinsic objective. The idea is novel, well motivated and seem to be supported by empirical evidence. However, some details are missing and there might be a confounding factor.

final_score: rating: 7: Good paper, accept
review_content: This paper presents and studies an unsupervised learning approach to the emergence of "meaningful/useful"
behaviour in a Deep RL setting, based on an algorithm which objective consists in minimizing surprise.
It is possible to see this paper as studying a Deep RL implementation of cognitive homeostasis, an old idea
recently popularized through the work on the free energy principle by Friston and colleagues.
The paper presents two versions of this algorithm, one without representation learning, and one with
representation learning (VAE), and compares it to Deep RL algorithms that maximize prediction error/novelty
from two perspectives: emergence of behaviours and utility as an auxiliary reward to solve sparse reward problems.
The paper discusses the relevance of the approach in environments that are said to be "unstable" (with an attempt
to formalize this concept). The paper also shortly presents an application of the algorithm for imitation learning.
Experiments are presented in a variety of environments.

Major strengths:
+ The paper is globally clear and well-written
+ The qualitative results showing and analyzing emergent behaviour are stimulating
+ Implementation of the free-energy principle in high-dimensional spaces is known to be challenging and this
paper contributes towards understanding how to do it (yet the precise formal articulation between SMIRL and the
FEP remains to be done, and this does not seem to be the primary objective of the paper)
+ The discussion of the complementarity of within-episode SMIRL and across-episode novelty seeking intrinsic
rewards is interesting
+ The shortly described application for imitation learning is promising
+ The are many and varied environments in experiments 

Major weaknesses:
- The environments chosen for experimentation are such that minimizing surprise aligns very well with either
producing "interesting" behaviour or maximizing an external reward. One could easily consider slight modifications
of most environments where a trivial solution to minimizing surprise could be found instead (e.g. giving the ability to push
on the "pause" button in the Tetris game or having safe rooms in opposite directions than enemies in Vizdoom/HauntedHouse).
In a real robot, this would lead a robot to hide in the corner of a room and just look at a uniform wall that does not move.
This problem of trivial solutions to surprise minimization approaches has been called the Dark Room problem in
the FEP literature. This is known to be a limit of this general approach when applied to the real world, and how to address it
operationnally (in a tractable manner) under this paradigm remains an open question.
- Related to the point above, the fact that different environments are used in different experiments of the paper does not
allow to get a good grasp of the general behaviour of the algorithm: I would recommend to run all experiments in all
environments (maybe to report in Annex), and at least to justify why environmnets change across experiments.
-  The paper is positioned in comparison with novelty seeking/prediction error maximizing agents, and motivates
the surprise minimizing approach by arguing that real-world situations are problematic for novelty seeking agents,
as the real world spontenously generates novelty through distractors that should be avoided rather than explored.
This is accurate, but an incomplete discussion of the literature in the area of unsupervised learning of behaviours:
approaches maximizing learning progress were precisely introduced to adress these limits of novelty seeking approaches
and to scale to real world environments with two families of distractors: novel unlearnable parts of the environments
(which is a problem for novelty seeking approaches) and trivial low-entropy parts of the environments (which are a problem
in principle for surprise minimization approaches). Thus, the paper should position itself in comparison with LP-based
approaches and show experiments with at least some of the existing LP-based algorithms (e.g. Schmidhuber, 1991; Lopes et al., 2012;
Kim et al., 2020).
- As the authors acknowledge, the version of SMIRL using represention learning is not fully "within episode surprise minmization",
as the learned representation depends on past episodes: as a consequence, it is unclear what one should conclude about the relation
between within vs across episode mechanisms and the properties of emergent behaviours
- The paper lacks discussion of related work aiming to implement surprise minimization approaches that scale to high-dimensional
spaces, especially in a RL framework, e.g. Tschantz et al., 2020; Annabi et al., 2020
- The paper does not provide code to enable reproducibility of results (and does not say it will)

Overall, the topic of this paper is interesting and the work could make a valuabe contribution, especially from the perspective
of studying incentives to emergent structured behaviour in unsupervised learning,. However, the weaknesses mentionned above need
to be addressed to better establish the contributions of this paper, both in terms of understanding the generality of the
results and the contributions in relation to the existing litterature.

References:

Annabi, L., Pitti, A., & Quoy, M. (2020). Autonomous learning and chaining of motor primitives using the Free Energy Principle. arXiv preprint arXiv:2005.05151.
Kim, K. H., Sano, M., De Freitas, J., Haber, N., & Yamins, D. (2020, January). Active world model learning in agent-rich environments with progress curiosity. In International Conference on Machine Learning (ICML).
Lopes, M., Lang, T., Toussaint, M., & Oudeyer, P. Y. (2012). Exploration in model-based reinforcement learning by empirically estimating learning progress. In Advances in neural information processing systems (pp. 206-214).
J. Schmidhuber, “Curious model-building control systems,” in Proc. Int. Joint Conf. Neural Netw., Singapore, 1991, vol. 2, pp. 1458–1463.
Tschantz, A., Baltieri, M., Seth, A. K., & Buckley, C. L. (2020, July). Scaling active inference. In 2020 International Joint Conference on Neural Networks (IJCNN) (pp. 1-8). IEEE.
Tschantz, A., Millidge, B., Seth, A. K., & Buckley, C. L. (2020). Reinforcement Learning through Active Inference. arXiv preprint arXiv:2002.12636.

review_title: Interesting approach, but several issues need to be addressed

Meta-review: The paper is studying a new intrinsic motivation RL setup in a dynamic environment, where the authors minimize the state entropy instead of the common approach of maximizing it. The resulting idea is simple but also surprising that it works so well. All reviewers appreciated the new problem formulation of using dynamic environments and found the idea very promsing. In addition, they identified the following strengths of the paper:
- The experiments are exhaustive, identifying many domains where the approach can be applied
- The presented results are compelling
- The paper is well written
- The paper introduces a new problem setup that has not been studied before

I agree with the reviewers that this paper contains many interesting contributions and therefore recommend acceptance.


Decision: Accept (Oral)