Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces a novel latent state-space method for unsupervised reinforcement learning in dynamic partially-observed visual environments. The key insight of the paper is that minimizing the entropy of the belief over the latent state does not incentivize agents to construct a "niche", i.e. modify the environment such that it becomes more easily controlled. As a remedy, the authors propose to instead minimise the entropy of the belief over latent state visitation. The authors discuss a variety of reward incentivised constructed from the latent-space model and introduce a separate exploration policy. The authors evaluate on three separate single-agent environments, including one with continuous action spaces. It is found that the novel latent-visitation based reward incentives outperform existing methods on a variety of metrics.
main_review: The paper's fundamental idea about minimizing the entropy of the belief order latent-state visitation is original, and clearly marks a step change over previous approaches. The ideas are subsequently studied in appropriate detail and evaluated on a broad set of toy environments. The quality of both theoretical and empirical developments is high.
The paper is written very clearly. 

Overall, I believe this paper is very solid and deserves acceptance.





limitations_and_societal_impact: In Table 1 (OneRoomCapture3D), how is it possible that Niche Expansion score significantly lower in entropy, while Niche Creation+Infogain wins wrt both obj capture and obj visible metrics?

The analogy with Maxwell's demon may deserve further detailed explanation: It is not clear to me what exactly the "potential energy"-analogy constitutes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a new intrinsic reward signal for partially observable environments with dynamic objects that can potentially be controlled by an agent. This reward seeks to minimize the entropy of the agent's state visitation as estimated by a latent-space model.
To do so it makes use of a latent state estimation approach using a VAE and an ensemble of dynamics models, and specifies 4 reward functions that can be seen as optimizing the above objective through different means.

These proposed reward functions are evaluated on three domains with partial observability and dynamic objects that the agent's actions can somehow affect and potentially control. The experiments show that one or the other of the proposed reward functions lead to the agent controlling these dynamic objects, whereas previously proposed rewards fail to do so. Of note in the VizDoom domain, the paper shows that one of the proposed rewards leads to maximization of the extrinsic reward without learning to do so.
main_review: ## Technical Positives:
* Taking the problem formulation used for intrinsic control to the harder domain of partial observability with dynamic objects is a step forward towards more complex domains.
* This paper identifies what intrinsic control would mean in such an environment and contrasts it with approaches that have been used in the fully observable setting.
* It then proposes reward functions that would lead to such control being exerted by the agent, by manipulating the entropy of its state belief and visitation.
* These rewards seem to be validated in the three domains the paper evaluates on, though in practice it seems like the Niche expansion and Niche Creation + Infogain objectives are the ones that show results that are different and improvements on the baselines.

## Questions, Remarks, and Drawbacks:
* Since the proposed objective seeks to reduce the entropy of its state visitation, if left to its own devices it seems like the approach would also learn to go to a part of the environment where it does not encounter any dynamic objects and not move. The control aspect seems to arise from a well trained latent state model whose components include the encoding from observations to the latent state (which is explained appropriately in the paper) but also an exploration policy that learns both an accurate estimate of the latent state as well as an accurate estimate of the uncertainty of the dynamic objects. This exploration policy seems to not have been emphasized and explained enough in the paper.
* Is the same exploration policy used to collect data for the baseline methods? Would those methods benefit from a similar exploration?
* SMiRL is proposed as an intrinsic reward in CMPs that proposes rewards on the states encountered. This paper argues that the equivalent in the CHMP would be rewards computed based on observations. However I would argue that a reasonable equivalent would be to compute the SMiRL reward on the state estimated by the latent state encoder. Why is considering the observations a more appropriate analog for SMiRL? If the state used for estimating the reward in SMiRL is the latent state, then do the differences between the proposed rewards and SMiRL become more nebulous?
* In Figure 4 (b), it would be helpful if the reward functions depicted are further labeled with the names in Section 5.1, so that correspondence can be drawn more readily.
limitations_and_societal_impact: Discussed in supplementary material.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a set of novel objectives for intrinsic motivation that aim to encourage an agent to learn to exercise control over its environment, despite only having partial observability. The key motivating idea is that the agent be encouraged to reduce "Belief Entropy" by limiting the ability of the environment to change, particularly when that part of the environment is not being observed. Four different objectives are presented that encourage the agent to learn policies that achieve this goal and results are demonstrated on a handful of toy environments meant to illustrate the central claims of the paper. Among the results is good performance on the DefendTheCenter VisDoom task, in which the agent learns to shoot enemies without specific instruction to do so, as this prevents the enemies from continuing to exist and "surprising" the agent in the future.
main_review: *High-level Comments*

Overall the paper is clearly written and motivates an interesting problem and proposes a handful of intrinsic objectives that are shown to encourage the desired behavior. Yet while the core narrative of the paper is mostly solid (excepting some of the more minor comments below), the results are lacking and somewhat unconvincing. In particular, the results section is too short, and the results themselves do not support the authors' narrative in a way that leads to clear recommendations about when their approach (or which of their approaches) will perform better than others and when. The discussion of the results (aside from the tables, figures and captions themselves) exists across only a single paragraph in , problematically short considering that a number of the results do not "speak for themselves". There are a few such instances:

- Across all the different environments shown, the Niche Creation + Infogain and the Niche Expansion objectives compete for the best performance, yet /why/ each objectives performs better or worse or how their behaviors differ in a way that could lead to a recommendation across environment types is not explored. In particular, the number of monster kills for the DefendTheCenter environment is unintuitively low compared to the Random Policy (and *zero* for the NC+I objective) despite that killing monsters is described as behavior that we might expect across all policies. This behavior should be discussed.
- Second, the performance on the TwoRoom-Large environment includes only the Niche Creation + Infogain results. This is strange, seeing as it is the only environment for which this is that case.
- Relatedly, the text highlights that the Niche Expansion performs well for the DefendTheCenter but it is unclear why this is the case or what the reader should hope to understand, important details to include if the goal is to prescribe how this approach can be used across environments.

Expanding on the results section to address these points and to better understand when and why each objective is well-suited to its task would greatly strengthen the paper.

*Smaller comments and suggestions*

- To the best of my knowledge, the nature of the "oracle policy" is not elaborated upon. It is unclear how an agent with "access to privileged about the environment state" would achieve good behavior on the objectives of interest, which seek to maintain control over the belief. This would help to better clarify the results and the relation of the proposed techniques against this oracle.
- The addition of the exploratory data and policy intuitively seems an important part of the ability of the chosen approach to succeed. Yet it appears that the approaches the authors compare against (SMiRL and Empowerment) do not include such a mechanism to augment data collection. It may help understanding if the authors comment on how this may influence the comparison between the various approaches.
- The limitations of the approach should not exist only in the supplementary material. In particular, some discussion should be included in the paper regarding in what environments the proposed approach would be expected to either perform poorly or result in undesirable behavior would be a welcome addition.

Edit: The authors have provided significant additional detail in response to my questions and I am satisfied by the depth of their investigation. While I still feel as if the approach has some limitations (and the authors have agreed to expand the discussion of these in the relevant section of their paper) I do not think these limitations diminish the quality of the work. I was also persuaded by the other reviews that I was perhaps too harsh on these grounds. I have updated my score accordingly.
limitations_and_societal_impact: Though the limitations and societal impact are touched upon in the supplementary materials, it is worth expanding on the impacts in a bit more detail. In particular, it would be meaningful to briefly expand on some example environments in which the proposed method would be likely to result in undesired or dangerous behavior.

Edit: The authors have agreed to address this point in their updated draft, and I am satisfy by their reply in this regard.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the problem of designing intrinsic rewards that encourages active exploration in partially observable dynamic environments. The authors observe that by minimizing the entropy of the latent visitation frequency, the agent is able to achieve both the active information gathering and prevent the environment from changing. Specifically, the authors introduce Believer, a novel framework based on variational latent Bayes filters with discrete latent states, and designed several rewards. In the experiments, Believer has achieved good performance on 3 different tasks.
main_review: I think this is an interesting paper with good insights. Reinforcement learning in partially observable environments is always a challenging task, and a good intrinsic reward can significantly improve the exploration and the overall performance of the system. Believer has made a good observation of using the latent visitation frequency for encouraging both the active information gathering and interacting with the environment. Furthermore, the discrete latent states make evaluating the visitation frequency possible.

Overall, I think the paper has made enough contributions. I have some minor suggestions about the presentation and the experiments. Please find detailed comments below.

limitations_and_societal_impact: 1/ I think the current experiments are all conducted in relatively simple domains. It would be good to see Believer applied to more complex domains, e.g., robot navigation in REALISTIC environments. The current experiments have small spaces, and it would be good to see if the proposed intrinsic reward is still good for much more complex environments. In addition, a minor point is that it is a bit unclear if the proposed LSSM is still sufficient for modeling the complex environments.

2/ Is it possible to visualize the learned latent states under different intrinsic rewards? I think it would be nice to have some visualizations of how the beliefs are being updated through time.

3/ How does the size of the latent belief affect the performance? Will a larger belief size keep improving the performance? Or is it possible that with a slightly bigger belief size, other variants also achieve comparable performance with Believer?

4/ Some notations are undefined. For example, in line 107 and 108, \hat{p}(o) and \hat{p}(s) are undefined.
review_title: null

Meta-review: Learning agents need to be able to explore and control their environment. The authors introduce a novel objective, the entropy of the state visitation, which combines these two desiderata and is tightly connected to information gain for controlling environments. 

The authors show promising results on a number of benchmarks created to highlight partial observability and clearly ablate the different components of their algorithm on these environments. 

The discussion focussed on a few key points: The authors provided a more detailed comparison to SMiRL, added additional experimental data as requested by the reviewers and clarified technical points. Overall, the reviewers felt that their concerns had been addressed appropriately by the authors.

Given the positive reviews, the discussion and the overall contributions of the paper, I recommend this paper to be accepted.

Decision: Accept (Poster)