Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this paper, the authors introduced Dynamic Bottleneck (DB) model that learns dynamics-relevant representations based on the Information Bottleneck principle. They further proposed DB-bonus for efficient exploration and established theoretical connections between the proposed DB-bonus and provably efficient bonuses. The experiments show that the proposed method outperforms several strong baselines in stochastic environments for self-supervised exploration.
main_review: Pros:
1. The information-based DB method is novel in combating noise in observation. 
2. The variational methods help to scale the method up to solve high dimensional problems.
3. The authors show the link of the DB bonus to two provably efficient cases: linear MDP and tabular MDP.
4. The ablation studies are sufficient that unveil interesting patterns latent space.

Cons:
1. I am a little bit confused about the definition of “noisy states”. In the introduction, the authors give an example: “For example, in autonomous driving tasks, the states captured by the camera may contain irrelevant objects, such as clouds, birds, and aircraft.” However, most of these objectives may would have their own dynamics, it is just those dynamics are irrelevant to rewards. Could the proposed method be able to handle disturbance with dynamics? If not, I suggest change this example and make it clear in the paper. If so, then maybe add experiments in this case.

Typos:
1. The “log” operators are missing in Eq. 7. 
2. There are some obvious grammar errors.
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new self-supervised exploration method for RL that encourages an agent to explore states which are more relevant in terms of information gain. To this aim, the Information-Bottleneck (IB) principle has been utilized to learn a model that can obtain dynamics-relevant information and discard dynamics-irrelevant features together. The paper evaluates the proposed technique in multiple Atari environments without accessing the extrinsic rewards. 
main_review: Overall, the main idea of the paper is interesting and it is well explained and well written. In addition, the experiments in this paper support the main claim of the paper and show better performance than some of the curiosity-driven methods. 

1) While the idea of the paper seems promising, utilizing information-bottleneck for exploration has been well-studied and it seems this paper has a good overlap with previous works such as [1] and [2]. The main difference that I see between this work and previous works is using contrastive learning. Is it fair? or are there many more differences between them?  

2) It is not clear to me why I([St, At]; Zt) should be minimized rather than maximized? Since the main claim of the paper is to acquire dynamics-relevant information and discard dynamics-irrelevant, why does minimizing I([St, At]; Zt) not result in discarding "relevant" information? I might miss something here, but can the authors elaborate on this?

3) There'd have been interesting if the authors had included experiments in which reward = DB-Bonus + extrinsic-reward for a game like Montezuma’s Revenge. The reason I mention this is because if combining these two works, it further supports the claim of this paper (i.e. acquiring dynamics-relevant information and discard irrelevant ones).

4) It is not clear how negative samples are collected for Eq 5? What does "sampling observation encodings randomly from the batch" [line 143] mean? Don't we do the same thing for regular samples ( i.e. positive ones)? 

5) Is it fair to say that Theory 1 is not that useful and relevant as it only holds without the contrastive objective? as such it shouldn't be claimed as a contribution in this paper.

6) SSE-DB seems only better than 12-13 tasks, not 15 tasks (Not better in Tennis, Crazy Climber, Centipede, Time Plot, MasPAcman). Refer to results in Figure 3. Is that right?

Minor comments:
- Explain the information-bottleneck principle in the paper or at least in the appendix. The paper should be self-contained.

- Explain what is "data processing inequality" in line 222.

- What is \mu in line 191?

- Use a different notation for variational approximation r(z_t), it can be mistaken with reward function.


[1] Curiosity-Bottleneck: Exploration By Distilling Task-Specific Novelty, Youngjin Kim, Wontae Nam, Hyunwoo Kim, Ji-Hoon Kim, Gunhee Kim

 [2] EMI: Exploration with Mutual Information 
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, Hyun Oh Song
limitations_and_societal_impact: This paper discusses a new method to encourage an RL agent to explore state-action pairs with high information gain which doesn't have a social impact. As such, I don't think this work has any negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper propose an exploration method for reinforcement learning by introducing a dynamics bottleneck (DB) model that captures dynamics-revelant information from the states. To this end, they propose to maximize the mutual information between $Z_{t}$ and $S_{t+1}$ but minimize the mutual information between $Z_{t}$ and $(S_{t}, a_{t})$ to compress representation. Intrinsic reward based on learned DB-model. The proposed method is evaluated on tasks from Atari Learning Environment and corrupted Atari tasks.
main_review: Proposed method is not surprising, but seems novel and reasonable, writing is clear, and its effectiveness is supported by extensive experimental results. I like the paper in general, but I have some minor concerns that could be addressed easily, and i'm willing to increase the score when the concerns below are resolved:

- Positioning of the proposed method in the paragraph 2 of introduction is a bit questionable. For example, representations learned by inverse dynamics model you mentioned as 'curiosity-driven explorations' should capture controllable features (relevant discussions are in [Pathak'17; Badia'20]), which in principle should not be affected by dynamics irrelevant objects like birds. While this aspect ​of capturing dynamics-relevant information is the very important characteristic of the proposed method and the method could be very good at doing it, it could not be its uniqueness compared to other related works. Could you elaborate on this and provide more intuitive explanation and supporting evidence why the proposed method should be better than the previous works?

- While I appreciate that experimental results are extensive, but important and more recent baselines for self-supervised exploration are missing, like VISR [Hansen'20], APT [Liu'21a], Plan2Explore [Sekar'20], which were all available at the time of submission (via conference, arXiv, OpenReview). As a note, much more recent works like APS [Liu'21b], RE3 [Seo'21], ProtoRL [Denis'21] are also relevant.

- How are hyperparameters ($\alpha_1, \alpha_2, \alpha_3$) are selected? Is the proposed method sensitive to the choice of these hyperparameters?

Minor
- line 38: discards -> discard
- line 137: plays
- line 292: observer - > observe

[Pathak'17] Curiosity-driven Exploration by Self-supervised Prediction

[Sekar'20] Planning to Explore via Self-Supervised World Models

[Hansen'20]  Fast Task Inference with Variational Intrinsic Successor Features

[Liu'21a] Behavior From the Void: Unsupervised Active Pre-Training

[Liu'21b] APS: Active Pretraining with Successor Features

[Seo'21] State Entropy Maximization with Random Encoders for Efficient Exploration

[Yarats'21] Reinforcement Learning with Prototypical Representations
limitations_and_societal_impact: The draft does not address limitations and potential negative societal impact in the main text. They should include the relevant discussion in the main text following the suggestion of NeurIPS conference.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce the dynamics bottleneck which uses the information bottleneck to discover the novelty of state-action pairs to introduce a reward bonus, used for exploration in RL tasks. The authors evaluate their method on top of PPO in the Atari domain. 
main_review: Strengths:

- Overall, the empirical results look good compared to related approaches.
- Theoretical results drawing the connection between the proposed approach and different bonuses in the exploration literature shows this approach is reasonable. 
- I thought the supplementary was very complete, reproducibility of the paper is very high. Supplementary includes an ablation study. 

Weaknesses:

- For a paper that aims to tackle the task of exploration, I thought the choice of environments was a bit lacking in specifically hard exploration tasks such as Montezuma's revenge or modified sparse reward environments. For example [1] lists (Gravitar, Montezuma’s Revenge, Pitfall!, Private Eye, Solaris, and
Venture) as hard exploration tasks. Out of these 6, the 2 in tested in this paper (Solaris and Gravitar) have mostly flat learning curves. Compared to the learning curves from Dopamine [2], the performance of SSE-DB doesn't seem to be much better than the randomly initialized policy on these tasks. ~~According to the PPO paper [3], on Gravitar (Solaris results were not included in this paper), vanilla PPO can achieve a performance of 500-750+ after 40M frames, which is than SSE-DB in fewer frames.~~ (Edit: not a valid concern)
- Clarity on novelty. One thing that wasn't too clear to me after the reading the paper is what of the approach was novel and what was based on prior work. For example 3.1 and 3.3 are also described in the CB paper [4] (with references to prior work), so it isn't clear to me if there is novelty here, or if the novelty was limited to viewing the information bottleneck over the dynamics. A similar concern for 3.2 as (as mentioned in the related work) there has been literature on contrastive learning in RL. 

Minor:
- In the section 5.2 on the experiment in the presence of noise, is SSE-DB was only using the learned reward bonus, or is the learned representation used in some more meaningful way? 

References:
- [1] Burda, Yuri, et al. "Exploration by random network distillation." 2018.
- [2] Castro, Pablo Samuel, et al. "Dopamine: A research framework for deep reinforcement learning." 2018.
- [3] Schulman, John, et al. "Proximal policy optimization algorithms." 2017.
- [4] Kim, Youngjin, et al. "Curiosity-bottleneck: Exploration by distilling task-specific novelty." 2019.
limitations_and_societal_impact: This is addressed. Although I would encourage the authors to move their comments from the checklist to either the main paper or the supplementary material.
review_title: null

Meta-review: Paper presents a method for intrinsic exploration based on learning a representation of observation that can ignore distractors using the dynamics bottleneck principle. Optimality of the exploration bonus in the case of bandits is established. Results on standard benchmarks used in the exploration literature are convincing. Reviewers unanimously vote to accept the paper, which I agree with. 
Decision: Accept (Poster)