Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to learn robust representation from the replay buffer for reinforcement learning. The key idea is to leverage the concept of mutual information and the InfoNCE tool to compute the mutual information as a regularizer (a.k.a. DRIBO loss). The authors conducted experiments on some standard benchmarks (DeepMind Visual Control Suite and ProcGen). 
main_review: Strength:
- To me, the final experiment result on ProcGen is impressive.
- Supplementary material shows that the authors seem to have digged into the experiment data with some depth. 

Weakness:
- The writing is not so clean. Needs polishing or even rewriting.
  - The paper introduction is quite vague and covers very general ideas, lacking specificity and advantage to the proposed approach; 
  - In Eq (5) and (6), there abruptly comes the Lagrange multiplier. Many other places also show that the writing is very rough.
- The results on DeepMind Control Suite are somewhat marginal improvement. 
- While the results on ProcGen are impressive, the analysis and interpretation into these results are somewhat thin. Because the method involves many differences in comparison to the baselines, it is unclear whether the proposed main idea, mutual information loss, is the source of the major contributions. To me, the breakdown of contributions of each design choice (esp. including fair comparison with baselines with the same type and level of data augmentation) is important. 
summary_of_the_review: Although there might be some good core contribution, the paper in its current state is not quite ready for publication. See details in my Main Review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of generalization amidst visual distractors for control tasks. In particular, the distractors have no dependence on the optimal policy and thus clearly form a task-irrelevant component. The proposal is to use mutual information between two views as a proxy for how much task-relevant information is present in the constructed representation. This objective is adapted for RL to consider the long term sequential nature. Finally, the authors test this approach on ProcGen and DMC Suite with distractors, while also performing certain ablations. 
main_review: The paper is well motivated and the main problem is interesting in my opinion. The related work is well covered and most of the writing flows well. Experimenting over two benchmarks (ProcGen + DMC) is quite extensive relative to other similar papers. 

I start to have issues in understanding around Definition 1. I get the overall concept, that we can deconstruct the overall mutual information in a task-relevant and a task-irrelevant terms. However, why should we maximize the sum of both (as noted in the 5th line below Eq. 2)? You say maximizing I(O, S) gets you a sufficient representation. That would mean maximizing the first term as well. But you also say that the first term needs to be minimized. This seems contradictory. I understand Theorem 1 and also get the derivation of the components of Eq. 5 and 6. However in Eq. 5, do we want to maximize both terms? One is the task irrelevant term (aka minimality) while the other is the task relevant term (aka sufficiency). They should have opposite signs right (as is in Equation 7)? 

Finally, regardless of the derivation of the DRIBO loss, my main concern is the empirical results here. The scores reported for DMC are not accurate in my opinion. I have run RAD on the distractor suite and it gets much better performance than what is reported. Why is RAD performance this low? DRIBO is using augmentations from RAD to generate the multi-view observations, and so as a first baseline, it should be performing better than RAD. Note that RAD achieves the same score (roughly) as reported in the original paper even when run with distractors. Can the authors clarify this?

Moreover, the performance reported for the proposed method is not good enough. In particular, running SAC with a reward prediction head achieves similar or even better performance than of the proposed method. In light of these two observations, I am not convinced that DRIBO really is doing more than a simple baseline, and certainely not doing so much better than RAD.

For the ProcGen experiments, why don't you compare with DAAC [1] and IDAAC [1] since I believe they are the state of the art methods on ProcGen currently.

Why compare the t-sne with CURL? We already know that CURL does not do a good job and that RAD is much better and even simpler. Also, in Figure 3, CURL is outperforming RAD in some of the tasks, which I think is again almost certainly not the case (based on the experiments I have run with RAD).

References:

[1] Decoupling Value and Policy for Generalization in Reinforcement Learning
summary_of_the_review: The paper is motivated by an interesting problem to the community, proposes a fairly novel method for leraning better representations, but there remain certain gaps in my understanding currently. I hope the authors can clarify this during the rebuttal. In any case, the experimental results are not correct in my prior experience of running these exact methods. For this reason, I am advocating for a reject. I'm happy to revise my review if these issues can be addressed. 


-------------------------- Post Rebuttal ----------------------------------

The new results provided by the authors convince me fairly that the method is somewhat useful. I am still skeptical about the RAD results and the overall scores reported for DMC tasks, since the absolute numbers are not state-of-the-art. This might be due to a different evaluation setting where the distractor videos are changing across training, and at test time a video is drawn from a different distribution. I am unsure how much this sort of an evaluation scheme affects RAD's results, which should produce much better numbers than reported. I am giving the benefit of doubt here to the authors and hoping that they have done a fair evaluation for the baseline RAD method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of pixel-based control with reinforcement learning. The authors categorize observation changes into task-relevant and task-irrelevant changes. They define task-irrelevant changes as those do not have casual relations with actions, and introduce a conditional prior to compress task-irrelevant information, such that agent can focus on task-relevant information. The authors evaluated the proposed DRIBO method on pixel-based control tasks (DM Control) with natural video background and the procgen suite.
main_review: Strengths:
* The paper shows a technically approach to compress task-irrelevant information, making the agent focus on task-relevant information.
* The paper shows good empirical improvements on DM Control with natural video background and procgen, compared to existing methods in system-wise comparisons.
* Comprehensive ablation studies on DM Control that identify the importance of history length and information compression.

Weaknesses:
* It’s not clear to me whether there are as many task-irrelevant observation changes in procgen as in DM Control with video background. The generalization requirement in procgen seems to be different. I would like to see if the authors can clarify how the procgen experiment align with the main story of this paper. 
* Many of DRIBO’s design choices (using RSSM, long history length, etc) are different to existing methods. The system-wise comparison on procgen (table 1) doesn’t provide much insight on why DRIBO improves generalization on procgen. 
* The authors claim that $I(S_t^{(1)};S_t^{(2)}|S_{t-1},A_{t-1})$ can be maximized by maximizing $I(S_t^{(1)};S_t^{(2)})$. I don't think this is mathematically correct without making certain assumptions.

Smaller presentation issues:
* In Sec.3, the author states that "task-irrelevant information does not contribute to the choice of actions". I think it should be "task-irrelevant information does not affected by choice of actions".
* The author parameterizes mutual information as like $I_\theta$. Mutual information is a true measurement and can't be parameterized. Only MI estimates can be parameterized.
* CEB is a generic information bottleneck. It's incorrect to say that it cannot be applied to sequential data.
* The appendix shows in the main paper file.
summary_of_the_review: The approach is novel. The paper shows good empirical results on DM Control with natural video background and procgen. However, it's not clear to me why the procgen-type of generalization and the procgen experiment are relevant to the main story and I have some concerns about the mathematical correctness. I'm slightly leaning to acceptance at this point, but I hope the authors would be able to clarify.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Motivated by the fact that RL agents are sensitive to unseen environments, this paper proposes a method to separate task-relevant and task-irrelevant information from observations based on unsupervised multi-view settings. They use data augmentation methods to create multi-view data for calculating contrastive Multi-View Information Bottleneck (MIB) and use it as the objective to increase the generalization of model representation.
main_review: 1. Strengths and Weaknesses:

\+ The perspective of using mutual information to increase the robustness is interesting. The authors provide clear analyses of how to use mutual information to learn task-irrelevant information based on multi-view settings. 

\+ The proposed method is a plug-in penalty that can be integrated into many modern RL algorithms. 

\- The representation encoder and the estimator of mutual information (using InfoNCE) are borrowed from existing works, which may limit the contribution of this work.

\- The proposed methods show advantages in DMC and ProcGen environments. However, the settings in both environments are very similar: changing the background image in the same task. I am not sure if there are any real-world tasks that can be better solved by the proposed method. It would be great if the author can conduct experiments on other realistic tasks.


2. General questions

(1)	How is the data augmentation conducted to create the multi-view setting? For example, in Figure 1, is the entire image rotated or only the background rotated? If the entire image is rotated, will the state change and the action not be consistent with the state?

(2)	How to create the multi-view setting for layouts in the ProcGen suite? In my understanding, the layout of environments is corresponding to the difficulty level. How to conduct image augmentation on such difficulty levels? Or, do the author only change the background image in ProcGen? 

(3)    Based on (2), I am also wondering if this method can be extended to more general settings to increase the robustness?
summary_of_the_review: In general, I think the proposed idea of this paper is interesting, and the theoretical analysis seems correct to me. However, I still have questions about the details of how to implement the multi-view settings (In my general question (1)). Also, the experiment part seems a little weak since the two environment suites are very similar. 

I think this paper is marginally above the borderline. I may raise my score if the author provides reasonable answers to my question.
review_title: null

Meta-review: The authors introduce a method that improves the representation learned by RL agents, making them more robust to visual distractions. In particular, their approach proposes to use mutual information between two views as a proxy for that objective. This is clearly a borderline paper that required many discussions among the reviewers and the authors. The reviewers mention that the approach is novel, addresses an important problem of robustness in RL and some of the experiments provided are impressive. On the other hand, the reviewers point out that the baselines seem to achieve lower results than previously reported, writing could be improved and some of the results don't show significant improvement over baselines. 

Given that some of the results cause confusion around the evaluation protocol (it's still not 100% clear why the performance of baselines is lower than expected) and other doubts expressed by the reviewers, I encourage the authors to continue working on the paper and resubmit. I believe that with a little bit of extra work and clarifications this can be a very strong submission.
Decision: Reject