Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a generic saliency-guided method (SGQN) for visual reinforcement learning, aiming at improving the generalization capabilities and robustness of deep reinforcement learning policies. This is based on two mechanisms introduced by authors, both depending on masking the input image based on a computed attribution (saliency) map, and therefore inducing the agent to pay more attention where it is looking. The first consists in regularization of the value function with a consistency term (computed form Q values of original and masked input) that encourage the value function to depend mainly on pixels with high attribution. The second is exploiting data augmentation while introducing an auxiliary self-supervised learning objective where the agent trains to predict its own Q-value's saliency maps, based on features encoding information about pixels relevant to make decisions. 
strengths_and_weaknesses: The paper overall quality is good and the contribution is significant. The context of the work is clearly illustrated with a proper literature review, and specific contributions are clearly described compared to existing approaches. While building up over existing methods, such as regularization and data augmentation, the paper introduces an original and significant architecture for saliency-guided RL. Differently from the majority of current approaches, in which saliency maps are used as tool for interpreting the RL agents behavior, this work incorporates them in the learning process. 
The approach is experimentally validated and compared with state-of-the-art methods for generalization in continuous actions RL, demonstrating its significance in improving training efficiency, generalization capabilities, and providing a straightforward way for policy interpretability that does not require to compute the actual saliency map. Results are reported in a clear and complete way, and they are exhaustively discussed.
An advantage of the proposed method is that it can be combined with other techniques for attribution map computation and image augmentation, value function learning objective, and it is suited for both continuous and discrete action spaces, therefore fostering reusability.

questions: - In Background and related work, when methods to improve generalization capabilities are presented, there are many acronyms that are never explicitly expressed. I think it would help readability writing the full name at least one.
- In the experimental comparison with other methods for generalization, and considering the background paragraphs, SAC and SODA belong to the Data augmentation approach. To which approach do RAD and SVEA belong (it is stated they all include data augmentation in one of their stages)? From my understanding, SGQN mix some aspect of data augmentation (which is used to generate the attribution mask), attribution and regularization methods. Could you say something about its advantages and limitations compared to methods that are not considered in the comparison, such as methods based on representation learning?
- How is the image augmentation function defined (at line 256 it says random overlay augmentation)? This means initial saliency maps are randomly generated? It seems to me that the mechanism works if initial saliency maps are properly defined, and therefore if there is a previous knowledge about which pixels are relevant to the task. What happens if the initial map is not good enough?
- In Fig. 6, I would make the caption more descriptive, specifying explicitly what is shown in each subplot.
- line 318: from Table 1 it seems SGQN outperforms SODA on 5 (not 3) out of 5 video hard environments.
limitations: Authors adequately address limitations and perspectives of their research in the Conclusion section. I share their view about the fact that the main limitation in the use of saliency maps to guide the RL learning process is that these strongly rely on a human point of view (and prior knowledge about the task).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents saliency-guided Q networks (SGQN), which use saliency maps of the Q network to guide learning. The goal is to focus attention on pixels that are relevant to the task, while ignoring irrelevant ones, to aid in visual generalization at test time. SGQN is evaluated on the DM Control generalization benchmark with video backgrounds, and compared with SAC, RAD, SODA, and SVEA.
strengths_and_weaknesses: Strengths
* Visual generalization in RL is an important problem.
* The "video hard" evaluation seems like a challenging test of generalization.

Weakness
* To me, the method seems cyclic. The saliency maps from the Q network are used to influence how the Q network is trained.
* Some of the baseline numbers don't seem to align with what was reported in prior work. For example, in Table 1, the finger spin (easy) performance for SODA is 535, but in the SODA paper it is 695 (higher than the 609 reported for SGQN). Similarly, SVEA is 537 (Table 1) vs 808 (original paper). These numbers are averaged across 5 random seeds, right?
* The paper is poorly organized. For example, Section 3 (the approach) has no subsections or paragraphs, making it hard to follow.

-------
Review update:
I thank the authors for their responses and additional experiments.

I looked in more detail at saliency-guided training (e.g. in Ismail et al.) and this technique seems to have some merit. My concern about the cyclic nature of the method was that it is only applying large gradients to the Q network and discarding small ones, which seems like it could result in unstable training and/or suboptimal use of the model parameters. Expanding the discussion of saliency-guided training (such as including the findings from Ismail et al. in related work) would help mitigate these concerns.

L293: “SGQN yields sharp saliency maps.” This evaluation in particular is cyclic since SGQN used this saliency technique in training while baselines did not. This point should be mentioned, or even better a different attribution technique should be used for this evaluation. See discussion of Goodhart’s Law in Ismail et al.’s OpenReview forum: https://openreview.net/forum?id=x4zs7eC-BsI

Paper organization: I would say the background and the method’s motivation should be separated out from related work. Regarding the method section (3), I find myself having to jump around quite a bit. For example, if I want to understand how the image augmentations are used, I must look at  L161, L177, L218, Figure 1, and Algorithm 1 (though τ is not actually used in the algorithm definition). Instead, the approach section could have clear subsections or paragraph labels for each learning phase (consistency and self-supervised objectives). The parts of the text and figure 1 could more clearly refer to each other. Algorithm 1 could probably be moved to the appendix since most of it does not relate specifically to the method.

My baseline numbers and ρ values questions have been resolved.

The primary reason for my low rating was the concern about the cyclic nature of the saliency-based training, which as I mentioned has mostly been resolved (since it has shown up in prior work). Thus I have increased my overall, soundness, and contribution ratings. I still encourage the authors to improve the discussion and organization of the text in the ways I have described.
questions: I don't think changing the quantile value ρ hyperparameter for different tasks is a fair comparison (Appendix table 3). This would require doing hyperparameter search for any new task encountered. For results, just report for the same value across all tasks.

L224: "relying on the assumption that the initial saliency maps are reasonably good." Why is this a good assumption? Some issues with saliency maps have been discussed in prior work, e.g. "Sanity Checks for Saliency Maps" from Adebayo et al.

Minor comments (no need to respond to these):
* I would not say that Deep RL has "outstanding efficiency in simulated visual control tasks" (L1). 
limitations: There should be discussion of the limitations of the approach, which might talk about drawbacks of saliency maps, computational cost/speed, etc.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a saliency-guided Q-networks (SGQN) method that guides the policy to focus on important pixels for improving generalization sample efficiency. SGQN has two major components: 1) force the saliency of the augmented image to be close to that of the original image, 2) force the Q values to be close. The experiments show that it outperforms existing state-of-the-art methods on DMC-GB.
strengths_and_weaknesses: strengths:
1. This work shows that the learned policy could largely improve the generalization ability in the DMC-GB with self-supervised learning.
2. The learned policy could be interpretable with the use of saliency map.
3. The paper is well-written.

weaknesses:
1. The proposed method is only applied on DMC-GB. It is not sure that the proposed method is useful in other tasks other than the DMC-GB.
2. SGQN is a DrQ based algorithm since the batch size data is augmented by random shift. Then there are various places the authors can edit and use DrQ instead of SAC.
3. SODA+SVEA can be a stronger baseline since you implement two loss functions that are similar to the losses in the two papers respectively. To better illustrate the superiority, a comparison is preferred: the improvement does come from the saliency representation rather than any representation or embeddings with these two losses.
4. ‘One can note that agents trained with our self-supervised learning objective also outperform SODA agents (Table 1) on 4 out of 5 video easy environments and on 3 out of 5 video hard environments.’ This comparison result seems problematic according to the Table 1(SODA) and Table 2 (SAC+Self learning).
5. Missing references:

Fu, X., Yang, G., Agrawal, P., & Jaakkola, T. (2021, July). Learning task informed abstractions. In *International Conference on Machine Learning*
 (pp. 3480-3491). PMLR.

Yuan, Z., Ma, G., Mu, Y., Xia, B., Yuan, B., Wang, X., ... & Xu, H. (2022). Don't Touch What Matters: Task-Aware Lipschitz Data Augmentationfor Visual Reinforcement Learning. *arXiv preprint arXiv:2202.09982*.Chicago
questions: 1. The authors pick a large number for thresholding the binary mask. Why is this number selected? For a different task, do you need to choose a different number? 
2. How does the proposed method compare with SODA+SVEA?
3. How does the proposed method perform on other benchmarks?
limitations: The authors mention their limitations as future research directions such as explanations and theoretical analysis of the proposed method. We appreciate the authors for being upfront.
review_title: null

Meta-review: metareview: The paper proposes a method for ignoring task-irrelevant background information for RL algorithms trained directly from visual inputs. The method consists of two parts: (a) a consistency loss; (b) a self-supervised learning objective on masks. Reviewers unanimously agree that contributions made by the paper are significant, and I agree. The authors should address the remaining minor concerns of the reviewers about the presentation and keep the promises made in the rebuttal in the camera-ready version. 
confidence: Certain
Decision: Accept