Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning
Authors: David Bertoin
Adil Zouitine
Mehdi Zouitine
Emmanuel Rachelson
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement learning
Generalization

Abstract: Deep reinforcement learning policies, despite their outstanding efficiency in simulated visual control tasks, have shown disappointing ability to generalize across disturbances in the input training images. 
Changes in image statistics or distracting background elements are pitfalls that prevent generalization and real-world applicability of such control policies.
We elaborate on the intuition that a good visual policy should be able to identify which pixels are important for its decision, and preserve this identification of important sources of information across images. 
This implies that training of a policy with small generalization gap should focus on such important pixels and ignore the others. 
This leads to the introduction of saliency-guided Q-networks (SGQN), a generic method for visual reinforcement learning, that is compatible with any value function learning method. 
SGQN vastly improves the generalization capability of Soft Actor-Critic agents and outperforms existing state-of-the-art methods on the Deepmind Control Generalization benchmark, setting a new reference in terms of training efficiency, generalization gap, and policy interpretability.

Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning
David Bertoini
Adil Zouitinet IRT Saint-Exuper ISAE-SUPAERO Toulouse, France
IRT Saint-Exupéry
IMT, INSA Toulouse ANITI
adil. www.:otiiieerrrrtaaattteepeepppp .oom
Toulouse, France
david. perroinnlrrt-saattexperyy com
Mehdi Zouitinet
Emmanuel Rachelson
IRT Saint-Exupéry
ISAE-SUPAERO
IMT, Université Paul Sabatier
Université de Toulouse
Toulouse, France
ANITI
mehdi wtt.:iooottieeeitttraeetatteeeee
Toulouse, France
emmanuel. simmauull.aachlloooddssaa---ppge
Abstract
Deep reinforcement learning policies, despite their outstanding efficiency in simu- lated visual control tasks, have shown disappointing ability to generalize across disturbances in the input training images. Changes in image statistics or distract- ing background elements are pitfalls that prevent generalization and real-world applicability of such control policies. We elaborate on the intuition that good visual policy should be able identify which pixels are important for its decision, and preserve this identification of important sources of information across images. This implies that training a policy with small generalization gap should focus such important pixels and ignore the others. This leads to the introduction of saliency-guided Q-networks (SGQN), a generic method for visual reinforcement learning, that is compatible with any value function learning method. SGQN vastly improves the generalization capability Soft Actor-Critic agents and outperforms existing oftth---atrtt methods the Deepmind Control Generalization bench- mark, setting a new reference in terms of training efficiency, generalization gap, and policy interpretability.
Introduction
Deploying reinforcement learning (RL) [Sutton and Barto, 2018] algorithms real-life situations requires overcoming number of still open challenges. Among these is the ability for the trained control policies to focus their attention on causal state features and ignore confounding factors [Machado et al., 2018, Henderson et al., 2018]. In visual RL tasks, this implies for instance being able to ignore the background and other distracting factors, even when they might be somehow correlated with progress within the task hand. Despite very active trend of research on the topic of closing the generalization gap for RL agents [Cobbe et al., 2019, 2020, Song et al., 2019, Hansen et al., 2021, Hansen and Wang. 2021], current algorithms are still rather brittle when it comes to filtering out such distracting factors, which hinders their applicability to real-life scenarios.
These authors contributed equally to this work
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In the present work, we propose a novel method which encourages the agent to identify efficiently crucial input pixels, and strengthen the policy's dependency on those pixels. In plain words, we encourage the agent to pay attention to, and be self-aware of where it looks in input images, in order to make its decision policy more focused on important areas and less sensitive to ambiguous or distracting pixels. This intention is expressed within the generic method of saliency-guided Q-networks (SGQN), which can be applied on any approximate value iteration based, deep RL algorithm. SGQN relies two core mechanisms. First, it regularizes the value function learning process with a consistency term that encourages the value function to depend in priority on pixels that are identified as decisive. The second mechanism pushes the agent to be self-aware of which pixels are responsible for making decisions, and encode this information within the extracted features. This second mechanism translates to superviie learning objective, where the agent trains to predict its own Q-value's ' saliency maps. In turn, this improves the regularization of the value function learning phase, which provide better labels for the self-supervised learning phase, overall resulting in a virtuous improvement circle.
SGQN is simple, generic method, that permits many variants in the way the two core mechanisms are implemented. In the present paper, we demonstrate that applying SGQN to soft actor-critic agents [Haarnoja et al., 2018] dramatically enhances their quality on the DMControl generaliza- tion benchmark [Hansen and Wang. 2021], a standard evaluation benchmark for generalization in continuous actions RL. SGQN already improves the training efficiency of such agents in domains without distractions. But most importantly, it sets a new state-of-the-aee in terms of generalization performance, in particular in especially difficult benchmarks where previous methods suffered from confounding factors. As a side benefit, it also provides explanations of its own decisions at run time, under the form of interpretable attribution maps, with overhead cost and need to compute ad hoc saliency maps, which is another desirable property in the pursuit of deployable RL.
Section 2 of this paper introduces the necessary background and state-of-the-art in closing the generalization gap in RL, as well as attribution methods, leading to the key intuitions underpinning SGQN. Section 3 introduces the method itself and implements it within soft actor-critic agents. Section evaluates SGQN's s training efficiency, generalization capabilities, and policy interpretability. It also discusses the different design choices made along the way and some foresceable limitations. Section 5 summarizes and concludes this paper.
2 Background and related work
Reinforcement learning (RL). RL [Sutton and Barto, 2018] considers the problem of learning a decision making policy for an agent interacting over multiple time steps with a dynamic environment. At each time step, the agent and environment are described through a state E S, and an action a E A performed; then the system transitions a new state s' according to probability P(s'|s, a). while receiving reward R(s, a)). The tuple M = (S, P, PR) forms a Markov Decision Process (MDP) [Puterman, 2014], which is often complemented with the knowledge of an initial state distribution (s).). A decision making policy parameterized by 0 is a function Ty (als) mapping states to distributions over actions. Training a reinforcement learning agent consists finding the policy that maximizes the discounted expected return (TT)) = E EI R'R(st, az)].
Poor generalization in RL. Despite the recent progress of (deep) RL algorithms in solving complex tasks, a number of studies have pointed out their poor generalization capabilities. Using a grid-world maze environment, Zhang et al. (2018c] demonstrate the ability of deep RL agents to memorize non-trivial number of training levels with completely random rewards. Using attribution methods, Song et al. [2019] highlight what they define as observational overfitting i.c., the propensity of RL agents to base their decision on background uninformative elements observed during training, instead of the semantic pieces of information one could intuitively expect such as object positions or relations. Zhang et al. [2018a] measure the generalization error in continuous control environments by training and testing agents on different sets of seeds. Zhao al. [2019] define generalization in RL as robustness to a distribution of environments, and samples environments from this distribution to learn a robust policy. Overall, these works illustrate the lack of generalization abilities of vanilla deep RI. algorithms, either to states that were not encountered during training, or to variations in the transition dynamics. In the present work, we aim to shape the policy learning process, so that it relies on meaningful features that permit such generalization and robustness.

---

Evaluating generalization in RL. Under the impetus of these works and the need for benchmarks with separate training and testing environments [Whiteson et al.. 2011, Machado et al., 2018, Hender- son et al., 2018], original benchmarks for evaluating the generalization capacities of RI. agents have been designed. Packer et al. [2018] propose modified version from control problems in OpenAl Gym [Brockman et al., 2016] and Roboschool [Schulman et al., 2017] that lets the user change the system dynamics. Machado et al. [2018] propose a modified version of the ALE environments [Belle- mare et al., 2013] allowing one to change modes and difficulties. Without modifying the underlying transition model, Zhang et al. [2018b], Grigsby and Qi [2020]. Stone et al. [2021] add distracting elements (e.g., addition of real images or videos in the background, change of colors) to the ALE environments and the Deepmind control suite. Even if the modifications to the original environments do not alter the semantic information, they already appear to be challenging for agents prone to observational overfitting. Cobbe et al. [2019, 2020), Juliani et al. [2019] use procedural content generation to design highly randomized sets of environments with different level layouts, game assets, and objects locations, letting the user study robustness to several independent variation factors. One may note that the diversification of learning environments is in itself first practical way to induce generalization [Tobin et al., 2017, Cobbe et al., 2019, 2020] and also permits curriculum-base learning [Jiang et al., 2021]. Nevertheless, when the diversity of training scenarios is lacking, three sets of methods are generally employed, as detailed below.
Regularization. Farebrother et al. [2018], Cobbe et al. [2019, 2020] demonstrated the beneficial effects of popular regularization methods from the supervised learning literature. Igl et al. [2019] mitigate the adverse effect that classical regularization may have on the gradient quality with selective noise injection and combine it with an information bottleneck regularization. Inspired by mixup [Zhang et al., 2018d), Wang al. [2020] use mixtures of observations stimulate linearity in the policy's 's outputs in-between states.
Data augmentation. Laskin et al. [2020a] evidence the benefits of training RL agents with augmented data (RAD). Yarats et al. [2020] average both the value function and its target over multiple image transformations (DrQ). Hansen et al. [2021] only apply data augmentation in Q-value estimation without augmenting Q-targets used for boostrapping. Raileanu et al. [2021] combine the previous method with UCB [Auer, 2002] to pick the most promising augmentation, and apply it to PPO [Schulman et al., 2017]. Yuan et al. [2022] propose a aask- Lipschitz data augmentation method (TLDA) to augment task irrelevant pixels. Fan al. [2021] use weak data augmentation train an expert without hindering its performance and distill its policy to a student trained with substantial data augmentation. Besides augmentations of raw inputs, other augmentations operate directly within the agents' network. Lee et al. [2020] introduce a random convolutions layer at the earliest level of the agent's network modify the texture of the visual observations. Zhou et al. [2020] adapt mixup [Zhang et al., 201 8d] with style statistics encoded in early instance normalization layers to increase data diversity. Bertoin and Rachelson [2022] apply channel-consistent local permutations of the feature map to induce robustness to spatial spurious correlations. Finally, data augmentation can also be used in an auxiliary loss to promote invariance to distributional shift in representations. Hansen and Wang [2021] propose a soft-data augmentation method (SODA) by adding an auxiliary self-supervised learning phase to SAC [Haarnoja et al., 2018], similar to BYO [Grill et al., 2020].
Representation learning. Higgins et al. [2017b] demonstrate zero-shot adaptation to unseen con- figurations in testing environments, using 8-VAE [Higgins et al., 2017a] to learn disentangled representations. Fan and Li [2021] jointly maximize the mutual information between sequences of observations to remove the task-irrelevant information. Fu et al. [2021] learn a disentangled world model that separates reward-correlated features from background. Wang et al. [2021] extract, using visual attention, the observation foreground to provide background invariant inputs to the policy learner. Raileanu and Fergus [2021] separate the actor from the critic in the agent's network architecture and add an adversarial auxiliary objective on the actor's representations to remove the information needed to estimate the value function that is not irrelevant to a general policy. Zhang et al. [2020] train an encoder to project states so that their distances match with the bisimulation distances in state space. Other recent works use behavioral similarities combined with contrastive learning [Aarraa et al., 2020] or clustering [Mazoure et al., 2022] to map behaviorally similar observations to similar representations.
Att Attributing decisions to inputs. Although not directly aiming at generalization, a related topic is that of attribution, where one wishes to identify which parts of an input are responsible for major changes in the output of function. Intrinsically, computing attributions boils down to computing (some trans-
3

---

formation of) the gradient of the function's output with respect to the input's 's components. Computa- tional graphs of differentiable functions, such as neural networks, are particularly suited to computing attributions by using the back propagation algorithm [Simonyan et al., 201 4, Springenberg et al., 2015, Smilkov et al., 2017, Selvaraju et al., 2017, Chattopadhay et al., 2018]. When these methods are applied to images, one obtains map which is known as a saliency map or attribution map. Such attribution maps indicate which input pixels are determining for a policy's s output (or the Q-value of action a) and thus permit interpretation of the function itself, rather than its point-wise decision alone. Mousavi et al. [2016], Greydanus et al. [2018], Atrey et al. [2020] have used saliency maps to analyze and explain the behavior of RL agents. Rosynski et al. [2020] indicates in particular that guided backpropagation ISpringenberg et al., 2015] provides good visualizations of RL policies across a span of environments. Most existing works, however, only exploit saliency maps tools for interpretation. Closely related to our contribution is that of Ismail et al. [2021], who incorporate attributions into their training process in supervised learning. Their procedure iteratively uses binary mask computed from attributions to remove features with small and potentially noisy gradients while maximizing the simi- larity of model outputs for both masked and unmasked inputs. This saliency guided regularization im- proves the quality of gradient-bast based saliency explanations without interferring with training stability,
This contribution. The rationale of the method we introduce in the next section is to encourage the agent to generalize to new states, based on which pixels are identified as important decision factors. For this purpose, we perform pixel-level masking the input image, depending on the computed attribution, and regularize the value function learning process with the difference in Q-values. This way, we encourage the value function to focus specifically the pixels with high attribution. Leveraging data augmentation, self-supervised learning methods have demonstrated their ability to induce features that are insensitive to lighting, background, and nigh-frrquency noise. The method we propose does not directly use data augmentation in the value function or policy update phase. Instead, it is introduced during an auxiliary phase where the augmented state's encoding is used to predict the attribution mask of the original state. This way, the encoder is encouraged to preserve information that is useful for predicting which pixels were important in the agent's 's decision-making. A parallel with SODA can thus be made by considering that the projector used in the BYOL objective here replaced by surrogate of the derivative of the value function, allowing to refine the quality of the projection and to learn which pixels and visual features are consistently important across states to predict the Q-values, The value function regularization and the self- supervipe learning objective are mutually beneficial: the former outputs sharper saliency maps from the value function, that serve as better labels for the auxiliary self-supervised learning task, which in turn induces better features and better attributions. In short, we encourage the agent to pay attention to where it is looking, with the intention that this triggers more efficient learning and more interpretable output.
3 Saliency-guided Q-networks
We propose a generic saliency-guided Q-networks (SGQN) method, for visual deep reinforcement learning. In a nutshell, SGQN considers the application of the binarized attribution map as a mask over the input state and regularizes the value function learning objective with consistency term between the Q-values of the masked and the original state images. also defines an auxiliary self-supervised learning task that aims to match a prediction of the attribution map on an augmented image, with the attribution map of the original image. Such an auxiliary task orients the gradient descent towards features that are shared across states, as illustrated by the work of Hansen and Wang [2021]. SGQN can be combined with any value function learning objective, any attribution map computation technique, any image augmentation method for self-supervised learning, and is suited for both discrete and continuous actions. We first present SGQN as a generic enhancement of approximate value iteration methods. Then we derive specific version built on SAC [Haarnoja et al., 2018] and on the guided backpropagation algorithm [Springenberg et al., 2015].
A vast number of deep RL algorithms belong to the family of approximate value iteration methods. Such methods build a sequence of (QR)eeN and (T,, )ne functions that aim to asymptotically tend to Q" and π." Qn+1 is defined as minimizer of LQ = I|Q T=lQu where 7 is the Bellman evaluation operator with respect to n. Then T is defined by applying a greediness operator G to Qu+1 and the process is iterated. Geist et al. [2019] showed how one could introduce regularization within the expression of T.q. yielding the class of regularized MDPs. The classical DQN algorithm [Mnih et al., 2015] approximates the solution to LQ by taking number of gradient steps with respect

---

M2(Rs S.0
fa
Qc
fo
Le
fø
M2
fe
R
((S)
Malfa, (())
Self supervised objective
Consistency regularization objoctive
Figure 1: SGQN losses. The Lst self-supervised loss trains fo that Me(fe (T((s)r(s)) a) predicts Mp(Qu o fa, s, a). In turn, the Lo consistency loss pushes Qa o fe to only depend on salient pixels.
to a target network Qn and uses an arg max greediness operator. When actions are continuous, actor-critic methods introduce surrogate model of the Qu+1-greed policy, under the form of an actor network Tn+1 ((Q++1) obtained by gradient ascent. In what follows, we denote by (0) the loss minimized by generic learning procedure for Qs. based on the Bellman operator, independently of whether regularized, uses double critics, etc. Similarly, we note (0) the loss minimized to yield greedy policy To, when applicable.
We denote by M (Q, s, a) an attribution map for Q(s,, a). the space of images S. Vanilla grad [Si- monyan et al., 2014] for instance will compute such a map under the form M (Q, a) iQ(s, a)/Os, while guided backpropagation [Springenberg et al., 2015] will mask out negative gradients, yield- ing different attribution map. We note Mp(Q, s.a) the binarized value attribution map where (2( 8, a)j = if attribution pixel (Q. s, 0)5 belongs the p-quantile of highest values for M(Q, s, a), and 0 otherwise.
The proposed method is built on classical Q-network architecture. The value function is divided into 2 parts: an encoder fo S > 2 and a Q-function Qo Z x A - R built on top of this encoder. We add decoder function Mg after the features encoder fe. such that Mo(Je(s), a) aims predict the attribution map of Qs (f(((s)) a). Many algorithms require defining double critics [Fujimoto et al., 2018] or target networks fo and Q2 which are often updated with an exponential moving average of 0 [Polyak and Juditsky, 1992]. We omit them here for clarity, although their introduction in SGQN is straightforward. When needed, a policy head Ty Z -> A is built on top of the encoder fo define the actor network. The backbone architecture and training process are summarized in Figure 1. The SGQN training procedure involves two additional objectives: a consistency objective responsible for regularizing the critic update and an auxiliary supervised learning objective.
The consistency regularization objective (Figure 1 right) (c( ((oo((QQ(o((s))))) Qo(s o M,(Qe fe, s, a), ]]] (where @ denotes the Hadamard product), is added to the classical critic loss (f) during the critic update phase. This loss function encourages the Q-network Qo fo to make its decision based in priority the salient pixels in (Q, 8, a). hence promoting consistency between the masked and original images. The new critic objective function is thus defined as (o) + (f).
The self- self-supeviied learning phase (Figure left) updates the parameters of so so that given generic image augmentation function т, (fo(T())), a) contains enough information to accurately learning objective function LsL (0) = E.,a ( (fo(((s) a), Mp(Q9. s, a)), where BCE is the binary cross entropy loss, which could be replaced by any other measure of discrepancy between attribution maps.
reconstruct the attribution mask M,(Qo, a). This defines a
The interplay between these two phases acts as a virtuous circle. The consistency regularization loss, similar to that of Ismail et al. [2021], pushes the network to focus its decision on a selected set of pixels (hence relying on the assumption that initial saliency maps are reasonably good). This enhances the contrast between pixels in the gradient image, and thus yields sharp saliency maps, even before binarization. These maps serve as target labels during the self-supervised learning phase; since they are less noisy than without the consistency loss, they provide stronger incentive encode the information of which pixels are important, within fo- In turn, as exemplified by Hansen and Wang [2021] and Grill et al. [2020], the features obtained by the self-supervised learning procedure
5

---

tend to be insensitive to background, noise, or exogenous conditions, and provide features that are shared across observations. These features benefit from the better labels (less irrelevant pixels in the attribution map). Finally, this helps provide good pixel attributions that will be used in the minimization of the consistency loss during the critic phase. Appendix proposes an extended discussion on this virtuous circle.
Note that the thresholding operation performed to obtain Mp is not strictly necessary, either in the consistency loss or in the self-supervised learning one. Instead of a hard thresholding, one could turn to normalization of the attribution map, such as softmax for instance. Such a soft-attribution image remains fully compatible with SGQN. The choice to keep the thresholded p-quantile mask motivated by the arguments of Ismail et al. [2021] who extensively study such variations and conclude to the benefits of this binarized mask. One could also remark that SGQN does not require to use the target network during the eeff-supervised learning phase, which contrasts with the choices of SODA or BYO and makes the method somewhat more versatile.
Algorithm I presents the pseudo-code of combining SGQN with SAC, yielding an SG-SAC algorithm. Note that, for the sake of simplicity, we write 0 for the full set of network parameters, which are thus shared by the encoder, the Q-value head, the policy head, and the attribution reconstruction head.
Algorithm/ 1: Saliency-guided SAC (changes to SAC in blue)
Parameters: frequency of auxiliary updates NSL, attribution quantile value p. learning rate a, data augmentation function T. for each interaction time step do a, s ~ Tp(- fo(s)) P( s))a) B - BU {(, R (s.0) s')} {s;, (e,,a, 0z), [E B
// Sample a transition // Add transition to replay buffer // Sample a mini- of transitions
0-0- aV (0) + (L)
// Critic update phase
// Actor update // Self- Self-supervised learning Note: LQ and La are as defined by Haarnoja et al. [2018], temperature update, double critics and target network updates are omitted here for clarity.
0 --00aV
Every NSt. steps: 0
aVLL..
4 Experimental results and discussion
This section evaluates SGQN's training efficiency, generalization capabilities, and policy interpretabil- ity. It also discusses the different design choices made along the way. We compare our approach with current state-of-the-art methods for generalization in continuous actions RL (RAD [Laskin et al., 2020b], DrQ [Yarats et al., 2020], SODA [Hansen and Wang, 2021]. SVEA [Hansen et al., 2021]) on five environments from the DMControl Generalization Benchmark (DMControl-GB) [Hansen and Wang. 2021]. The DMControl-GB presents variety of vision-based continuous control tasks based on the Deepmind control suite. Agents are trained in a fixed background environment and evaluated under two challenging distribution shifts, consisting in replacing the training background with natural videos. Figure 2 illustrates the effects of both domain shifts. All the compared methods herein are variants of SAC, for which we use the same architecture for all agents. These methods all use data augmentation in one of their stages. Following the experimental protocol of the com- peting approaches, we used a random overlay augmentation [Hansen and Wang, 2021] (consisting in blending together original observations with random images from the Places365 dataset [7hou et al., 2017]) for all the methods except for RAD and DrQ, for which we used random crops and random shifts respectively, et al., 2020J. We trained all agents for 500 000 steps using the vanilla training environment with no visual variation. Appendix A, B. and E discuss all the hyperparameters. network architectures, and implementation choices used for this benchmark. Appendix C includes extra experimental results on DMControl-GE and Appendix D provides additional results
it is reported as producing the best results [Laskin et al., 2020b, Yarats
vision-based robotic environment.
SGQN improves value iteration in the training domain. We first compare the performance in the training domain, with no visual distractions, of SAC and SGQN, on five environments from the DMControl-GB (Figure 3). The SGQN agents outperform, by a considerable margin, the SAC agents both in terms of asymptotic performance and sample efficiency on 4 out of the 5 environments. In

---

(a) Training
(b) "Video easy" distribution shift
(c) "Video hard" distribution shift
Figure 2: Examples of training and testing environments
Benchmark
Fnvironment
DxQ
RAD
SODA
SCQON
A
Walker walk
245 165
747 21
60S 02
771 66
828 66
910 = 24
+82(10%)
Walker stand
359 + 131
926 2 30
879 + 64
965 7
966 + 5
955 2:
-11(1%)
Easy
Cartpolo
cup
398 60
177::133
380
459 :: 81
473 +: 54
158
742 73
753 + 45
761
+: 24
+: 28
+11(1%)
+8(1%)
Finger spon
206 169
599 +: 02
516 +: 113
783 51
723 + 98
956 4: 26
173(22%)
Average
622
836
+70[8%)
Walker
335((92%)
Walker stand
231 = 57
262 1: 57
229 +: 45
736 # 132
747 + 43
851 +: 24
+10((14%)
Hond
Ball
101 37
100
98 10
381 # 163
198 174
782 :: 57
+884(57%)
Cartpole
(141(35%)
Finger spin
13 1::+30 10
38 + 13
39 20
309 + 19
307 + 24
822 + 24
(166%
Averaage
125
129
119
430
468
748
(600%))%
Table 1: Performance on video easy and video hard testing levels. A difference with second best.
addition obtaining better results, the variance of the agents trained with SGQN is also significantly lower than that of the agents trained with SAC, demonstrating that the enhancements employed in SGQN have beneficial effect on the stability of the training, regardless of the ability to generalize across domains.
Walker walk
Walker stand
Cartpole
Ball in cup
Finger spin
800
SAC
600
400
200
0
Step
Step
Step
Step
Step
Figure Comparison SAC and SGQN training learning curves
SGQN improves generalization. We assess the zero-shot generalization ability of SGQN on the video easy and video hard benchmarks from the DMControl-GB. The easy version only replaces the background of the training image with a distracting image, while the hard version also replaces the ground and the shadows (Figure 2). We report the average sum of rewards after 500 000 training steps for the video easy benchmark in the top part of Table Agents trained with SGQN outperform agents trained with other state-of of-the-art methods on all tasks but one (where it is on par with other agents), thus demonstrating the generalization capabilities induced by the method. By removing the ground and shadows, the video hard benchmark (bottom part of Table 1) causes larger, more confusing, and more challenging distributional shift. All the competitors of SGQN experience radical decrease in their generalization performance. SGQN is significantly less impacted and outperforms all its competitors with an average margin over the second-best of 60% on all environments, and a gain range of 14 to 166% across environments. Figure 4 reports the evolution of each agent's 's video hard environments, along training. SODA and SVEA's scores drop drastically when the ground and shadows are removed. SGQN is less sensitive to this change, notably through the consistency loss, which encourages agents to make decisions based on the subsets of pixels they deem most interesting. Overall, the interplay of the phases of SGQN seems to be key to major leap forward in terms of generalization gap in difficult environments, setting a new reference state-of-the-ar
on the
SGQN yields sharp saliency maps. We use guided backpropagation visually compare the SGQN agents' ability to discriminate the essential information with that of other agents. Figure 5 shows an example of the binarized attribution maps for all agents in video hard state. While the other agents
7

---

Walker
Walker stand
Cartpolc
Ball cup
Finger spin
800
SAC
Score
600
DRO
SODA
SVEA
SGON
2
0
2
Step
Step
Step
Step
Step
Figure 4: Performance on video hard testing levels.
seem to be disturbed by background elements and retain some dependency on background pixels in their decision, the attributions the agent trained with SGQN are precisely located on the important information, hence suggesting better generalization potential.
(a) Observation
(b) SAC
(c) RAD
(d) SODA
(e) SVEA
(f) SGQN
Figure 5: Example of attributions in video hard levels
SGQN is interpretable by design. In all our experiments, we trained the SGQN agents using guided backpropagation. We emphasize that any other attribution method could be used instead. Some of these methods are expensive and require several forward (or backward) passes within the network (e.g., RISE [Petsiuk al., 2018], or the work of Fel al. [2021]) yield attribution maps which explain the agent's 's decision. In contrast, SGQN's auxiliary phase trains predictor to estimate the most important pixels according to the chosen attribution method. Therefore, this predictor is surrogate of the attribution method used during training. It allows identifying the essential image features that condition the agent's decision in the same forward pass as the prediction of the action itself, without incurring the cost of one or several costly additional backward or forward passes. Figure 6 illustrates the proximity between the actual saliency map and the attribution surrogate model Mg. This makes SGQN both self-aware (of its own saliency maps) and intrinsically interpretable from a human perspective, with no computational overhead at evaluation time.
(a) Observation
(b) (Q (,
(c) 8 c M,((u,s,a)
(d) Mo(s,
Figure Comparison between the true saliency map (b), the masked image (c) used the consistency regularization term, and the estimated saliency map (d) in a video hard level.
Ablation study. SGQN relies on two enhancements of vanilla approximate value iteration: the auxiliary self-supervised learning phase and the consistency regularization term in the critic's loss. We perform an ablation study to assess their individual contribution and Table 2 reports the average sum of rewards in the training domain and for zero-shot generalization in all environments (Appendix F takes different perspective and compares SGQN with the combination of SVEA and SODA). Individually, each of these features greatly improves both training and zero-shot generalization performance on all environments. The auxiliary self-supervised learning phase provides the most significant performance gains over vanilla SAC. The average training performance on all environments of agents trained with this auxiliary objective is more than 73% higher than that of agents trained with vanilla SAC. The same applies to the performance zero-shot generalization, which increases by more than 146% video easy and by more than 198% video hard environments. One can note

---

Environment
benchmark
SAC
SAC*Consitency
SAC4Self learning
SOQN
Walker walk
train
287 +: 165
449 100 (+-56%)
934 (2-5%))
937 12
easy
245 + 165
423 = 96 (473%)
910 21-271113113 %)
hard
122
314 87 (+182%)
(++85%)
T39 21
Walker stand
train
467 = 162
807 = (4-8%)
967
960 =9 (4106%)
easy
389 1: 131
846 1: 107 (+1117%)
944 14
955 19 (+145%)
hard
231 57
696
(+-001%)
769 20 34
851 4 (4-68%)
Ball cup
train
284 1. 329
755 1. (
967
971 11 (+242%)
eacy
440 (4-129%)
43
950 (4-399%)
hard
101 + 37
190 = 63 (488%)
203 (2-10%)))
782 57 (+670%)
train
850 +: 29
863 +: 9 (++2%)
857 + 201+1%)
839 + (- (-1%)
Cartpole
eas
+: (+67%%
(++63%)
761 (491%)
hard
158 1: 17
228 +: 62 (+444)
294 1: (++%))
544 1: 43 (+ (444)
Finger spin
train
829 21
985 :: 1 (+19%)
985 (111
985 : (419%)
easy
382
1 40
865 65 (+126%)
803 1: 65 (+110%)
956 1 (+150%)
hard
20 # 10
352 + 58 (+ 1660%)
385 - 45 (+ 1825%)
822 = 24 (+-4010)
Averuge
train
easy
543
321
647 (+- 102%)
781 44%)
789 (+- 146%)
(+73%)
906 182%)
938 (+7))
hard
126
362 187%)
375 (1-19%)
747 493%)
Table 2: Ablation study. Percentages indicate variations compared to vanilla SAC.
(a) Observation
(b) Without consistency loss
(c) With consistency loss
Figure 7: Comparison of SGQN attributions with and without consistency loss
that agents trained with our self-supervised learning objective obtain performance of the same order of magnitude as SODA agents (Table 1). Recall that SODA relies on the BYOL [Grill et al., 2020] self--uper feature learning procedure, whose target labels differ notably from those proposed herein. The reported performance, compared that of SODA suggests that attribution maps constitute good labeling function that could be considered in the more general context of self-supervised learning. To a slightly lesser extent, adding the consistency loss to SAC also significantly improves its performance. The average score obtained improves by more than 44% on the training domain and by 102% and 187% respectively on the video easy and video hard domains. Similarly to the results obtained by Ismail et al. [2021] in supervised learning, the regularization of the critic's 'ss loss with the consistency term sharpens the attribution maps obtained (Figure 7). In SGQN, these fine-grained attributions provide higher quality labels to the auxiliary self-supervised learning phase, thus yielding significant performance improvements in training (+73% on average, range up to +242%), video easy generalization (+182% average, range up to +399%), and video hard generalization (+493% on average, range up to +4010%).
5 Conclusion
The ability filter out confounding variables is long-standing goal in machine learning. For visual reinforcement learning, it is a pre-requisite for real-world deployment of learned policies, since we avoid at all costs situations where an agent makes the wrong decision due to distracting visual factors. In this work, we introduced saliency-guided Q-networks (SGQN), a generic method for visual reinforcement learning, that is compatible with any value function learning method. SGQN relies on the positive interaction of two core mechanisms of self-supervised learning and attribution consistency that jointly encourage the RL agent to be self-aware of the decisive factors that condition its value function. We implement an SGQN agent based on the soft actor-critic algorithm, and evaluate it on the DMControl generalization benchmark. This agent displays dramatically more efficient learning curve than vanilla SAC on the various environments it is trained on. Most importantly, the policy it learns closes the generalization gap on environments that include confusing and distracting visual features, setting new reference in terms of generalization performance. Since they rely on self-awareness of important pixels, SGQN agents are also very interpretable, the sense that they provide prediction of their own saliency maps, with computational overhead.

---

The introduction of SGQN is an exciting milestone in RL generalization, and we wish to conclude this contribution by highlighting some limitations and perspectives for research that we believe are beneficial to the community. Attribution maps seem to be an efficient proxy for encouraging causal relationships within policies, but they are strongly grounded in an anthropomorphic point of view of what visual policy should be. Extending their definition to more abstract notions of attribution is still a challenge and begs for important contributions, both theoretical and algorithmic. Similarly, such attribution maps appear to be relevant self-supervised learning targets, in order to learn good features for RL agents. Exploring whether this still holds for different tasks than RL is an open question. is likely that one could design variations of SGQN that perform better than SG-SAC. The extension of SGQN agents to discrete action agents (e.g. DQN), or policy gradient methods (e.g. PPO) is a promising perspective in itself. Although generic and grounded in sound algorithmic mechanisms, the losses introduced by SGQN lack formal connection to some measure of the generalization gap. Such connection could provide insights better self-aware, explainable agents with improved generalization capabilities. Finally, using SGQN as a building brick, among all those required to bridge the gap between simulation and real-worke applications, an exciting perspective.
Acknowledgments
The authors acknowledge the support of the DEEL, ENVIA and MINDS projects, the funding of the AI Interdisciplinary Institute ANITI funding, through the French "Investing for the Future PIA3" program under grant agreement 1NR-19-P13A-0004, This work benefited from computing resources from CALMIP under grant P21001.
References
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization reinforcement learning. In International Conference on Learning Representations, 2020.
Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. In International Conference on Learning Representations, 2020.
Peter Auer. Using confidence bounds for exploitationnexxllraiion trade-offs. Journal of Machine Learning Research, 3:397-422, 2002.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
David Bertoin and Emmanuel Rachelson. Local feature swapping for generalization in reinforcement learning. International Conference on Learning Representations, 2022.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1606.00 2016.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient- based visual explanations for deep convolutional networks. In IEEE winter conference on applications of computer vision (WACV), pages 839-847, 2018.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1282-1289, 2019.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048-2055 2020.
Jiameng Fan and Wenchao Li. Dribo: Robust deep reinforcement learning via multi-view information bottleneck. arXiv preprint arXiv:2102.13268 2021.
10

---

Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In International Conference Machine Learning, 2021.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. In NeurIPS Deep Reinforcement Learning Workshop, 2018.
Thomas Fel, Rémi Cadène, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis.
Advances in Neural Information Processing Systems, 34, 2021.
Xiang Fu, Ge Yang, Pulkit Agrawal, and Tommi Jaakkola. Learning task informed abstractions. In International Conference on Machine Learning, pages 3480-3491. PMLR, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pages 1587-1596, 2018.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference Machine Learning, pages 2160-2169, 2019.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International conference on machine learning, pages 1792-1801, 2018.
Jake Grigsby and Yanjun Qi. Measuring visual generalization in continuous control from pixels. arXiv preprint arXiv:2010.067440 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Valko Michal. Bootstrap your own latent-a new approach to seff-supee learning. Advances in Neural Information Processing Systems, 33: 21271-21286 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870, 2018.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmen- tation. In International Conference Robotics and Automation, 2021.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. In Conference on Neural Information Processing Systems, 2021.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017a.
Irina Higgins, Arka Pal, Andrei Rusu, Loie Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference Machine Learning, pages 1480-1490, 2017b.
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek. Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. Advances in Neural Information Processing Systems, 32:13978--3990, 2019.
Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi. Improving deep learning inter- pretability by saliency guided training. Advances Neural Information Processing Systems, 34, 2021.

---

Rishabh Jangir, Nicklas Hansen, Sambaran Ghosal. Mohit Jain, and Xiaolong Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, pages 1-1, 2022. doi: 10.1
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pages 4940-4950, 2021.
Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: generalization challenge vision. Control, and Planning. arXiv e-prints, 2019.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pages (639-5650, 2020a.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in Neural Information Processing Systems, 33, 2020b.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. International Conference on Learning Representutions, 2020.
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Bogdan Mazoure, Ahmed M Ahmed, R Devon Hjelm, Andrey Kolobov, and Patrick MacAlpine. Cross-traeectory representation learning for zero-shot generalization in RL. In International Conference on Learning Representations, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-55 533, 2015.
Sajad Mousavi, Michael Schukat, Enda Howley, Ali Borji, and Nasser Mozayani. Learning to predict where to look in interactive environments using deep recurrent q-leaming. arXiv preprint 1rXiv:1612.05553. 2016.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforeement learning. arXiv preprint arXiv:1810.1 1810.12210..222,,, 2018.
Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized input sampling for explanation of black-box models. In 29th British Machine Vision Conference, 2018.
Boris T Polyak and Anatoli B Juditsky, Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838-855, 1992.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. International Conference Machine Learning, pages 8787-8798. 2021.
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Matthias Rosynski, Frank Kirchner, and Matias Valdenegro-Toro. Are gradient-based saliency maps useful in deep reinforcement learning? Can't Believe It's Not Better!" NeurIPS workshop. 2020.
12

---

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06647, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In Proceedings of the IEEE international conference on computer vision, pages 61 8-626, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. In Workshop International Conference on Learning Representations, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint 17X.v:/706.038000 2017.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. In International Conference on Learning Representations, 2019.
J Springenberg, Alexey Dosovitskiy, Thomas Brox, and M Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (workshop track), 2015.
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite-a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. In NeurlPS, 2020.
Xudong Wang, Long Lian, and Stella X Yu. Unsupervised visual attention and invariance for reinforcement learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6677-6687, 2021.
Shimon Whiteson, Brian Tanner, Matthew E Taylor, and Peter Stone. Protecting against evalua- tion overfitting in empirical reinforcement learning. In IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pages 120-127, 2011.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In International Conference Learning Representations, 2020.
Zhecheng Yuan, Guozheng Ma. Yao Mu, Bo Xia, Bo Yuan, Xueqian Wang, Ping Luo, and Huazhe Xu. Don't touch what matters: Task-aware lipschitz data augmentationfor visual reinforcement learning. arXiv preprint wwXiv:220..09982 2022.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 118-833. Springer, 2014.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint 180Vv:1800607337, 2018a.
Amy Zhang, Yuxin Wu, and Joelle Pineau. Natural environment benchmarks for reinforcement learning. arXiv preprint arXiv: /0t:::///..00000 1.06032, 2018b.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. International Conference on Learning Representations, 2020.
13

---

Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint (et1:1804400833 2018c.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018d.
Chenyang Zhao, Olivier Sigaud, Freek Stulp, and Timothy M Hospedales. Investigating generalisation in continuous deep reinforcement learning. arXiv preprint arXiv: 190.0000900...11. 2019.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2020.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you's 're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "hCugTpwzFPP", "reviewer": "Reviewer_1cpS", "summary": "The paper presents saliency-guided Q networks (SGQN), which use saliency maps of the Q network to guide learning. The goal is to focus attention on pixels that are relevant to the task, while ignoring irrelevant ones, to aid in visual generalization at test time. SGQN is evaluated on the DM Control generalization benchmark with video backgrounds, and compared with SAC, RAD, SODA, and SVEA.", "questions": "I don't think changing the quantile value ρ hyperparameter for different tasks is a fair comparison (Appendix table 3). This would require doing hyperparameter search for any new task encountered. For results, just report for the same value across all tasks.\n\nL224: \"relying on the assumption that the initial saliency maps are reasonably good.\" Why is this a good assumption? Some issues with saliency maps have been discussed in prior work, e.g. \"Sanity Checks for Saliency Maps\" from Adebayo et al.\n\nMinor comments (no need to respond to these):\n* I would not say that Deep RL has \"outstanding efficiency in simulated visual control tasks\" (L1). ", "limitations": "There should be discussion of the limitations of the approach, which might talk about drawbacks of saliency maps, computational cost/speed, etc.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Visual generalization in RL is an important problem.\n* The \"video hard\" evaluation seems like a challenging test of generalization.\n\nWeakness\n* To me, the method seems cyclic. The saliency maps from the Q network are used to influence how the Q network is trained.\n* Some of the baseline numbers don't seem to align with what was reported in prior work. For example, in Table 1, the finger spin (easy) performance for SODA is 535, but in the SODA paper it is 695 (higher than the 609 reported for SGQN). Similarly, SVEA is 537 (Table 1) vs 808 (original paper). These numbers are averaged across 5 random seeds, right?\n* The paper is poorly organized. For example, Section 3 (the approach) has no subsections or paragraphs, making it hard to follow.\n\n-------\nReview update:\nI thank the authors for their responses and additional experiments.\n\nI looked in more detail at saliency-guided training (e.g. in Ismail et al.) and this technique seems to have some merit. My concern about the cyclic nature of the method was that it is only applying large gradients to the Q network and discarding small ones, which seems like it could result in unstable training and/or suboptimal use of the model parameters. Expanding the discussion of saliency-guided training (such as including the findings from Ismail et al. in related work) would help mitigate these concerns.\n\nL293: “SGQN yields sharp saliency maps.” This evaluation in particular is cyclic since SGQN used this saliency technique in training while baselines did not. This point should be mentioned, or even better a different attribution technique should be used for this evaluation. See discussion of Goodhart’s Law in Ismail et al.’s OpenReview forum: https://openreview.net/forum?id=x4zs7eC-BsI\n\nPaper organization: I would say the background and the method’s motivation should be separated out from related work. Regarding the method section (3), I find myself having to jump around quite a bit. For example, if I want to understand how the image augmentations are used, I must look at  L161, L177, L218, Figure 1, and Algorithm 1 (though τ is not actually used in the algorithm definition). Instead, the approach section could have clear subsections or paragraph labels for each learning phase (consistency and self-supervised objectives). The parts of the text and figure 1 could more clearly refer to each other. Algorithm 1 could probably be moved to the appendix since most of it does not relate specifically to the method.\n\nMy baseline numbers and ρ values questions have been resolved.\n\nThe primary reason for my low rating was the concern about the cyclic nature of the saliency-based training, which as I mentioned has mostly been resolved (since it has shown up in prior work). Thus I have increased my overall, soundness, and contribution ratings. I still encourage the authors to improve the discussion and organization of the text in the ways I have described.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "h76Wo7SYT-R", "reviewer": "Reviewer_S2Er", "summary": "This paper proposes a saliency-guided Q-networks (SGQN) method that guides the policy to focus on important pixels for improving generalization sample efficiency. SGQN has two major components: 1) force the saliency of the augmented image to be close to that of the original image, 2) force the Q values to be close. The experiments show that it outperforms existing state-of-the-art methods on DMC-GB.", "questions": "1. The authors pick a large number for thresholding the binary mask. Why is this number selected? For a different task, do you need to choose a different number? \n2. How does the proposed method compare with SODA+SVEA?\n3. How does the proposed method perform on other benchmarks?", "limitations": "The authors mention their limitations as future research directions such as explanations and theoretical analysis of the proposed method. We appreciate the authors for being upfront.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n1. This work shows that the learned policy could largely improve the generalization ability in the DMC-GB with self-supervised learning.\n2. The learned policy could be interpretable with the use of saliency map.\n3. The paper is well-written.\n\nweaknesses:\n1. The proposed method is only applied on DMC-GB. It is not sure that the proposed method is useful in other tasks other than the DMC-GB.\n2. SGQN is a DrQ based algorithm since the batch size data is augmented by random shift. Then there are various places the authors can edit and use DrQ instead of SAC.\n3. SODA+SVEA can be a stronger baseline since you implement two loss functions that are similar to the losses in the two papers respectively. To better illustrate the superiority, a comparison is preferred: the improvement does come from the saliency representation rather than any representation or embeddings with these two losses.\n4. ‘One can note that agents trained with our self-supervised learning objective also outperform SODA agents (Table 1) on 4 out of 5 video easy environments and on 3 out of 5 video hard environments.’ This comparison result seems problematic according to the Table 1(SODA) and Table 2 (SAC+Self learning).\n5. Missing references:\n\nFu, X., Yang, G., Agrawal, P., & Jaakkola, T. (2021, July). Learning task informed abstractions. In *International Conference on Machine Learning*\n (pp. 3480-3491). PMLR.\n\nYuan, Z., Ma, G., Mu, Y., Xia, B., Yuan, B., Wang, X., ... & Xu, H. (2022). Don't Touch What Matters: Task-Aware Lipschitz Data Augmentationfor Visual Reinforcement Learning. *arXiv preprint arXiv:2202.09982*.Chicago", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "V_8VrJWH6w", "reviewer": "Reviewer_zoMV", "summary": "The paper introduces a generic saliency-guided method (SGQN) for visual reinforcement learning, aiming at improving the generalization capabilities and robustness of deep reinforcement learning policies. This is based on two mechanisms introduced by authors, both depending on masking the input image based on a computed attribution (saliency) map, and therefore inducing the agent to pay more attention where it is looking. The first consists in regularization of the value function with a consistency term (computed form Q values of original and masked input) that encourage the value function to depend mainly on pixels with high attribution. The second is exploiting data augmentation while introducing an auxiliary self-supervised learning objective where the agent trains to predict its own Q-value's saliency maps, based on features encoding information about pixels relevant to make decisions. ", "questions": "- In Background and related work, when methods to improve generalization capabilities are presented, there are many acronyms that are never explicitly expressed. I think it would help readability writing the full name at least one.\n- In the experimental comparison with other methods for generalization, and considering the background paragraphs, SAC and SODA belong to the Data augmentation approach. To which approach do RAD and SVEA belong (it is stated they all include data augmentation in one of their stages)? From my understanding, SGQN mix some aspect of data augmentation (which is used to generate the attribution mask), attribution and regularization methods. Could you say something about its advantages and limitations compared to methods that are not considered in the comparison, such as methods based on representation learning?\n- How is the image augmentation function defined (at line 256 it says random overlay augmentation)? This means initial saliency maps are randomly generated? It seems to me that the mechanism works if initial saliency maps are properly defined, and therefore if there is a previous knowledge about which pixels are relevant to the task. What happens if the initial map is not good enough?\n- In Fig. 6, I would make the caption more descriptive, specifying explicitly what is shown in each subplot.\n- line 318: from Table 1 it seems SGQN outperforms SODA on 5 (not 3) out of 5 video hard environments.", "limitations": "Authors adequately address limitations and perspectives of their research in the Conclusion section. I share their view about the fact that the main limitation in the use of saliency maps to guide the RL learning process is that these strongly rely on a human point of view (and prior knowledge about the task). ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper overall quality is good and the contribution is significant. The context of the work is clearly illustrated with a proper literature review, and specific contributions are clearly described compared to existing approaches. While building up over existing methods, such as regularization and data augmentation, the paper introduces an original and significant architecture for saliency-guided RL. Differently from the majority of current approaches, in which saliency maps are used as tool for interpreting the RL agents behavior, this work incorporates them in the learning process. \nThe approach is experimentally validated and compared with state-of-the-art methods for generalization in continuous actions RL, demonstrating its significance in improving training efficiency, generalization capabilities, and providing a straightforward way for policy interpretability that does not require to compute the actual saliency map. Results are reported in a clear and complete way, and they are exhaustively discussed.\nAn advantage of the proposed method is that it can be combined with other techniques for attribution map computation and image augmentation, value function learning objective, and it is suited for both continuous and discrete action spaces, therefore fostering reusability.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept