A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations

Authors: wonil  song
hyesong  choi
kwanghoon  sohn
dongbo  min
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 01:32:04

Abstract: In the rapidly evolving domain of vision-based deep reinforcement learning (RL), a pivotal challenge is to achieve generalization capability to dynamic environmental changes reflected in visual observations.
Our work delves into the intricacies of this problem, identifying two key issues that appear in previous approaches for visual RL generalization: (i) imbalanced saliency and (ii) observational overfitting.
Imbalanced saliency is a phenomenon where an RL agent disproportionately identifies salient features across consecutive frames in a frame stack. 
Observational overfitting occurs when the agent focuses on certain background regions rather than task-relevant objects.
To address these challenges, we present a simple yet effective framework for generalization in visual RL (SimGRL) under dynamic scene perturbations.
First, to mitigate the imbalanced saliency problem, we introduce an architectural modification to the image encoder to stack frames at the feature level rather than the image level.
Simultaneously, to alleviate the observational overfitting problem, we propose a novel technique called shifted random overlay augmentation, which is specifically designed to learn robust representations capable of effectively handling dynamic visual scenes.
Extensive experiments demonstrate the superior generalization capability of SimGRL, achieving state-of-the-art performance in benchmarks including the DeepMind Control Suite.

Main paper: 
 introduction: 
 Deep reinforcement learning (RL) utilizing visual observations has achieved remarkable success across diverse domains, including robotic manipulation [21] , video games [25, 1, 38] , and autonomous navigation [24, 48] . However, acquiring generalizable RL policies across diverse environments remains challenging, mainly due to overfitting [44] in the high-dimensional observation space [32] . To obtain robust policies invariant to visual perturbations, a variety of approaches based on domain randomization [36, 28] and data augmentation [31, 19, 40, 29, 13, 12] have been widely proposed. These approaches operate under the assumption that exposing an agent to various augmentations during the training phase enhances its adaptability to unseen domains. Despite the encouraging results, performance still falls behind in challenging environments with dynamically changing backgrounds. with the movement of the task object. s and M ρ represent the stacked frames and attribution masks, respectively. Attribution masks in this figure were obtained using the critic trained by DrQ [40] .Fig. 1 (a) shows the notable performance degradation of existing approaches when comparing 'Video Hard' scenes with dynamically changing backgrounds to less dynamic 'Video Easy' scenes in the DeepMind Control Suite-Generalization Benchmark (DMControl-GB) [13] .Using a gradient-based attribution mask M ρ [2] , we first investigate the causes of the degradation in generalization in such challenging environments by examining salient regions across consecutive stacked frames used as an RL input. Based on our analysis, we empirically identified two phenomena, highlighting them as key causes of performance degradation: (i) what we refer to as imbalanced saliency and (ii) observational overfitting [32] . In the DMControl [35, 37] , Fig. 1 (b) depicts an example of the imbalanced saliency in a 'Cartpole, Swingup' task, where salient features across all stacked frames are biased to the regions corresponding to the task objects in the latest two frames. Fig. 1 (c) shows an example of the observational overfitting that occurs in a 'Cheetah, Run' task. In this case, the RL agent misidentifies the ground as more salient than the 'Cheetah' object. These two problems contribute to overfitting to the training environment, making generalization of the RL agent even more challenging.In image-based RL, the effectiveness of regularization has been demonstrated in numerous studies. DrQ [40] and RAD [19] , each employing random shift and random crop augmentations as data regularization on input images, achieved remarkable performance improvements over vanilla SAC [8] . Furthermore, [32] exhibited that regularization effects from architectural modifications such as overparameterization or residual connections can help avoid observational overfitting and reduce the generalization gap.In this context of research, we introduce a Simple yet effective framework for Generalization in visual RL (SimGRL) under dynamic scene perturbations. Firstly, we empirically found the image-level frame stack, commonly used in traditional vision-based model-free RL approaches [25, 18, 41, 40, 19] , to be the main factor causing the imbalanced saliency problem. To address this issue, we propose a simple architectural modification of an image encoder, which employs a feature-level frame stack instead of the image level. This involves extracting individual feature maps for each frame from the shallow layers of the encoder, stacking them along channels, and encoding the stacked feature maps through the remaining layers. Since this approach considers solely individual frames during the initial encoding, the agent is trained to focus on spatially salient features in each consecutive frame that are essential for a given task, alleviating the imbalanced saliency problem of Fig. 1(b ). Secondly, to address the observational overfitting problem by encouraging the agent to focus on the task object rather than backgrounds, we propose a new data augmentation called shifted random overlay, which is a modified version of random overlay augmentation [13] . This augmentation directly injects background dynamics irrelevant to the given task into the agent during the training phase by interpolating natural images moving in a random direction on each frame. By enabling the agent to implicitly learn to ignore task-irrelevant backgrounds and focus on task-relevant pixels, this approach alleviates the observational overfitting problem of Fig. 1(c ). Furthermore, this augmentation enables the agent to adapt to test environments with dynamic perturbations in the surrounding backgrounds.We verify that these strategies can remarkably improve generalization in challenging environments, achieving state-of-the-art performance without using any additional auxiliary losses or networks.Furthermore, utilizing the attribution mask M ρ [2] , we introduce novel metrics, called Task-Identification (TID) metrics, consisting of TID score and TID variance, to quantitatively evaluate the discrimination ability on salient regions. With the proposed metrics, we quantitatively analyze the two problematic phenomena in the existing approaches, demonstrating the excellent discrimination ability of SimGRL. Moreover, we show a tendency of positive correlation between the TID score and RL generalization performance, emphasizing the importance of accurately identifying salient features in input images.Our contributions include the following aspects:• By utilizing gradient-based attribution masks, we highlight the two core issues of imbalanced saliency and observational overfitting, which hinder the generalization of visual RL for most model-free RL settings. Additionally, we propose TID metrics to measure the discrimination ability of an RL agent on task objects, providing insights into these issues. • To address these problems, we propose architectural and data regularization methods through a modification to an encoder structure and an introduction of new data augmentation. • We achieve state-of-the-art performances across video benchmarks of DMControl-GB [13] ,DistractingCS [34] , and robotic manipulation tasks [14] . 
background: 
 Visual RL and Generalization We consider a partially observable Markov decision process (POMDP) problem M = (S, O, A, P, r , γ), where S is state space, O is observation space, A is action space, P : S × A → S is the transition function that defines the conditional probability distribution P(s t+1 |s t , a t ) over next states given a state s t ∈ S and an action a t ∈ A taken at time t, r : S × A → R is a reward function, and γ ∈ [0, 1) is a discount factor. In the POMDP problem, such as the visual RL, because only the high-dimensional observations o t ∈ O can be observable [15] , a state is defined as a sequence of the k consecutive image frames s t = (o t-k +1 , ..., o t-1 , o t ) [25] . Without loss of generality, we will set k = 3 for the sake of notational simplicity. RL aims to learn a policy π : S → A that maximizes the expected sum of discounted rewards E π [ T t=0 γ t r(s t , r t )]. In this work, we focus on the generalization problem to POMDPs M = ( S, O, A, P, r , γ), where the states ŝ ∈ S are constructed from the perturbed observations ô ∈ O, and a POMDP M is sampled from the space of POMDPs M, M ∼ M.Deep Q-Learning and Soft Actor-Critic Deep Q-learning [25] is a common model-free RL algorithm that aims to learn a parameterized state-action value function Q θ (s t , a t ), where θ is the deep neural network parameters of the Q-function and action is greedily selected as the one with the maximum value a t = argmax a Q θ (s t , a) at time t. The training of the Q-function is achieved by minimizing a mean squared error of the Bellman residuals:EQUATIONwhere θ is the parameters of the target Q network Q tgt and B is a replay buffer. To increase the stability, the parameter of the target Q network is slowly updated by the exponential moving average (EMA) θ = λθ + (1 -λ) θ with λ ≪ 1 [22] . For the continuous action space, rather than the greedy sampling, a parameterized actor π ϕ (s t ) is employed as the policy, where ϕ is the neural network parameters of the actor. Soft Actor-Critic (SAC) [8] is a common actor-critic algorithm with a state-action value Q θ (s t , a t ) and a stochastic policy π ϕ (a t |s t ), and a temperature parameter α, which aims to optimize a γ-discounted maximum-entropy objective [49] .In visual RL, a parameterized encoder f θ : R C×H×W → R d is employed to compress the highdimensional image inputs and shared by both the actor and critic, where d is a dimension of the encoded feature. Consistent with the prior visual RL approaches [41, 18, 19, 40, 12] , we jointly train the encoder with the critic and freeze it during actor updates, where the encoder learns representations for RL tasks by the critic loss. For notational simplicity, we denote all parameters updated by the critic loss as θ and the actor parameters as ϕ.Gradient-based Attribution Mask An attribution map, also known as a saliency map, is designed to visualize the salient pixels in input images for given tasks. A common approach to computing the attribution map is a gradient-based method [30, 3] , which indicates how sensitive the task prediction is to perturbations in the input pixels. Consistent with SGQN [2] , we employ guided backpropagation [33] to compute the attribution map M (Q θ , s t , a t ) = ∂Q θ (st,at) ∂st , where s t and M (Q θ , s t , a t ) ∈ R C×H×W . Then, the binarized attribution mask M ρ (Q θ , s t , a t ), referred to as ρ-quantile attribution mask, is computed by thresholding M (Q θ , s t , a t ) by the ρ-quantile, where M ρ (Q θ , s t , a t ) (i,j,k) = 1 if the pixel value of M (Q θ , s t , a t ) (i,j,k) belongs to the ρ-quantile of highest values for M (Q θ , s t , a t ), and 0 otherwise. We use this attribution mask to investigate the cause of the degradation of the generalization performance of existing methods in challenging environments such as Video Hard [13] . Furthermore, to justify our analysis, we will introduce new metrics that quantitatively measure the ability to identify salient pixels in Section 4.4. Note that we leverage the attribution mask only for analysis without involving it in the training process.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [No] Justification: The paper has limitations, but those are not discussed in the paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
pitfalls within conventional practices in visual rl generalization: 
 The generalization ability of RL agents is often degenerated in challenging environments characterized by dynamic perturbations and different structures compared to the training environment. In this section, we investigate the causes of the overfitting to the training environment observed in existing visual RL methods on the DMControl Generalization Benchmark (DMControl-GB) [13] .Conventional practices used to train the visual RL agent for generalization [13, 2, 42, 4] include frame stack and data augmentation. First, to reflect the temporal structure of the input state, Q-value is predicted using stacked frames [25, 18, 41, 40, 19] instead of a single image as follows:EQUATIONwhere θ is a neural network parameter,f θ (•) is a convolutional neural network (CNN) image encoder, Q θ (•, •) is a critic head, and [•] is a concatenation operator along a channel dimension.Subsequently, to learn the representations robust to visual perturbations, data augmentation to the visual observations is leveraged when training the encoder. Specifically, s t in Eq. ( 2) can be substituted with τ (s t ) = (τ (o t-2 ), τ (o t-1 ), τ (o t )), τ ∼ T , where τ (•) is a sampled transformation function from the transformation space T . However, we empirically found that these practices can cause the saliency imbalance between the stacked frames and easily fall into observational overfitting, resulting in performance degradation in unseen challenging environments.Pitfall 1: Imbalanced Saliency Fig. 2 (a) illustrates an example of the imbalanced saliency by SVEA [12] in the 'Cartpole' task. As described in Section 2, the attribution masks were acquired by thresholding the gradient value of the critic function with respect to input image frames [2] . In this example, an RL agent recognizes the regions of the salient objects in the two latest frames as having high saliency across all stacked frames. This makes the agent misidentify as salient pixels of o t-2 its background parts corresponding to the positions of the 'Cartpole' object in o t-1 and o t . As a result, o t-2 provides the agent with redundant information that is unnecessary for the task, leading to overfitting to training data [45, 5] . In a test environment with complicated backgrounds, the background information that is considered salient from o t-2 may act as noise that interferes with the decision-making process of the policy. As the latest frames are more closely related to the subsequent decision-making process, the agent tends to identify spatially consistent saliency maps based on these recent frames, leading to imbalanced saliency maps. We hypothesize that this phenomenon occurs because the encoder extracts features throughout concatenated images along the channel as in Eq.(2), forcing it to capture the same spatial saliency for all stacked images.Pitfall 2: Observational Overfitting Observational overfitting [32] can arise when certain background elements move in synchronization with task-relevant objects. It is prone to occur particularly when the motion of the task object is not substantial, such as in the 'Cheetah, Run' task. Although data augmentation is leveraged during training, we found that observational overfitting can still occur. Furthermore, as the same augmentation is applied uniformly to all consecutive frames, it does not effectively prevent the agent from erroneously focusing on background elements. For instance, Fig. 2 (b) shows an example of observational overfitting that occurs in the 'Cheetah, Run' task by SVEA, where the critic assigns more significant saliency to the ground than to the 'Cheetah' object. In a test environment, if there is a lack of correlation between objects and backgrounds present in the training environment, the agent may struggle to make accurate decisions, thereby leading to challenges in generalization.where f 1 θ (•) is an image encoder to encode individual frames, and f 2 θ (•) is a feature encoder for the stacked feature maps. While f 2 θ encodes both spatial and temporal structures from inputs, f 1 θ conducts only spatial encoding of each frame. This simple modification allows the critic to be implicitly trained to focus on the spatially salient pixels of individual frames, enabling the agent to distinctly identify the salient pixels of each frame. In our experiments, we will verify that this simple modification of the encoder is highly beneficial for generalization, especially in challenging test environments.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
shifted random overlay augmentation: 
 The random overlay augmentation [13] used in existing approaches augments input images through linear interpolation with a natural image ε ∈ R C×H×W randomly sampled from the Places [46] dataset that contains 1.8M diverse scenes: where s is a state that consists of a sequence of images (o 1 , ..., o n ), o i ∈ R C×H×W , D is a dataset and a common choice for the interpolation coefficient is α = 0.5. The original random overlay augmentation uses the same natural image ε for all stacked frames. Contrarily, we introduce a shifted random overlay (SRO) augmentation to inject the task-irrelevant dynamics into the training images, which is depicted in Fig. 3 . Considering a maximum shift length l for each shifting and a stacked frame number n, this method first samples a natural image ε ∈ R C× H× W from the dataset D, where ( H, W ) = (H + 2(n -1)l , W + 2(n -1)l ). Then, we crop n shifted patches ε c i ∈ R C×H×W from ε to augment each frame in a shifted way. Given an upper left corner coordinate (h 1 , w 1 ) of H × W sized center crop at the center of ε, the coordinates (h i , w i ) of the upper left corner of ε c i are selected as follows:EQUATIONEQUATIONwhere dh, dw ∼ unif{-l, l}. Finally, using the cropped images ε c i from (h i , w i ) in ε, the shifted random overlay augmentation is defined as:EQUATIONwhere we adopt α = 0.5. This augmentation provides two implicit advantages when training RL agents. Firstly, the inclusion of background elements in motion, independent of the task, enables the RL agent to perceive that rewards are solely associated with changes in the genuine task object. Consequently, the agent is trained to concentrate on the task object, disregarding the movement of background elements, thereby alleviating the issue of observational overfitting. Secondly, this augmentation method generates training data akin to real environments with dynamic backgrounds, enabling the agent to be robust in such conditions and enhancing generalization capability.Built on the SVEA [12] framework that leverages the data-mixing strategy between weak and strong data augmentations and computes the target Q-value using clean images for stability in SAC, we propose a Simple framework for Generalization in visual RL (SimGRL) under dynamic scene perturbations, which integrates the proposed two strategies. For the strong augmentation, we utilize the shifted random overlay instead of the original random overlay without shifting while the random shift [40] is employed as the weak augmentation. Considering that the clean images s t and s t+1 already have weak augmentation, i.e., random shift, applied by default, the critic loss for SimGRL is defined as follows:EQUATIONwhere q θ (•, •) is computed by Eq. (3), q tgt = r t + γq θ (s t+1 , a ′ ), a ′ ∼ π ϕ (•|f θ (s t+1 )), and θ is the parameter of the target network that is updated by the exponential moving average (EMA). The strong augmentation τ SRO (•; •, •) is the shifted random overlay in Eq. ( 6). Consistent with the SVEA, we adopt the data-mixing coefficient β = 0.5. When training the actor π ϕ , we leverage solely clean images like the existing methods [12, 13] . The overall framework of SimGRL is illustrated in Fig. 4 and the algorithm is summarized in Appendix C. To quantitatively evaluate the capability to identify task-relevant objects in each stacked frame as salient, we introduce novel metrics, referred to as Task-Identification (TID) score and variance, based on the ρ-quantile attribution mask M ρ (Q θ , s t , a t ) [2] described in Section 2.TID Score TID score measures how much the model identifies the task object's pixels across stacked frames, which is defined as:EQUATIONwhere N obj is the number of task object's pixels in input images, N M is the number of pixels in attribution masks M ρ , N obj M is the number of task object's pixels included in M ρ , and ρ is a quantile value for thresholding the attribution map. Note that all numbers consisting of the TID score are counted across the full consecutive frames. In Eq. ( 8), the first term quantifies the model's ability to identify the pixels of the task object, while the second term quantifies how accurately the model identifies the task object's pixels. These two terms trade-off depending on the size of the quantile value and are upper-bounded by 1, thus leading to an upper-bounded TID score by 1. With a model that perfectly identifies all task pixels, this upper-limit value can only be achieved along with an optimal ρ value, which is the quantile of the number of the task object's pixels in frames. This ρ value can be computed by ρ = 1 -N obj (n×C×H×W ), where n is the number of frame stack. We explain the impact of the ρ value on this TID score in Appendix F.2.TID Variance TID variance measures how discriminatively the model distinguishes the task object's pixels in each frame, which is defined as:EQUATIONwhere In this section, we present the experimental results of SimGRL on the DMControl-GB [13] video benchmarks ('Video Easy' and 'Video Hard') at 500K simulated training frames to demonstrate the generalization capability under dynamic scene perturbations. To tackle the vision-based control tasks with continuous action space, we employ the SAC [8] as a backbone RL algorithm, and compare SimGRL against current state-of-the-art Table 1 : Generalization performance on video benchmarks from DMControl-GB [13] . We report the results on mean and standard deviation over 5 seeds. The scores of the comparison methods were taken from their respective papers, and in cases where the scores were unavailable, they were obtained through our implementation using the official codes. ∆ indicates the difference with second best.T ID i S = (N i obj M ) 2 N i obj ×N i M , N i obj , N i M ,DMControl-GB SAC RAD DrQ SODA SVEA TLDA SGQN EAR CG2A SimGRL ∆ Video Easy Walker, Walk 245±165 608±92 747±21 768±38 819±71 868±63 910±24 913±38 918±20 910±21 -8 (0.8%) Walker, Stand 389±131 879±64 926±30 955±13 961±8 973±6 955±9 970±23 968±6 973±4 0 Ball In Cup, Catch 192±157 363±158 380±188 875±56 871±106 855±56 950±24 911±40 963±28 964±7 +1 (0.1%) Cartpole, Swingup 398±60 473±54 459±81 758±62 782±27 671±57 761±28 762±88 788±24 838±35 +50 (6%) Finger, Spin 206±169 516±113 599±62 695±97 808±33 744±18 956±28 717±51 912±69 983±2 +27 (3%) Cheetah, Run 73±18 153±7 270±16 268±10 251±17 336±57 289±35 334±56 314±49 317±16 -19 (6%) Video Hard Walker, Walk 122±47 80±10 121±52 312±32 385±63 292±133 739±21 383±59 687±18 773±31 +34 (5%) Walker, Stand 231±57 229±45 252±57 771±83 834±46 595±56 851±24 744±62 895±35 932±17 +37 (5%) Ball In Cup, Catch 101±37 98±40 100±40 327±100 403±174 304±58 782±57 320±48 806±44 902±19 +96 (12%) Cartpole, Swingup 158±17 152±29 136±29 429±64 393±45 308±44 544±43 375±37 472±24 727±23 +183 (34%) Finger, Spin 13±10 39±20 38±13 302±41 335±58 256±25 822±24 277±62 819±38 864±12 +42 (5%) Cheetah, Run 75±14 21±9 49±13 130±24 112±12 67±23 157±69 91±46 168±16 301±7 +133 (79%)methods for visual RL generalization including RAD [19] , DrQ [40] , SODA [13] , SVEA [12] , TLDA [42] , SGQN [2] , EAR [4] , and CG2A [23] . For weak augmentation, RAD and SODA utilize random crop, while the others employ random shift, except for SAC, which does not use any data augmentations. In addition, all competitors leverage the original random overlay augmentation without shifting for strong augmentation except for RAD and DrQ, which use only weak augmentations. As default for SimGRL, we used the random shift augmentation [40] and denoted the images with only this weak augmentation as clean images with the data distribution of the training data. Additionally, we employed the proposed shifted random overlay (SRO) for strong augmentation in SimGRL and denoted images together with this augmentation as augmented ones. Implementation details are described in Appendix A, and further experimental results on other benchmarks of Distracting Control Suite [34] and robotic manipulation [14] are provided in Appendices B.7 and B.8, respectively. 
results on dmcontrol-gb: 
 We evaluated the zero-shot generalization performance on 6 tasks in 'Video Easy' and 'Video Hard' benchmarks from DMControl-GB [13] . As illustrated in Fig. 5 , the easy version shares certain structures with the training environment such as the ground and shadow, while the hard version shares nothing other than the agent's object by replacing all backgrounds with distracting videos. In Table 1 , SimGRL demonstrates state-of-the-art performance in 4 out of 6 tasks in the Video Easy benchmark and achieves comparable performance in the remaining two tasks, including 'Walker, Walk' and 'Cheetah, Run'. On the other hand, our approach shows outstanding performance in all 6 tasks at the Video Hard level, where SimGRL achieves performance gain by 15% on average compared to SGQN. Specifically, SimGRL outperforms existing methods by a significant margin in 'Cartpole, Swingup' and 'Cheetah, Run', which were previously difficult to solve. We suggest that the reason for this lies in the fact that 'Cartpole, Swingup' requires detailed identification of salient objects, while 'Cheetah, Run' needs to address the observational overfitting. Both issues are mitigated in SimGRL. We provide the training curves in Appendix B.6. Ablation Study To verify the effectiveness of the proposed regularizations, we compare the ablation variants with the SVEA baseline. Fig. 7 shows training curves for zero-shot test performances averaged across the 6 tasks of DMControl-GB at the Video Hard benchmark as a function of the number of stepped training frames. This result indicates that the additions of each regularization to SVEA can remarkably improve the generalization ability of the model. In particular, we emphasize that each of the two regularizations leads to sufficient improvement, where all variants of SimGRL achieve, on average, nearly 82% better performance than SVEA. Full experimental results for each task are provided in Appendix B.1 and further in-depth discussions on the effectiveness of each regularization are described in Appendix B.2.To analyze the two potential overfitting problems, namely imbalanced saliency and observational overfitting, we evaluate the TID metrics, including the TID score and variance. For clarity, we compare three methods of DrQ [40] , SVEA [12] , and SimGRL as the former serve as baselines for the latter. We analyze two representative tasks, namely 'Cartpole, Swingup' and 'Cheetah, Run', where each problem is prominently observed. Full evaluations for all algorithms and tasks are provided in Appendix F.3. The left plot in Fig. 8 depicts that the overall distribution of training scores of the TID metrics is divided into three regions. In this plot, both DrQ and SVEA exhibit results divided into two regions, either 'low score and low variance' or 'middle score and high variance', depending on the task, indicating that different phenomena appear in the two tasks. The 'low score and low variance' region can be interpreted as observational overfitting, which does not identify the correct object in all frames. On the other hand, the 'middle score and high variance' area suggests that the correct object is accurately recognized only in certain frames, implying imbalanced saliency. We note that although SVEA [12] incorporates a data-mixing strategy with strong augmentation to enhance the generalization capability of DrQ [40] , both problems are still observed with only minor improvements over DrQ. In contrast, SimGRL reports comparatively high scores and low variances for both tasks, indicating that it can effectively identify task-relevant objects regardless of the type of task. The middle plot shows a positive correlation between training and test TID scores at Video Hard from DMControl-GB [13] . This suggests that a high discrimination ability during training can result in a high discrimination ability during testing. Finally, the right plot shows the generalization gaps, which are computed by the performance differences between training and testing, against the TID scores. This suggests that a high discrimination ability of the task object can lead to improved generalization capability.  
related works: 
 The field of domain generalization for RL has garnered considerable attention in recent years where various approaches aim to enhance the robustness of policies against visual changes. A promising approach is adapting the policy to a test domain. For example, PAD [11] suggests adopting a self-supervised task to obtain a free training signal during deployment. On the other hand, one method, proposed in [36] , involves using randomly simulated RGB images. Similarly, [27] train domain-adaptive policies by randomizing dynamics during the training phase. Several works explore data augmentation techniques to improve policy generalization capacity [19, 40, 29] . For instance, RAD [19] and DrQ [40] achieve significant improvement through random crop and shift, while DrAC [29] automatically identifies the most effective augmentation with regularization terms for the policy and value function. Instead of relying solely on augmented data for policy learning, SODA [13] aims to decouple augmentation from policy learning by using soft-augmented data for policy learning and strong-augmented data for auxiliary representation learning. In a recent development, SVEA [12] designs a stabilized Q-value estimation framework to address instability issues under strong data augmentation in off-policy RL, while DBC [43] uses bisimulation metrics to learn a representation that disregards task-irrelevant information. VAI [39] extracts a universal visual foreground mask to provide an invariant observation to RL. Similarly, several methods leverage salient masks to encourage the agent to focus on import pixels. For example, TLDA [42] attempts to only augment the task-irrelevant pixels using masks obtained from the Lipschitz constant of the policy while SGQN [2] proposes saliency-guided self-supervised learning using a gradient-based saliency mask. EAR [4] attempts to learn environment-agnostic representations to enhance the robustness of policies against visual perturbations. On the other hand, several methods such as TIA [7] , DRIBO [6] , and RePo [47] , leverage model-based RL through a recurrent state-space model (RSSM) [10, 9] to explicitly learn latent representations that focus on task-relevant features while discarding task-irrelevant ones. Our work focuses on improving the generalization ability of the RL agent based on implicit regularization approaches. 
conclusion: 
 In this paper, we identified two key issues that degrade generalization in vision-based RL, particularly under dynamic scene perturbations, by employing a gradient-based attribution mask. To resolve these issues, we proposed a simple framework involving an architectural modification to the encoder and a new data augmentation. The proposed SimGRL algorithm has achieved state-of-the-art results on Video Hard environments in DMControl-GB, outperforming existing methods that have yet to address these challenges. Additionally, we introduced TID metrics to quantitatively evaluate the ability to discriminate task-relevant objects of RL agents. Using these metrics, we demonstrated that improving the identification of task-relevant objects can enhance the generalization capability of the vision-based RL agent. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The paper includes experiments compute resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no safeguards risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Creators or original owners of assets used in the paper, are properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this section, we describe the implementation details used for SimGRL. For a fair comparison, we followed the same architectural designs and hyperparameters used in [13] , with only a minor difference in the encoder. For the feature-regularized encoder, we used 3 layers for the image encoder f 1 with 16 channels and 8 layers with 32 channels for the feature encoder f 2 , leading to the encoder with a total of 11 layers that are the same layer number as the encoder used in [13] . Subsequently, we projected the convolutional features of the last layer to 100-dimensional linear vectors, which are fed into actor and critic heads with 1024-dimensional hidden layers. We updated the actor parameters by freezing the encoder, and the target network parameters using an exponential moving average (EMA) with a rate of 0.01 for the critic head and 0.05 for the encoder. Both actor and target networks were updated every 2 critic updates. We used a minibatch size of 128, stacked 3 image frames each of size 3 × 84 × 84 as input for RL, and employed the Adam optimizer [17] . For the maximum shift length L in the shifted random overlay augmentation, we employed L = 6, which leads to 3 × 108 × 108 size of a natural image ε before shifted cropping. In Fig. We provide detailed results of the ablation study in Table 3 . For DrQ, we utilized [K = 1, M = 1] [40] . In this table, by introducing weak augmentation using random shift into the vanilla SAC [8] , DrQ demonstrates highly improved performance in image-based RL at the training level. While DrQ reports the best results in most tasks at the training level, the performance drastically decreases in testing levels, implying overfitting to the training environments. SVEA [12] introduces a datamixing strategy to enhance training stability under strong data augmentation, thereby improving DrQ's generalization capability in distracting backgrounds using random overlay [13] . However, the performance of SVEA still falls behind in challenging environments such as Video Hard. To address this issue, we proposed the feature-level frame stack and shifted random overlay augmentation, resulting in significant improvements compared to the SVEA baseline in Table 3 , especially in the Video Hard environments. It is noteworthy that each of the proposed regularizations demonstrates significant improvements compared to SVEA, where SimGRL-F and SimGRL-S apply the featurelevel frame stack and the shifted random overlay augmentation to SVEA, respectively. Especially, all of SimGRL's variants reach the saturated performance in several tasks at the Video Hard level except for 'Walker, Walk' and 'Cheetah, Run'. For more complicated tasks such as 'Walker, Walk' and 'Cheetah, Run' tasks where performance saturation has not yet been reached, the integration of the regularizations (i.e., SimGRL) clearly shows the performance improvement. Fig. 10 shows the training curves of ablations for each task at Video Hard. Train SAC ✓ 257±49 861±23 148±114 807±10 640±117 214±26 DrQ ✓ ✓ 943±3 973±2 975±5 878±2 982±4 463±11 SVEA ✓ ✓ ✓ 923±14 972±4 974±5 872±4 976±15 296±10 SimGRL-F ✓ ✓ ✓ 903±25 972±3 972±4 865±7 982±4 310±24 SimGRL-S ✓ ✓ ✓ 901±36 973±3 974±3 868±13 982±2 314±18 SimGRL ✓ ✓ ✓ 923±17 971±2 969±3 873±5 986±2 322±23 Video Easy SAC ✓ 245±165 389±131 192±157 398±60 206±169 73±18 DrQ ✓ ✓ 747±21 926±30 380±188 459±81 599±62 270±16 SVEA ✓ ✓ ✓ 819±71 961±8 871±106 782±27 808±33 251±17 SimGRL-F ✓ ✓ ✓ 873±22 971±4 947±4 844±15 975±3 290±37 SimGRL-S ✓ ✓ ✓ 878±32 973±5 962±10 826±25 962±21 301±23 SimGRL ✓ ✓ ✓ 910±21 973±4 964±7 838±35 983±2 317±16 Video Hard SAC ✓ 122±47 231±57 101±37 158±17 13±10 75±14 DrQ ✓ ✓ 121±52 252±57 100±40 136±29 38±13 49±13 SVEA ✓ ✓ ✓ 385±63 834±46 403±174 393±45 335±58 112±12 SimGRL-F ✓ ✓ ✓ 729±39 928±7 907±6 719±34 878±21 264±13 SimGRL-S ✓ ✓ ✓ 727±36 918±9 861±36 708±4 886±4 269±11 SimGRL ✓ ✓ ✓ 773±31 932±17 902±19 727±23 864±12 301±7We discuss the impacts of the proposed regularizations based on the attribution masks and the TID metrics. First, we investigate the impacts in terms of the imbalanced saliency in the 'Cartpole, Swingup' task. In the left plot of Fig. 11 , both SimGRL-F and SimGRL-S show increased TID scores compared to SVEA. However, they exhibit different effects on TID variance as SimGRL-F significantly reduces it but SimGRL-S rather increases it. This indicates that the imbalanced saliency remains in SimGRL-S while SimGRL-F alleviates this issue. Therefore, the performance improvement of SimGRL-F can be interpreted as a mitigation of the imbalanced saliency problem, while the performance improvement of SimGRL-S is a bit ambiguous. For the reason of the result with SimGRL-S, we suggest that despite the persistence of imbalanced saliency, the explicit injection of dynamic background elements might have contributed to improved generalization in such conditions. Furthermore, even though it does not distinctly discriminate task objects between the stacked frames, the increased TID score of SimGRL-S can be attributed to its improved accuracy in identifying recognized objects in certain frames, as depicted in Fig. 12 . This implies that the shifted overlay augmentation implicitly encourages the model to identify task objects more accurately. As a result, SimGRL, which integrates both architectural and data regularization, demonstrates that its agent can identify task-relevant objects across stacked frames with greater accuracy and discrimination.In the right plot of Fig. 11 , all of the SimGRL variants demonstrate remarkably increased TID scores in the 'Cheetah, Run' task. Since the 'Cheetah, Run' task has less movement of task objects than the 'Cartpole, Swingup' task, differences in TID variance values for this task are not effective. Therefore, the core issue in this task is the observational overfitting [32] to ground or shadow rather than imbalanced saliency between consecutive frames. The increase in the TID score implies an enhanced ability to identify task-relevant objects, which is demonstrated in Fig. 12 . As expected, this shows that the application of the shifted random overlay augmentation encourages the agent to focus on the task object while disregarding background elements. On the other hand, it is noteworthy that the frame stack at the feature level also contributes to solving this problem. We suppose this is because encoding from a single image without temporal information allows the model to concentrate on each image's most salient visual features. Consequently, by integrating those strategies, the proposed approach, SimGRL, effectively addresses the two problems highlighted in this paper that hinder generalization in visual RL. While maintaining the number of layers in the entire encoder at 11, we conducted the ablation study for the number of layers in the image encoder. To consider only the impact of the layer number, we leveraged the SimGRL-F model which employs the original random overlay without shifting for the strong augmentation rather than the shifted version. Depending on the number of layers in the image encoder, we investigated test rewards, generalization gaps, and throughputs in the 'Walker, Walk' task, where the Video Hard domain was employed for testing environments. The generalization gap was evaluated by the difference between the training and testing performances, and the throughput was computed by the FPS processed in the simulator on a single NVIDIA TITAN RTX GPU. The results presented in Fig. 13 demonstrate that the incorporation of the feature-level frame stack, facilitated by the image encoder, enhances the model's generalization capabilities in dynamic scenes. It is noteworthy that even if the number of image encoder layers is only 1, the generalization gap is remarkably reduced. Additionally, the generalization gap decreases when the number of layers in the image encoder is increased, indicating that incorporating feature maps with higher abstraction enables the improvement of generalization. However, despite the improvement in generalization performance, there does not seem to be a significant increase in practical test performance, implying a reduction in the training performance through the mitigation of overfitting. This can also be attributed to the diminished depth of the feature encoder, which is crucial for encoding temporal information. Moreover, increasing the layer number in the image encoder results in decreased throughput, primarily due to the increased computational burden associated with processing individual frames. Considering both generalization capability and computational efficiency, we selected 3 layers for the image encoder and 8 layers for the feature encoder for the experiments. To investigate the impact of differently augmenting the stacked frames, we provide additional experiments under two types of additional augmentations: (1) randomly cropped patches from the same image and (2) completely different images. We denote the first method as irregularly shifted random overlay (I-SRO) and the second method as arbitrarily stacked random overlay (A-SRO).While the SRO augmentation crops patches in a regularly shifted manner as Eq. ( 5), I-SRO irregularly crops patches from a sampled natural image ε. In contrast to Eq. ( 5), the coordinates (h i , w i ) of the upper left corner of each cropped patch become as follows:EQUATIONwhere dh i , dw i ∼ unif{-l, l}. On the other hand, A-SRO samples different natural images ε i ∼ D with H × W sizes and augments each RL frame using the different images. The processes of I-SRO and A-SRO are illustrated in Fig. 14 .For clarity of the effect for each augmentation, we do not include the feature-level frame stack and compare them in the video hard level with SRO (i.e., SimGRL-S in the paper) that leverages regularly shifted cropped patches from the same image for data augmentation. Table 4 shows that augmentations using irregularly shifted patches (I-SRO) or completely different patches (A-SRO) across the consecutive frames can achieve comparable performance with SRO using the regularly shifted patches, implying that augmenting each frame in a different manner (i.e., employing SRO, I-SRO, and A-SRO) can achieve significant performance improvement over augmenting each frame uniformly (i.e., RO) for visual RL generalization under dynamic scenes. Among such augmentations, we have employed the (regularly) shifted random overlay (i.e., SRO) as our primary approach for data augmentation considering its superior averaged performance. According to its architectural simplicity and absence of additional auxiliary losses beyond the actor and critic losses, we adopted the SVEA [12] algorithm as our baseline to clarify our contribution. However, our approach that utilizes two types of regularizations can be seamlessly integrated with any other in a plug-and-play manner. On the zero-shot evaluation in the video hard test environments, we present the experimental results of SGQN [2] + F (Feature-level frame stack), SGQN + S (Shifted random overlay), and SGQN + S + F in Table 5 . This table shows that the proposed regularizations can also contribute to improved generalization for other methods, such as SGQN, demonstrating the generality of our method. We provide the additional benchmark results on the Distracting Control Suite (DistractingCS) [34] , where camera pose, background, lighting, and colors continually vary throughout an episode. In To verify the generalization capabilities of the proposed approach across various domains, we conducted additional experiments for robotic manipulation tasks [14] , where the agents are trained for 250K steps. In these experiments, since the test environments introduce color and texture changes rather than structured distractions, SGQN [2] employed random convolution [20, 19] for strong augmentation. However, for SimGRL, we still utilized the shifted random overlay. In addition, for shifting, we employed two frames as an input rather than a single image setup as in SGQN. Fig. 20 exhibits examples of two robotic manipulation tasks, including 'Reach' and 'Peg in Box', and each of the testing environments. In these examples, SimGRL demonstrates its ability to appropriately capture salient regions, characterized by the robot arm and target positions. Algorithm 1 A Simple Framework for Generalization in visual RL (SimGRL) applied to SAC. a t ∼ π ϕ (•|f θ (s t ))▷ Sample action from actor (= policy) for i = 1...N do 9:s i = τ (s i ; ν i ), s ′ i = τ (s ′ i ; ν ′ i ), ν i , ν ′ i ∼ V▷ Apply random shift augmentation 10:q tgt i = r(s i , a i ) + γQ θ (f θ (s ′ i ), a ′ i ), a ′ i ∼ π ϕ (•|f θ (s ′ i )) ▷ Compute target Q-value 11:s aug i = τ SRO (s i ; l, ε i ), ε i ∼ D ▷ Apply SRO augmentation 12:end for 13:L Q (θ) = 1 N N i=1 [β(Q θ (f θ (s i ), a i ) -q tgt i ) 2 + (1 -β)(Q θ (f θ (s aug i ), a i ) -q tgt i ) 2 ]14:▷ Compute SimGRL loss 15:θ ← θ -η∇ θ L Q (θ) ▷ Optimize critic for SimGRL loss 16:if every M step then 17:ϕ ← ϕ -η∇ ϕ L π (ϕ)▷ Optimize actor for SAC's actor loss 18:θ ← λθ + (1 -λ) θ ▷ Update target network using EMA 19:end if 20: end for 21: Note: In this paper, we demonstrated that it is important for the RL model not to be disturbed by taskirrelevant backgrounds for better policy generalization. To do this, we assumed that the RL agent should at least focus more on the task object than on the background, and employed the attribution masks to analyze whether the trained RL models correctly distinguish the task objects and backgrounds. Additionally, to quantitatively analyze how well RL models can distinguish between the task object and the background, we proposed the TID metrics.f θ (s t ) represents f 2 θ ([f 1 θ (o t-2 ), f 1 θ (o t-1 ), f 1 θ (o t )]), where s t = (o t-2 , o t-1 , o t ).In practice, RL policies may not make decisions based on the entire object. Instead, they could care about a few key points within such objects that are most predictive of the reward. To investigate whether the ability to distinguish between task objects and backgrounds also helps in identifying the key points that influence decisions, we provide attention maps obtained from the softmax of the attribution maps M (Q θ , s t , a t ) = ∂Q θ (st,at) ∂st before thresholding for the binarized masks. Fig. 21 shows that the identification ability for the task objects of SimGRL also leads to focus on the most important key point parts such as the pole and body of the 'Cartpole' or legs of the 'Cheetah' in the attention maps. On the other hand, the SVEA baseline incorrectly considers the background features as key points. In response to the recommendation of the reviewers to investigate whether the pitfalls also apply to robust RL algorithms such as DBC [43] , TIA [7] , DRIBO [6] , and RePo [47] , we provide comparative experimental results for these methods. While our work focuses on addressing the generalization issue in a purely model-free vision-based RL setting, which requires the frame stack to encode temporal information, the robust RL approaches aim to explicitly learn robust representations against background distractions by utilizing model-based RL. To encode temporal information, these approaches, except for DBC, train transition dynamics by leveraging complex recurrent neural network (RNN) encoders which take only a single frame as input to the encoder, thus applying a concept similar to feature-level frame stacking. On the other hand, DBC uses a CNN encoder that takes stacked frames and trains a dynamics model that predicts the latent states of the next stacked frames. Fig. 22 shows that methods like TIA, DRIBO, and RePo, which use RNN-based encoders, do not struggle with the imbalanced saliency problem, successfully identifying each salient object across consecutive frames like SimGRL. In contrast, despite employing robust representation learning, DBC still struggles with the imbalanced saliency problem, showing a bias toward identifying pixels from the object in the most recent frame as salient across consecutive frames. 23 shows the curves of zero-shot test performances for the 'Cartpole, Swingup' task in the video-hard environment according to the number of training frames. For a fair comparison, we applied the random overlay augmentation to the robust RL algorithms where following to [13] , the data augmentation was applied only during the representation learning stage. As anticipated, robust RL algorithms like DRIBO and RePo, which can distinctly identify task-relevant objects, achieved better test performance (except for TIA) than methods that struggle with imbalanced saliency, such as SVEA and DBC. We suggest that the reason for the inferior performance of TIA is that the TIA agent struggles to train the Distractor Model, which explicitly captures the background, due to the more diverse background augmentations compared to the relatively limited video backgrounds from the Kinetics dataset [16] used in its original experimental setup. Notably, despite its straightforward structure and absence of additional representation learning stages, the proposed SimGRL achieved both sample efficiency and strong test performance, outperforming robust RL approaches like DRIBO and RePo that rely on model-based representation learning strategies.To fairly compare TID metrics of comparison methods on the same images, we constructed a dataset for the evaluation in advance. As illustrated in Fig. 24 , we excluded samples that were excessively static because the side effect of imbalanced saliency is diminished in such samples. Additionally, we excluded samples where the task object was not fully visible. Instead, we included dynamic samples to fully evaluate the impact of imbalanced saliency. To this end, as the samples obtained from random actions were less dynamic, we selected the images by running SVEA's policies, our baseline algorithm, up to 100K steps to obtain a variety of dynamic samples. Using those images, as depicted in Fig. 25 , we constructed 60 pairs of images and ground truth (GT) masks of task objects for each training and testing environment in Video Hard. These GT masks are employed to count N obj and N obj M in Eq. ( 8). Here, N obj represents the pixel number in the GT mask while N obj M is determined by the pixel number in the overlapping regions between the GT and attribution masks. In Fig. 26 , the impact of quantile ρ values for the ρ-quantile attribution mask [2] is illustrated. Additionally, the effect of the ρ value on the TID score can be explained as follows. When the value of ρ is too small, the region of a mask is enlarged, thus making it easier for task objects to be included in the attribution mask. This corresponds to the increased N obj M , leading to an increasedN obj MN obj in the first component of the TID score in Eq. (8) . Simultaneously, this also increases N M , balancing the TID score by decreasingN obj MN M that is the second component of the TID score. Conversely, a large value of ρ reduces the size of the mask N M as in Fig. 26 , consequently reducing N obj M . On the other hand, the decreased N M might increase N obj M N M , balancing the TID score. Therefore, for the quantile value used to evaluate the TID score and variance, we used the optimal quantile value computed by ρ = 1 -N obj (3×C×H×W ) rather than a hand-designed value. However, we used ρ = 0.95 in our qualitative illustrations of the attribution masks, as it empirically provided the most plausible visual representation of a masked region. We present comprehensive results of the TID evaluation, including a comparison of training TID variance, test TID score, and generalization performance against the training TID score. For the 'Walker' agent, we evaluated the TID scores on the 'Walk' task rather than 'Stand'. Fig. 27 (a) reports higher TID scores and lower TID variances of our approach, indicating that SimGRL can distinctly identify the salient pixels across the stacked frames for all tasks. In particular, for the 'Ball In Cup' agent, measuring the TID scores accurately is difficult due to the relatively small size of the salient object. Nevertheless, SimGRL shows remarkably higher TID with lower variance scores than the competitors. These results demonstrate that SimGRL can effectively alleviate the two highlighted issues, 'imbalanced saliency' and 'observational overfitting'. In Fig. 27 (b), we can observe that the high identification ability of the task object in the training environment can lead to a similar capability in test environments. Finally, Fig. 27(c ) shows a positive correlation between generalization performance and the TID score, where smaller generalization gaps correspond to higher generalization performance. For the distributions in Figures 27(b ) and 27(c), we present Pearson correlation coefficients [26] in Table 7 . These results imply that the high identification capability for task-relevant objects across stacked frames in input, as represented by the high TID score, can contribute to improving generalization performance. For the existing state-of-the-art visual RL approaches for generalization, we provide rich examples of the attribution masks and the corresponding masked observations, where we used the images employed in the TID evaluations. Additionally, we used the quantile ρ = 0.95 across all examples. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The reviewers advocate accepting this paper, and the AC agrees with this evaluation. During the rebuttal phase, the authors clarified the reviewers' confusion on the technical aspects and the experimental results. The reviewers found that the rebuttal adequately resolved the weaknesses. One minor note, the authors should update their references. Specifically, the authors cited the arXiv version instead of the conference version, e.g. [37] has been accepted to ICLR.

Reviews: review: paper_summary: paper_summary: The paper introduces SimGRL, a framework aimed at enhancing generalization in vision-based deep reinforcement learning (RL) under dynamic scene perturbations. It addresses two critical issues in existing visual RL methods: imbalanced saliency and observational overfitting. SimGRL employs architectural modifications to the image encoder and a novel shifted random overlay augmentation technique. Extensive experiments demonstrate SimGRL's superior generalization capabilities, achieving state-of-the-art performance on the DeepMind Control Suite and other benchmarks.
strength_weakness: strength_weakness: Dependency on Augmentation Quality: The effectiveness of the shifted random overlay augmentation relies on the quality and diversity of the natural images used for augmentation, which may limit its performance in certain real-world scenarios where such images are not available or appropriate.
Ad Hoc approach: the authors offer specific fixes to observed shortcomings in vision-based deep RL in simulated environments, and it remains to be seen if these fixes are an hand-crafted overfitting to specific tasks that won't generalize to other tasks.
Finally, while the method shows impressive results in simulated environments, its applicability and effectiveness in more realistic real-world scenarios have not been thoroughly validated.
questions: questions: How does the diversity of natural images used in the shifted random overlay augmentation impact the generalization performance of the RL agents? Are there specific types of images that are more effective?
Are there any plans to test SimGRL in real-world environments, and what challenges are anticipated in such scenarios? How does the method handle real-world dynamic perturbations that are not present in simulated benchmarks?
What are the effects of using different numbers of frames in the feature-level frame stack on the performance and generalization capability of SimGRL?
How sensitive is the performance of SimGRL to the choice of hyperparameters such as the number of layers in the image encoder and the maximum shift length in the shifted random overlay augmentation?
limitations: limitations: The architectural modifications and data augmentations, while effective, still introduce some computational overhead, which may impact training times and computational resources required.
The method is specifically designed for vision-based RL and may not directly apply to other types of RL tasks without visual components.
The shifted random overlay augmentation relies on the quality and diversity of natural images used for augmentation, which may limit its performance in scenarios where such images are not available or appropriate.
paper_summary: paper_summary: The paper first identify two pitfalls with the visual RL via gradient-based attribution mask, i.e., imbalanced saliency and observational overfitting. To address these two pitfalls, the paper proposes two novel modifications. One is for the encoder where encode each frame and stack encoded representations instead of encoding stacked frames directly. The other is a newly proposed shifted  randomly overlay augmentation method to address the observational overfitting.
strength_weakness: strength_weakness: - One concern I have about the paper is the first pitfall where the imbalanced saliency is specifically applied to visual RL algorithms that apply stacked frames. However, many robust RL algorithms, e.g., DBC, SLAC, TiA, DRIBO, RePo, take only single frame as the input to the encoder which apply similar ideas as the feature-level frame stack. These algorithms also explicitly take previous frame stack as input with an encoder $p_\theta(s_t | s_{t-1}, a_{t-1}, o_t)$. It would be interesting to investigate whether these methods also suffer from the imbalanced saliency and compare the proposed feature-level stack method with them in terms of performance and efficiency.
- Though the paper identifies the pitfalls with attribution methods, it would also be interesting to see attribution masks of SimGRL in different settings, i.e., clean, video easy and video hard. This would help validate that the identified pitfalls are sufficiently addressed by SimGRL.
- Fig 5 and demos showed along the paper may be misleading that the policy is purely learned from the clean environment without any background perturbation. However, the proposed randomly shifted overlay augmentation explicitly introduces background perturbations into the training process.
questions: questions: - Could you comment on whether the pitfalls also apply to robust RL algorithms like DBC, SLAC, TiA, DRIBO and RePo?
- Could you comment on how different attribution methods may affect the identified pitfalls and TID metrics?
limitations: limitations: Yes, the author addressed the limitations in the paper.
paper_summary: paper_summary: This paper proposes two simple, yet crucial modifications to the Generalization in the Visual RL pipeline where during the test time, the visual observation consists of various degrees of dynamically varying backgrounds.

Concretely, the authors identify two issues that inhibit RL agents from generalization: (i) imbalanced saliency and (ii) observational overfitting. To overcome these issues they propose two simple, yet effective solutions -- (a) Feature level frame stack where instead of stacking frames at the observation level, they stack at a feature level after obtaining spatial features of the images and (b) Shifted Random Overlay Augmentation where instead of applying data augmentation to the stack of frames, a shifted version of the data augmentation is applied to simulate the dynamically moving background effect which in turn helps learn more generalizable policy.

Through their experiments on the DM-Control (and robotic manipulation in the appendix) with varying backgrounds, the authors show the generalization capability of their proposed SimGRL (Simple framework for Generalization in visual RL) framework.  

Additionally, the authors also propose Task-Identification (TID) Metrics to evaluate how much of the policy can capture the task-relevant objects in each stacked frame.
strength_weakness: strength_weakness: 1. I do find myself in confusion with respect to the Attribution masking argument. I believe that the proposed solutions of Feature-level frame stack and Shifted Random overlay augmentation are both valid solutions to a generalizable policy irrespective of the authors inclusion of Attribution masking. My specific concern is as follows: 

    What is the guarantee that a clear thresholded gradient masking of the critic wrt the input, which consists of the complete segmentation of the relevant objects and agent in the mask will lead to a better generalizable policy? It can be possible that a generalizable policy would care about only a few key points that are predictive of the reward -- and hence be able to accurately predict the Value of a state, and in that sense, an accurate segmentation of the attribution mask need not be necessary. To give an example, in the robotic manipulation task of reach -- the reward function is typically shaped as some function of the Euclidean distance between the robot's end-effector and the goal position. In such a situation, the Value function shouldn't really be bothered about where the other parts of the robot are.

2. The weakness in (1) ties into my second concern where if only a few parts of the agent are responsible for achieving (and predicting) reward, then I don't see the value of the proposed TID metric. This assumes that my entire object and agent in the scene is responsible for the policy, which as argued in (1) is not the case in most scenarios.

3. [Relatively minor concern, not taken into account for the final scoring] It would be beneficial to the reader if the proposed components in Figure 4 could be highlighted with say a different color and a distinction between the existing pipeline of SVEA be made.

4. On the experimentation end, I feel that the evaluation is very limited to DM-control tasks. While SimGRL performs quite well on Video-Easy and Video-Hard settings of DM-Control, and 2 other tasks of Robotic manipulation, I would have liked to see a more comprehensive evaluation of various visual RL environments. Specifically, I'd recommend performing evaluation on RL-ViGen [1] (https://gemcollector.github.io/RL-ViGen/)  which consists of a collection of diverse visual RL tasks. This would further strengthen this work's results.

5. On L263, where authors say: 
> SimGRL demonstrates state-of-the-art performance in 4 out of 6 tasks in the Video Easy benchmark

    One has to be careful while making such claims especially when the difference on one such task is +1 -- which in DM-Control doesn't really mean anything. I would ask the authors to look at IQM and Probability of Improvement metrics to compare SimGRL to other methods from [2] which has shown to be more reliable than taking an average over seeds. This does not involve re-running the experiments, the existing checkpoints and returns can be directly used over their wrapper.
questions: questions: 6.  Observational Overfitting: For this, one of the core arguments that authors provide is that because the same augmentation is applied uniformly to all frames the agent can tend to still focus on the background. I'm wondering how a baseline for this where you have different data augmentations for every frame in the history of stacked frames(say jittering, slight rotation, blur, overlay, random convolution, random shift) performs?

7. In the Robotic manipulation results what does Test {1, 2, 3} mean? Does it mean running an evaluation on a _single episode_ for 5 seeds?

----
**References**

[1] RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization, Zhecheng Yuan et al., NeurIPS 2023.	

[2] Deep Reinforcement Learning at the Edge of the Statistical Precipice, Rishabh Agarwal et al., NeurIPS 2021.

----
**Reason for current rating**: Because of my two major concerns around Attribute masking (and TID metrics), as well as lack of extensive experimentation on diverse RL environments, I am leaning towards a weak reject. However, my final decision would be based on authors' rebuttal, other reviewers comments and the discussion with authors during the rebuttal period. I'd encourage the authors to ask me any clarification questions for any of the experiments if they have any during the rebuttal/discussion period.

-----

**Post rebuttal update** -- As the authors have addressed all my major concerns, I have decided to increase my score to **Weak Accept**.
limitations: limitations: The authors don't mention any limitations of their work. It would be good to have a couple of lines highlighting where their method may fail. For instance, potentially large changes in background can lead to failure or their Feature stacking would work only in situations where Frame stacking is used -- model-based works like Dreamer do not use any frame stacking because of a recurrent network (such as GRU) that handles the history of states.
paper_summary: paper_summary: This paper introduces SimGRL for vision-based DRL tasks. It tackles two challenges: the imbalanced saliency, where an agent repeatedly favors saliency maps from the recent one in stacked frames, and the issue of overfitting to particular background areas rather than concentrating on elements crucial to the task at hand. To overcome the first challenge, the proposed solution modifies the image encoding process, treating each frame separately before combining their features. To deal with the second issue of observational overfitting, this research uses an innovative method based on shifted random overlay augmentation. This method trains the agent to associate rewards with changes in the actual object of interest, thus learning to ignore background motions. Experiments were carried out using the DMControl-GB video benchmarks, evaluated by the TID score and variance metrics. The results show that SimGRL outperforms existing methods and the ablation study reveals each regularization technique independently contributes to notable enhancements.
strength_weakness: strength_weakness: The paper could benefit from an explicit articulation of its motivation and contributions. While multiple mentions are naturally made where the two challenges are being addressed, situating the motivation behind the work and a summary of contributions in a dedicated paragraph would enhance the strength of the paper. Although the devised SimGRL algorithm achieves top-tier performance on the Video Hard benchmark, the analysis of its performance on the Video Easy benchmark is not explored in depth, which presents an area for more detailed examination.
questions: questions: 1) The findings concerning Video Easy presented in Table 1 appear to be of lesser importance. Could you provide a more detailed explanation of these findings?

2) Furthermore, Table 1 reveals that 'Cartpole, Swingup' and 'Cheetah, Run' exhibit noteworthy outcomes, although the significance is not observed across all tasks. While you have offered explanations for the strong performance in 'Cartpole, Swingup' and 'Cheetah, Run', can you clarify why there weren't substantial improvements in some other tasks? Does this indicate that the impact of your study might be constrained?
limitations: limitations: No. The paper has limitations, but those are not discussed in the paper. The societal impact is not applicable.

Comments: title: Thank you for your positive comment!
Thank you for your positive comment!
Thank you for your positive comment!
Thanks for addressing my major concerns and for the clarification
Thank you
Thank you for your detailed response!
comment: Dear Reviewer kgwp,

We sincerely appreciate your thoughtful response and your dedicated time to review our paper.
We are strongly encouraged by your recognition of our work.
We will incorporate your suggestions and insights into the revised manuscript. 
Specifically, we agree with your comments that DBC and DRIBO need to be compared with SimGRL under the setting using the same data augmentation and we will add a section in the Appendix of the revised version to discuss the robust RL algorithms including the second part of the results of the rebuttal.
Additionally, we will make an effort to include results comparing recent robust RL methods, such as TiA [2] and RePo [3], in the revised version.
Thank you once again for your thorough review and your positive evaluation.

Sincerely,

Authors

***

**References**

[2] "Learning task informed abstractions." ICML (2021).

[3] "Repo: Resilient model-based reinforcement learning by regularizing posterior predictability." NeurIPS (2023).
Thank you for the further clarification!

The results on applying random overlay augmentation on DBC and DRIBO are particularly helpful. I would rather disagree with the setting where DBC and DRIBO are compared with SimGL but trained under the clean setting. Both DBC and DRIBO did not claim transferability of robust RL agents trained under the clean setting. I would recommend author to include the second part of results in the revised version.

In addition, I would strongly recommend authors to compare with more recent robust RL methods in the revised version.

I would like to raise my score to 6 and this is a borderline paper to me. I think the major contribution of the paper lies on the random overlay augmentation and the TID metric. However, more comparison with other robust RL methods can further strengthen the paper.
Dear Reviewer t66L,

We sincerely appreciate your thoughtful response and the time you've dedicated to reviewing our paper.
We are strongly encouraged by your recognition of our work.
Thank you once again for your thorough review and your positive evaluation.

Sincerely,

Authors
Dear Reviewer wvnF,

We sincerely appreciate your thoughtful response and the time you've dedicated to reviewing our paper.
We are strongly encouraged by your recognition of our work. We will incorporate your suggestions and insights into the revised manuscript. 
Specifically, we will add some sentences to avoid confusion and enhance the clarity of our claims.
Thank you once again for your thorough review and your positive evaluation.

Sincerely,

Authors
We thank you for your constructive feedback and your eagerness to engage in further discussions regarding our work!

We address your comments in the following.

***

> **[Q1] Are DBC and DRIBO trained with the same video augmentation as SimGL or trained in the clean setting?**

**(A)** To clarify whether robust RL methods suffer from the imbalanced saliency problem, and since both DRIBO and DBC did not originally use strong augmentation such as random overlay, clean images were used during training. 
This implies that even when methods like DRIBO, which learn robust representations, do not suffer from issues such as imbalanced saliency, training on diverse backgrounds remains important for generalization to unseen environments.

***

> **[Q2] Can existing robust RL methods also benefit from the random overlay augmentation?**

**(A)**  To investigate whether existing robust RL methods also benefit from augmentations like random overlay, we provide experimental results on the 'Cartpole, Swingup' task in the 'Video Hard' test environment for DBC and DRIBO when using random overlay augmentations.
Following SODA [1], we applied random overlay augmentations solely to the representation learning losses of the encoder for stability in training.

||Train	|No Aug.	|Random Overlay	|Shifted Random Overlay|
|-|-|-|-|-|
|**DBC**	|805.2	|83.7	|169.7	|170.4|
|**DRIBO**	|818.6	|134.7	|414.8	|532.3|

The above table indicates that DBC showed marginal effects even with the application of random overlay, while DRIBO demonstrated clear improvements. Additionally, DRIBO was able to obtain additional performance gain when using the shifted random overlay augmentation. For DRIBO, these results clearly suggest that robust RL algorithms can benefit from data augmentation techniques such as random overlay, further strengthening our contribution. On the other hand, DBC's results suggest that a specific augmentation method may not always be well-suited for a particular learning algorithm. Alternatively, this could be due to issues with hyperparameter tuning in methods with complex representation learning processes, where the marginal performance might be attributed to not finding the optimal hyperparameters for applying the augmentation. Our approach, with its simple structure and minimal hyperparameter tuning, could be a solution to address these drawbacks.

***
**References**

[1] "Generalization in reinforcement learning by soft data augmentation." ICRA (2021).
Thanks to the authors for their rebuttal.

1. The clarification on the attribution masking is now helpful for me to understand the claims they are making and helped clear my confusion. I'd request the authors to add maybe a sentence or two in the main paper as well (either for camera ready if accepted or for future submissions).

2. Thanks for showing the IQM over 5 runs.

3. Re: Q1 -- I was hoping for an empirical evidence but I do get the authors' point. By _random augmentation_, I was implying the typical computer vision augmentation (color jitter, crop, rotate etc) but independently for each frame in the stack rather than applying the same augmentation for the entire 3 frames. But in hindsight looking back at this comment I do think that even the random shift would be better as that is a stronger form of data augmentation.

4. Thanks for the clarification on the test1, 2, 3 and for mentioning the number of episodes on which the models were evaluated.

Based on the rebuttal, I'm convinced that this paper is in a good quality for publication and hence am increasing my score to Weak Accept. Once again, thank you to the authors for their rebuttal.
I thank the author for answering my questions. The score remains unchanged.
I would like to first thank authors for the detailed response and new comparison results. These are super helpful.

Most  of my concerns have been addressed except W1 and Q1. I would love to raise my score if authors can help better understand W1 and Q1 with the following questions.

## W1 and Q1

Thank you for providing the comparison results with DBC and DRIBO. These results are very helpful.

- A clarification question, are DBC and DRIBO trained with the same video augmentation as SimGL or trained in the clean setting? I am asking because Fig. 1(b) in the response letter showed that both DBC and DRIBO does not perform well during testing. I am wondering if author have any comments on it.

- The new results addressed my concern of whether existing robust RL methods addressed imbalanced saliency issue. Since the other major contribution of the paper is the random overlay augmentation, I am wondering if existing robust RL methods can also benefit from the random overlay augmentation.